# How to tokenize using map

**URL:** <https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473>\
**Category:** 🤗Datasets\
**Created:** [April 13, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473 "2021-04-13T20:02:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](https://discuss.huggingface.co/u/Neel-Gupta)\
**Post date:** [April 13, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473/1 "2021-04-13T20:02:16Z")

</div>

This is a problem that I have gotten: full info on this [thread](https://discuss.huggingface.co/t/error-when-fine-tuning-pretrained-masked-language-model/5386/10) I had a quick question -

I have constructed `Dataset` object using numpy arrays, however when I use this:-

```auto
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000)
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)

```

and I explore my `train_encoded_dataset`, I see that when trying to view the source sequence:

```auto
>>> train_encoded_dataset
>>>Dataset({
    features: ['attention_mask', 'input_ids', 'src', 'tgt'],
    num_rows: 4572
})
>>> train_encoded_dataset['src'][0]

```

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call `tokenizer.tokenize`

So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 13, 2021, 9:35pm UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473/2 "2021-04-13T21:35:45Z")

</div>

hey @Neel-Gupta, could you share a minimal example of the `Dataset` object you’re working with?

> [@Neel-Gupta](#):
>
> The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call `tokenizer.tokenize`
> 
> So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything

just so i understand, you’re saying that the tokenizer _is not_ tokenizing the strings in the `src` field right?

---

<div class="post-metadata">

**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](https://discuss.huggingface.co/u/Neel-Gupta)\
**Post date:** [April 13, 2021, 10:04pm UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473/3 "2021-04-13T22:04:55Z")

</div>

Thanks a lot for the quick reply! Yeah, the tokenizer is not tokenizing those strings, maybe because I didn’t call the `tokenize` method?

Anyways, plugging these variables do the trick for reproduction

```auto
train_text = np.array(['a foxy', 'b ball', 'c cats r bad'])
train_label = np.array([1,2,3])
val_text = np.array(['a foxy', 'r c cats'])
val_label = np.array([1, 2])

```

if you can’t reproduce the issue with the pre-trained `longformer (base)` tokenizer, then I can provide you my model - but I doubt it won’t since I have used that same one 🤗

`tokenizer = AutoTokenizer.from_pretrained(".....", use_fast=True, truncation=True, padding=True, max_length=2000)`

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 14, 2021, 12:53am UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473/4 "2021-04-14T00:53:58Z")

</div>

The result of the tokenization is not in the `"src"` field, but in `'input_ids'` and `'attention_mask'`. The `'src'` field is just the same as before.

---

<div class="post-metadata">

**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](https://discuss.huggingface.co/u/Neel-Gupta)\
**Post date:** [April 14, 2021, 3:33pm UTC](https://discuss.huggingface.co/t/how-to-tokenize-using-map/5473/5 "2021-04-14T15:33:07Z")

</div>

so definitely, my tokenizer does tokenize the input right? this means the problem is in the fine-tuning code if I am correct. would you have a clue as to where the problem can be?  
Previously, before using datasets it I was able to fine-tune it with the same code 😞  
Im getting `IndexError: index out of range in self`