# Encoder Decoder Loss

**URL:** <https://discuss.huggingface.co/t/encoder-decoder-loss/4335>\
**Category:** 🤗Transformers\
**Created:** [March 12, 2021, 4:55am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335 "2021-03-12T04:55:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [March 12, 2021, 4:55am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/1 "2021-03-12T04:55:10Z")

</div>

Hi all,  
I was reading through the [encoder decoder](https://huggingface.co/transformers/model_doc/encoderdecoder.html) transformers and saw how loss was generated. But I’m just wondering how it is internally generated?

Is it something like the following: Suppose I have the following pair: `("How are you?", "I am doing great")`. In this case, is it calculating the cross entropy loss for the four output tokens and then averaging them?

---

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [March 15, 2021, 7:22am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/2 "2021-03-15T07:22:49Z")

</div>

I posted a longer version of above with reproducible code: [deep learning - Sequence to Sequence Loss - Stack Overflow](https://stackoverflow.com/questions/66633813/sequence-to-sequence-loss)

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [March 15, 2021, 7:37am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/3 "2021-03-15T07:37:18Z")

</div>

Hi @sachin

the `EncoderDecoder` model calculates the standard auto-regressive cross-entropy loss using the `labels` i.e the output sequence. It just shifts the `labels` inside the models before computing the loss.

It’s the same loss used in other seq2seq models like BART, T5, and decoder models like GPT2.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [March 16, 2021, 12:34am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/4 "2021-03-16T00:34:21Z")

</div>

perfect. I forgot to shift the labels.

Just to extend the question though, I looked at the source code and managed to replicate the loss by huggingface, but just wondering shouldn’t it be:

```auto
# current loss calculation
output_logits = logits[:,:-1,:]
output_mask = mask[:,:-1]
label_tokens = output_tokens["input_ids"][:, 1:].unsqueeze(-1)
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
huggingface_loss = -select_logits.mean()

# proposed loss instead:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss = -seq_loss.mean()

```

Happy to create a PR if you agree.

The biggest downside of existing loss IMO is that if there is large variation in a batch output lengths it will focus on padding after the token.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [March 16, 2021, 10:26am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/5 "2021-03-16T10:26:32Z")

</div>

padding tokens in the labels should be replaced by -100 so the `cross_entriopy` loss ignores the pad tokens when computing the loss.

and the loss is actually computed like this

```auto
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous() # prediction_scores is logits
labels = labels[:, 1:].contiguous()
loss_fct = CrossEntropyLoss()
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

```

---

<div class="post-metadata">

**Author:** ![stephnn7](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@stephnn7](https://discuss.huggingface.co/u/stephnn7)\
**Post date:** [September 14, 2021, 5:39pm UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/6 "2021-09-14T17:39:26Z")

</div>

Hello, I am unsure if I should ask this here or if I should create a separate post but I was looking that the way the loss is computed and it seems really confusing to me how the logits are shifted and why is it done in such a way? I have been looking online and I haven’t managed to find a proper explanation, so could you please help me by explaining why and how the logit shifting is done?

---

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](https://discuss.huggingface.co/u/sachin)\
**Post date:** [October 14, 2021, 8:54am UTC](https://discuss.huggingface.co/t/encoder-decoder-loss/4335/7 "2021-10-14T08:54:04Z")

</div>

Hey, sorry for not replying earlier. The basic reason is because when the tokenizer encodes it, it will do something like `"<START> My decoded sentence <END>"`. The output of the decoder transformer will only predict for `"My decoded sentence <END>"`.

So the logits predict for the tokens shifted by one (without the token). And the reason we look at logits except for last one is the one value it predicts after is non-sensical, so we simply drop it.