# Train Bart for Conditional Generation (e.g. Summarization) **URL:** \ **Category:** Models\ **Created:** [November 5, 2020, 3:45pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904 "2020-11-05T15:45:54Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\ **Post date:** [November 5, 2020, 3:45pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/1 "2020-11-05T15:45:54Z")
Hi everybody I ran into some issues when trying to fine-tune bart for summarization using the `BartForConditionalGeneration` model. The issue evolved around properly masking and ignoring the padding tokens when training. Without the following fix the loss went down but the model produced bad summaries. I post the solution here in case anyone else runs into similar problems. ```auto from transformers import BartTokenizer, BartForConditionalGeneration from transformers import Trainer, TrainingArguments from transformers.modeling_bart import shift_tokens_right dataset = ... # some Datasets object with train/validation split and columns 'text' and 'summary' tokenizer = BartTokenizer.from_pretrained('facebook/bart-large') model = BartForConditionalGeneration.from_pretrained('facebook/bart-large') def convert_to_features(example_batch): input_encodings = tokenizer.batch_encode_plus(example_batch['text'], pad_to_max_length=True, max_length=1024, truncation=True)) target_encodings = tokenizer.batch_encode_plus(example_batch['summary'], pad_to_max_length=True, max_length=1024, truncation=True)) labels = target_encodings['input_ids'] decoder_input_ids = shift_tokens_right(labels, model.config.pad_token_id) labels[labels[:, :] == model.config.pad_token_id] = -100 encodings = { 'input_ids': input_encodings['input_ids'], 'attention_mask': input_encodings['attention_mask'], 'decoder_input_ids': decoder_input_ids, 'labels': labels, } return encodings dataset = dataset.map(convert_to_features, batched=True) columns = ['input_ids', 'labels', 'decoder_input_ids','attention_mask',] dataset.set_format(type='torch', columns=columns) training_args = TrainingArguments( output_dir='./models/bart-summarizer', num_train_epochs=1, per_device_train_batch_size=1, per_device_eval_batch_size=1, warmup_steps=500, weight_decay=0.01, logging_dir='./logs', ) trainer = Trainer( model=model, args=training_args, train_dataset=dataset['train'], eval_dataset=dataset['validation'] ) ``` The `convert_to_features` function makes sure that the decoder inputs are correctly shifted and still include the padding tokens while in the labels the padding tokens are replaced by -100 such they are ignored in the model loss. ---
**Author:** ![Jeremias](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@Jeremias](https://discuss.huggingface.co/u/Jeremias)\ **Post date:** [January 18, 2021, 1:30pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/2 "2021-01-18T13:30:43Z")
Hi, I tried to use your fix but I’m wondering whether this is still up-to-date: `transformers.modeling_bart` doesn’t exist. There is only `transformers.models.bart.modeling_bart` but its `shift_tokens_right` function requires a `torch.Tensor` object while huggingface’s `datasets` object only consists of lists (plus it needs an additional `decoder_start_token_id`). This also leads to an error in `labels[labels[:, :] == model.config.pad_token_id] = -100` because this is numpy syntax. Also, `batch_encode_plus` is deprecated. It would be nice if there was a statement from huggingface if this is still necessary and if not, could it please be removed from the BART page? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [January 19, 2021, 5:57am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/3 "2021-01-19T05:57:49Z")
Hi @Jeremias In recent versions all models now live under their own dir, so bart is now in `models.bart` > huggingface’s `datasets` object only consists of lists `datasets` can return any type (`list`, `numpy array`, `torch tensor`, `tf tensor`), by default it returns `list`, you need to explicitly set the format for it to return tensors, it’s explained in the datasets intro [colab](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb#scrollTo=NzOXxNzQvSVo), also, you won’t need to manually call `shift_tokens_right` to prepare `decoder_input_ids`, if you just pass `labels` the model will prepare the `decoder_input_ids` by correctly shifting them. We have Bart training examples in `examples/seq2seq` [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) , which should help you fine-tune bart. Hope this helps. ---
**Author:** ![Kusuma](https://avatars.discourse-cdn.com/v4/letter/k/8797f3/32.png) [@Kusuma](https://discuss.huggingface.co/u/Kusuma)\ **Post date:** [May 16, 2021, 12:55pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/4 "2021-05-16T12:55:10Z")
HI @valhalla , I am searching for examples to train Bart and not unable to access the link mentioned here. Could you please share the latest one. Thanks ---
**Author:** ![niksqwerty](https://avatars.discourse-cdn.com/v4/letter/n/74df32/32.png) [@niksqwerty](https://discuss.huggingface.co/u/niksqwerty)\ **Post date:** [September 16, 2021, 4:57am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/5 "2021-09-16T04:57:54Z")
> [@lvwerra](#): > > ```auto > training_args = TrainingArguments( > output_dir='./models/bart-summarizer', > num_train_epochs=1, > per_device_train_batch_size=1, > per_device_eval_batch_size=1, > warmup_steps=500, > weight_decay=0.01, > logging_dir='./logs', > ) > > ``` As per the documentation, labels field is only for “Labels for computing the masked language modeling loss.”, Not specifying ‘labels’ doesn’t provide the loss values, how to compute loss if I want to train only for text summarization? I am sending encoder\_input\_ids as paragraph, decoder\_input\_ids as summary also adding attention\_mask\_target for both. ---
**Author:** ![Deema](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deema/32/40150_2.png) [@Deema](https://discuss.huggingface.co/u/Deema)\ **Post date:** [November 8, 2021, 8:58pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/6 "2021-11-08T20:58:14Z")
Hi, ellipsis’ object has no attribute ‘map’, what to do in this case? best, ---
**Author:** ![Zhengyao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zhengyao/32/3372_2.png) [@Zhengyao](https://discuss.huggingface.co/u/Zhengyao)\ **Post date:** [December 5, 2021, 7:00am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/7 "2021-12-05T07:00:27Z")
You can ignore the specifying labels index when initial the loss function. ---
**Author:** ![vikasy95](https://avatars.discourse-cdn.com/v4/letter/v/e47c2d/32.png) [@vikasy95](https://discuss.huggingface.co/u/vikasy95)\ **Post date:** [June 27, 2022, 8:01pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/8 "2022-06-27T20:01:12Z")
> [@valhalla](#): > > here CC’ing @lvwerra @Jeremias Just checking if anyone can post an example of the dataset. The above example says: dataset = … # some Datasets object with train/validation split and columns ‘text’ and ‘summary’ so do we have to provide dataset = {‘text’:[list of all the text,…], ‘summary’:[list of all the corrsponding summaries]} Thanks, -V ---
**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\ **Post date:** [June 28, 2022, 8:28am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/9 "2022-06-28T08:28:35Z")
You could for example use the DailyMail/CNN dataset: [cnn\_dailymail · Datasets at Hugging Face](https://huggingface.co/datasets/cnn_dailymail) ```python from datasets import load_dataset dataset = load_dataset("cnn_dailymail") ``` Note that the columns in this dataset are called `"article"` and `"highlights"`. ---
**Author:** ![Tiezheng](https://avatars.discourse-cdn.com/v4/letter/t/b487fb/32.png) [@Tiezheng](https://discuss.huggingface.co/u/Tiezheng)\ **Post date:** [October 25, 2022, 10:43pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/10 "2022-10-25T22:43:51Z")
Hi @lvwerra Can you remember how bad is it without the preprocessing you post? For example, how many points of ROUGE-1 drop? ---
**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\ **Post date:** [November 22, 2022, 3:31pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/11 "2022-11-22T15:31:08Z")
I cannot, sorry 🙂 Maybe 5-10% but I am just guessing. ---
**Author:** ![Tiezheng](https://avatars.discourse-cdn.com/v4/letter/t/b487fb/32.png) [@Tiezheng](https://discuss.huggingface.co/u/Tiezheng)\ **Post date:** [November 22, 2022, 5:45pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/12 "2022-11-22T17:45:45Z")
Thanks for sharing 😆 ---
**Author:** ![maciejskorski](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maciejskorski/32/57000_2.png) [@maciejskorski](https://discuss.huggingface.co/u/maciejskorski)\ **Post date:** [June 27, 2023, 9:29pm UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/13 "2023-06-27T21:29:03Z")
The shared example doesn’t work anymore (not because of typos). I think this adapted code snippet can do the job (but review params before use carefully!) ```python from transformers import AutoTokenizer, BartForConditionalGeneration from transformers.generation import GenerationConfig from transformers import Trainer, TrainingArguments from transformers.models.bart.modeling_bart import shift_tokens_right from transformers import DataCollatorForSeq2Seq model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn", attention_dropout=0.1) tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn") seq2seq_data_collator = DataCollatorForSeq2Seq(tokenizer, model=model) def get_features(batch): input_encodings = tokenizer(batch["text"], max_length=1024, truncation=True) with tokenizer.as_target_tokenizer(): target_encodings = tokenizer(batch["summary"], max_length=256, truncation=True) return {"input_ids": input_encodings["input_ids"], "attention_mask": input_encodings["attention_mask"], "labels": target_encodings["input_ids"]} dataset_ftrs = dataset.map(get_features, batched=True) columns = ['input_ids', 'labels', 'input_ids','attention_mask',] dataset_ftrs.set_format(type='torch', columns=columns) training_args = TrainingArguments( output_dir='./models/bart-summarizer', num_train_epochs=1, per_device_train_batch_size=1, per_device_eval_batch_size=1, warmup_steps=500, weight_decay=0.01, logging_dir='./logs', ) model.config.output_attentions = True model.config.output_hidden_states = True training_args = TrainingArguments( output_dir='./models/bart-summarizer', num_train_epochs=1, warmup_steps=500, per_device_train_batch_size=1, per_device_eval_batch_size=1, weight_decay=0.01, logging_steps=10, push_to_hub=False, evaluation_strategy='steps', eval_steps=500, save_steps=1e6, gradient_accumulation_steps=16, ) trainer = Trainer( model=model, args=training_args, tokenizer=tokenizer, data_collator=seq2seq_data_collator, train_dataset=dataset_ftrs["train"], eval_dataset=dataset_ftrs["test"], ) trainer.train() ``` ---
**Author:** ![Raghavender](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@Raghavender](https://discuss.huggingface.co/u/Raghavender)\ **Post date:** [October 4, 2023, 5:14am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/14 "2023-10-04T05:14:40Z")
Hi I tried to use above code but facing below error. Can someone advise Below part of the code is raising error dataset\_ftrs = dataset.map(get\_features, batched=True) ## Error details: Map: 0% 0/287113 [00:00\() ----\> 1 dataset\_ftrs = dataset.map(get\_features, batched=True) 8 frames /usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py in **getitem** (self, key) 268 269 def **getitem** (self, key): → 270 value = self.data[key] 271 if key in self.keys\_to\_format: 272 value = self.format(key) KeyError: ‘text’ ---
**Author:** ![lele](https://avatars.discourse-cdn.com/v4/letter/l/e5b9ba/32.png) [@lele](https://discuss.huggingface.co/u/lele)\ **Post date:** [November 22, 2023, 8:51am UTC](https://discuss.huggingface.co/t/train-bart-for-conditional-generation-e-g-summarization/1904/15 "2023-11-22T08:51:34Z")
Hello! In the code above, it seems that you did not apply padding to the source and target. What difference would it make compared to the results when padding is used?