# Roberta vs Bart perplexity calcuation **URL:** \ **Category:** Models\ **Created:** [February 4, 2022, 7:35pm UTC](https://discuss.huggingface.co/t/roberta-vs-bart-perplexity-calcuation/14419 "2022-02-04T19:35:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![bivouac0](https://avatars.discourse-cdn.com/v4/letter/b/3d9bf3/32.png) [@bivouac0](https://discuss.huggingface.co/u/bivouac0)\ **Post date:** [February 4, 2022, 7:35pm UTC](https://discuss.huggingface.co/t/roberta-vs-bart-perplexity-calcuation/14419/1 "2022-02-04T19:35:52Z")
I need to create a perplexity score for Bart and I’ve looked at a few examples (ie… [https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling) run\_mlm.py for roberta) Unfortunately this doesn’t work for Bart and gives very high perplexity scores. The root of the issue seems to be the use of -100 ids for all non-masked label tokens. The only way I have found around this is to keep the labels id as tokenized (no -100 masking) and then manually extract the logits for the specific mask locations and do a little math. This gives me about a 20 perplexity which seems high considering Roberta gives about 6 with the code from the link above. Anyone know of code to do this calculation for Bart or have advice on the simple/correct way to do it? ---
**Author:** ![bivouac0](https://avatars.discourse-cdn.com/v4/letter/b/3d9bf3/32.png) [@bivouac0](https://discuss.huggingface.co/u/bivouac0)\ **Post date:** [February 5, 2022, 1:54pm UTC](https://discuss.huggingface.co/t/roberta-vs-bart-perplexity-calcuation/14419/2 "2022-02-05T13:54:27Z")
Looks like the trick is to pass in manually created `decoder_input_ids` to the model. If these aren’t passed in Bart creates them from `labels` and since most of those are -100, that messes up the decoding process. Also note that I think the run\_mlm.py script isn’t correctly placing the bos/eos tokens. To get Bart to score properly I had to tokenize, segment for length and then manually add these tokens back into each batch sequence. ---
**Author:** ![Yiping](https://avatars.discourse-cdn.com/v4/letter/y/ccd318/32.png) [@Yiping](https://discuss.huggingface.co/u/Yiping)\ **Post date:** [February 11, 2022, 3:24am UTC](https://discuss.huggingface.co/t/roberta-vs-bart-perplexity-calcuation/14419/3 "2022-02-11T03:24:52Z")
I don’t have experience particularly calculating perplexity by hand for BART. Below is the code snippet I used for GPT-2. Setting all the padded tokens (or tokens you don’t want to include in the perplexity) to -100 works. ```auto tgt_len = claim_inp.input_ids.size(1) input_ids = torch.cat([evidence_inp.input_ids, claim_inp.input_ids], axis=-1).to(device) target_ids = input_ids.clone() # mask the evidence so they're not considered when calculating the perplexity target_ids[:, :-tgt_len] = -100 with torch.no_grad(): outputs = model(input_ids, labels=target_ids) ``` Also there’s a link from Huggingface that I found very useful: > **[Perplexity of fixed-length models](https://huggingface.co/docs/transformers/perplexity)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![bivouac0](https://avatars.discourse-cdn.com/v4/letter/b/3d9bf3/32.png) [@bivouac0](https://discuss.huggingface.co/u/bivouac0)\ **Post date:** [February 11, 2022, 1:30pm UTC](https://discuss.huggingface.co/t/roberta-vs-bart-perplexity-calcuation/14419/4 "2022-02-11T13:30:52Z")
Thanks for the info. Just FYI and for anyone else interested… What works for GPT-2 (and the solution in the fixed length models link) doesn’t work for Bart. With Bart you need to pass in the decoder\_input\_ids instead of letting the model create them from the labels. The code is something like… ```auto input_ids = [[bos] + sample + [eos] for sample in samples] decoder_input_ids = [[dst] + iids[:-1] for iids in input_ids] # shift_tokens_right ``` where samples are the tokenized and chunked (ie… length = 512) text without the bos/eos tokens added. You could then manually mask the input\_ids and create the ‘labels’ but there’s also a collator that will do this for you. You can just use… DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm\_probability=0.15). The output from the DataLoader will then have the randomly masked input\_ids and the labels with -100 in the appropriate locations. Pass in these and the decoder\_input\_ids to the model and use perplexity = math.exp(statistics.mean(losses)). Here’s the full code [Bart Token Level Perplexity](https://gist.github.com/bjascob/7cece1eb654301925f79eb567af097f8)