# Benchmarking LLMs

**URL:** <https://discuss.huggingface.co/t/benchmarking-llms/61007>\
**Category:** 🤗Transformers\
**Created:** [November 4, 2023, 1:16am UTC](https://discuss.huggingface.co/t/benchmarking-llms/61007 "2023-11-04T01:16:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![thomas-ferraz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomas-ferraz/32/20894_2.png) [@thomas-ferraz](https://discuss.huggingface.co/u/thomas-ferraz)\
**Post date:** [November 4, 2023, 1:16am UTC](https://discuss.huggingface.co/t/benchmarking-llms/61007/1 "2023-11-04T01:16:09Z")

</div>

Hi, I am new to decoder-only models (often referred as LLMs). Can someone point me on a code example for the correct way to benchmark a LLM (that is on hugging face hub) on a dataset (also on the hub)? Like how is it run the Open LLM leaderboard? The classical way of batching with data collator (that works on encoder-decoder models) is running into some problems because of batching with padding. As decoder-only models are casual models, it makes no sense to add pads, right? I tested with Llama 2 and Mistral and if i add pad before the answer it changes the answer. So now I am running with test batch size 1.

---

<div class="post-metadata">

**Author:** ![hubsnippetai](https://avatars.discourse-cdn.com/v4/letter/h/e495f1/32.png) [@hubsnippetai](https://discuss.huggingface.co/u/hubsnippetai)\
**Post date:** [August 20, 2024, 3:59am UTC](https://discuss.huggingface.co/t/benchmarking-llms/61007/2 "2024-08-20T03:59:41Z")

</div>

AI generated:  
Benchmarking a decoder-only model on a dataset on Hugging Face Hub can be a bit tricky due to the casual nature of these models. I’d be happy to guide you through a correct way to do it.

One popular approach to benchmarking LLMs is to use the [OLDS (Open LLMs Benchmarking System)](https://huggingface.co/transformers/main_classes/oldevaluation/evaluation_tasks.html#oldevaluation-open-llevaluation-benchmarking-system). However, you can also use the Hugging Face library’s integration with the [tensorflow benchmark] to run the benchmark directly.

Here’s a straightforward example of how to benchmark a decoder-only model (e.g., Llama 2 or Mistral) on a dataset (also on the Hugging Face Hub):

```python
# You need to download the dataset before running the benchmark
from datasets import load_dataset, load_metric
import transformers

dataset_name = 'your_dataset_name'
dataset = load_dataset('your_dataset_name')

model_name = 'the name of the model (e.g. llaama2/7base or mrgm903/mistral-3.2b)'
model = AutoModelForCausalLM.from_pretrained(model_name)

batch_size = 1 # As you mentioned, this is a good starting point for causal models

# Benchmarking loop
metric = load_metric('accuracy')
for batch in dataset['train'].batched(batch_size=batch_size):
    input_ids = batch['input_ids']
    with torch.no_grad():
        outputs = model(input_ids=input_ids)
        logits = outputs.last_hidden_state
        predictions = torch.argmax(logits, dim=2)
        metric.add jednary_results(predictions, batch['label'])
    torch.cuda.empty_cache()

print(metric.compute())

```

Another approach is using [Open LLM Evaluation (OLDEVAL)], which uses techniques such as **fast text decoding and subtraction of futures bias** (essentially equivalent to OpenAI’s [model riggeresteios](https://arxiv.org/abs/2101.11118) in different optimization objective) to provide a more accurate way to benchmark fast-, slow, 1-for-all and max-per