Using huggingface transformers library, I see different outputs when generating text with model.generate with and without use_cache argument. Is this intended and how can I combat this? The scores when I use cache (from the second token generated and onwards) are different. AFAIK use_cache is an op…

Similar case? [image] Transformer KV-Cache Produces Worse Output Than Normal Generation – Why? Beginners Hi everyone, I implemented a transformer model from scratch based on Andrej Karpathy’s video. After that, I added some improvements, such as parallelized multi-head at…

This case talks about a problem with his custom implementation. I am facing a problem with the official model.generate function in huggingface transformers.

I found a cause. Possible Bug with KV Caching in Llama (original) model opened 07:50PM - 09 Aug 23 UTC (UTC) closed 08:07AM - 09 Jan 24 UTC (UTC) [image] maximkha …

Model.generate use_cache=True generates different results than use_cache=False

Intermediate

John6666 March 4, 2025, 1:24pm 2

Similar case?

Topic		Replies	Views
Outputs change if re-using KVCache (past_key_values) for model.forward and generation 🤗Transformers	4	652	January 22, 2025
Transformer KV-Cache Produces Worse Output Than Normal Generation – Why? Beginners	1	625	March 3, 2025
generate() produces wrong output when past_key_values is an externally-built DynamicCache 🤗Transformers	1	73	July 29, 2026
What is the purpose of 'use_cache' in decoder? 🤗Transformers	5	25255	July 4, 2023
IndexError: index -1 is out of bounds for dimension 0 with size 0 🤗Transformers	3	324	November 7, 2025

Model.generate use_cache=True generates different results than use_cache=False

Related topics