Zen0 commited on
Commit
31cbca7
·
1 Parent(s): f47aa21

Fix model generation: disable KV cache to avoid DynamicCache error

Browse files

Root cause of 0% accuracy found!
All predictions were empty because model.generate() was throwing:
AttributeError: 'DynamicCache' object has no attribute 'seen_tokens'

This is a transformers version incompatibility with Phi-3 models.

Solution: Add use_cache=False to model.generate() to disable KV cache.
This avoids DynamicCache entirely and allows generation to succeed.

Side effect: Slightly slower generation (no KV cache), but models will work.

Files changed (1) hide show
  1. app.py +2 -1
app.py CHANGED
@@ -287,7 +287,8 @@ def evaluate_single_model(model_name, tasks, use_4bit=True, temperature=0.7, max
287
  temperature=temperature,
288
  do_sample=True,
289
  top_p=0.9,
290
- pad_token_id=tokenizer.eos_token_id
 
291
  )
292
 
293
  if i == 0:
 
287
  temperature=temperature,
288
  do_sample=True,
289
  top_p=0.9,
290
+ pad_token_id=tokenizer.eos_token_id,
291
+ use_cache=False # Disable KV cache to avoid DynamicCache compatibility issues
292
  )
293
 
294
  if i == 0: