Spaces:
Sleeping
Sleeping
Zen0
commited on
Commit
·
31cbca7
1
Parent(s):
f47aa21
Fix model generation: disable KV cache to avoid DynamicCache error
Browse filesRoot cause of 0% accuracy found!
All predictions were empty because model.generate() was throwing:
AttributeError: 'DynamicCache' object has no attribute 'seen_tokens'
This is a transformers version incompatibility with Phi-3 models.
Solution: Add use_cache=False to model.generate() to disable KV cache.
This avoids DynamicCache entirely and allows generation to succeed.
Side effect: Slightly slower generation (no KV cache), but models will work.
app.py
CHANGED
|
@@ -287,7 +287,8 @@ def evaluate_single_model(model_name, tasks, use_4bit=True, temperature=0.7, max
|
|
| 287 |
temperature=temperature,
|
| 288 |
do_sample=True,
|
| 289 |
top_p=0.9,
|
| 290 |
-
pad_token_id=tokenizer.eos_token_id
|
|
|
|
| 291 |
)
|
| 292 |
|
| 293 |
if i == 0:
|
|
|
|
| 287 |
temperature=temperature,
|
| 288 |
do_sample=True,
|
| 289 |
top_p=0.9,
|
| 290 |
+
pad_token_id=tokenizer.eos_token_id,
|
| 291 |
+
use_cache=False # Disable KV cache to avoid DynamicCache compatibility issues
|
| 292 |
)
|
| 293 |
|
| 294 |
if i == 0:
|