Add missing `torch_dtype` in commented snippet in `README.md`
#9
by
alvarobartt HF Staff - opened
Without setting the torch_dtype the snippet will fail with RuntimeError: FlashAttention only support fp16 and bf16 data type, as it defaults to torch.float32.
@Qwen Please merge this is needed to make it work
Still not merged and I've also encountered this issue recently...😔
You can also set it to "torch_dtype":"auto"
this will reference config.json and load it as bfloat16