Hugging Face's logo

RedHatAI
/
Qwen3.5-397B-A17B-FP8-dynamic

Image-Text-to-Text
Transformers
Safetensors
qwen3_5_moe
fp8
vllm
llm-compressor
compressed-tensors
conversational
Model card Files
xet
 Community
8

W4A16 quant

#1
by timroethig - opened
Discussion
timroethig

Thanks for providing these quants, are you by chance also working on a W4A16 quant? Could make a lot of sense for a sparse MoE model like this no?
ekurtic
 Red Hat AI org

yes, other quant schemes (int4 and fp4) are coming very soon
llmDau

thanks for your work.
hopefully i can run it on two h100 :)
aabbccddwasd

thanks for your work.
hopefully i can run it on two h100 :)

maybe you need gguf q3 or autoround w2a16🥲
traphix
•
edited Mar 2

yes, other quant schemes (int4 and fp4) are coming very soon

I'm very much looking forward to your INT4(w4a16) quantized model!

Or do you have a quantization Python script for Qwen3.5-397B that we can try ourselves?
traphix

Any progress?

Sign up or log in to comment