QuantDiff

Best Qwen3.5 9B quant for a 24GB GPU

Short answer: Q8_0 from unsloth . Of the 11 Qwen3.5 9B files we tested that fit fully in 24GB with a 8192-token context, it has the smallest diff from the BF16 model (mean KLD 0.0025, same top token 98.4%).

Smallest diff that fits 24GB
Q8_0 by unsloth
File size
8.87 GB
KLD (mean)
0.0025
Same top token
98.4%
Tool calls
91.5%
Generation
66 t/s full GPU
Qwen3.5-9B-Q8_0.gguf
Runner-up
Q6_K by unsloth
File size
6.95 GB
KLD (mean)
0.0043
Same top token
97.5%
Tool calls
92.5%
Generation
82 t/s full GPU
Qwen3.5-9B-Q6_K.gguf
Fastest that fits
UD-Q2_K_XL by unsloth
File size
3.84 GB
KLD (mean)
0.2071
Same top token
81.4%
Tool calls
82.0%
Generation
133 t/s full GPU
Qwen3.5-9B-UD-Q2_K_XL.gguf

Other budgets: 8GB12GB16GB ยท all 11 Qwen3.5 9B quants

How we picked

We estimate VRAM need as file size + f16 KV cache for 8192 tokens + about 0.6GB of compute buffers, and keep 0.5GB free for your desktop. Among files that fit, we rank by mean KL divergence against the BF16 model's logits, then by tool-call score. If you need a longer context, step down one quant level or quantize the KV cache (-ctk q8_0 -ctv q8_0). Methodology.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Cloud GPU pods with 24-80GB cards (RTX 4090 / 5090 / A100 / H100) for big quants or BF16 reference runs.

Rent on RunPod
Vast.ai

GPU marketplace for renting a 24-48GB card for an evening of testing.

Rent on Vast.ai
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff

Referral links: QuantDiff may earn credit if you sign up. Referral partners have no say in scores or rankings.