QuantDiff

Best Qwen3.5 9B quant for your GPU

11 quantized files of Qwen/Qwen3.5-9B (8.95B), each diffed against the BF16 original.

Qwen/Qwen3.5-9B (Apache-2.0). Hybrid architecture: 32 layers, of which 8 use full attention and 24 linear attention, so the KV cache is small. The GGUF quants are text-only (vision tower not included). Tool calls tested with thinking disabled.

Recommendation by VRAM

"Best" = smallest diff (lowest mean KLD) among files that fit fully on the GPU (file + 8192-token KV cache), with tool-call score as tie-breaker.

8GB GPU

Smallest diff that fits
Q5_K_M by bartowski
File size
6.62 GB
KLD (mean)
0.0093
Same top token
96.4%
Tool calls
92.5%
Generation
88 t/s full GPU
Qwen_Qwen3.5-9B-Q5_K_M.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 133 t/s, KLD 0.2071)

12GB GPU

Smallest diff that fits
Q8_0 by unsloth
File size
8.87 GB
KLD (mean)
0.0025
Same top token
98.4%
Tool calls
91.5%
Generation
66 t/s full GPU
Qwen3.5-9B-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 133 t/s, KLD 0.2071)

16GB GPU

Smallest diff that fits
Q8_0 by unsloth
File size
8.87 GB
KLD (mean)
0.0025
Same top token
98.4%
Tool calls
91.5%
Generation
66 t/s full GPU
Qwen3.5-9B-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 133 t/s, KLD 0.2071)

All Qwen3.5 9B quants

QuantPublisherSizeFitsKLD meanKLD p99Same topTool callsPP t/sGen t/sRun
Q8_0 unsloth 8.87 GB 12GB+ 0.0025 0.014 98.4% 91.5% 4985 66 settings & logs
Q6_K unsloth 6.95 GB 12GB+ 0.0043 0.032 97.5% 92.5% 4010 82 settings & logs
Q5_K_M bartowski imatrix 6.62 GB 8GB+ 0.0093 0.082 96.4% 92.5% 4687 88 settings & logs
UD-Q4_K_XL unsloth imatrix 5.56 GB 8GB+ 0.0206 0.225 94.4% 90.5% 4719 97 settings & logs
Q4_K_M bartowski imatrix 5.75 GB 8GB+ 0.0219 0.232 94.0% 92.0% 4652 98 settings & logs
IQ4_XS bartowski imatrix 5.12 GB 8GB+ 0.0332 0.345 92.9% 89.5% 5358 111 settings & logs
i1-IQ4_XS mradermacher imatrix 4.72 GB 8GB+ 0.0379 0.403 92.4% 89.5% 5327 114 settings & logs
Q4_K_M unsloth 5.29 GB 8GB+ 0.0462 0.475 92.4% 91.5% 4700 101 settings & logs
i1-Q4_K_M mradermacher imatrix 5.24 GB 8GB+ 0.0481 0.522 92.0% 92.0% 4707 101 settings & logs
IQ3_M bartowski imatrix 4.64 GB 8GB+ 0.0899 0.991 88.0% 88.0% 4903 112 settings & logs
UD-Q2_K_XL unsloth imatrix 3.84 GB 8GB+ 0.2071 2.178 81.4% 82.0% 3932 133 settings & logs

GB = GiB (1024³ bytes). Speeds from llama-bench on Ghost (RTX 5070 12GB, Ryzen 7 5800X, 64GB DDR4). On a different GPU, expect generation speed to scale roughly with memory bandwidth.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Per-second GPU pods. RTX 4090 / 5090 / A100 / H100 for big quants or BF16 reference runs.

Link coming soon
Vast.ai

Marketplace GPUs, often the cheapest way to try a 24-48GB card for an evening.

Link coming soon
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff