QuantDiff

Best Qwen3.5 4B quant for your GPU

8 quantized files of Qwen/Qwen3.5-4B (4.21B), each diffed against the BF16 original.

Qwen/Qwen3.5-4B (Apache-2.0). Same hybrid design as the 9B (8 of 32 layers full attention). Text-only GGUF quants. Tool calls tested with thinking disabled.

Recommendation by VRAM

"Best" = smallest diff (lowest mean KLD) among files that fit fully on the GPU (file + 8192-token KV cache), with tool-call score as tie-breaker.

8GB GPU

Smallest diff that fits
Q8_0 by unsloth
File size
4.17 GB
KLD (mean)
0.0019
Same top token
98.3%
Tool calls
86.5%
Generation
114 t/s full GPU
Qwen3.5-4B-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 187 t/s, KLD 0.2425)

12GB GPU

Smallest diff that fits
Q8_0 by unsloth
File size
4.17 GB
KLD (mean)
0.0019
Same top token
98.3%
Tool calls
86.5%
Generation
114 t/s full GPU
Qwen3.5-4B-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 187 t/s, KLD 0.2425)

16GB GPU

Smallest diff that fits
Q8_0 by unsloth
File size
4.17 GB
KLD (mean)
0.0019
Same top token
98.3%
Tool calls
86.5%
Generation
114 t/s full GPU
Qwen3.5-4B-Q8_0.gguf

Fastest that fits: UD-Q2_K_XL (unsloth, 187 t/s, KLD 0.2425)

All Qwen3.5 4B quants

QuantPublisherSizeFitsKLD meanKLD p99Same topTool callsPP t/sGen t/sRun
Q8_0 unsloth 4.17 GB 8GB+ 0.0019 0.011 98.3% 86.5% 8442 114 settings & logs
Q6_K bartowski imatrix 3.54 GB 8GB+ 0.0037 0.024 97.1% 86.5% 7188 130 settings & logs
UD-Q4_K_XL unsloth imatrix 2.71 GB 8GB+ 0.0220 0.191 93.3% 85.0% 7928 151 settings & logs
Q4_K_M bartowski imatrix 2.81 GB 8GB+ 0.0258 0.213 92.8% 86.5% 7774 152 settings & logs
Q4_K_M unsloth 2.55 GB 8GB+ 0.0308 0.251 92.4% 87.0% 7917 156 settings & logs
i1-Q4_K_M mradermacher imatrix 2.52 GB 8GB+ 0.0339 0.314 91.8% 89.5% 7785 158 settings & logs
IQ3_M bartowski imatrix 2.31 GB 8GB+ 0.1002 0.930 86.3% 87.5% 8197 161 settings & logs
UD-Q2_K_XL unsloth imatrix 1.81 GB 8GB+ 0.2425 2.036 78.6% 76.0% 6685 187 settings & logs

GB = GiB (1024³ bytes). Speeds from llama-bench on Ghost (RTX 5070 12GB, Ryzen 7 5800X, 64GB DDR4). On a different GPU, expect generation speed to scale roughly with memory bandwidth.

GPU too small? Rent one.

Run a 32B model at Q6/Q8, or a 70B, on a cloud GPU for an hour or two instead of buying hardware.

RunPod

Per-second GPU pods. RTX 4090 / 5090 / A100 / H100 for big quants or BF16 reference runs.

Link coming soon
Vast.ai

Marketplace GPUs, often the cheapest way to try a 24-48GB card for an evening.

Link coming soon
Your GPU cloud here

Sponsor slot. Sponsors never influence scores or rankings.

Sponsor QuantDiff