RedditPP Mascot

RedditPP

r/unsloth • … • u/silenceimpaired

How to decide which quant to pick?

Which do you choose? *UD-Q2\_K\_XL* or *UD-IQ3\_XXS*? You would think that choosing a quant is as easy as selecting the largest bit rate that can run on your system. But then there is the designation Unsloth gives of XL / XXS...

I've seen a general rule of thumb which states, "Choose **XXS** when VRAM is extremely limited or maximum speed is prioritized; choose **XL** for the best balance of quality and size, as it retains more accuracy than medium or small variants without the massive file size of lossless formats."

That seems to challenge the age old advice to use a higher bitrate across the entire model. I assume it is stated in the context of the same bitrate *UD-Q2\_K\_XXS vs* *UD-Q2\_K\_XL* ... but I cannot find that qualifier anywhere.

What get's sacrificed with XXS models that isn't sacrificed with XL models? I wish at this point I could easily tell you. Alas, my interest is in a model that I can barely fit onto my computer (and many of you might not be able to at all - GLM-5.3-Flash-GGUF) so I cannot easily benchmark the two versions.

Bonus question... how likely is it I'll have to re-download GLM-5.3-Flash-GGUF if I want to use it with the mainline llama.cpp branch once GLM 5.3 Flash is fully supported there.

Anyone from Unsloth here with an easy answer? Has anyone taken the time to compare situations like this?

Whether you have an answer or not, thanks for visiting, and sorry I had none for you...

EDIT: There is a chart for Qwen 3.8 27b which seems to indicate the next bitrate up is better... not sure if it's better in all ways, but it was how I decided. Still no answer if downloading gigabytes of a model might be a waste of resources from Unsloth.

— 0