To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175
Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
haven't tried it yet though.
You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.