K-Quant subjective quality

[LostRuins]

So I can see from the perplexity scores that the K-quants generally seem to perform more size efficiently compared to earlier formats.

But I wanna see if anyone else agrees - outputs from Q4_K_M and Q4_K_S actually feel worse than Q4_0 subjectively despite having a better perplexity score.

I dunno if I am dreaming
my responses from q4_0 just... feel? better?

Just wanna hear some anecdotes on this.

[KerfuffleV2]

How do the high quality non-k-quants feel? Like q5_1, q8_0?

Maybe you just like LLaMAs that are a bit buzzed.

[LostRuins]

I used to use q5_1 previously and it definitely does feel better for some reason.

[mounta11n]

Hmm, in my subjective experience, q4.0 performed best within the old quant method anyway, although higher perplexity than q5 and q8.. no idea why this is so.. and currently I haven't made so many comparisons yet, but subjectively I have the impression that q4km is about the same or even minimally better than q4.0

[KerfuffleV2]

Check out my post here: #406 (comment)

q4_k_m is lower perplexity than q5_0 for all models I have data except 13b (it's just a little worse in that case, but significantly smaller size). It's always lower perplexity than q4_0.

[mounta11n]

Oh interesting! Thanks for the link. So this confirms my subjective impression actually

[viperwasp]

For the first time I am switching from 5_1 to Q5K_M I think it's called. Does anyone still agree or disagree that Q5K_M should be better than 5_1? I care mostly about accuracy/perplexity. I heard that Q5K_M should give better prompts than the old 5_1. But be faster at generation than 5_1 as well? Now that people have had more time to subjectively test this? Any opinions in new quant methods?