Thanks for sharing. A nice addition would be a two axis graph showing the score and the VRAM required for model and say 128k KV cache (i know, the cache part differs wildly between models and also the impact of cache quantization affects them differently).
Other good graph would be score vs total time , or time per task.
Also take into account the harness impact - especially affected are dumber models. I remember seeing even 10% difference between say Claude Code and OpenCode for a good model.
I built my own local LLM benchmark and ended up benchmarking some of the most used local LLM configs. I learned a lot about their capabilities, how they compare and how quantization affects their performance.
Thanks for sharing. A nice addition would be a two axis graph showing the score and the VRAM required for model and say 128k KV cache (i know, the cache part differs wildly between models and also the impact of cache quantization affects them differently).
Other good graph would be score vs total time , or time per task.
Also take into account the harness impact - especially affected are dumber models. I remember seeing even 10% difference between say Claude Code and OpenCode for a good model.
Good suggestions on the graphs. I will add them. The total VRAM would be nice even though it is not a hard constraint on the MoE models.
Trying different harnesses is a different matter as it requires re-running the entire benchmark so it is not something that scales.
I built my own local LLM benchmark and ended up benchmarking some of the most used local LLM configs. I learned a lot about their capabilities, how they compare and how quantization affects their performance.
Great work!