> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.