Eleven AI models in twenty days! Releases have outrun anyone’s ability to test them

Eleven new AI fashions have shipped this month, from seven completely different suppliers. The checklist is price studying in sequence, as a result of the tempo is the purpose.

Meta launched Muse Spark 1.2 on August 6. xAI launched Grok 4.6 the identical day, then Grok Think about Picture 2.0 on August 8. ByteDance launched Seed 2.1 Turbo on August 10. Google launched Gemini 3.7 Flash on August 13, three weeks after Gemini 3.6 Flash. Z.AI launched GLM-5.3 on August 14.

That’s six in 9 days, from 4 international locations, spanning textual content, picture and video technology. Late July had already produced Moonshot’s Kimi K3, a 2.8-trillion-parameter open-weight mannequin, and the month earlier than carried GPT-5.6 and Claude Opus 5.

Evaluating a frontier mannequin correctly takes weeks. Impartial benchmarking organisations need to receive entry, run standardised suites, verify for contamination, and publish. Purple-teaming for safety-relevant behaviour takes longer, as a result of the fascinating failures are uncommon and need to be hunted.

A 3-week hole between a mannequin and its successor is shorter than that course of. Gemini 3.7 Flash arrived earlier than most unbiased evaluation of Gemini 3.6 Flash had been revealed — which implies the comparability being drawn is between a brand new mannequin’s vendor-reported numbers and an previous mannequin’s vendor-reported numbers.

The result’s a market the place buying choices are made on self-reported benchmarks, as a result of nothing else is obtainable in time.

Why Self-Reported Numbers Are A Downside

This isn’t an accusation of fabrication. The difficulty is extra mundane and tougher to repair.

A laboratory selecting which benchmarks to publish is selecting those its mannequin does effectively on. Each laboratory does this, and each laboratory is telling the reality concerning the numbers it prints. The distortion is within the choice, not the arithmetic.

Benchmark contamination compounds it. Fashions educated on internet-scale knowledge could have absorbed the take a look at units, and detecting that requires entry the seller doesn’t all the time grant. Impartial analysis exists exactly to catch these results, and unbiased analysis is what the discharge schedule is outpacing.

There may be latest precedent for a way badly this may go. Meta confronted sustained criticism over benchmark presentation earlier this 12 months, and the episode demonstrated that the hole between a broadcast rating and a person’s expertise may be substantial sufficient to develop into a company disaster.

The compression will not be arbitrary. It displays the place the competitors has moved.

Gemini 3.7 Flash gained sixteen factors on DeepSWE v1.1 in three weeks — a leap inconsistent with a brand new base mannequin and fully per post-training refinement of an present one. A lot of what’s being shipped as a brand new launch is a brand new fine-tune, which is genuinely quicker to provide and genuinely cheaper to serve.

In the meantime the differentiators being marketed have shifted from functionality to supply. Google led on output pace and introductory value. OpenAI answered on the identical day with a quicker serving tier. Moonshot competes on open weights and self-hosting. These are distribution arguments, not intelligence arguments.

When competitors strikes to distribution, launch frequency turns into a advertising and marketing instrument in itself. Delivery usually indicators momentum to clients and buyers no matter what every particular person launch accommodates.

For anybody choosing a mannequin, the sensible response is unglamorous.

Deal with vendor benchmarks as a declare concerning the vendor’s priorities moderately than a measure of health. The place unbiased numbers exist, weight them extra closely even when they’re a technology behind. Run an inside analysis on the precise activity, which is the one benchmark that displays the workload in query and the one one that can not be gamed by another person.

And observe which distributors publish analysis methodology alongside outcomes, and which publish solely outcomes. That distinction has develop into extra informative than the scores.

Eleven fashions in twenty days is a exceptional engineering achievement throughout the trade. Additionally it is extra new software program than any buyer can responsibly assess, arriving quicker than the establishments constructed to evaluate it may work — and that hole is being crammed, at current, by the sellers.