Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier

This mid August is as hot as July when it comes to LLM releases: Grok 4.6, Gemini 3.7 Flash, DeepSeek V4 Pro 0813 as well as open-weight models: Qwen3.8 Max, Qwen3.8 27B, and GLM-5.3.

Since these models are smart, I decided to rerun the Baba Is Bench, to see how the models fare on a puzzle game. Even though the game is popular, we check for spoilers - and to our surprise, there are no signs of models knowing solutions ahead - unlike in SWE-bench Verified.

Results are stunning for Grok, Gemini Flash or DeepSeek. Don’t get deceived by minor version changes - they all give qualitatively different results! For both GLM and Qwen the progress was more gradual.

As previously, we do it in two rounds. First, asking to play the initial stage the Intro, then for those who pass at least 7/8 levels, also the next one the Lake.

Stage 0: The Intro

We show the models we analyze, against dimmed previous ones.

The results are nothing short of stunning, as both results went up and prices went down significantly.

• Gemini 3.7 Flash now solves every trial, and is over 20x cheaper than its predecessor, Gemini 3.6 Flash.
• Grok 4.6 solved all but one attempt (vs Grok 4.5 that missed 6 attempts and one level), while being over 3x cheaper.
• DeepSeek V4 Pro 0813 solved 7/8 levels vs its previous version that solved only 3/8 - and while being 4x cheaper.

With the GLM and Qwen family there is less success:

• GLM-5.3 got exactly the same result as GLM-5.2.
• Qwen3.8 Max is better than Qwen3.7 Max (way more attempts solved), but still far from completing all levels.
• Qwen3.8 27B has worse results than Qwen3.6 27B - though, all within experimental noise.

I was surprised that it didn’t do better. Especially as Qwen3.8 27B scores 52 on Artificial Analysis. From what I’ve seen in the transcripts, it tried to solve in one go, without actually exploring the board. It matches Simon Willison’s experience that Qwen3.8 27B defaults to wildly overthinking things. While Baba Is You clearly needs thinking, it is a balance of thinking and testing. Compare and contrast with previous Grok and Gemini Flash models that were doing a lot and thinking too little.

It is interesting to look at the chart of agent turns and price. Grok and Gemini Flash moved from trigger-happy models, to more moderate ones, resulting in a significantly reduced cost.

Stage 1: The Lake

For models that passed at least 7 out of 8 intro levels, we run them on the next stage, the Lake. Only one attempt per level, as it is costly. Or at least - it used to be costly with the previous models, as now it is getting significantly better.

Time

First, let’s look at the wall time, for models run via APIs. While it might change with providers, or model bandwidth at a given time, it gives a ballpark estimate of effective model speed. In everyday software engineering it is what we care for, anyway.

And wow, Gemini 3.7 Flash is fast - not only per token (which is widely cited but largely irrelevant), but per problem solved. Here, for most levels, only Claude Fable 5 is faster.

Grok 4.6, while 2x slower than GPT-5.6 Sol, still manages to solve 13 of 15 levels (GPT-5.6 Sol solves 14). It’s a promotion to the league of top models.

DeepSeek V4 Pro 0813 might be the slowest of the top models - but it is the first open-weight model that finally reached the same top league. It is way beyond GLM-5.2 or Kimi K3 results.

GLM-5.3 was almost the same as GLM-5.2, to the point that two lines overlap; though, it solved one more level.

Cost

Now let’s look at costs. It’s beautiful.

Grok 4.6 is cheaper than Fable 5, but more expensive than GPT-5.6 Sol or Opus 5.

While Gemini 3.6 Flash was ridiculously expensive, Gemini 3.7 Flash is one of the cheapest models. At 12 levels, it is twice as cheap as GPT-5.6 Sol, the previous cheapest model. At 13 tasks it is still twice as cheap. Only vs DeepSeek Pro is it slightly more expensive.

What happens with DeepSeek V4 Pro 0813 is a pure wonder. Not only is it open-weight, but also the cheapest of all.

GLM-5.3 was 2x cheaper than its predecessor.

Some lessons about harnesses

In previous experiments we used various harnesses. Here we used Harbor-default Terminus-2, except DeepSeek, which we ran on Claude Code. While there are Grok Build and DeepSeek Harness, results were slightly worse (Grok) or vastly inferior (DeepSeek). That said, these are newer tools, and likely to get improved in the future. If there is interest, we’re happy to share more insight on running with various harnesses — they influence both cost and results.

Conclusion

Progress is fast - with more players joining the frontier game, and open-weight models reaching frontier-level results at a much lower price. And I am looking forward to seeing more models in the coming months. And to the first local model solving the Lake.

As my colleague Piotr Grabowski noticed:

“Second-tier labs caught up with the frontier” matches my experience: last few days I’ve been using Grok 4.6 or GLM-5.3 for easy/medium difficulty tasks and couldn’t see major difference between them and Opus 5/GPT-5.6 Sol (for harder tasks yes, there’s still gap).

And what is your experience?