Can AI beat Baba Is You?

2
stages: 22
levels: 27
models: 75%
pass rate
Updated: 20 Aug 2026
Source: stared/baba-is-harbor

The benchmark

LLM agents play Baba Is You, a puzzle game where you win by rewriting the rules, compared on pass rate, speed, and cost. Levels and game logic run in the Harbor framework through a text interface. Tasks never name the game or the level. OpenAI mentioned this benchmark when discussing their ARC-AGI-3 results.

Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost? (16 Jul 2026)Kimi K3 is Open, Opus 5 is Good, DeepSeek V4 Flash is Cheap: LLMs on Baba Is You (31 Jul 2026)Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier (20 Aug 2026)

Stage 0: The Intro

8 levels, 3 attempts per model per level, Terminus-2 harness via OpenRouter, 60-minute limit per attempt.

Model	pass@1 ↓	pass@3 ↓	Turns ↓	Output tokens ↓	Total cost ↑
Gemini 3.7 Flash	100%	100%	53	22k	$5.38
Claude Opus 5	100%	100%	7	23k	$17.15
GPT-5.5	100%	100%	8	15k	$13.62
Claude Fable 5	100%	100%	6	29k	$40.98
GPT-5.6 Sol	100%	100%	11	10k	$11.91
Grok 4.6	96%	100%	25	52k	$14.01
GLM-5.3	96%	100%	13	47k	$6.20
Kimi K3	96%	100%	9	29k	$12.56
Claude Opus 4.8	96%	100%	35	46k	$63.14
GLM-5.2	96%	100%	12	75k	$6.20
Gemini 3.1 Pro	92%	100%	8	38k	$12.41
Gemini 3.6 Flash	88%	100%	92	82k	$124.31
Gemini 3.5 Flash	88%	100%	111	74k	$98.31
GPT-5.6 Terra	88%	100%	41	55k	$34.28
DeepSeek V4 Pro 0813	75%	88%	18	130k	$2.94
DeepSeek V4 Flash	75%	100%	55	190k	$2.82
Grok 4.5	75%	88%	86	68k	$50.66
DeepSeek V4 Flash 0731	75%	88%	22	156k	$1.16
Claude Sonnet 5	67%	88%	22	151k	$42.21
GPT-5.6 Luna	63%	88%	118	129k	$59.32
Qwen3.8 Max	58%	75%	9	66k	$10.75
Hy3	54%	75%	25	76k	$3.23
MiniMax M3	46%	63%	96	173k	$34.43
Qwen3.6 27B	29%	38%	24	82k	$5.59
DeepSeek V4 Pro	29%	38%	37	183k	$11.49
Qwen3.7 Max	25%	38%	21	132k	$16.98
Qwen3.8 27B	21%	38%	9	134k	$9.07

solvedtimeoutwrong answer

Models that solved every intro level at least once (pass@3 = 100%, one miss starred), by how many turns they took and what they cost:

Stage 1: The Lake

15 levels, including 2 bonus ones, run once per model with a dedicated harness and effectively no time limit. The Human line is a first-time playthrough by the Twitch streamer Vinny.

Model	Harness	Solved	Total time	Output tokens	Total cost
Claude Fable 5	Claude Code	14	2h 36m	10.7M	$53.25
Claude Opus 5	Claude Code	14	6h 46m	16.3M	$61.96
Gemini 3.7 Flash	Terminus-2	14	9h 11m	1023.1M	$55.76
GPT-5.6 Sol	Codex	14	9h 16m	16.4M	$31.42
GPT-5.5	Codex	13	8h 57m	50.1M	$102.08
Grok 4.6	Terminus-2	13	20h 16m	38.6M	$46.20
DeepSeek V4 Pro 0813	Claude Code	13	22h 7m	75.6M	$8.48
Claude Opus 4.8	Claude Code	12	9h 7m	49.9M	$107.76
GLM-5.3	Terminus-2	12	24h 57m	9.6M	$16.85
GLM-5.2	Claude Code	11	23h 41m	62.9M	$37.51
Kimi K3	kimi-cli	11	29h 34m	129.5M	$78.15
Gemini 3.1 Pro	Terminus-2	8	5h 58m	41.8M	$58.17
Gemini 3.5 Flash	Terminus-2	7	12h 35m	909.6M	$221.39
Gemini 3.6 Flash	Terminus-2	6	7h 46m	1118.1M	$269.48
DeepSeek V4 Flash 0731	Claude Code	6	24h 11m	92.8M	$4.31

solvedstoppedfailed

Run it yourself

The harness is open source at stared/baba-is-harbor. It extracts levels and the game engine from your local installation of Baba Is You, and needs uv and Docker:

uv tool install harbor
git clone [email protected]:stared/baba-is-harbor.git
cd baba-is-harbor
uv run extract.py --extract  # copy levels from your game installation
uv run scripts/generate_tasks.py
bash scripts/setup_tasks.sh  # bundle the game engine into each task

Then verify the setup with the scripted oracle agent (no API key needed) and run any model available on OpenRouter (-k is attempts per level):

harbor run -p tasks/ -i stage-00-level-00 -a oracle  # expects reward 1.0
export OPENROUTER_API_KEY=...
harbor run -p tasks/ -i "stage-00-level-0[0-7]" -a terminus-2 \
  -m openrouter/moonshotai/kimi-k3 -k 3 -n 8 -y

Can AI beat Baba Is You?

The benchmark#

Stage 0: The Intro#

Stage 1: The Lake#

Run it yourself#

The benchmark

Stage 0: The Intro

Stage 1: The Lake

Run it yourself