I participated in the Open Kernel Challenge, a competition to make Meta’s Muse Spark Open run faster, with $100,000 in prizes. With GPU kernels 2x faster than the original, I landed around 500th place. So I contacted the top 50 to learn the tricks from the best.
The challenge is run by Paradigm with Meta and Andromeda; Daniel Gross announced it as “a $100,000 challenge to see who can make Muse Spark run fastest”. For two weeks, about 2,000 people rewrote the GPU kernels behind Muse Spark Open’s inference, timed against SGLang. That was the kernel round. The second round, serving the whole model, is where the $100,000 goes, and it runs until Nov 19.
My naive weekend approach was yelling at Claude Code with Opus 5.5 while hiking in Pinnacles. It cost me $150 for an H200 rented through Vast.ai, plus the weekly limits of my Claude Max 20x and ChatGPT Team Premium plans. I threw all my credentials onto a box in Japan and hoped for the best. After the first weekend, I peaked at #22, and I could have climbed higher by allocating more compute and tokens.
Clearly, many people achieved better speedups than mine.
| Speedup over original | Decode layer | Projections | Attention | MoE | Prefill layer |
|---|---|---|---|---|---|
| #1 | 2.61× | 1.73× | 3.34× | 2.84× | 2.15× |
| #10 | 2.50× | 1.67× | 3.01× | 2.68× | 2.04× |
| #50 | 2.40× | 1.60× | 2.81× | 2.53× | 1.95× |
| Me (~500th) | 2.00× | 1.48× | 2.41× | 2.22× | 1.78× |
| Original (SGLang) | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× |
Leaderboard as of the kernel round’s close on Nov 5. The organizers are still reviewing submissions, so ranks may move.
On the decode layer, the whole top 50 fits within 9%. So on the last day of the kernel round, Nov 4, I asked the top 50 to share their setups. Twelve replied with their agents, models, GPUs and spend; a few of them slipped out of the top 50 later as the board moved.
How much you paid to play
Most of the out-of-pocket bill was GPU, because tokens came from flat-rate plans (Claude Max 20x, ChatGPT Pro, the $500 Codex plan) priced far below API rates. Shishir Pai’s model usage came to about $68k at API cost.
I estimated roughly the GPU cost and how far from the winner.
The effect of bigger spend on results is smaller than I thought. On inspection, I was the sucker paying $4 an hour for one H200 on Vast.ai, or $8 for two, while preemptible Nebius spot instances are often available for less than $1 an hour. Modal and Daytona offer more granular per-second billing, which can squeeze out more, as agents mostly think, plan and code, and need GPUs only to test.
Some reported that NVIDIA Nsight profiling helps a lot, but its kernel profiler, Nsight Compute, needs GPU performance counters. Only some platforms, such as Nebius, AWS and Google Cloud, fully support them; others, like some Lium machines, support them partially. Container platforms such as RunPod and Vast.ai block them.
The intelligence behind agents
Almost everyone used frontier models from Anthropic and OpenAI. Nobody ran on Fable 5.1: the one entrant who tried it dropped it for cost and speed.
- Claude only
- Claude and OpenAI models
- OpenAI only
- Not stated
- Claude Code
- Custom
- Codex
- Not stated
Even with a stock harness, most reports described several agents working together: sub-agents, a small parallel team or a council of reviewer models.
Four ran custom harnesses:
- Aleksey Smolenchuk (5%) launched a fresh Codex worker for every attempt, up to six in parallel, with one integrator combining the wins. Long-lived agents “would stall, repeat narrow experiments, or need nudging”.
- Shiyi Cao (4%) used a supervisor from Shiyi’s NeurIPS 2026 work that decides how many of up to six Opus 5.5 agents run in each iteration.
- Shishir Pai (6%) ran oh-my-pi sessions that spawn other sessions, about 1,100 threads in total, while an advisor agent watched threads, costs and the leaderboard and patched the shared rules live.
- Colin Snyder (9%) “built my own custom evolutionary search harness”.
Three of the four replies closest to #1 (4 to 6% behind) ran custom harnesses; the fourth didn’t say. There is an edge in custom, but customizing Claude Code/Codex can also get you far.
What we can learn from the top 50
On the harness and the loop
- “The biggest lesson for me was that the harness and evaluation loop mattered as much as the individual agents.” Aleksey Smolenchuk (5% behind #1)
- “The most impactful thing throughout the challenge was the ability to monitor and improve the harness live, based on any problem that occurred.” Shishir Pai (6% behind #1)
- “More agents didn’t automatically mean more progress. The hard part was turning their ideas into measured improvements.” UnderAndOver (12% behind #1)
- “I honestly don’t know every detail of what happens inside.” meganpark (9% behind #1), on Claude orchestrating its own sub-agents and Codex
On cost and compute
- “That phase cost about $4.8k in model usage, produced 86.5% of the total gain and brought me to #6 overall.” Then: “Model usage reached about $68k at API cost.” Shishir Pai
- “At the higher ranks I think it was just a matter of pure compute, both with agents and with GPU access.” Aamir Ahmed (8% behind #1)
On agents and expertise
- “You can get very far with no kernel knowledge […] Beyond a point though you still need kernel expertise.” George Pantazis (7% behind #1)
- “I found a lot of success by having the agent write some basic ‘proof’ kernels that established top-lines for mem bandwidth, compute, latency, etc.” Andrew Pullin (7% behind #1)
On models
- “Opus did most of the heavy lifting. Fable did not justify its output quality for the cost and speed. GPT Sol and Astra were weak at hill climbing for this challenge, but they did a good job reviewing code.” Shishir Pai
- “I tried Codex/Astra several times but it wasn’t effective on this challenge.” George Pantazis
And one I can’t verify
- “gpt-5.6 does noticeably better work if you randomly put bible quotes into your prompts. Serious.” Andrew Pullin
What I would do differently
Next time I would rent a preemptible Nebius H200, do all the work there and push to git, so it can resume elsewhere. I would install Nsight and every other perf tool I can find, and fetch all the knowledge the agents can use. Instead of one agent per problem, I would point a Claude Code swarm with distinct roles, Codex as one subagent, at one problem, then switch problems. I would stack several seat-based plans so tokens never run out, and keep it running for the whole competition, checking in from my phone.
Where is the alpha for humans
Agents will make a big difference in hardware choice. Many companies standardized on NVIDIA because porting kernels to another chip took scarce, expensive specialists. The training moat might stay, but the inference moat will shrink. Challengers compete on speed (d-Matrix, Cerebras) or price (AMD, Google TPUs, AWS Trainium). The largest labs are already on several chips: in its S-1, Anthropic says it runs Claude natively on Trainium, TPUs, NVIDIA GPUs and AMD GPUs, even for training.
The competition resulted in a ranking, not just a few clusters. Humans made the difference in how they allocated resources, organized the work and sometimes overrode the agents.
Though agents produced superior results, a lot of their code will be hard to merge back into SGLang: the organizers say many entries run past 10,000 lines, with separate variants for each tested shape. The agents clearly succeeded at a well-defined, easily measurable goal, but humans still have an edge in balancing trade-offs. What is hard to learn during RL, such as long-term maintainability, is still a blind spot for LLMs. It usually comes with an unclear reward function: how do you score how maintainable an architecture is?
Agents speeding up the software that runs AI sounds like recursive self-improvement (RSI). To me it feels like self-driving in 2005. A year earlier no car finished the DARPA Grand Challenge; that year five did. Ever since, full self-driving has been a few years away. Twenty-one years later, the vast majority of cars still have drivers. Progress in AI can be extremely fast and extremely slow at once. Predicting S-curves is hard.
