A lesson about retries, hidden in the DeepSeek-V4 paper
A warning hidden in the DeepSeek-V4 paper says retrying interrupted LLM requests is mathematically incorrect — it introduces length bias. I reproduced it on 100,000 poems.
Benchmarks, token economics and research on AI coding agents
Get new posts by email
Subscribe via RSS
Do Qwen3.6 27B quantizations break the pelican?
We tested Qwen3.6 27B quantizations by Unsloth on Hugging Face, with pelicans on bikes, gears, Terminal-Bench 2.1, and AIME-120.
HN I burned all my tokens researching how to save tokens
How I burned a full Claude limit in 30 minutes and built my own deep research pipeline instead: 3 subscriptions, shared memory, a clear role for each model. You can build the same from what you already pay for.
Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost?
We ported the puzzle game Baba Is You to the Harbor framework, and benchmarked current models, including Claude, GPT, Gemini, GLM, and DeepSeek. A human Twitch streamer is 4x faster than Claude Fable 5.
Featured Tokenflation: When “Hi” triggers 33 tool calls
Tokenflation: simple tasks consuming ever more context, reasoning, and tool calls without more useful output. We benchmarked 14 models; one answered “Hi” with 33 tool calls and an unsolicited commit.
HN Qwen3.6 27B is the sweet spot for local development
Qwen3.6 27B is finally a smart model we can use for coding on MacBook or NVIDIA RTX - with llama.cpp and OpenCode.
Compare harnesses, not models: Blitzy vs. GPT-5.4 on SWE-Bench Pro
An independent audit of agentic scaffolding and harnesses. We analyze how agent workflows, codebase documentation, and test verification impact performance compared to raw base models like GPT-5.4, Gemini 3.1 Pro, and Claude Code.
Reviving the 20-year-old puzzle game Chromatron with Ghidra and AI
Decompiling the classic laser puzzle game Chromatron from Windows XP and PowerPC executables into Rust. A pixel-perfect port using Claude Code, Opus 4.6, Cursor, GPT-5.2-Codex, and Ghidra.
HN We hid backdoors in ~40 MB binaries and asked AI + Ghidra to find them
BinaryAudit benchmarks AI agents using Ghidra to find backdoors in compiled binaries of real open-source servers, proxies, and network infrastructure.
Featured Reverse engineering River Raid with Claude, Ghidra, and MCP
Connecting Claude to Ghidra via MCP to reverse engineer River Raid. A test of AI agents against 6502 assembly, memory mapping, and 80s game logic.
HN Benchmarking OpenTelemetry: can AI trace your failed login?
A lot of vendors pitch AI SRE. We tested 14 models across 11 programming languages; even the best ones struggle with instrumenting code with the leading open-source standard, OpenTelemetry.
Vibe coding needs git blame
Prompts are specs, not code. This influences git workflows for vibe coding, tracking LLM prompts in GitHub repositories, managing commit messages, and debugging non-deterministic AI outputs.
How 2025 took AI from party tricks to production tools
AI reasoning models like DeepSeek-R1, agentic coding tools like Claude Code, and image generation with Nano Banana Pro set daily software engineering standards.
Migrating CompileBench to Harbor: standardizing AI agent evals
Standardizing AI agent evaluation with Harbor: an open-source framework for reproducible benchmarks, reinforcement learning, and collaborative evals.
Antigravity feels heavy and Claude Skills are light
Comparing Google Antigravity and Claude Code for AI-assisted workflows, and why custom Claude Skills might be the better approach.