A lesson about retries, hidden in the DeepSeek-V4 paper

The DeepSeek-V4 paper contains an unexpected lesson for anyone running LLM benchmarks: sometimes it’s not correct to retry failures.

So I decided to check it myself. 100,000 AI poems later, here’s what I found.

DeepSeek’s warning

Here’s the paragraph from the DeepSeek-V4 paper that made me curious:

Importantly, it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias. Because shorter responses are more likely to survive interruption, regenerating from scratch makes the model more prone to producing shorter sequences whenever an interruption occurs.

Adding retries is an obvious way to fix issues with reliability. But as DeepSeek noticed, longer requests have a higher chance of being interrupted. When you retry a failed long request, there’s a chance you’ll get a short response as a replacement.

Retries make AI poems shorter

Experiment

Since statistical biases can often be hard to grasp, let’s see how this affects a real run.

I asked DeepSeek-V4-Flash to generate 100,000 poems, haikus, or other literary works for me:

Write a complete piece of literature in one randomly chosen form: a one-line poem, a haiku, a novel chapter, or a limerick.

It cost me $6.06 (V4-Flash is cheap!) and generated a large variety of responses:

On average it generated a 74-word text, but the results varied widely:

• 73.2% haikus. Very short: 15 words on average.
• 11.5% novel chapters. 34 times longer: 503 words on average.

Simulating failures

Now let’s simulate what would happen if the LLM inference were really unreliable and 10% of requests failed by being interrupted randomly during generation.

I used a statistical model called a Poisson process to model this behavior; an average time between interruptions of 31 seconds makes 10% of requests fail in my experiment.

As the DeepSeek paper noted, longer requests are affected more often. You can use this formula for a Poisson process to calculate it:

$$P(\text{failure during request}) = 1 - e^{-\text{request time}/\text{mean time between interruptions}}$$

For example, haikus take 2.38 seconds on average to generate, so their failure rate is $1 - e^{-2.38/31} \approx 7.4\%$. But novel chapters are longer (11.52 seconds), so their failure rate is higher: $1 - e^{-11.52/31} \approx 31.0\%$.

You can see how the failure rate rises for longer requests:

Adding retries

So let’s see what would happen if I added retries. I artificially simulate failures and perform retries on the failed requests until they succeed.

After running the simulation, the resulting dataset looks noticeably different! Just as the DeepSeek authors warned us:

	Without failures	Failures + retries	Change
Average word count	73.51	59.40	−19.2%
Responses ≥ 600 words	2,904	1,961	−32.5%
Novel chapters	11,499	8,919	−22.4%

The average response is now noticeably shorter and there are fewer longer-form responses. To better understand why it changed so much, let’s take a look at the requests that failed and what happened when they were retried:

After retries, (would-be) long responses often become shorter responses. The right-hand side of the chart gets affected the most.

Looking at it with a statistics toolset, adding retries changed the final distribution of the data. What we’re seeing here is a form of selection bias: the retried sample was not representative - long responses were overrepresented in it. Survivorship bias is also a good perspective: the final dataset includes only responses that survived some filtering process - in this case, interruptions that penalized long requests.

Conclusion

Shorter poems might sound innocent, but the same problem could be dangerous in a real benchmark. A longer request might be stuck in an endless reasoning loop or headed down the wrong path, and retrying it might give the model a second chance - raising the score.

Armed with that knowledge, we have 3 ways to deal with the issue:

• The DeepSeek authors architected their system to resume interrupted requests rather than regenerate them from scratch.
• Failures that happen truly randomly (independently of request length) can be safely retried.
• This problem can be important for benchmarks or research, but for most consumer-facing applications the difference doesn’t really matter.

What started as a curious warning in the DeepSeek-V4 paper became a surprisingly intuitive lesson in statistics for me.