The Cost of Agent Retries: When Slow Inference Kills Reliability

Every agent fails sometimes. A model returns malformed JSON. A tool call targets a function that doesn't exist. A confidence check rejects the output. Retries are how agents handle these failures without crashing. But retries have a cost, and when inference is slow, that cost adds up fast enough to make your agent functionally unusable.

This post is about how to think about retry budgets in agentic systems, why slow inference forces you into bad tradeoffs, and how inference speed changes what retry strategies are actually viable.

Why agents retry

An agent step can fail in several distinct ways, and each one calls for a different retry strategy.

Structural failures are the most obvious. The model produces JSON that doesn't parse, or calls a tool with the wrong argument types. These are deterministic failures: the output is not usable as-is, and the agent needs to try again. Usually a retry with a richer error message ("you called search_web with an integer, but the query parameter must be a string") will fix the problem in one pass.

Tool execution failures happen when the model's call is structurally valid but the tool itself fails. The web search times out. The database throws a connection error. The code sandbox returns a non-zero exit code. These are often transient. A retry with the same call will succeed after the transient clears, or fail again for a reason worth escalating.

Validation failures are subtler. The model returns output that is valid JSON and passes schema validation, but a downstream check rejects it as wrong. A fact-checker finds a claim that contradicts the source documents. A test runner shows the generated code throws an exception. The agent needs to retry, but this time with the validation error folded back into the context so the model knows what it got wrong.

Confidence-based retries are a pattern some orchestrators use: generate the same step multiple times (or generate a score alongside the output), and retry if confidence falls below a threshold. This is more expensive than deterministic retries but is useful when the error signal is not explicit -- when the output could be wrong without the agent noticing at the structural level.

Each of these failure modes has different expected retry counts. Structural and transient failures usually resolve in one or two retries. Validation failures may need two or three. Confidence-based retries, by their nature, generate multiple candidates before selecting. In a multi-step agent with a pipeline of ten steps, the cumulative retry load is not trivial.

The math of retries at slow inference speeds

Suppose each of your agent's steps succeeds 90% of the time on the first attempt. That sounds reasonable. But in a five-step pipeline, the probability that all five steps succeed on the first try is 0.9^5 = 59%. Without retries, your agent fails four times out of ten.

One retry per failed step changes the math. If each step has a 90% first-attempt success rate and you allow one retry, the per-step success rate becomes 1 - (0.1 * 0.1) = 99%. Five steps in series at 99% per step gives roughly 95% end-to-end success. That is viable.

But now add timing. If each inference call takes 5 seconds, and a single step fails once before succeeding on retry:

First attempt: 5 seconds, fails
Retry: 5 seconds, succeeds
Total for this step: 10 seconds

With five steps in a pipeline, and assuming roughly one failure across the pipeline (consistent with a 90% per-step success rate), the expected total latency becomes approximately 27.5 seconds: five steps at 5 seconds each for first attempts, plus one 5-second retry. With bad luck -- multiple steps failing -- you could easily be looking at 40 to 50 seconds.

If inference takes 500 ms instead of 5 seconds, the same pipeline runs in under 3 seconds including the expected retry. That is not a marginal difference; it is the gap between a feature and a usable product.

Retry budgets: capping latency instead of counting attempts

Most retry logic is written as a maximum attempt count: "retry up to 3 times, then give up." This is simple to implement but is the wrong abstraction for agentic systems.

The problem is that maximum attempt count does not compose well across steps. If each of ten steps retries up to 3 times, and your inference calls take 4 seconds each, your worst-case latency is 10 * 3 * 4 = 120 seconds. That failure mode will not show up in development; it takes load testing with realistic failure injection to observe.

A better abstraction is a retry budget: a time limit for the entire agent run, not per step. Each attempt consumes from the budget, and when the budget is exhausted, the agent fails fast rather than continuing to retry.

import time

class RetryBudget:
    def __init__(self, total_seconds: float):
        self.deadline = time.monotonic() + total_seconds

    def remaining(self) -> float:
        return max(0.0, self.deadline - time.monotonic())

    def is_exhausted(self) -> bool:
        return time.monotonic() >= self.deadline

    def check(self):
        if self.is_exhausted():
            raise TimeoutError("agent retry budget exhausted")

A step that respects the budget looks like:

MINIMUM_RETRY_TIME = 1.0  # don't attempt a retry if less than 1s remains

def run_step_with_budget(step_fn, budget: RetryBudget, max_attempts: int = 3):
    last_error = None
    for attempt in range(max_attempts):
        budget.check()  # fail fast if time is up
        try:
            return step_fn()
        except RecoverableError as e:
            last_error = e
            if budget.remaining() < MINIMUM_RETRY_TIME:
                break  # not enough time to retry meaningfully
    raise last_error

The budget is shared across all steps in the pipeline. A step that consumes 15 seconds of retries leaves less time for subsequent steps to retry. This creates backpressure: a pipeline with one very flaky step naturally gets fewer retries later, rather than compounding latency without bound.

This pattern also makes it easier to reason about SLAs. If your product promises a 30-second response time, set the budget to 25 seconds and you have a 5-second margin for overhead.

How inference speed changes the retry math

The budget framing makes the connection between inference speed and retry strategy explicit.

With a 30-second budget and 5-second inference:

You can fit at most 6 inference calls total
In a 5-step pipeline, that means one retry across all steps
Any step that fails twice exhausts the retry allocation for the whole pipeline

With a 30-second budget and 500 ms inference:

You can fit up to 60 inference calls
In a 5-step pipeline, each step could retry 11 times before hitting the budget
You can afford more aggressive confidence thresholds and more liberal retry policies

The improvement is not just that each call is faster. The slack you gain from faster inference directly funds additional retries, which funds higher reliability. When inference is fast enough, retries become nearly free and you can design your agent to retry aggressively without worrying about the latency cost. When inference is slow, every retry is a real cost you are passing on to the user.

Failure modes that slow inference amplifies

Beyond raw retry counts, there are a few specific failure patterns that slow inference makes worse.

Cascading timeouts. Many agent steps call external tools with their own timeouts. If your inference call for a step takes 8 seconds, and the downstream tool has a 10-second timeout, you have 2 seconds of headroom for the tool to execute. A tool that normally takes 3 seconds will time out, triggering a retry. With faster inference, that same tool has 9.5 seconds of headroom and completes without incident.

Context length growth during retries. Each retry typically appends the failed attempt's output and an error message to the context before retrying. After two or three retries, the context has grown substantially. Longer contexts take longer to prefill on the next attempt, which slows inference further, which makes the next retry take longer still. This compounding effect is easy to miss in testing (where you rarely hit three consecutive failures on the same step) but shows up in production under load.

User abandonment. This is the non-technical failure mode. An agent running under a budget-aware retry strategy will still sometimes run slowly, especially under load. Users who see no progress for 15 to 20 seconds tend to close the tab or restart the session. This creates a category of failures that your retry logic never gets to handle because the request is already gone. Faster inference reduces the base latency, which gives retries more room to happen before the user gives up.

Designing retry logic for a multi-step agent

Given all of this, here are the practical choices that matter most when building retry logic for a production agent.

Distinguish retryable from non-retryable errors. Tool execution failures are usually retryable. Structural failures in the model's output are retryable with error feedback. Permission errors and missing tools are not retryable and should escalate immediately. Burning two retries on a non-retryable error is pure waste.

Pass error context on retry. A retry that sends the same prompt as the original attempt will often fail the same way. The retry should include, at minimum, the error that caused the failure and a corrective instruction. For validation failures, include the specific assertion that failed. For tool argument errors, include the expected schema.

def run_with_error_context(agent_step, context: list, error_feedback_template: str):
    for attempt in range(3):
        result = call_model(context)
        error = validate(result)
        if not error:
            return result
        # fold the error back into context for the next attempt
        context = context + [
            {"role": "assistant", "content": result.raw_output},
            {"role": "user", "content": error_feedback_template.format(error=error)},
        ]
    raise ValueError("step failed after 3 attempts")

Set a deadline at the top of the pipeline, not at each step. Per-step max attempt counts compose badly. A shared budget propagates the constraint correctly and makes the worst-case latency predictable and auditable.

Track retry counts in observability. The average number of retries per step, and the distribution across percentiles, will tell you which steps are flaky and need attention. A step that retries 40% of the time is consuming nearly twice as much inference budget as a step with a 2% retry rate. Log it, alert on it, fix the underlying issue.

Invest in faster inference before investing in more complex retry logic. Exponential backoff, jitter, circuit breakers -- these add real engineering cost and operational complexity. Faster inference solves much of the same problem with less code, because the retry budget runs out less often when each attempt is cheaper.

Measuring the impact

If you are not sure how much retries are costing your agent, the measurement is straightforward. Add a counter to each agent step that tracks the number of attempts before success (or final failure). Log it alongside per-step latency. Then look at the distribution.

If your median step runs in one attempt but your 95th percentile runs in three, retries are adding significant latency to a small fraction of requests. Those are the requests your users notice most. Reducing per-attempt latency (by using faster inference) compresses the 95th percentile faster than anything you can do in retry logic alone.

If retries are common -- say, 20% or more of steps need at least one retry -- you have a reliability issue that retry logic alone will not fix. You need to either improve the model's accuracy on the task, improve the quality of error feedback on retry, or reduce the complexity of the step so it is less likely to fail in the first place. Retries are the recovery mechanism, not the primary reliability mechanism.

Agent reliability and inference speed are more closely coupled than they appear from the outside. The model accuracy numbers are the same regardless of how fast inference runs. But the retry budget your architecture can afford, the confidence thresholds you can set, and the failure modes you can absorb -- all of these scale with how much inference you can fit into the time you have.

General Compute serves open models at very high tokens-per-second on an OpenAI-compatible API. If your agent is spending a significant fraction of its wall time on retries, the fastest way to test what changes is to run the same agent against a faster backend and measure the retry distribution directly. The API is at generalcompute.com.