We raised $15M to build the world's fastest neocloud.Read
agentsinferencelatencybenchmarkingevaluation

Evaluating Agent Performance: Latency as a First-Class Metric

General Compute·

When teams benchmark AI agents, the focus almost always lands on accuracy: did the agent complete the task correctly, call the right tools, produce valid outputs? These are important questions. But they leave out something that matters just as much in production deployments -- how long did it take?

For single-turn LLM calls, latency is straightforward to measure. You send a request, you track time-to-first-token and total generation time, and you have what you need. For agents, the picture is more complicated. An agent might make a dozen LLM calls interleaved with tool calls, retrieval operations, and state management. Latency compounds across every step, and a slow model that scores well on accuracy benchmarks is often less useful in practice than a faster one that scores slightly lower.

This post covers why latency belongs in your evaluation framework, how to measure it properly, and what a practical benchmarking setup looks like.

Why accuracy alone is misleading

Accuracy benchmarks for agents are typically run offline: you give the agent a fixed dataset of tasks, compare outputs to ground truth, and compute a score. This works for measuring capability but tells you nothing about what users actually experience.

Consider a coding agent evaluated on SWE-bench. It might resolve 40% of GitHub issues correctly. That is a reasonable signal for capability. But the benchmark says nothing about whether each task takes 8 seconds or 80 seconds. For a developer waiting on a code suggestion, those 72 extra seconds are not an acceptable trade-off for the same accuracy score.

The deeper problem is that agent latency and agent accuracy are orthogonal axes. A model that is fast but slightly less accurate and a model that is slow but slightly more accurate can score identically on a capability benchmark while having wildly different suitability for a given use case. Without latency data alongside accuracy data, you cannot make a principled choice between them.

For real-time applications like voice agents, interactive coding assistants, and customer-facing chatbots, latency sets a hard ceiling on what is even possible. For batch processing like overnight report generation or background research, throughput and cost matter more than interactive latency. Your evaluation framework should reflect which mode you are actually building for, and accuracy alone does not encode that distinction.

What to measure

Agent latency is not a single number. There are several distinct metrics worth tracking, and each one tells you something different.

End-to-end task latency is the wall-clock time from when the user submits a task to when the agent returns a final result. This is the number users experience directly. It should be your primary production metric.

Per-step LLM latency measures how long each individual inference call takes. This includes time-to-first-token (TTFT), which determines how quickly the agent starts responding, and tokens-per-second (TPS), which determines how quickly it finishes. When an agent makes 10 LLM calls, per-step latency compounds directly into end-to-end latency. A model that is 300ms faster per call saves 3 full seconds across a 10-step pipeline.

Tool call latency covers time spent executing tools: web searches, code execution, database queries, external API calls. Inference providers have no control over this, but it is often where agents spend surprising amounts of time. Separating tool latency from inference latency in your instrumentation lets you identify where the time is actually going before deciding what to optimize.

Step overhead is the dead time between steps, spent on parsing LLM output, building the next prompt, and updating state. This is usually small but worth tracking to catch runaway prompt construction or expensive serialization.

P50 vs P99 latency deserves particular attention. Agents are far more variable than single-turn calls because they take different numbers of steps on different inputs. A task that averages 5 steps might take 20 in the worst case. P99 latency tells you what users experience on the hardest inputs, and it is often three to five times the median. An agent with a fast median and a catastrophic P99 will still generate support tickets.

Instrumentation

Collecting these metrics requires explicit instrumentation. Most agent frameworks do not emit detailed timing data by default.

A simple approach wraps your LLM client to capture per-call timing and accumulates traces across the run:

import time
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class StepTrace:
    step_index: int
    step_type: str  # "llm" | "tool" | "overhead"
    start_time: float
    end_time: float
    ttft: Optional[float] = None
    tokens_generated: Optional[int] = None

    @property
    def duration_ms(self) -> float:
        return (self.end_time - self.start_time) * 1000

@dataclass
class AgentTrace:
    task_id: str
    start_time: float
    end_time: Optional[float] = None
    steps: List[StepTrace] = field(default_factory=list)

    @property
    def total_duration_ms(self) -> float:
        if self.end_time is None:
            return 0.0
        return (self.end_time - self.start_time) * 1000

    @property
    def llm_duration_ms(self) -> float:
        return sum(s.duration_ms for s in self.steps if s.step_type == "llm")

    @property
    def tool_duration_ms(self) -> float:
        return sum(s.duration_ms for s in self.steps if s.step_type == "tool")

    @property
    def llm_fraction(self) -> float:
        if self.total_duration_ms == 0:
            return 0.0
        return self.llm_duration_ms / self.total_duration_ms

Wrapping your LLM client to record step traces looks like this:

class InstrumentedClient:
    def __init__(self, client, trace: AgentTrace):
        self._client = client
        self._trace = trace
        self._step_index = 0

    def complete(self, messages, **kwargs):
        step = StepTrace(
            step_index=self._step_index,
            step_type="llm",
            start_time=time.perf_counter(),
            end_time=0.0,
        )
        self._step_index += 1

        first_token_time = None

        def on_first_chunk():
            nonlocal first_token_time
            if first_token_time is None:
                first_token_time = time.perf_counter()

        result = self._client.complete(messages, on_chunk=on_first_chunk, **kwargs)

        step.end_time = time.perf_counter()
        if first_token_time is not None:
            step.ttft = (first_token_time - step.start_time) * 1000
        step.tokens_generated = result.usage.completion_tokens
        self._trace.steps.append(step)

        return result

Once you are collecting traces, aggregating across runs gives you the distribution:

import statistics

def summarize_traces(traces: List[AgentTrace]) -> dict:
    completed = [t for t in traces if t.end_time is not None]
    durations = sorted(t.total_duration_ms for t in completed)
    n = len(durations)

    return {
        "p50_ms": durations[n // 2],
        "p95_ms": durations[int(n * 0.95)],
        "p99_ms": durations[int(n * 0.99)],
        "mean_ms": statistics.mean(durations),
        "mean_steps": statistics.mean(len(t.steps) for t in completed),
        "mean_llm_fraction": statistics.mean(t.llm_fraction for t in completed),
    }

The mean_llm_fraction is particularly useful. If it is high (above 0.70), switching to a faster inference provider will have a large effect on end-to-end latency. If it is low, the bottleneck is tool calls or overhead, and you will need to look elsewhere.

Designing a latency benchmark

A good latency benchmark has a few structural requirements.

A representative task set covers the actual range of tasks your agent handles. If your coding agent mostly does small bug fixes but your benchmark is all complex feature additions, you will overestimate latency for most real traffic. Stratify tasks by complexity so you can report latency separately for easy, medium, and hard inputs.

Controlled infrastructure means running your benchmark with consistent hardware, consistent network conditions, and a consistent load on the inference provider. Agent latency is noisy. Run each task multiple times -- three to five runs is usually enough -- and use the median to reduce variance.

Accuracy and latency measured on the same tasks. You need both axes to make principled model selection decisions. A scatter plot with task-level accuracy on one axis and task-level latency on the other shows the full picture more clearly than any single aggregate score.

Step count alongside latency. If slow tasks are slow because they take more steps, that is a different problem from slow tasks being slow because each step is slow. Step count variation usually points to model reasoning quality: can the model plan efficiently enough to avoid unnecessary tool calls? Per-step latency variation points to the inference provider.

A minimal benchmark runner:

import asyncio
from typing import Callable, Awaitable

async def run_benchmark(
    tasks: list[dict],
    agent_fn: Callable[[dict], Awaitable[dict]],
    runs_per_task: int = 3,
) -> list[dict]:
    results = []

    for task in tasks:
        for run in range(runs_per_task):
            trace = AgentTrace(task_id=task["id"], start_time=time.perf_counter())
            try:
                output = await agent_fn(task, trace=trace)
                success = True
            except Exception as e:
                output = {"error": str(e)}
                success = False

            trace.end_time = time.perf_counter()

            results.append({
                "task_id": task["id"],
                "difficulty": task.get("difficulty", "unknown"),
                "run": run,
                "latency_ms": trace.total_duration_ms,
                "step_count": len(trace.steps),
                "llm_fraction": trace.llm_fraction,
                "success": success,
                "correct": task.get("evaluator", lambda o: None)(output),
            })

    return results

The accuracy-latency tradeoff in practice

Once you have latency data alongside accuracy data, you can reason about model selection more concretely. The tradeoff is not arbitrary -- there is typically a Pareto frontier where smaller models are faster but less accurate, and larger models are more accurate but slower.

Where the optimal point sits depends on what you are building:

For interactive agents where users are waiting in real time, you generally want end-to-end task latency under a few seconds for most tasks. A 5% accuracy drop that saves 8 seconds per task is usually worth taking. The user who gets a slightly less accurate answer in 2 seconds has a better experience than the user who waits 10 seconds for a slightly better one.

For background agents that run asynchronously and deliver results later, optimize for cost and accuracy. Latency tolerance is much higher when users are not blocking on the result.

For streaming-capable agents that can push intermediate results to the UI, TTFT matters more than total latency. If the first token arrives within 300ms, users perceive the agent as responsive even if the full output takes several seconds to finish.

These distinctions are invisible if you only benchmark accuracy.

How inference speed multiplies across steps

In a single-turn LLM call, inference speed affects one interaction. In a multi-step agent, it affects every step. An agent that makes 10 LLM calls has 10 opportunities for inference latency to accumulate.

If your current setup takes 800ms per LLM call, a 10-step agent run costs at least 8 seconds from inference alone -- not counting tool calls or overhead. Switching to an inference provider where calls take 150ms brings that to 1.5 seconds, a reduction that shows up directly in the end-to-end latency distribution your benchmark produces.

The multiplier also works the other way. If your inference gets 2x slower -- for example, because you swap to a larger model or your provider has degraded performance -- your agent latency doubles. Monitoring per-step LLM latency in production, not just end-to-end latency, lets you catch this kind of regression before it shows up as user complaints.

What your evaluation checklist should include

When setting up agent evaluation, make sure you are tracking:

  • End-to-end task latency (p50, p95, p99) across your full task set
  • Per-step LLM latency and TTFT, broken out from tool call latency
  • Step count distribution by task and by task difficulty level
  • Accuracy measured on the same tasks as latency, so you can correlate them
  • Separate breakdowns for different task types: simple, complex, error recovery cases

With these numbers, you can make informed decisions about model selection, inference provider, and whether optimizing for accuracy or for speed has more leverage for your specific workload.

If you are not tracking latency at all right now, start with end-to-end task latency and step count. Those two numbers, measured over a representative task set, will tell you more about your agent's production behavior than any accuracy benchmark alone.

General Compute's API runs popular open-source models -- including the ones commonly used in LangGraph, CrewAI, and AutoGen -- at high tokens-per-second on an OpenAI-compatible endpoint. If you want to run latency-aware agent benchmarks against fast infrastructure, the setup is a one-line base URL change:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key",
)

From there you can instrument the client with the trace patterns from this post and measure what your agent's latency distribution actually looks like. See the GeneralCompute docs for model options and API details.

ModeHumanAgent