We raised $15M to build the world's fastest neocloud.Read
agentic systemai agentsinferenceproductionobservabilityrate limitingscaling

Agentic AI in Production: Inference Requirements for Multi-Step Workflows

General Compute·

The difference between a working agentic AI demo and a reliable production system is mostly invisible on the happy path. Both complete tasks, both produce reasonable outputs, both look great in a demo. The problems show up under load, at the edges, and in the places you didn't instrument.

This post covers what actually changes when you move an agentic system from prototype to production: the inference requirements, the observability gaps, the error handling patterns, and what happens when you scale from one agent to a thousand.

What Makes an Agentic System Different

A traditional LLM API call is a single request-response. You send a prompt, you get a completion, you move on. An agentic system is a loop: the model generates an action, the system executes it, the result comes back, the model generates the next action. Repeat until done.

This loop changes everything about how you think about inference:

  • Latency compounds. A 10-step agent with 500ms per LLM call has 5 seconds of pure inference time, before any tool execution. Shave 200ms off each call and you save 2 full seconds per task.
  • Errors propagate. A failed tool call mid-chain doesn't just lose one response. It can corrupt the agent's context, trigger unnecessary retries, or leave external systems in a partial state.
  • Context grows. Each iteration adds tokens. A naive implementation that appends every tool call result to the context window can hit token limits several steps in.
  • Concurrency multiplies costs. One agent doing 10 LLM calls costs the same as 10 agents each doing 1. When you scale to 100 agents, you're suddenly making 1,000 concurrent inference requests.

Observability: You Can't Debug What You Can't Trace

The first time a production agent fails in a non-obvious way, you'll wish you had traced every step.

A good observability setup for an agentic system captures:

  • The full prompt and response at each step
  • Tool calls and their results
  • Token counts per step (input and output separately)
  • Wall-clock latency per LLM call
  • The agent's current goal or plan state if you're using an explicit planning step
  • Any errors and the full context at the time they occurred

OpenTelemetry is a reasonable choice here. Most inference providers expose request IDs you can attach to spans. The pattern looks like this:

import opentelemetry.trace as trace

tracer = trace.get_tracer("agent")

def run_agent_step(messages, tools):
    with tracer.start_as_current_span("agent_step") as span:
        span.set_attribute("input_tokens", count_tokens(messages))

        response = client.chat.completions.create(
            model="qwen-coder-32b",
            messages=messages,
            tools=tools
        )

        span.set_attribute("output_tokens", response.usage.completion_tokens)
        span.set_attribute("finish_reason", response.choices[0].finish_reason)
        return response

For longer-running agents, you'll also want to log state checkpoints. If an agent is running for 10 minutes and fails at step 47, you don't want to replay from the beginning. Periodic snapshots of the agent's context and memory let you resume or debug from a known point.

One common mistake: logging only the final result. When things go wrong, the final result tells you almost nothing. You need the intermediate steps, especially the tool call arguments and results that led to the failure.

Error Handling: Retries Are Not Enough

Agentic systems fail in ways that chat apps don't. Here are the failure modes worth designing around explicitly.

Tool call failures. The agent asks a tool to do something, the tool fails, and now the agent has to decide what to do. If you don't handle this explicitly, the model will often try the same call again, fail again, and spin. You need explicit error handling at the tool call level that communicates useful information back to the model: what failed, why, and what it should try instead.

def execute_tool(tool_name, args):
    try:
        result = tools[tool_name](**args)
        return {"success": True, "result": result}
    except ToolTimeoutError:
        return {
            "success": False,
            "error": "Tool timed out after 10s. Try a simpler query or break it into smaller parts."
        }
    except PermissionError as e:
        return {
            "success": False,
            "error": f"Permission denied: {e}. Check that the required credentials are configured."
        }

The quality of your error messages to the model matters a lot here. Vague errors like "tool failed" give the model nothing to work with. Specific messages with actionable context let the model adapt its plan.

Context overflow. Long-running agents will eventually hit token limits. Handle this before it becomes an error. Monitor token usage at each step and implement a summarization step when you approach the limit. A common pattern: when the context hits 70-80% of the model's limit, summarize the history into a compact "progress so far" block and replace the full history with it. You lose some detail, but the agent can continue.

Hallucinated tool calls. Models sometimes call tools with invalid arguments, or call tools that don't exist. Validate tool call arguments against your schemas before executing, and return a clear error message when the model generates an invalid call. Most models self-correct when given explicit feedback about what went wrong.

Loops and stuck states. An agent can get into a loop where it repeatedly tries the same approach without making progress. Track the last N tool calls and detect when the pattern is repeating. If the agent is stuck, inject a system message prompting it to try a different approach, or surface it to a human for review.

Rate Limiting: Scaling From 1 Agent to 1,000

A single agent making 10 LLM calls per task is manageable. A hundred agents making 1,000 calls per minute is a different situation.

Rate limits are usually expressed as requests per minute (RPM) and tokens per minute (TPM). At scale, you'll hit both. Here's how to handle them.

Token estimation before calls. Count your input tokens before making each call. This lets you predict whether you're approaching TPM limits and schedule calls accordingly. Tiktoken works for most models:

import tiktoken

def estimate_tokens(messages, model="gpt-4"):
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for msg in messages:
        total += len(enc.encode(msg["content"])) + 4  # per-message overhead
    return total

Request queuing. Don't fire all agent requests simultaneously. Use a queue with a rate limiter. The asyncio.Semaphore pattern works well in Python:

import asyncio

class RateLimitedClient:
    def __init__(self, max_concurrent: int, rpm_limit: int):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.rpm_limit = rpm_limit
        self.request_times = []

    async def create_completion(self, **kwargs):
        async with self.semaphore:
            now = asyncio.get_event_loop().time()
            self.request_times = [t for t in self.request_times if now - t < 60]

            if len(self.request_times) >= self.rpm_limit:
                wait = 60 - (now - self.request_times[0])
                await asyncio.sleep(wait)

            self.request_times.append(now)
            return await self.client.chat.completions.create(**kwargs)

Exponential backoff on 429s. Even with rate limiting, you'll occasionally get 429 responses. Retry with exponential backoff and jitter:

import random
import asyncio

async def call_with_backoff(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return await fn()
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(wait)

Prioritization. When you're running a mix of agent types, not all requests are equally urgent. A user-facing agent responding in real time needs low latency. A background batch job can wait. Build priority levels into your queue so high-priority requests jump ahead of lower-priority ones.

Scaling: The Architectural Decisions That Matter

Scaling an agentic system to thousands of concurrent agents requires a few architectural choices.

Async by default. Synchronous agent execution blocks threads. If you're running agents in Python, use asyncio throughout. Each agent step involves network I/O for both the LLM call and typically one or more tool calls -- both excellent candidates for async execution.

Stateless agent workers. Don't store agent state in memory on the worker process. Put it in a fast key-value store like Redis. This lets you run any number of stateless worker processes, restart them without losing in-progress agents, and distribute work across machines.

Worker 1: pick up agent_id=42 from queue, load state from Redis, run step, save state, enqueue next step
Worker 2: pick up agent_id=43 from queue, load state from Redis, run step, save state, enqueue next step

Parallel tool execution. When an agent can make multiple tool calls that don't depend on each other, run them in parallel. If the agent wants to call a search API and a database lookup simultaneously, there's no reason to do them sequentially. This requires parsing the tool calls to identify independent ones, but the latency savings are significant for agents with wide tool use.

Agent timeouts at every level. Set timeouts on individual LLM calls, on tool executions, and on the total agent run. Without these, a hung LLM call can hold up an entire worker indefinitely. The total task timeout should be aggressive enough to surface problems quickly.

Observability Infrastructure at Scale

When you're running 1,000 concurrent agents, you can't read traces individually. You need aggregated metrics that tell you the health of the system at a glance:

  • P50/P95/P99 latency per agent step -- watch P99 for outliers that affect user-facing tasks
  • Token usage per agent run -- spot agents that are consuming excessive context
  • Error rate by tool -- identify tools that are failing more than expected
  • Stuck agent count -- how many agents have been running longer than the expected maximum
  • Retry rate -- high retry rates indicate either rate limiting or flaky tools

Feed these into a dashboard with alerting. A sudden spike in P99 latency or error rate often indicates an infrastructure issue with your inference provider or a newly deployed change that's causing more LLM calls than expected.

Inference Provider Requirements for Agentic Systems

The inference provider you choose affects almost everything above. For production agentic systems, the requirements are more demanding than for simple chat apps.

Low and consistent latency. High variance in response times is worse than uniformly higher latency for agentic systems. When step 3 takes 800ms but step 7 takes 5 seconds, it's hard to build reliable timeout logic. Look for providers with low p99 latency, not just low median.

High rate limits. At 1,000 concurrent agents making 10 calls each, you need a provider that can handle sustained high throughput without degrading. Most providers offer enterprise tiers with higher limits -- verify before you're blocked in production.

Reliable function calling. Agentic systems depend on the model reliably generating valid tool calls. Test your specific tool schemas with your target model before committing to it. Some models handle complex nested schemas better than others.

Good error messages. When the API returns an error, the message should tell you what happened and what to do. Cryptic error codes require looking up docs at the worst possible moment; good error messages let you handle failures programmatically.

Putting It Together

A production-ready agentic system needs:

  • Distributed tracing across all agent steps
  • Explicit error handling at the tool call level, with informative messages back to the model
  • Context management to avoid token limit failures mid-run
  • Rate limiting and request queuing to handle concurrent agents
  • Stateless workers backed by persistent state storage
  • Timeouts at every level of the stack

Most of these aren't difficult to implement individually. The challenge is getting all of them in place before you need them, because the failure modes are subtle and often only appear under load.

If you're scaling an agentic system and running into inference bottlenecks, General Compute's API is worth evaluating. The combination of low latency and high throughput limits reduces the rate limiting complexity and makes the concurrency math more manageable. You can get started at generalcompute.com.

ModeHumanAgent