Long-Running Agents: Managing Inference Costs Over Hours of Execution

Markdown body

Most agent use cases are short. A user asks a question, the agent makes a few tool calls, and the interaction finishes in under a minute. The engineering concerns at that scale -- TTFT, per-step latency, prompt construction -- are well understood.

Long-running agents are different in character. Think of a research agent that autonomously gathers sources, synthesizes findings, and refines its conclusions over several hours. Or a background worker that monitors a codebase, runs tests, opens pull requests, and responds to review feedback over a full workday. Or a data pipeline agent that ingests, transforms, and validates records across a multi-hour batch window. These agents make hundreds or thousands of LLM calls, accumulate context that can push against model limits, and generate inference costs that add up in ways that are easy to miss until the bill arrives.

This post covers the engineering patterns that matter specifically when your agent runs for hours: context management, cost accounting, model routing, and the checkpointing strategies that let you recover from failures without losing work or spending twice.

## How costs compound at long time scales

In a short agent interaction, inference cost is mostly determined by the number of steps and the size of each LLM call. Both are bounded by the task. A 10-step coding agent with modest context costs a predictable amount.

For long-running agents, the cost dynamics change in two important ways.

First, context grows. Most agents include prior conversation history in each LLM request. Over hundreds of steps, that history can grow to tens of thousands of tokens. Since inference providers charge per token in and out, sending a 50,000-token context on every step is expensive -- and the cost per step grows over time as context accumulates, even if the step itself produces a small output.

Second, the error budget for inefficiency shrinks. When an agent makes 15 LLM calls, a 2x cost inefficiency costs you twice the price of a short run. When an agent makes 2,000 LLM calls over 8 hours, the same inefficiency represents a large absolute dollar figure. The cost of ignoring prompt size, model selection, and token overhead at small scale becomes the cost of a failed project at large scale.

A rough estimate for a research agent running for 6 hours with a capable frontier model, if you are not careful about context management:

- 600 LLM calls averaging 8,000 input tokens each = 4.8M input tokens
- At a typical frontier model price of $3-5 per million tokens, that is $14-24 in input tokens alone
- Plus output tokens, plus retries, plus tool call parsing

That is not always prohibitive, but it is easy to blow past budgets when agents are running continuously and no one is watching.

## Context management over long runs

The central engineering problem in long-running agents is what to do with the accumulating conversation history. You have four main options, each with different cost and quality tradeoffs.

**Sliding window**: Keep only the last N tokens of history in each prompt. Simple to implement, zero extra cost, no state management overhead. The downside is that the agent forgets everything outside the window. For tasks where early context matters -- a research plan established in the first hour, a constraint stated at the start -- a sliding window will cause the agent to contradict itself or repeat work.

**Summarization**: At regular intervals (or when the context exceeds a threshold), have the agent summarize the conversation history into a compact representation, then replace the raw history with the summary. This preserves the gist of prior work at much lower token cost. The tradeoff is that summaries lose detail, and if the agent summarizes too aggressively, it may lose important nuance. A good heuristic: summarize every 30-50 steps, keeping the full raw history for the last 10-15 steps and a summary for everything before.

**External memory**: Store conversation history and intermediate results outside the prompt -- in a vector database or a structured state store -- and retrieve relevant pieces at each step via semantic search. This is the most flexible approach for very long runs. It requires more infrastructure and adds retrieval latency, but it keeps prompt sizes bounded regardless of how long the agent has been running. It works especially well for tasks with structured intermediate state, like a research agent that accumulates sources and findings.

**Hierarchical summarization**: A refinement of summarization, where you maintain summaries at multiple levels of detail -- a one-paragraph high-level summary updated every 50 steps, a more detailed recent-events summary updated every 10 steps, and raw history for the last 5 steps. The agent includes all three in its prompt, with the high-level summary providing continuity and the recent history providing specificity. This preserves more context than a single summary while keeping token counts manageable.

Here is a basic sliding-window with summarization implementation in Python:

```python
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class ContextManager:
    max_recent_tokens: int = 8000
    summarize_every_n_steps: int = 30
    
    recent_messages: List[dict] = field(default_factory=list)
    running_summary: Optional[str] = None
    steps_since_summary: int = 0
    
    def add_message(self, role: str, content: str):
        self.recent_messages.append({"role": role, "content": content})
        self.steps_since_summary += 1
    
    def build_prompt_messages(self, system: str) -> List[dict]:
        messages = [{"role": "system", "content": system}]
        
        if self.running_summary:
            messages.append({
                "role": "user",
                "content": f"Summary of prior work:\n{self.running_summary}"
            })
            messages.append({
                "role": "assistant",
                "content": "Understood. Continuing from where we left off."
            })
        
        messages.extend(self.recent_messages[-20:])
        return messages
    
    def should_summarize(self) -> bool:
        return self.steps_since_summary >= self.summarize_every_n_steps
    
    def apply_summary(self, new_summary: str):
        self.running_summary = new_summary
        self.recent_messages = self.recent_messages[-10:]
        self.steps_since_summary = 0
```

The summarization call itself is worth doing with a smaller, cheaper model rather than your main reasoning model. Summarizing 2,000 tokens of history does not require the same capability as planning the next research step.

## Model routing to control cost

Not every step in a long-running agent requires your most capable model. A research agent, for example, might use a large frontier model for initial planning and synthesis, but most of its work is routine: fetching a URL, extracting structured data from a document, checking whether a source is relevant. These steps can be done with a smaller, cheaper model at equivalent quality.

Routing by step type is straightforward to implement if you structure your agent around named step types:

```python
MODEL_ROUTING = {
    "plan": "large-frontier-model",
    "synthesize": "large-frontier-model",
    "summarize": "small-fast-model",
    "extract": "small-fast-model",
    "classify": "small-fast-model",
    "write": "medium-model",
}

def get_model_for_step(step_type: str) -> str:
    return MODEL_ROUTING.get(step_type, "medium-model")
```

The cost savings from routing can be substantial. If 70% of your agent's steps are routine extraction and classification tasks, and you route those to a model that is 10x cheaper per token, you reduce your overall inference cost by a large amount without meaningful quality loss.

One pattern that works well is a two-stage approach for each step: run the cheap model first, and escalate to the expensive model only if the cheap model signals uncertainty or produces an output that fails validation. This keeps costs low for the majority of well-formed steps while ensuring quality on the edge cases.

## Cost accounting and budget enforcement

For a 10-minute agent, you can mentally track costs. For an 8-hour agent, you need instrumentation.

At minimum, track tokens in and out for every LLM call and accumulate totals in your agent's state. Check the running total before each step and halt if you have exceeded a configured budget:

```python
@dataclass
class CostTracker:
    input_token_cost_per_m: float
    output_token_cost_per_m: float
    budget_usd: float
    
    total_input_tokens: int = 0
    total_output_tokens: int = 0
    
    def record_call(self, input_tokens: int, output_tokens: int):
        self.total_input_tokens += input_tokens
        self.total_output_tokens += output_tokens
    
    @property
    def cost_usd(self) -> float:
        return (
            self.total_input_tokens / 1_000_000 * self.input_token_cost_per_m +
            self.total_output_tokens / 1_000_000 * self.output_token_cost_per_m
        )
    
    def check_budget(self):
        if self.cost_usd >= self.budget_usd:
            raise BudgetExceeded(
                f"Agent exceeded budget of ${self.budget_usd:.2f}. "
                f"Spent ${self.cost_usd:.2f} over {self.total_input_tokens + self.total_output_tokens} tokens."
            )
```

A hard budget ceiling prevents runaway cost from loops, unexpected context growth, or model errors that cause the agent to spin. Budget enforcement should be checked before each LLM call, not just at the end, so the agent can save its state and exit gracefully rather than failing mid-step.

Logging cost breakdowns by step type also helps you identify which steps are most expensive. Often 80% of cost comes from 20% of step types, and that is where optimization effort is best spent.

## Checkpointing for resumability

A long-running agent that fails at hour 6 of an 8-hour run has wasted everything if it cannot resume. Checkpointing serializes agent state to durable storage at regular intervals, so a restart can pick up from the last checkpoint rather than from scratch.

What to include in a checkpoint:

- The running summary of prior work (or full history if manageable)
- Intermediate results that are expensive to reproduce (gathered sources, generated artifacts)
- The current step index and any step-specific state
- The cost tracker state, so budget enforcement survives restarts
- A task queue or plan, with completed steps marked

Checkpoint frequency is a tradeoff between checkpoint overhead (each checkpoint is an I/O write plus potentially a summarization call) and the amount of work lost on failure. For most long-running agents, checkpointing every 10-20 steps is reasonable.

```python
import json
from pathlib import Path

def save_checkpoint(state: dict, checkpoint_path: Path):
    tmp = checkpoint_path.with_suffix(".tmp")
    tmp.write_text(json.dumps(state, indent=2))
    tmp.rename(checkpoint_path)  # atomic rename avoids partial writes

def load_checkpoint(checkpoint_path: Path) -> Optional[dict]:
    if checkpoint_path.exists():
        return json.loads(checkpoint_path.read_text())
    return None
```

The atomic rename pattern (write to a temp file, then rename) avoids corrupting the checkpoint if the process is killed mid-write.

For agents running in distributed environments, use a durable store -- S3, a database, or a distributed cache -- rather than local files. The checkpoint key should include a run ID so that parallel runs do not overwrite each other.

## Rate limiting and throttling

Inference providers have rate limits, typically expressed in requests per minute and tokens per minute. A long-running agent that runs at full speed can hit these limits, which causes requests to fail or queue. Unexpected rate limit errors mid-run can break agent logic if not handled gracefully.

For long-running agents, it is often worth deliberately throttling below the rate limit ceiling, especially during periods when the agent's work is not time-sensitive. Spreading 2,000 LLM calls over 8 hours at a steady rate of 4 per minute is much less likely to cause problems than bursting to the rate limit and triggering backpressure.

Exponential backoff with jitter is the standard pattern for handling rate limit responses:

```python
import asyncio
import random

async def call_with_retry(client, messages, max_retries=5):
    for attempt in range(max_retries):
        try:
            return await client.complete(messages)
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(wait)
```

If your agent has natural phases -- a planning phase, a research phase, a synthesis phase -- you can also throttle between phases rather than between individual steps. This keeps each phase fast while keeping the overall rate manageable.

## Monitoring during long runs

For short agent runs, you can check the output after the fact. For long runs, you want visibility while the agent is executing, both to catch runaway cost and to monitor progress.

A minimal monitoring setup emits structured events from the agent loop to a log stream:

```python
import logging
import json

logger = logging.getLogger("agent")

def log_step(step_index, step_type, duration_ms, tokens_in, tokens_out, cumulative_cost):
    logger.info(json.dumps({
        "step": step_index,
        "type": step_type,
        "duration_ms": duration_ms,
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
        "cumulative_cost_usd": cumulative_cost,
    }))
```

From there, you can stream these logs to any observability platform and set alerts on cumulative cost, error rate, or steps per hour. If the agent slows down significantly (steps per hour dropping), that usually indicates context growth driving up per-step latency, and it is a good trigger for a manual summarization pass.

## Putting it together

The engineering for long-running agents is not exotic -- it is mostly careful accounting. Track context size and summarize before it gets out of hand. Route cheap steps to cheaper models. Enforce a budget ceiling and checkpoint often enough that failures are recoverable. Throttle to avoid rate limit failures. Log structured events so you can tell whether the agent is making progress.

The cost and reliability properties that are acceptable to ignore at small scale become the main engineering constraints at long runtime scale. Getting these patterns in place before your agent runs for the first time at multi-hour scale is much easier than diagnosing a $200 failed run after the fact.

---

General Compute's API supports OpenAI-compatible requests for all major open-source models, with high tokens-per-second throughput that keeps per-step latency low across long runs. For long-running agents, the practical benefit is that each individual step finishes faster, which means the agent makes more progress per hour for the same wall-clock budget. To get started:

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key",
)
```

See the [GeneralCompute docs](https://generalcompute.com/docs) for available models and rate limit tiers.