Agentic RAG: When Retrieval and Generation Both Need to Be Fast

Standard RAG has a simple structure: the user asks a question, your system retrieves relevant chunks, and the LLM generates a response conditioned on those chunks. One retrieval step, one generation step, done. The latency budget is bounded and predictable.

Agentic RAG looks different. The agent retrieves, generates, reasons about what it found, decides what else it needs, retrieves again, generates again, and continues until it has enough to produce a final answer. This loop can iterate anywhere from two times to twenty or more, depending on the task. Each iteration adds retrieval latency plus generation latency. By the time the agent finishes, you have not paid once for retrieval and once for generation -- you have paid for both at every step of the loop.

That is the core problem this post addresses. When retrieve-then-generate runs in a loop, the latency at each stage multiplies by the number of iterations. Slow retrieval and slow generation, which are individually manageable in single-turn RAG, become the dominant engineering constraints in agentic RAG.

What the loop actually looks like

A concrete example helps. Consider a research agent tasked with answering a complex question about a large codebase or document corpus:

1. The agent receives the question and retrieves the most relevant chunks from a vector store.
2. It generates an initial reasoning step, deciding what it knows and what it still needs.
3. Based on that reasoning, it formulates follow-up queries and retrieves again.
4. It generates another reasoning step, integrating the new information.
5. This continues until the agent determines it has enough to answer, at which point it generates the final response.

A five-iteration loop with 400ms retrieval and 800ms generation per step takes at least 6 seconds of wall-clock time, assuming no parallelism and ignoring overhead. Ten iterations takes 12 seconds. And these numbers assume retrieval and generation are fast -- many production RAG pipelines are significantly slower than that.

The other thing that changes with loops is context growth. Each iteration adds retrieved content and generated reasoning to the conversation history. By iteration five or six, you are sending a substantially larger prompt than you were at iteration one. Since inference providers charge per token and longer prompts take longer to process, your per-step cost and latency both increase as the loop progresses.

Where the time goes in retrieval

Retrieval latency has several components, each of which matters at loop scale.

Embedding the query: Before you can search a vector store, you need to embed the query text. Embedding models have their own latency -- typically 20-100ms for a single query depending on the model size and infrastructure. Across ten iterations, that is 200ms-1000ms in embedding alone, before any search happens.

Vector search: The search itself against a populated vector store is usually fast -- most managed vector databases return results in 10-50ms for typical collection sizes. This is rarely the bottleneck, but it is not free.

Document loading and chunking overhead: After identifying relevant chunk IDs, many RAG systems need to fetch the actual document text from a separate store. If that store is a remote database or object storage, this adds a round-trip -- often 20-100ms. If your chunks are large (1,000-2,000 tokens), the volume of text being loaded also grows.

Reranking: Many production RAG pipelines add a reranker after the initial vector search to improve precision. Rerankers are cross-encoders that score each retrieved chunk against the query, which is much slower than vector search -- typically 50-300ms depending on the model and number of candidates. In single-turn RAG, this is often worth the cost. In a ten-iteration agentic loop, an extra 150ms per retrieval step adds 1.5 seconds to the total.

The practical takeaway: for agentic RAG, you want each retrieval step to be fast, which usually means accepting slightly lower recall in exchange for lower latency -- or finding ways to parallelize retrieval with other work.

Where the time goes in generation

On the generation side, the main variables are time-to-first-token (TTFT) and tokens-per-second throughput.

TTFT is the time between sending a request and receiving the first token of the response. In a streaming setup, this is what determines when the user (or the next stage of the pipeline) starts getting output. For agentic RAG, TTFT matters per step -- a 600ms TTFT across ten steps is 6 seconds in TTFT overhead alone, even before you count the time to generate the actual tokens.

Throughput matters when the agent is generating substantial reasoning or when the final answer is long. If your model generates 30 tokens/second and each reasoning step produces 200 tokens, each generation step takes about 6.7 seconds. Across ten steps, that is over a minute of generation time.

Context growth amplifies both of these. As the conversation history accumulates retrieved chunks and generated reasoning, the prompt that goes into each inference call gets longer. Longer prompts take more time to process, driving up TTFT. If you are using a model with a fixed context window and you are not truncating or summarizing, you will eventually hit that limit.

Patterns that help

Several patterns reduce the compound latency of agentic RAG. Most of them require either parallelizing work, reducing unnecessary overhead, or making smarter decisions about when to retrieve at all.

Parallel retrieval

If the agent needs to retrieve from multiple queries or multiple data sources at a given step, retrieve them in parallel rather than sequentially. This is the most straightforward optimization and often has a large impact. A step that would take 400ms + 400ms = 800ms in serial takes 400ms when both retrievals run simultaneously.

import asyncio

async def parallel_retrieve(queries: list[str], retriever) -> list[list[dict]]:
    tasks = [retriever.aretrieve(q) for q in queries]
    return await asyncio.gather(*tasks)

The agent's reasoning step can identify multiple information needs at once, submit them as parallel retrieval queries, and integrate all results in the next step. This trades sequential latency for a small amount of over-retrieval, which is usually a good tradeoff.

Prefetch the likely next query

If the agent's reasoning pattern is predictable -- for example, it always follows a broad retrieval with a narrower clarification retrieval -- you can start the next retrieval before the current generation step has finished. While the LLM is generating its reasoning about the current retrieved chunks, you can begin embedding and searching for the likely follow-up.

This requires some prediction of what the agent will want next, which is domain-specific. But even rough prefetching can eliminate full round-trips from the critical path.

Stream generation and act on partial output

If your agent framework allows it, start processing the generated output as it arrives rather than waiting for the complete response. This is particularly useful when the agent's output includes explicit signals that can be parsed incrementally -- a structured JSON output with query fields that appear early in the stream.

With streaming, you can begin the next retrieval query as soon as the agent has emitted the query text, even if it has not finished generating the rest of its reasoning. This overlaps retrieval latency with generation latency rather than sequencing them.

async def streaming_step_with_prefetch(client, messages, retriever):
    buffer = ""
    next_query = None
    retrieval_task = None

    async for chunk in client.chat.completions.create(
        model="your-model",
        messages=messages,
        stream=True,
    ):
        delta = chunk.choices[0].delta.content or ""
        buffer += delta

        # As soon as we can parse a retrieval query from the stream, kick it off
        if next_query is None:
            parsed = try_parse_query(buffer)
            if parsed:
                next_query = parsed
                retrieval_task = asyncio.create_task(retriever.aretrieve(parsed))

    results = await retrieval_task if retrieval_task else []
    return buffer, results

Reduce reranking to where it matters

Reranking is worth doing on the first retrieval, where recall is highest and precision is lowest. On later iterations where the agent is doing targeted lookups for specific information, the initial vector search is often precise enough without reranking. Skip the reranker on follow-up retrievals to save 50-300ms per step.

Bound context growth explicitly

The retrieved chunks that accumulate in the conversation history are often not all equally useful. Later steps do not need the full text of chunks that were retrieved in early steps -- they need the agent's reasoning about those chunks, which is more compact.

Instead of appending raw retrieved chunks to the history at each step, have the agent summarize what it learned from each retrieval. This keeps the prompt size from growing linearly with iterations. Paired with a sliding window that keeps only the last few raw retrieval results, you can keep prompt size roughly constant across the loop, which holds TTFT and per-step cost stable.

def build_iterative_context(steps: list[dict], max_recent: int = 2) -> list[dict]:
    messages = []
    
    # Older steps: include only the summary, not the raw chunks
    for step in steps[:-max_recent]:
        messages.append({"role": "user", "content": step["summary"]})
        messages.append({"role": "assistant", "content": step["reasoning"]})
    
    # Recent steps: include the full retrieved context
    for step in steps[-max_recent:]:
        chunks_text = "\n\n".join(c["text"] for c in step["chunks"])
        messages.append({"role": "user", "content": f"Retrieved:\n{chunks_text}"})
        messages.append({"role": "assistant", "content": step["reasoning"]})
    
    return messages

Use a smaller model for reasoning, larger for synthesis

Not every step in an agentic RAG loop requires your most capable model. Intermediate reasoning steps -- deciding what to retrieve next, summarizing what was retrieved, checking whether a source is relevant -- are often well within the capability of a smaller, faster model.

Route to a larger model only for the final synthesis step where the agent is generating the response the user will actually see. This can cut inference cost and per-step latency substantially across the middle steps of a long loop.

STEP_MODEL_ROUTING = {
    "plan": "large-model",
    "reason": "small-fast-model",
    "summarize": "small-fast-model",
    "synthesize": "large-model",
}

Measuring what matters

To optimize an agentic RAG system, you need to measure the right things. A single end-to-end latency number for the full agent run hides where the time is actually going.

Track these per step:

• Retrieval latency (embedding time + search time + document load time)
• TTFT for each generation call
• Output token count per step (to estimate generation time)
• Prompt token count per step (to track context growth)
• Which step in the loop you are on (to identify whether later steps are slower)

With this data, you can identify whether your bottleneck is retrieval or generation, and whether latency is growing across iterations (context growth) or staying flat (you have bounded it effectively).

A simple structured log per step is enough to start:

import time
import logging
import json

logger = logging.getLogger("rag_agent")

def log_step(step_idx, retrieval_ms, ttft_ms, output_tokens, prompt_tokens):
    logger.info(json.dumps({
        "step": step_idx,
        "retrieval_ms": retrieval_ms,
        "ttft_ms": ttft_ms,
        "output_tokens": output_tokens,
        "prompt_tokens": prompt_tokens,
    }))

Plot prompt_tokens over step index. If it is growing steeply, context is accumulating and you need summarization or truncation. If retrieval_ms dominates total step time, focus on parallel retrieval and reranker reduction. If ttft_ms is the main factor, faster inference is the lever.

The asymmetry of optimization

One useful observation about agentic RAG: retrieval and generation have different optimization levers, and they are not equally easy to pull.

Retrieval latency is largely determined by infrastructure choices -- which vector database you use, how you size your embedding model, whether you run reranking. These are one-time architectural decisions that are hard to change once a system is in production, but they are also choices you can make deliberately at design time.

Generation latency is determined by model size and inference throughput. The fastest way to reduce generation latency per step is to use a faster inference provider. A model that generates 200 tokens/second versus 30 tokens/second cuts generation time by 6x without any changes to your prompts, retrieval logic, or agent design. For agentic RAG with many generation steps, fast inference has a linear impact on total wall-clock time.

This is why the two need to be considered together. Optimizing retrieval while leaving generation slow, or vice versa, still leaves you with a compound latency problem. The fastest step in a sequential chain does not matter if the other steps remain slow.

---

General Compute provides inference at high tokens-per-second throughput across major open-source models, with low TTFT that keeps each step in an agentic loop fast. The OpenAI-compatible API means you can swap in a faster endpoint with minimal code changes:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key",
)

For agentic RAG workloads, see the GeneralCompute docs for current model options and latency benchmarks.