AI Inference Costs in 2025: How to Calculate and Reduce Your LLM API Spend

Markdown body

If you've run a production AI application for more than a few weeks, you've probably been surprised by an inference bill. LLM API costs compound in ways that aren't obvious from the per-token pricing alone: output tokens cost more than input tokens, long context windows multiply your spend, and an agentic workflow that makes 50 API calls per task can blow past 100x the cost of a single call.

This guide covers how LLM API pricing actually works, walks through three real-world cost calculations, and gives you eight concrete strategies to reduce your spend without degrading quality.

## How LLM API Pricing Works: The Fundamentals

### Input Tokens vs Output Tokens: Why Output Costs 3-5x More

Every major inference provider prices input and output tokens separately, and output tokens almost always cost more -- typically 3 to 5 times more. The asymmetry exists because output generation is more compute-intensive. During the decode phase, the model generates tokens one at a time in a memory-bandwidth-bound loop that's hard to parallelize. The prefill step that processes your input, by contrast, can run in parallel across all input positions at once.

For most chat applications, your input tokens outnumber output tokens in raw count (long system prompts, conversation history), but the higher output price per token keeps the bill higher than you'd expect from input count alone.

See [LLM Inference Explained](/blog/llm-inference-explained) for a deeper look at the prefill/decode split and why it shapes pricing the way it does.

### Context Window: The Hidden Cost Multiplier

Every token in a request -- system prompt, conversation history, retrieved documents -- counts as input tokens. A 4,000-token system prompt sent on every request at 1M requests/day is 4 trillion input tokens per day. Multi-turn conversations are especially tricky. If you naively append the full conversation history on each turn, your input token count grows linearly with conversation length. A 20-turn conversation has roughly 10x the input cost of the first turn.

### Model Tiers: How Capability Affects Price

Providers tier their pricing by model capability. A frontier model like GPT-4o costs 10 to 50x more per token than a smaller model like Llama 3.1 8B on a managed API. For many tasks -- classification, summarization, simple Q&A -- a smaller model at a fraction of the cost performs nearly as well. The biggest cost wins often come from routing requests to the smallest model that can handle them reliably.

### Batch Discounts and Committed Use Plans

Most providers offer 40-50% discounts for asynchronous batch processing (jobs that don't need real-time responses). If you have a document processing pipeline that runs overnight, batch mode can cut your bill in half. Committed use agreements typically kick in at $1,000-$5,000/month and can get you 20-40% off list pricing.

---

## AI Inference Cost Calculator: Real-World Formulas

### Formula 1: Simple Single-Turn Request

```
cost = (input_tokens / 1_000_000 * input_price) + (output_tokens / 1_000_000 * output_price)
```

For a request with 500 input tokens and 200 output tokens using GeneralCompute ($0.10/1M input, $0.30/1M output):

```
cost = (500 / 1_000_000 * 0.10) + (200 / 1_000_000 * 0.30)
     = $0.000050 + $0.000060
     = $0.000110 per request
```

### Formula 2: Multi-Turn Conversation Cost

With naive history concatenation, input tokens grow with each turn:

```
turn_input_tokens[n] = system_prompt_tokens + sum(all_prior_messages) + current_user_message
total_cost = sum(cost(turn) for turn in conversation)
```

A 10-turn conversation with a 200-token system prompt, 150-token user messages, and 100-token assistant replies accumulates roughly 12,250 input tokens and 1,000 output tokens total.

At GeneralCompute pricing: ($0.00123 + $0.00030) = **$0.00153 per conversation**

### Formula 3: Monthly Production Estimate

```python
def monthly_cost(daily_requests, avg_input_tokens, avg_output_tokens,
                 input_price_per_1m, output_price_per_1m):
    monthly_requests = daily_requests * 30
    input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * input_price_per_1m
    output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * output_price_per_1m
    return input_cost + output_cost

# Example: 33,333 requests/day, 400 input, 150 output
monthly_cost(33_333, 400, 150, 0.10, 0.30)
# Returns approximately $72/month on GeneralCompute
```

### Example A: Chatbot with 100,000 Users per Month

Assumptions: 5-turn average conversation, 400 input tokens per turn (system prompt + history), 150 output tokens per turn.

- Total input tokens: 100,000 users x 5 turns x 400 tokens = 200M tokens
- Total output tokens: 100,000 users x 5 turns x 150 tokens = 75M tokens

| Provider | Input cost | Output cost | Total/month |
|---|---|---|---|
| GeneralCompute | $20 | $22.50 | **$42.50** |
| Groq | $30 | $33.75 | $63.75 |
| Together AI | $40 | $45.00 | $85.00 |
| GPT-4o | $500 | $750 | $1,250 |

### Example B: Coding Agent, 10,000 API Calls per Day

Assumptions: 800 input tokens per call (system prompt + code context), 400 output tokens (suggestions or edits), 30 days.

- Total input tokens: 10,000 x 30 x 800 = 240M tokens
- Total output tokens: 10,000 x 30 x 400 = 120M tokens

| Provider | Input cost | Output cost | Total/month |
|---|---|---|---|
| GeneralCompute | $24 | $36 | **$60** |
| Groq | $36 | $54 | $90 |
| Together AI | $48 | $72 | $120 |
| GPT-4o | $600 | $1,200 | $1,800 |

### Example C: Document Processing Pipeline, 1M Pages per Month

Assumptions: 600 tokens per average page, 120-token summary output per page.

- Total input tokens: 1M x 600 = 600M tokens
- Total output tokens: 1M x 120 = 120M tokens

| Provider | Input cost | Output cost | Total/month |
|---|---|---|---|
| GeneralCompute | $60 | $36 | **$96** |
| Groq | $90 | $54 | $144 |
| Together AI | $120 | $72 | $192 |
| GPT-4o | $1,500 | $1,200 | $2,700 |

For batch-eligible workloads like document processing, async batch mode cuts these numbers by up to 50%.

---

## Provider Cost Comparison (Q3 2026)

### Price per 1M Input Tokens

| Provider | Small model | Large/frontier model |
|---|---|---|
| GeneralCompute | $0.10 | $0.50 |
| Groq | $0.15 | $0.60 |
| Fireworks AI | $0.20 | $0.90 |
| Together AI | $0.20 | $0.80 |
| OpenAI (gpt-4o-mini) | $0.15 | -- |
| OpenAI (gpt-4o) | -- | $2.50 |
| Anthropic (claude-3-haiku) | $0.25 | -- |

### Price per 1M Output Tokens

| Provider | Small model | Large/frontier model |
|---|---|---|
| GeneralCompute | $0.30 | $1.50 |
| Groq | $0.45 | $1.80 |
| Fireworks AI | $0.50 | $2.70 |
| Together AI | $0.60 | $2.40 |
| OpenAI (gpt-4o-mini) | $0.60 | -- |
| OpenAI (gpt-4o) | -- | $10.00 |
| Anthropic (claude-3-haiku) | $1.25 | -- |

Prices change frequently. Always verify on provider pricing pages before committing to a budget.

### Cost Efficiency: Tokens Per Second Per Dollar

Speed affects your total infrastructure cost too. A slower provider means your application servers hold open connections longer, handling fewer requests per hour. The table below normalizes output throughput against blended cost for Llama 3.1 70B-class models.

| Provider | Output TPS | Blended cost/1M tokens | TPS per $ |
|---|---|---|---|
| GeneralCompute | 180 | $0.90 | **200** |
| Groq | 150 | $1.13 | 133 |
| Fireworks AI | 110 | $1.60 | 69 |
| Together AI | 95 | $1.40 | 68 |

See [What Is Tokens Per Second (TPS)?](/blog/what-is-tokens-per-second-tps) for more on how throughput affects your overall cost model.

---

## The Hidden Costs No One Talks About

### Rate Limit Penalties: The Cost of Retries

When you hit a rate limit, your application retries. Each retry burns more input tokens (resending the full prompt) and adds latency. A retry loop that fires five times before succeeding costs 5x the token spend for that request. Exponential backoff with jitter prevents retry storms, but the deeper fix is choosing a provider with headroom in their concurrency limits.

### Cold Start Penalties for Batch APIs

Batch APIs are cheaper but can have cold start delays of 30 seconds to several minutes for the first requests in a job. If your pipeline has sequential steps where step 2 consumes step 1's output, cold starts compound. Truly async offline workloads absorb this fine, but pipelines with intermediate dependencies should factor in the latency cost.

### Engineering Overhead: The True Cost of Self-Hosting

Self-hosting looks cheap on paper if you only count GPU rental. A single A100 instance at cloud prices runs around $2.50-$3.50/hour, or $1,800-$2,520/month. But that assumes 100% utilization, zero downtime, no engineering time for setup and maintenance, and no on-call burden when something breaks at 2am. Self-hosting a production inference server typically requires 0.5 to 1 full-time engineer to maintain. At $150k/year loaded cost, that's $12,500/month in engineering overhead -- more than most startups pay in managed API fees.

---

## 8 Proven Strategies to Reduce Your AI Inference Costs

### 1. Use Smaller Models for Simpler Tasks (Model Routing)

Not every request needs a 70B model. A classification task (is this message in-scope or out-of-scope?) can run on a 7B or 8B model at 1/10 the cost. Route requests by complexity: detect intent, then pick the smallest model that can handle it reliably. A router that correctly identifies 60% of requests as simple can reduce costs by 40-60%.

### 2. Implement Prefix Caching for Repeated System Prompts

If your system prompt is the same across all requests, prefix caching lets you avoid re-processing it on every call. Providers that support it charge 0-10% of normal input price for cached tokens. A 1,000-token system prompt sent on 1M requests/month costs $100 normally; with prefix caching it's nearly free after the first call. See [Prefix Caching](/blog/prefix-caching-why-repeated-prompts-shouldnt-cost-you-twice) for implementation details.

### 3. Use Quantized Models (INT4/FP8)

Quantized models run faster and often at lower cost since providers can fit more of them on the same hardware. A FP8 checkpoint of Llama 3.1 70B performs within 1-2% of BF16 on most benchmarks but generates tokens 30-50% faster. Providers pass some of this efficiency back as lower prices. See [Quantization Explained](/blog/quantization-explained-int4-gguf-gptq) for the full quality tradeoff breakdown.

### 4. Compress Prompts to Reduce Input Token Count

Audit your system prompts and conversation history management. Common sources of waste: redundant instructions repeated in every turn, full conversation history when only the last 3-5 turns are relevant, whitespace and verbose formatting that adds tokens without adding information. Tools like LLMLingua can reduce input token count by 2-4x for information-dense prompts with minimal quality loss.

### 5. Batch Non-Urgent Requests with Async APIs

Batch endpoints offer 40-50% discounts on most platforms. Document summarization, data extraction pipelines, evaluation runs, and embedding generation are all good candidates. The tradeoff is latency: batch jobs process within minutes to hours rather than seconds.

### 6. Cache Deterministic Outputs at the Application Layer

If the same input always produces the same output (FAQ answers, fixed classifications, template fills), cache the result and skip the API call entirely. A Redis cache in front of your inference calls can eliminate 20-40% of requests for applications with repetitive query patterns.

### 7. Negotiate Volume Discounts Above $1,000/Month

Most providers have unpublished volume tiers. If you're spending over $1,000/month, contact their sales team. Discounts of 20-40% are common for committed use agreements at $5,000+/month. Providers prefer predictable revenue over maximizing per-token margin.

### 8. Choose Faster Providers to Reduce Latency Overhead

A faster provider reduces [TTFT](/blog/what-is-time-to-first-token-ttft) and end-to-end latency, which means your application infrastructure can process more requests per hour with fewer resources. A provider that's 2x faster can let you halve the number of application servers needed to meet a given p95 latency target -- a real infrastructure cost savings on top of the per-token savings.

For a full treatment of these and other optimization techniques, see [Fast AI Inference: 8 Proven Techniques](/blog/fast-ai-inference).

---

## Self-Hosting vs Managed API: The Real Cost Comparison

### GPU Cloud Rental at Scale

An H100 80GB instance costs roughly $3.50-$4.50/hour on major cloud providers ($2,520-$3,240/month). Running Llama 3.1 70B in BF16 requires at least two H100s for reasonable throughput, putting base hardware at $5,000-$6,500/month -- before storage, networking, or reserved capacity premiums.

At GeneralCompute pricing, $5,000/month buys roughly 5.5 billion output tokens. Most applications don't come anywhere close to that volume, so self-hosting loses on cost unless you're operating at significant scale.

### Operational Overhead: Engineers, Maintenance, On-Call

Beyond hardware: monitoring, auto-scaling, model updates, security patching, and 24/7 on-call coverage. That operational load typically costs $10,000-$20,000/month in loaded engineering compensation once you factor in the people responsible for keeping it up.

### The Crossover Point: When Does Self-Hosting Make Sense?

Self-hosting starts to make economic sense when you have sustained utilization that requires at least 4-8 GPUs running 24/7, along with a team large enough to absorb the operational work. In practice that's usually $50,000+/month in managed API spend before the total cost of ownership math flips. Below that threshold, managed APIs almost always win.

---

## FAQ

### How expensive is AI inference per request?

For a typical chat request with 500 input tokens and 300 output tokens, cost ranges from about $0.00011 (GeneralCompute small model) to $0.00425 (GPT-4o). At 1M requests/month, that's $110 to $4,250/month.

### Is GPT-4o cheaper than open-source alternatives?

No. GPT-4o costs $2.50/1M input tokens and $10/1M output tokens. Open-source models like Llama 3.1 70B on managed APIs run $0.50-$0.90/1M input and $1.00-$1.80/1M output. For many tasks, open-source models at 10-20% of the price match or exceed GPT-4o performance.

### What is the cheapest LLM API in 2025?

For small to mid-size models, GeneralCompute offers some of the lowest prices alongside fast inference speeds, which also reduces your infrastructure overhead. For large frontier models, open-source alternatives on managed APIs are consistently cheaper than proprietary frontier APIs.

---

If you want to see what your specific workload would cost on GeneralCompute's infrastructure, the [API](https://generalcompute.com) is OpenAI-compatible -- swap in your existing code with a one-line base URL change and run the same requests. Check the [fast inference techniques guide](/blog/fast-ai-inference) for optimization strategies that apply regardless of provider.