We raised $15M to build the world's fastest neocloud.Read
ai inference pricinggroqfireworks aitogether aiinference apillm cost

Inference API Pricing Guide: Groq vs Fireworks vs Together vs GeneralCompute

General Compute·

The per-token rate on a provider's pricing page is a starting point, not the whole picture. When you're picking an inference API for a production workload, the advertised number is only one factor in what you'll actually pay each month.

This guide breaks down pricing across four popular inference providers -- Groq, Fireworks AI, Together AI, and GeneralCompute -- covering listed rates, effective cost, and the factors that make your actual bill differ from the estimate you built in a spreadsheet.

How Inference API Pricing Works

Every provider charges separately for input tokens and output tokens. Input tokens are what you send (the prompt, system instructions, conversation history, tool definitions), and output tokens are what the model generates. Output tokens are almost always priced higher than input tokens because generating a token is computationally more expensive than processing one.

A typical pricing entry looks like this:

| Model | Input (per 1M tokens) | Output (per 1M tokens) | |-------|----------------------|------------------------| | Llama 3.1 8B | $0.06 | $0.09 | | Llama 3.1 70B | $0.50 | $0.75 |

The ratio between input and output prices varies by provider, but a 1.5x to 2x output multiplier is common.

For a quick mental model: if your app generates an average of 300 output tokens per request with a 500-token input, and you're running one million requests per month, you're using 500M input tokens and 300M output tokens monthly. At $0.50/M input and $0.75/M output, that's $250 + $225 = $475/month on one model and one provider. Context length and output verbosity scale that number fast.

Provider Pricing Breakdown

Prices change frequently -- always verify on each provider's current pricing page before making a decision. The figures below reflect rates as of mid-2026.

Groq

Groq uses custom LPUs (Language Processing Units) designed specifically for token generation throughput. Their pricing model is simple: pay per token, no seat licenses, no compute reservations required.

Representative rates:

| Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 1B | $0.04 | $0.04 | | Llama 3.1 8B | $0.05 | $0.08 | | Llama 3.3 70B | $0.59 | $0.79 | | Mixtral 8x7B | $0.24 | $0.24 |

Groq's main differentiator is speed -- throughput on their LPU infrastructure is consistently faster than GPU-based providers at the same model size. The tradeoff is model selection. Their catalog covers the most popular open models but it's narrower than Fireworks or Together. If the model you need is available, Groq is often the fastest option at a competitive price.

Rate limits are a common friction point on Groq's free tier. On paid plans, limits are more generous, but high-volume workloads may still encounter per-minute token caps during peak usage. They offer a batch pricing tier for workloads that don't need real-time turnaround.

Fireworks AI

Fireworks offers a broad model catalog with support for fine-tuned model hosting. Their pricing structure splits models into tiers based on parameter count, and they use flat input/output pricing on most models (same rate for both directions).

Representative rates:

| Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 1B Instruct | $0.01 | $0.01 | | Llama 3.1 8B Instruct | $0.20 | $0.20 | | Llama 3.1 70B Instruct | $0.90 | $0.90 | | Mixtral 8x22B | $1.20 | $1.20 | | Qwen 2.5 72B | $0.90 | $0.90 |

Flat input/output pricing simplifies cost estimation for workloads where you don't know the exact token ratio upfront -- you just multiply total tokens by the rate.

One notable feature: Fireworks supports serving your own fine-tuned LoRA adapters on top of their base models. If you've fine-tuned Llama 3.1 70B for your use case, you can deploy that adapter through Fireworks without managing your own GPU cluster. There's an adapter storage cost and a per-request surcharge for adapter inference, worth factoring in if custom model serving is part of your plan.

Fireworks also offers dedicated deployment options alongside their serverless tier. Dedicated gives you a reserved GPU instance for consistent latency, but you pay for compute reservation time whether you're using it or not. This changes the cost model significantly compared to pay-per-token serverless.

Together AI

Together AI runs both serverless inference and dedicated GPU instances across a catalog of 100+ models. Fine-tuning and inference are handled through the same platform, which is useful if you're doing both.

Representative rates:

| Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.2 3B Instruct | $0.06 | $0.06 | | Llama 3.1 8B Instruct | $0.18 | $0.18 | | Llama 3.1 70B Instruct | $0.54 | $0.54 | | Llama 3.1 405B Instruct | $3.50 | $3.50 | | Qwen 2.5 72B Instruct | $0.72 | $0.72 | | DeepSeek V3 | $1.25 | $1.25 |

Together uses flat input/output pricing for most models, similar to Fireworks.

For offline processing jobs and bulk generation workloads, Together offers discounted batch inference that can cut costs by 40-60% compared to real-time serverless rates. If you're running nightly pipelines, data processing tasks, or evaluation jobs, the batch tier is worth looking at.

Fine-tuning pricing is separate from inference -- they charge by compute hour during training, then per token for serving the resulting model. If you're comparing total platform cost (training plus inference), run the math on both stages.

GeneralCompute

GeneralCompute runs custom ASIC infrastructure built specifically for LLM inference, with a focus on throughput and low latency. The API is OpenAI-compatible, so migration is usually a base_url swap and an API key change.

Representative rates:

| Model | Input (per 1M) | Output (per 1M) | |-------|---------------|----------------| | Llama 3.1 8B | $0.05 | $0.07 | | Llama 3.1 70B | $0.45 | $0.65 | | Llama 4 Scout | $0.20 | $0.30 | | Llama 4 Maverick | $0.60 | $0.80 | | Qwen3-Coder | $0.50 | $0.70 | | DeepSeek R1 | $0.55 | $0.75 |

For latency-sensitive workloads (streaming chat, real-time voice, interactive coding tools), infrastructure speed affects the practical value of each token you're paying for. A slower provider at a lower per-token rate can end up costing more in user experience than a faster provider at a similar rate -- especially in agentic systems where inference happens multiple times per user action.

Beyond the Per-Token Rate: What Drives Your Actual Bill

Context Window Bloat

All providers charge for the complete input on every request, including system prompts, conversation history, tool definitions, and any other context you're passing. A 2,000-token system prompt costs 2,000 tokens on every single request.

At 1 million requests per month with a 10,000-token system prompt, you're spending 10 billion input tokens on the system prompt alone. At $0.45/M input, that's $4,500/month before any user messages or model output.

Some providers support prefix caching, which caches a repeated prompt prefix and avoids rebilling those tokens on subsequent requests. Check whether your provider supports this and whether it's automatic or requires explicit opt-in.

Rate Limits

Providers handle rate limits in different ways. Some throttle at the RPM (requests per minute) level, others at the TPM (tokens per minute) level, and some apply both simultaneously. Hitting a rate limit in production means either queuing delays or dropped requests -- both have downstream costs.

Before committing to a provider for a high-volume workload, test their actual rate limit behavior. What happens when you hit the cap -- do requests queue, or do they return a 429 immediately? How quickly do limits reset? The advertised limits and the production behavior are not always the same thing.

Minimum Token Billing

Some providers apply a minimum token charge per request, even for very short completions. If your workload involves many small requests (intent classification, structured extraction, routing decisions), this can meaningfully inflate your effective cost per call. Check the terms before building a high-frequency request pattern.

Variable Output Length

Listed pricing assumes a fixed output length, but real workloads produce variable outputs. If you're comparing providers for tasks where the model tends to be verbose (detailed code generation, long explanations, multi-step reasoning), a small difference in output price compounds quickly across millions of requests.

Track your actual input/output token ratio from your production API responses before using the pricing page numbers to build a budget.

Calculating Your Real Monthly Cost

Use this formula:

monthly_cost = (avg_input_tokens * monthly_requests * input_price_per_token)
             + (avg_output_tokens * monthly_requests * output_price_per_token)

Example: Llama 3.1 70B on GeneralCompute at $0.45/M input, $0.65/M output.

  • Average input: 800 tokens
  • Average output: 400 tokens
  • Monthly requests: 500,000
input_tokens_monthly  = 800 * 500_000          # 400M tokens
output_tokens_monthly = 400 * 500_000          # 200M tokens

input_cost  = 400_000_000 * (0.45 / 1_000_000)  # $180
output_cost = 200_000_000 * (0.65 / 1_000_000)  # $130

total = 310  # $310/month

Run this against each provider for your actual model and token counts.

A few adjustments to add to the base estimate:

  • Add 15-20% for overhead tokens (tool definitions, structured output schemas, conversation history management)
  • Factor in failed requests if you're running retries
  • Check for monthly minimums or enterprise plan requirements at your expected volume

Cost Optimization Strategies

Use the smallest model that meets your quality bar. An 8B model costs 5-10x less per token than a 70B model. For classification, routing, and simple extraction tasks, 8B models often perform close to 70B. Test both before defaulting to the larger one.

Cache repeated context. If your system prompt is long and static, prefix caching is the highest-leverage optimization available. Check whether your provider supports it and enable it.

Batch non-real-time workloads. For nightly pipelines and offline processing, batch inference tiers (available on Together and Groq) can cut costs by 40-60% compared to real-time serverless rates.

Use structured outputs. Getting the model to return only the data you need (a JSON object rather than a paragraph of explanation) cuts output tokens and reduces downstream processing.

Monitor token usage in production. Every provider returns token counts in the API response. Log prompt_tokens and completion_tokens per request. If your average input is growing over time, something in your prompt construction is accumulating tokens -- find it and trim it.

Evaluate dedicated vs. serverless at high volume. At very high request rates, dedicated GPU instances can be cheaper than pay-per-token serverless. Run the break-even math at your actual request rate before assuming serverless is always the better deal.

Provider Strengths at a Glance

| Factor | Groq | Fireworks | Together | GeneralCompute | |--------|------|-----------|----------|----------------| | Raw inference speed | Fastest | Good | Good | Fastest | | Model selection | Narrow | Broad | Broadest | Growing | | Fine-tuned model serving | No | Yes | Yes | No | | Batch pricing | Yes | No | Yes | No | | API compatibility | OpenAI | OpenAI | OpenAI | OpenAI | | Dedicated instances | No | Yes | Yes | Yes |

There's no single best provider for every workload. Groq and GeneralCompute lead on latency. Together and Fireworks lead on model breadth and fine-tuning support. At the 70B level, pricing across all four providers is within 2x of each other, which means your total monthly cost is more sensitive to your token usage patterns than to which provider you pick.

Getting Started

The most reliable way to compare actual costs is to run your workload against a few providers and measure both the bill and the latency. All four providers here use OpenAI-compatible APIs, so switching is straightforward -- change the base_url, swap the key, and run your existing code.

If you're starting fresh or want to benchmark, GeneralCompute's API is available with a free tier and no SDK changes required. Their documentation covers migration from OpenAI and other providers, typically a few lines of code to get running.

ModeHumanAgent