What Is an AI Inference Server? Cloud API vs Self-Hosted vs On-Premise Explained

An AI inference server is a process (or set of processes) that sits between your application and a machine learning model. It accepts incoming requests -- typically an HTTP POST with a prompt and parameters -- loads the model into GPU memory, runs the forward pass, and returns a completion. Everything else in the landscape of "how to run LLMs" is a variation on that core function.

What Is an AI Inference Server?

The Core Function: Accepting Requests and Returning Completions

When your code calls an LLM API with a prompt, something has to receive that request, schedule it against available hardware, run the model, and stream back tokens. The inference server is that thing. It handles:

• Receiving and validating requests
• Queuing and batching requests efficiently
• Running the model forward pass on GPU (or other accelerator)
• Streaming tokens back to the caller as they're generated
• Tracking usage, errors, and health metrics

How an Inference Server Differs from a Training Cluster

Training clusters are optimized for throughput over many GPU-hours, running large batches repeatedly with gradient updates. Inference is different: requests come in at unpredictable intervals, context lengths vary widely, and latency matters to end users. A training cluster that finishes a job in 3 hours is fine; an inference server that takes 3 seconds to return the first token is not.

This is why training and inference infrastructure are typically separate. Training optimizes for maximizing GPU utilization on a fixed workload. Inference optimizes for minimizing latency while handling variable concurrent load.

Key Components: Model Loading, Request Queue, Batching Engine, API Layer

A production inference server has four layers:

1. Model loading layer: loads model weights from storage into GPU memory, handles quantization, sharding across multiple GPUs if needed
2. API layer: exposes HTTP endpoints (usually OpenAI-compatible), validates requests, handles auth
3. Request queue: buffers incoming requests, manages concurrency limits, handles backpressure
4. Batching engine: groups multiple requests together to maximize GPU utilization -- in modern servers, continuous batching means requests are batched at the token iteration level rather than the request level, so GPUs rarely sit idle between requests

The Three Deployment Models: Cloud API, Self-Hosted, On-Premise

There are three fundamentally different ways to run inference: use a managed cloud API, self-host on rented cloud GPUs, or run entirely on-premise hardware.

Managed Cloud APIs: How They Work Under the Hood

With a managed API (like GeneralCompute, Groq, Together AI, or OpenAI), you send HTTP requests to a provider's endpoint. The provider handles everything: hardware procurement, model loading, scaling, monitoring, and maintenance. You get an API key, you call the API, and you pay per token.

Under the hood, these providers run large GPU or ASIC clusters that are always warm, with models pre-loaded and ready to serve. They invest in serving optimizations -- continuous batching, prefix caching, speculative decoding -- that would take your team months to implement and tune.

Self-Hosted Inference Servers: You Run the Infrastructure

Self-hosting means you rent GPUs (from AWS, GCP, CoreWeave, Lambda Labs, etc.) and run your own inference stack. You install vLLM, SGLang, or TensorRT-LLM, load the model yourself, and expose an API. You control everything and pay for the underlying compute.

Self-hosting gives you model flexibility (you can run any open-weights model, fine-tuned or not) and potential cost savings at scale. The tradeoff is operational burden: you're responsible for uptime, scaling, hardware failures, CUDA driver updates, and optimization work.

On-Premise Inference: Air-Gapped and Fully Controlled

On-premise means your own physical hardware in your own data center or colocation facility. No cloud provider in the picture. This is the only option for organizations with strict data sovereignty requirements or compliance mandates that prohibit sending data to third-party clouds.

The economics are hardest here: hardware is expensive, utilization is difficult to maximize across variable demand, and you need dedicated infrastructure teams.

Hybrid Approaches: Cloud Burst + On-Prem Baseline

Many large organizations run baseline on-premise capacity for predictable load (on-prem is cheapest per GPU-hour when fully utilized) and burst to cloud APIs during traffic spikes. This requires careful routing logic but can optimize cost while maintaining compliance for sensitive workloads.

Popular Open-Source Inference Server Options

If you're self-hosting, you'll choose from a small set of mature open-source frameworks.

vLLM: The Production Standard

vLLM is the most widely deployed open-source inference server. Its key contributions are PagedAttention (efficient KV cache management that virtually eliminates memory waste) and continuous batching (requests are batched at the token iteration level, so the GPU almost never sits idle between requests).

vLLM supports most major model architectures -- Llama, Mistral, Qwen, Gemma, DeepSeek, and more -- and runs on NVIDIA GPUs with solid ROCm support for AMD. It exposes an OpenAI-compatible API, so migrating existing code is usually a base_url swap.

Typical throughput with vLLM on a single H100: 2,000-4,000 tokens/sec for a 70B model at FP16, depending on batch size and sequence length.

SGLang: Best for Agents and Multi-Turn Workloads

SGLang was built by researchers at UC Berkeley and MIT with agentic workloads in mind. Its main differentiation is RadixAttention: a KV cache data structure that efficiently reuses cached attention states across requests that share a common prefix.

For multi-turn conversations or agent workflows where many requests share a system prompt and conversation history, SGLang can achieve substantially higher effective throughput than vLLM. If your workload is chatbot-heavy or you're running agents, it's worth benchmarking against vLLM directly before committing.

TGI (Text Generation Inference): HuggingFace's Option

HuggingFace's Text Generation Inference is the runtime used by their Inference Endpoints product. It's production-tested, has strong support for the HuggingFace Hub model format, and integrates well with the broader HF ecosystem.

TGI supports tensor parallelism (spreading a model across multiple GPUs) and flash attention. Performance is generally competitive with vLLM for single-request scenarios but less optimal for high-concurrency workloads.

TensorRT-LLM: Maximum Performance on NVIDIA Hardware

TensorRT-LLM is NVIDIA's own inference library, built on top of TensorRT. It compiles models into optimized CUDA kernels for a specific GPU architecture, which means setup is more involved (you build engine files per GPU type) but the resulting throughput can be 20-30% higher than vLLM for compute-bound workloads.

It's the right choice when you have a specific NVIDIA GPU target, a stable model you're running at scale, and the engineering time to manage the compilation workflow. It's not the right choice for rapid model iteration or broad architecture support.

Managed Inference APIs: The Cloud-Hosted Alternative

Rather than running your own stack, several providers offer managed inference with no infrastructure overhead.

GeneralCompute: ASIC-Powered, Fastest TTFT

GeneralCompute runs on custom ASIC infrastructure optimized specifically for token generation. The result is industry-leading time to first token (TTFT) -- critical for voice AI and interactive applications -- and high tokens per second (TPS) throughput across major open models including Llama 4, DeepSeek, and Qwen. The API is OpenAI-compatible, so integration is typically a one-line change to existing code.

Groq: LPU-Based Fast Inference

Groq uses their Language Processing Unit architecture, which excels at the sequential token-by-token generation phase of decoding. Groq's throughput numbers are strong on supported models, though model selection is narrower than GPU-based providers.

Together AI: Broad Model Selection

Together AI offers a wide catalog of open models via API and supports custom fine-tuned model deployments. Their strength is breadth: if you need to try many different models quickly without managing GPUs, it's a reasonable starting point.

Fireworks AI: Fine-Tuning and Custom Models

Fireworks specializes in deploying fine-tuned models and has tooling for LoRA fine-tuning on their platform. If your workflow involves frequent model iterations (fine-tune, deploy, test, repeat), Fireworks simplifies the deployment side.

Cloud API vs Self-Hosted: A Comprehensive Comparison

Latency: Which Approach Is Actually Faster?

Counterintuitively, managed APIs often beat self-hosted setups on latency, because providers invest in optimizations that most teams don't implement: speculative decoding, prefix caching, optimized kernels, hardware-level tuning. A freshly deployed vLLM instance with default settings will typically underperform a well-tuned managed API.

That said, self-hosted can win on latency when you dedicate a full GPU or cluster to a single model with no sharing, and you're geographically co-located with your application. Dedicated capacity eliminates queuing latency from shared infrastructure.

Cost at Scale: The Real Break-Even Analysis

Managed APIs charge per token. Self-hosting charges per GPU-hour, regardless of actual utilization. The break-even depends on your utilization rate.

A single H100 costs roughly $2.00-2.50/hr on spot pricing via providers like CoreWeave or Lambda Labs. At 80% utilization (which is high for most teams), that's $1.60-2.00/hr of effective compute. An H100 can generate roughly 2,000 tokens/sec at 70B scale (Llama 4 FP8), so at 80% utilization you're producing around 1,600 tokens/sec of output.

At a typical managed API price of $0.50/1M tokens for a 70B model, you'd spend $1.60-2.00 generating 3.2-4.0M tokens. Over an hour at 1,600 tokens/sec, you'd generate 5.76B tokens self-hosted. Self-hosting wins decisively at high utilization -- but most teams don't sustain 80% utilization. At 20% utilization, the effective self-hosted cost per token is roughly 4x higher than managed.

A rough rule: self-hosting becomes cheaper than managed APIs at sustained utilization above 30-40%, assuming you're not paying significant DevOps overhead.

Operational Burden: Engineering Hours Required

Self-hosting a production inference server requires ongoing engineering time: monitoring, scaling, CUDA driver updates, model version management, debugging latency regressions, handling hardware failures. Budget at least 0.5-1 full-time engineer equivalent for a small cluster running one or two models. For teams that are primarily building applications rather than infrastructure, managed APIs let them stay focused on the product.

Model Availability: What You Can and Cannot Run

Managed APIs support whatever models the provider chooses to host. If you need a specific fine-tuned checkpoint, a research model, or a model the provider doesn't carry, you'll need to self-host or use a provider that supports custom deployments.

Compliance and Data Privacy: Who Controls the Data?

Prompts sent to a managed API leave your network. For applications handling sensitive data (medical records, financial information, legal documents), this requires either a BAA with the provider or self-hosting. Check specific certifications before assuming compliance: SOC 2 Type II, HIPAA BAAs, and FedRAMP authorization vary by provider.

When Does Self-Hosting Make Economic Sense?

GPU Rental Math: Breakeven at What Monthly Volume?

A single H100 at $2.25/hr (spot) costs roughly $1,620/month. Running a 70B model at 40% average utilization, you produce approximately 2.8B tokens/month. At $0.50/1M output tokens managed, that volume costs $1,400 from a provider (plus input tokens). Self-hosting starts to look cheaper here -- but add 20-30% for DevOps time and it's closer than the raw numbers suggest.

For most teams, the crossover is somewhere between 2-5B tokens/month per model. At 10B+ tokens/month, the math strongly favors self-hosting unless you have high engineering overhead or compliance requirements that favor managed options.

Latency SLA Requirements That Force Self-Hosting

Some use cases require latency guarantees that shared managed APIs can't provide. Real-time voice AI targeting sub-150ms TTFT at P99, or edge inference that needs single-digit millisecond response times, may require dedicated hardware. When you're sharing infrastructure with other customers, you're subject to their traffic patterns.

Regulatory Requirements: HIPAA, GDPR, FedRAMP

Healthcare applications handling PHI need a HIPAA BAA. Financial services in certain jurisdictions need data to stay within specific geographic boundaries. Government applications often require FedRAMP authorization. These constraints may force self-hosting or on-premise regardless of cost analysis, unless you're working with a managed provider who holds the relevant certifications.

How to Choose the Right Inference Server Architecture

Decision Framework: 5 Questions Before Choosing

1. What's your current monthly token volume? Under 1B tokens/month, managed APIs almost always win on total cost. Over 10B, self-hosting is worth doing the math.
2. Do you need models not available via managed APIs? Fine-tuned models, experimental architectures, or specific open-weights variants may require self-hosting.
3. What are your compliance requirements? If data can't leave your network, you're self-hosting or on-prem regardless of other factors.
4. What's your latency SLA? Under 200ms TTFT at P95 is achievable with managed APIs; under 100ms at sustained load may require dedicated hardware.
5. How much engineering time can you allocate to infrastructure? Be honest about this. Inference infrastructure takes real time to operate well.

For Startups: Start Managed, Move Later

Unless you have a specific compliance or model-flexibility requirement, start with a managed API. You'll ship faster and have better observability from day one. Revisit the architecture when your token spend exceeds $5,000-10,000/month and your utilization patterns are well-understood.

For Enterprises: Compliance First, Then Optimize

Enterprises typically have to answer the compliance question before anything else. If you need data residency or a specific certification, that narrows your options immediately. Once you know what's allowed, optimize for cost and latency within that constraint -- not before.

FAQ

What is the best open-source LLM inference server?

vLLM is the production standard for most use cases: broad model support, strong throughput via continuous batching and PagedAttention, and an active community. SGLang is worth evaluating for multi-turn or agentic workloads where KV cache reuse matters. TensorRT-LLM delivers the highest raw throughput on NVIDIA hardware if you have the engineering bandwidth for its compilation workflow.

Can I run a self-hosted inference server on a single GPU?

Yes. A single H100 (80GB) can run Llama 4 70B at FP8 quantization with room for KV cache. A single A100 (80GB) handles 70B models at FP8 as well, though throughput is lower. For 7-8B models, a single 24GB consumer GPU (RTX 4090) works for development and low-traffic production use.

How many requests per second can an inference server handle?

It depends on model size, sequence length, hardware, and concurrency. A single H100 running Llama 3.1 70B at FP8 with continuous batching can handle 50-150 concurrent requests depending on output length, which translates to roughly 20-50 completed requests per second at typical chatbot sequence lengths. Smaller models (7-8B) can handle 5-10x more RPS on the same hardware.

---

If you're evaluating inference options and want to skip the infrastructure work, GeneralCompute's API gives you access to fast open-model inference with no cluster to manage. The ASIC-optimized infrastructure delivers strong TTFT and TPS numbers across Llama 4, DeepSeek, Qwen, and other major models, on an OpenAI-compatible API. Check the documentation to get started, or review our AI inference cost guide to run the economics before deciding on an architecture.