Whisper API for Real-Time Transcription: Alternatives to OpenAI Whisper

Markdown body

OpenAI's Whisper is the model most teams reach for when they first need speech-to-text. The weights are open, accuracy is strong across languages and accents, and the managed API requires no infrastructure. For transcribing a recording, it is a reasonable starting point. For real-time transcription of a live call or voice agent, it is the wrong tool, and the teams that discover this late spend a lot of time working around a constraint that a different choice would have avoided entirely.

This post covers what the OpenAI Whisper API actually provides, where it stops working for live use cases, and how Faster-Whisper, Deepgram, and AssemblyAI compare as alternatives. The comparison covers accuracy, latency, pricing, and which use cases each one fits best.

## What the OpenAI Whisper API Provides

The `whisper-1` model is available through OpenAI's audio transcriptions endpoint. You send a complete audio file, and you get back a transcript. The interface is simple:

```python
from openai import OpenAI

client = OpenAI(api_key="your_openai_key")

with open("recording.mp3", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
        response_format="text"
    )

print(transcript)
```

This works well for batch use cases: uploading a recorded meeting, transcribing a podcast episode, processing a support call after it ends. You pay per minute of audio (currently $0.006 per minute), and you get a complete transcript back after a few seconds of processing time.

The constraints show up the moment you try to use it for anything interactive.

**No streaming support.** The API requires a complete audio file as input. You cannot open a WebSocket connection and feed chunks of live audio as someone is speaking. You have to wait for the utterance to complete, upload it, wait for the API to respond, and then process the result. That round trip is tolerable for short clips but makes it impossible to show partial transcripts while someone is still talking.

**Variable latency.** Transcription time through the managed API is not as consistent as a locally-hosted model. Under load, API response times can vary from a couple of seconds to notably longer, which is hard to plan around when you need predictable latency.

**File size and format limits.** The API accepts files up to 25 MB. For long recordings this requires chunking, which introduces its own stitching logic.

**Rate limits.** At higher volumes, the API rate limits require requesting increases. For a product that can spike unexpectedly, the ceiling is an operational risk.

None of these are criticisms of the model itself, which is genuinely accurate. The managed API is designed for file-based transcription. If that is your use case, it is a fine choice. If you need real-time transcription, you need something else.

## Option 1: Faster-Whisper (Self-Hosted)

Faster-Whisper is a reimplementation of the same Whisper model weights on top of CTranslate2, a C++ inference library. Running the same large-v3 model, it is typically four times faster than the reference PyTorch implementation, and it supports INT8 quantization to reduce memory usage further.

The key difference from the managed API is that you host it yourself, which means you control the model, the hardware, and the concurrency. You can build a streaming interface on top of it by running inference on rolling audio buffers:

```python
from faster_whisper import WhisperModel
import numpy as np

model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")

buffer = np.array([], dtype=np.float32)

for chunk in audio_stream:
    buffer = np.concatenate([buffer, chunk])
    
    segments, _ = model.transcribe(buffer, beam_size=1, language="en")
    partial = " ".join(s.text for s in segments)
    emit_partial(partial)
    
    if utterance_ended(buffer):
        emit_final(partial)
        buffer = np.array([], dtype=np.float32)
```

Running `beam_size=1` uses greedy decoding instead of beam search, which cuts latency meaningfully in exchange for a small accuracy dip that is usually not noticeable in real-time use.

**Where it makes sense:** teams that need to process high volumes of audio, want control over the model and infrastructure, or have data residency requirements that rule out sending audio to a third-party API. The operational overhead is real -- you're managing GPU instances, handling failures, and monitoring model performance -- but the per-minute cost at scale is much lower than any managed API.

**Where it doesn't:** teams that want a fully managed service with no infrastructure to operate. Faster-Whisper is a library, not an API. You have to build the serving layer yourself.

For accuracy, it matches the OpenAI API exactly because it uses the same weights. Word error rate on standard benchmarks (LibriSpeech clean, CommonVoice) is the same between the two implementations.

## Option 2: Deepgram

Deepgram is a purpose-built speech-to-text API company with models trained specifically for transcription rather than adapted from a general-purpose audio model. Their current production model, Nova-2, is the default for most use cases.

The key capability for real-time use is their streaming API, which uses WebSocket connections to accept audio chunks and return transcripts with very low latency. Partial transcripts arrive as the person is speaking, and final transcripts are emitted at speech boundaries:

```python
import asyncio
from deepgram import DeepgramClient, LiveTranscriptionEvents, LiveOptions

async def transcribe_stream(audio_source):
    dg_client = DeepgramClient(api_key="your_deepgram_key")
    connection = dg_client.listen.asynclive.v("1")

    async def on_transcript(self, result, **kwargs):
        alternatives = result.channel.alternatives
        if alternatives:
            text = alternatives[0].transcript
            is_final = result.is_final
            if text:
                if is_final:
                    print(f"Final: {text}")
                else:
                    print(f"Partial: {text}", end="\r")

    connection.on(LiveTranscriptionEvents.Transcript, on_transcript)

    options = LiveOptions(
        model="nova-2",
        language="en-US",
        smart_format=True,
        interim_results=True,
        endpointing=300,
    )

    await connection.start(options)

    async for chunk in audio_source:
        await connection.send(chunk)

    await connection.finish()
```

The `endpointing` parameter (in milliseconds) controls how long a pause the model waits before treating an utterance as complete and emitting a final transcript. Tuning this trades off responsiveness (lower value) against stability (higher value, fewer false endpoints).

**Latency.** Deepgram's streaming API returns first results in roughly 300ms from the start of speech in normal conditions, with final results following once the endpoint is detected. This is competitive for live voice applications.

**Accuracy.** Nova-2 is competitive with Whisper large-v3 on clean English speech, and in some benchmarks on accented or noisy speech it measures better, likely because it was trained on a more diverse audio corpus. For domain-specific vocabulary (medical, legal, technical), Deepgram offers custom model training, which can meaningfully improve accuracy on terms the base model handles poorly.

**Pricing.** Streaming transcription with Nova-2 runs at approximately $0.0059 per minute, close to the OpenAI API price for batch. There are volume discounts at higher monthly usage.

**Additional features.** Speaker diarization (who said what), automatic punctuation, filler word removal, and custom vocabulary are available as parameters on the same API call, which simplifies pipelines that previously required post-processing steps.

## Option 3: AssemblyAI

AssemblyAI is another managed transcription API, positioned similarly to Deepgram but with a different model architecture and a slightly different feature emphasis. Their production model is called Universal-2.

They offer both an async API for file transcription and a real-time streaming API:

```python
import assemblyai as aai

aai.settings.api_key = "your_assemblyai_key"

def on_data(transcript: aai.RealtimeTranscript):
    if not transcript.text:
        return
    if isinstance(transcript, aai.RealtimeFinalTranscript):
        print(f"Final: {transcript.text}")
    else:
        print(f"Partial: {transcript.text}", end="\r")

def on_error(error: aai.RealtimeError):
    print(f"Error: {error}")

transcriber = aai.RealtimeTranscriber(
    sample_rate=16_000,
    on_data=on_data,
    on_error=on_error,
)

transcriber.connect()

# stream PCM audio frames
with open("audio.raw", "rb") as f:
    while chunk := f.read(3200):
        transcriber.stream(chunk)

transcriber.close()
```

**Latency.** AssemblyAI's streaming latency is in a similar range to Deepgram, generally 300-500ms from speech start to first partial results. Their final transcript latency depends on the endpointing configuration.

**Accuracy.** Universal-2 performs comparably to Deepgram Nova-2 on standard benchmarks. For specialized content (lecture transcription, interview recordings), AssemblyAI publishes benchmark numbers on their site that show competitive WER against alternatives.

**Pricing.** Real-time streaming transcription is priced at approximately $0.01 per minute, which is higher than Deepgram and the OpenAI API. They offer discounts for higher volumes under enterprise agreements.

**Where AssemblyAI is differentiated.** They have invested in speaker diarization quality, audio intelligence features (topic detection, sentiment analysis, content moderation), and an async API that handles long files cleanly. If your application needs any of those capabilities alongside transcription, the combined offering is worth evaluating rather than stitching together separate services.

## Comparison at a Glance

| | OpenAI Whisper API | Faster-Whisper | Deepgram Nova-2 | AssemblyAI Universal-2 |
|---|---|---|---|---|
| Real-time streaming | No | Yes (self-built) | Yes | Yes |
| Latency to first partial | N/A (file-based) | ~50-100ms (local) | ~300ms | ~300-500ms |
| Accuracy (WER, clean) | Low (large-v3) | Same as OpenAI | Competitive | Competitive |
| Pricing (per minute) | $0.006 | Infra cost only | ~$0.0059 | ~$0.010 |
| Infrastructure to manage | None | Yes (GPU required) | None | None |
| Custom vocabulary | No | No | Yes | Yes |
| Speaker diarization | No | No | Yes | Yes |

Accuracy numbers here reflect general benchmarking on English speech. Non-English performance varies significantly by model and language. If your use case involves a specific language or accent, benchmark on real audio from that distribution rather than relying on aggregate WER numbers.

## Choosing Between Them

**Use the OpenAI Whisper API** if your use case is genuinely batch transcription: processing recorded files after the fact, with no latency requirement and acceptable per-minute pricing at your volume.

**Use Faster-Whisper** if you have infrastructure you can run it on, need data residency or data privacy guarantees, or have volume high enough that the cost difference versus managed APIs is worth the operational overhead. Self-hosting also gives you precise control over model version and quantization level.

**Use Deepgram** if you want a managed real-time streaming API with good accuracy, competitive pricing, and useful features like diarization and smart formatting included. It is the most straightforward choice for teams building voice agents or live transcription features who don't want to manage their own infrastructure.

**Use AssemblyAI** if you need the audio intelligence features (topic detection, PII redaction, content moderation) alongside transcription, or if their speaker diarization quality is important for your use case. The higher per-minute price reflects the broader feature set.

## Fitting Transcription Into a Voice Pipeline

Transcription is usually the front of a longer pipeline. In a voice agent, the flow goes: microphone audio, to speech-to-text, to a language model, to text-to-speech, back to the speaker. Each stage adds latency, and the user hears the sum.

For that kind of pipeline, the transcription stage's job is to produce a final transcript quickly enough that the language model has time to respond and TTS can start speaking before the pause becomes awkward. A rough budget for a conversational response that feels natural is around 800ms to one second from the moment the user stops speaking. Transcription at 300-500ms through a streaming API leaves only a few hundred milliseconds for everything else, which means the LLM generation stage needs to return a first token very quickly.

This is where the inference provider for the language model matters as much as the transcription choice. A transcription step that takes 300ms paired with an LLM that takes 700ms to first token means the user waits over a second before hearing anything -- and that is before TTS adds its own latency. Serving the LLM on infrastructure tuned for low time-to-first-token is what keeps the full loop inside the window.

If you're building a voice pipeline with Deepgram or a self-hosted Faster-Whisper front end and need a fast LLM backend, you can point any OpenAI-compatible client at [General Compute](https://generalcompute.com) to handle the generation step:

```python
from openai import OpenAI

gc_client = OpenAI(
    api_key="your_gc_api_key",
    base_url="https://api.generalcompute.com/v1"
)

def generate_response(transcript: str) -> str:
    stream = gc_client.chat.completions.create(
        model="llama-4-scout",
        messages=[
            {"role": "system", "content": "You are a helpful voice assistant. Keep responses concise."},
            {"role": "user", "content": transcript}
        ],
        stream=True,
        max_tokens=200
    )
    
    full_response = ""
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            full_response += delta
            # send to TTS as tokens arrive
            send_to_tts(delta)
    
    return full_response
```

Streaming the LLM output directly to TTS as tokens arrive, rather than waiting for the complete response, shaves additional latency off what the user hears. The first word of the response starts playing before the model has finished generating.

## Getting Started

The fastest path to a working real-time transcription setup depends on your constraints. If you want no infrastructure: sign up for Deepgram and run their streaming quickstart against a microphone. If you have a GPU available: install Faster-Whisper and run a rolling-buffer streaming loop. Either approach takes an afternoon to get working end-to-end.

The harder part, and the one where teams more often underestimate the work, is making the full pipeline feel fast. Transcription that works in isolation but sits in front of a slow LLM still produces a product that feels sluggish. Measure each stage independently and then measure the full loop -- the sum is usually worse than expected.