Embedding Model Benchmarks: Speed vs Quality for RAG Retrieval

When building a RAG pipeline, the embedding model is often chosen quickly and then never revisited. A lot of teams default to whatever model was in the tutorial they followed, or whichever API was already set up. That's understandable, but it leaves a meaningful performance gap on the table, because the tradeoffs in embedding models are not symmetric: going from a 22M parameter model to a 335M parameter model costs you about 10x in inference compute while delivering only a few points of retrieval improvement. But going from an open-source 335M model to a well-trained API model can jump retrieval quality by 10+ NDCG points with no change to your infrastructure.

This post works through those tradeoffs concretely, using BEIR benchmarks as the quality measure and query latency as the speed measure.

What "Quality" Means for Retrieval

For RAG retrieval specifically, the relevant metric is NDCG@10 (Normalized Discounted Cumulative Gain at 10 results) on the BEIR benchmark suite. BEIR is a collection of 18 retrieval datasets covering different domains: web search, scientific papers, news articles, financial questions, biomedical queries, and more. NDCG@10 measures whether relevant documents appear near the top of your retrieved set.

MTEB's retrieval leaderboard aggregates across BEIR datasets. The average NDCG@10 score is what you see on the leaderboard. For a given RAG application, you care more about the subset of BEIR that resembles your data domain -- a legal QA system should weight legal retrieval datasets more than financial ones. But the average is a reasonable proxy for general-purpose RAG.

A few practical notes on reading these numbers:

BEIR scores are measured with cosine similarity, which matches how most vector databases work by default
Scores are on a 0-100 scale (or 0.0-1.0 depending on the source -- we use 0-100 here)
The difference between 53 and 55 is real but may not matter for your application; the difference between 42 and 55 almost certainly does

What "Speed" Means for Embedding

Embedding has two distinct latency profiles depending on how you're using it.

Query-time embedding is synchronous with user requests. When someone sends a query, you embed it, run the vector search, and then generate the response. This needs to be fast -- under 50ms ideally. For a 384-dimension model running on an inference API, this is usually easy. For a 1024-dimension model served on shared infrastructure, it can be a bottleneck.

Indexing throughput matters when you're building or rebuilding your vector index. If you're ingesting 10 million documents, even a 2x throughput difference changes whether indexing takes 2 hours or 4 hours. Here, batch size and GPU memory matter more than individual query latency.

For this comparison, we focus on single-query latency (the query-time case) since that's the bottleneck for interactive applications.

The Benchmark Comparison

Here's how the main open-source and API embedding models compare across model size, dimensions, BEIR NDCG@10, and approximate query latency on GPU inference:

| Model | Params | Dims | BEIR NDCG@10 | Query latency (approx) | |-------|--------|------|-------------|------------------------| | all-MiniLM-L6-v2 | 22M | 384 | ~42 | <5ms | | bge-small-en-v1.5 | 33M | 384 | ~52 | <5ms | | bge-base-en-v1.5 | 109M | 768 | ~53 | ~10ms | | nomic-embed-text-v1.5 | 137M | 768 | ~53 | ~12ms | | bge-large-en-v1.5 | 335M | 1024 | ~54 | ~25ms | | e5-large-v2 | 335M | 1024 | ~56 | ~25ms | | bge-m3 (multilingual) | 568M | 1024 | ~55 | ~40ms | | text-embedding-3-small | API | 1536 | ~62 | ~30-60ms (API) | | text-embedding-3-large | API | 3072 | ~65 | ~40-80ms (API) | | Cohere embed-v3-english | API | 1024 | ~56 | ~30-60ms (API) |

Latency numbers are approximate and depend heavily on hardware, batching, and network proximity to the API. The open-source numbers assume GPU serving (A100 class) with individual queries. Self-hosted models generally have lower p50 latency than API calls because there's no network round-trip.

Where the Quality Cliff Actually Is

The numbers above reveal a few patterns worth paying attention to.

The all-MiniLM gap is real. Jumping from all-MiniLM-L6-v2 to bge-small-en-v1.5 improves BEIR NDCG@10 by roughly 10 points while adding almost no compute overhead -- they're both sub-5ms on GPU and both use 384-dim vectors. There's very little reason to use all-MiniLM for a production RAG system at this point. bge-small or nomic-embed-text-v1.5 (with its matryoshka support for variable-dimension outputs) are better defaults.

Small to large open-source yields diminishing returns. Going from bge-small (33M, ~52 NDCG) to bge-large (335M, ~54 NDCG) costs 10x the parameters for roughly 2 NDCG points. Whether 2 NDCG points matters depends on your application -- for a medical information system, it might; for an internal document search, probably not.

The gap between open-source large and API models is substantial. text-embedding-3-large scores around 65 NDCG@10 on BEIR vs. ~54-56 for the best open-source models. That's a 9-11 point gap. If your RAG pipeline is retrieval-limited (meaning retrieval quality is the primary driver of answer quality), this matters. If your pipeline is generation-limited (the LLM step is where quality falls down), improving embeddings may have less impact than you'd expect.

Cohere embed-v3 occupies an interesting middle ground. At ~56 NDCG, it matches or slightly exceeds the best open-source dense models and includes built-in reranking support. For teams that don't want to manage their own infrastructure, it's a reasonable API option at a lower cost than text-embedding-3-large.

The Reranker Option

One pattern that doesn't show up in the table above: retrieve with a smaller, faster embedding model and then rerank the top results with a cross-encoder reranker.

Rerankers (models like bge-reranker-v2-m3 or Cohere Rerank) take a (query, document) pair and produce a relevance score. They're slower than embedding-based retrieval because they process each pair separately, but they're much more accurate because they can use the full query-document context rather than independent embeddings.

A typical two-stage approach:

Embed with bge-small-en-v1.5 and retrieve the top 50 candidates
Rerank those 50 with bge-reranker-large and return the top 5-10

This combination often matches or exceeds the retrieval quality of text-embedding-3-large at lower overall latency and cost, because the reranker only runs on 50 documents rather than the full index. The downside is complexity: you now have two models to serve and a more complex pipeline.

Here's a rough latency breakdown for the two-stage approach vs. single-stage:

Single-stage (bge-large):

Embed query: ~25ms
Vector search: ~5-20ms depending on index size
Total: ~30-45ms

Two-stage (bge-small + reranker on top-50):

Embed query: ~5ms
Vector search: ~5-20ms
Rerank 50 docs: ~100-300ms (batch on GPU)
Total: ~110-325ms

The two-stage approach adds latency but can significantly improve final answer quality. Whether that tradeoff is acceptable depends on your latency budget. For asynchronous or batch pipelines, it's almost always worth doing.

Dimension Reduction and Matryoshka Models

Some modern embedding models support "matryoshka" training, where the model learns embeddings that remain useful even when truncated to fewer dimensions. nomic-embed-text-v1.5 and text-embedding-3-small/large both support this.

Practically, this means you can embed at full precision (1536 dims for text-embedding-3-small) and then truncate to 512 or 256 dims for the vector index to save storage and speed up similarity search, with a modest quality tradeoff. The exact quality curve varies by model and by how aggressively you truncate.

For high-volume RAG systems where vector storage costs matter, matryoshka models give you a knob to tune the storage/quality tradeoff without re-embedding your entire corpus.

Which Model to Use

If you're deploying open-source and want a strong baseline: Start with nomic-embed-text-v1.5 or bge-large-en-v1.5. Both score around 53-54 NDCG@10, support 1024 dims, and run well on a single GPU. nomic-embed-text has matryoshka support and an Apache-2.0 license; bge-large has slightly better raw retrieval on some BEIR subsets.

If you want the best retrieval quality without managing infrastructure: text-embedding-3-large (OpenAI) or text-embedding-3-small if cost is a factor. The ~10 NDCG point gap vs. open-source is real and shows up in end-to-end RAG quality for complex queries.

If you have multilingual documents: bge-m3 handles 100+ languages and scores competitively on English retrieval too. It's the clearest open-source choice for multilingual RAG.

If query latency is tight (under 20ms for the embed step): bge-small-en-v1.5 at 384 dims is fast enough for most infrastructure and significantly better than all-MiniLM on quality. The compute savings from using 384 dims vs. 1024 also show up in vector search latency as your index grows.

If you're building a high-quality production system and have the latency budget: combine bge-small or bge-base for first-stage retrieval with a cross-encoder reranker on the top-50. This often beats single-stage retrieval with any individual embedding model and gives you a quality floor that's harder to get with embeddings alone.

Running Your Own Benchmarks

BEIR scores are useful priors, but they don't guarantee your specific data will follow the same ordering. Legal documents, code, scientific papers, and product catalogs all have different retrieval characteristics.

The most reliable way to evaluate is to run a subset of your own data through the pipeline. Concretely:

Pick 100-500 representative query/document pairs from your corpus
Embed with each candidate model
Compute NDCG@10 or Recall@10 on your held-out set
Check whether the BEIR rankings hold

For many domains, especially those covered by BEIR (financial, biomedical, technical documents), BEIR scores are reasonably predictive. For more specialized data (chat transcripts, code comments, niche terminology), domain-specific evaluation often tells a different story.

Building this evaluation harness takes a few hours upfront but pays off every time you want to upgrade your embedding model or tune your retrieval pipeline. Store the results and re-run them when new models come out -- the embedding model landscape has moved quickly and will continue to.

General Compute's inference API supports the major open-source embedding models (including bge-large-en-v1.5 and nomic-embed-text-v1.5) with the throughput needed for both query-time serving and bulk indexing runs. If you're evaluating models and want to process a large corpus without provisioning dedicated GPU infrastructure, the API is a practical way to run the comparison before committing to a deployment setup.