Coding Model Benchmarks: Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2

Picking a coding model used to be simple: you grabbed CodeLlama and moved on. That's no longer the case. The coding-specific model landscape has changed substantially in the past two years, and the differences between the top options are large enough to actually matter for production use cases.

This post benchmarks Qwen 2.5 Coder, DeepSeek Coder V2, CodeLlama, and StarCoder 2 across common code tasks, organized by size bucket so you can see how the options compare at the compute tier you're actually considering.

What Makes a Coding Model Different

Coding models start from a general-purpose base and receive additional pretraining on large code corpora, often paired with code-specific fine-tuning for instruction following. The training mix differs significantly between projects: some emphasize breadth across programming languages, others concentrate on high-quality data from fewer sources, and some are specifically optimized for code completion (fill-in-the-middle) rather than generation from a description.

The result is that benchmark performance on tasks like HumanEval correlates more strongly with training data quality and volume than with raw parameter count. A well-trained 7B model can outperform a poorly-trained 70B on coding tasks.

The Benchmark Suite

The standard benchmarks for code models:

HumanEval: 164 Python function completion problems from OpenAI. Tests whether a model can generate correct function bodies given a docstring and signature. Reports pass@1 (fraction that pass unit tests on the first attempt).
MBPP (Mostly Basic Python Problems): 500 Python problems from Google, a bit more diverse in structure than HumanEval. Often used alongside HumanEval to confirm results aren't an artifact of dataset-specific optimization.
MultiPL-E: Translates HumanEval and MBPP to 18+ other languages (Java, JavaScript, TypeScript, C++, Rust, Go, etc.). Useful for measuring how well coding capability transfers outside Python.
SWE-Bench: A harder benchmark derived from real GitHub issues and their corresponding pull requests. Tests whether a model can produce a patch that actually resolves the issue and passes CI. More representative of real engineering work than HumanEval.

HumanEval and MBPP measure whether a model can write clean, correct short functions. SWE-Bench measures something closer to what a coding agent does. The gap between these is worth paying attention to -- models can score well on HumanEval and still struggle with multi-file edits or context-dependent bugs.

Model Overview

| Model | Sizes | Architecture | Context | Code Training | |---|---|---|---|---| | Qwen 2.5 Coder | 0.5B to 72B | Dense | 128K | 5.5T tokens | | DeepSeek Coder V2 | 16B, 236B | MoE | 128K | Based on DeepSeek-V2 | | CodeLlama | 7B to 70B | Dense (Llama 2 base) | Up to 100K | ~1T code tokens | | StarCoder 2 | 3B, 7B, 15B | Dense | 16K | The Stack v2 (600+ langs) |

A few notes on these numbers. DeepSeek Coder V2 is a MoE model: the 16B Lite has 16B total params with fewer active, and the full 236B model has 236B total params with 21B active. This affects inference cost considerably. StarCoder 2's 16K context window is a real constraint for longer codebases. CodeLlama has a 100K window in its code infilling mode, but the base models default to shorter contexts.

Benchmark Results by Size

Sub-16B Models

At the 7-8B range, this is where most teams run inference for latency-sensitive code tasks.

| Model | HumanEval pass@1 | MBPP pass@1 | |---|---|---| | Qwen 2.5 Coder 7B Instruct | ~88% | ~83% | | DeepSeek Coder V2 Lite (16B MoE) | ~81% | ~75% | | CodeLlama 7B | ~34% | ~44% | | CodeLlama 13B | ~36% | ~49% | | StarCoder 2 7B | ~35% | ~54% | | StarCoder 2 15B | ~46% | ~60% |

Qwen 2.5 Coder 7B Instruct is the clear winner at this size tier. Scores in the high 80s on HumanEval put it ahead of DeepSeek Coder V2 Lite despite the latter being a 16B model (with lower active params). The gap over CodeLlama 7B and StarCoder 2 7B is large -- roughly 50 percentage points on HumanEval.

CodeLlama and StarCoder 2 at these sizes were competitive when released in 2023, but the training data and instruction tuning improvements since then have shifted the baseline significantly.

30-70B Range

| Model | HumanEval pass@1 | MBPP pass@1 | |---|---|---| | Qwen 2.5 Coder 32B Instruct | ~93% | ~90% | | Qwen 2.5 Coder 72B Instruct | ~92%+ | ~89%+ | | DeepSeek Coder V2 (236B, 21B active) | ~96% | ~92% | | CodeLlama 34B | ~49% | ~62% | | CodeLlama 70B | ~68% | ~74% |

At larger sizes, DeepSeek Coder V2's full 236B model (with its MoE architecture keeping active params to ~21B) achieves the highest scores in this category. Qwen 2.5 Coder 32B is within a few points and is meaningfully smaller to serve.

CodeLlama 70B does improve substantially over its smaller versions, reaching ~68% on HumanEval -- a legitimate result for a model from 2023 -- but it's well below the modern alternatives.

What These Numbers Mean in Practice

HumanEval pass@1 in the 90s sounds like it should translate to a model that never makes mistakes. It doesn't. HumanEval problems are short, self-contained, and well-specified. Production code is longer, under-specified, and depends on context in other files.

The roughly 15-point gap between Qwen 2.5 Coder 32B (93%) and CodeLlama 70B (68%) does reflect a genuine quality difference: the newer model handles edge cases better, follows instructions more reliably, and produces code that compiles more often on the first try. But both will make mistakes on hard problems, ambiguous specifications, and unfamiliar APIs.

Code Completion: Fill-in-the-Middle

For IDE-style code completion (the cursor is in the middle of a function and you want a suggestion), models need fill-in-the-middle (FIM) support. This is a distinct capability from instruction following -- the model sees code above and below the cursor and must generate what fits in between.

All four families support FIM, but the quality varies:

Qwen 2.5 Coder: Strong FIM performance across sizes. The 7B model is a practical choice for low-latency autocomplete. The 32B model produces higher quality completions but at higher latency.
DeepSeek Coder V2: Competitive FIM performance, particularly for long completions where the model needs to reason about what comes after the completion.
CodeLlama: Has solid FIM support, and was one of the first widely available models to implement it properly. The 34B version is commonly used for this use case.
StarCoder 2: Also supports FIM. The 15B model is a reasonable option if you need wide programming language coverage and are willing to accept lower benchmark ceilings.

For autocomplete specifically, latency matters as much as benchmark score. A model that generates tokens 40ms faster is meaningfully better for interactive use even if it scores slightly lower on HumanEval. This is worth measuring on your actual serving infrastructure.

MultiPL-E: Beyond Python

HumanEval and MBPP are Python-only. If your codebase is in TypeScript, Go, or Rust, the benchmark rankings can shift.

Qwen 2.5 Coder maintains strong scores across languages on MultiPL-E. Its training corpus covers 92 programming languages, and the multilingual performance is more consistent than most models at equivalent sizes.

StarCoder 2 was specifically designed for broad language coverage (600+ languages in The Stack v2), but the depth of coverage varies a lot by language. For commonly used languages (Python, JavaScript, Java, C++), it's competitive. For less common ones, the training signal is thinner.

CodeLlama's multilingual performance is more uneven than the newer models. Strong Python, decent JavaScript and C++, weaker on more recent languages like Rust or Swift that were underrepresented in its training data.

DeepSeek Coder V2 performs well across most major languages, with particular strength in languages common in its training data.

SWE-Bench: Real Repository Tasks

SWE-Bench scores are considerably lower than HumanEval for every model -- this is expected, because resolving a real GitHub issue in an unfamiliar codebase is much harder than writing a function given a docstring.

The Lite version of SWE-Bench (a curated 300-issue subset) has become the standard for comparing models on this task. Scores above 30% resolve rate are considered strong. To put this in context:

Models in the 40-50% range on HumanEval often achieve only 5-15% on SWE-Bench Lite
Models at 90%+ on HumanEval achieve 25-45% on SWE-Bench Lite
Getting past 50% on SWE-Bench Lite requires careful scaffolding in addition to a capable model

For SWE-Bench specifically, DeepSeek Coder V2 (full 236B) and Qwen 2.5 Coder 72B are among the stronger open-source options. Neither is dramatically better than the other when matched with equivalent scaffolding. The practical takeaway: model quality matters, but the agent framework around the model matters roughly as much for complex, multi-file repair tasks.

Inference Cost by Size

For teams running these models in production, the size-to-quality curve matters.

For the 7B range: Qwen 2.5 Coder 7B is the best option available. It outperforms much larger models from two years ago and runs efficiently on a single A100 or H100 with room for reasonable batch sizes.

For the 14-32B range: Qwen 2.5 Coder 32B is strong here and is the default recommendation for teams that need higher quality than a 7B provides. DeepSeek Coder V2 Lite (16B MoE) is competitive on benchmarks but the MoE architecture adds routing overhead that affects latency at low batch sizes.

For the 70B+ range: DeepSeek Coder V2's 236B model (21B active params) achieves the highest benchmark scores and is cost-efficient to serve relative to its active parameter count. Qwen 2.5 Coder 72B is the dense alternative at competitive quality.

CodeLlama and StarCoder 2 are no longer competitive with these options for most use cases. They remain useful if you have specific requirements -- CodeLlama if you need a model that was released under a permissive license with a well-established deployment history, StarCoder 2 if you specifically need broad language coverage in the BigCode OpenRAIL-M license.

Which Model to Use

For interactive code completion (autocomplete, FIM): Qwen 2.5 Coder 7B Instruct. Low latency, high quality at size, good multilingual coverage. Upgrade to 14B if quality on complex completions matters more than inference speed.

For code generation from description (chat interface, function generation): Qwen 2.5 Coder 32B Instruct or DeepSeek Coder V2 (full). The 32B is easier to serve; the V2 236B scores higher on harder problems.

For coding agents on real repositories: Either Qwen 2.5 Coder 72B or DeepSeek Coder V2 (full) with appropriate agent scaffolding. The model is necessary but not sufficient -- the scaffolding is where most of the SWE-Bench performance comes from.

If you need broad language coverage and aren't CPU-bound on quality: StarCoder 2 15B is a reasonable option for less common programming languages where the newer models have less training data.

If license terms are the primary constraint: All four families offer models under their respective licenses. Qwen and DeepSeek have commercial-friendly licenses for most use cases; StarCoder 2 uses BigCode OpenRAIL-M; CodeLlama uses Meta's CodeLlama license. Check the specifics for your situation.

A Practical Comparison on Your Workload

Benchmark scores are useful for filtering options, but they don't replace testing on your actual code and prompts. Different codebases, languages, and problem types produce different rankings in practice. A model that tops HumanEval may underperform on your TypeScript monorepo, and vice versa.

The most reliable evaluation is running your top two or three candidates on a representative sample of your real tasks -- ideally tasks where you already know the correct answer -- and measuring pass@1 or edit distance from the target. This takes a few hours and produces much more actionable data than any third-party benchmark table.

All four model families are available via General Compute's inference API with OpenAI-compatible endpoints. You can benchmark them on your workload without infrastructure setup by changing a base URL and model name in your existing code. Run your eval suite against each option to find the right fit for your specific use case.

Ready to benchmark coding models on your workload? General Compute supports Qwen 2.5 Coder, DeepSeek Coder V2, and other top coding models with low-latency inference and no rate limits. Swap the model name in your OpenAI integration and run your evaluation in minutes.