How to Build a Real-Time Coding Assistant With Open-Source Models

Building a coding assistant LLM from open-source components is practical today. The models are good enough, the tooling has matured, and the OpenAI-compatible API format means most of the plumbing is already written. This guide walks through the full stack: picking the right model, indexing your codebase for context retrieval, and wiring everything into VS Code.

The reason teams build their own instead of using commercial tools like GitHub Copilot typically comes down to three things: code stays on your infrastructure, you can fine-tune on proprietary patterns, and at scale the cost difference is meaningful.

Choosing a Model

The two strongest open-source coding models right now are Qwen3-Coder and DeepSeek Coder V2. They have different strengths, and for a real-world coding assistant you'll likely want both.

Qwen3-Coder

Qwen3-Coder is Alibaba's latest coding model and performs well at:

• Fill-in-Middle (FIM) completion -- the format that powers single-line autocomplete, where the model sees code before and after the cursor and fills the gap
• Multi-language support -- TypeScript, Python, Rust, Go, and Java all work reliably
• Following explicit formatting constraints, which matters when generating code that needs to match surrounding style

The 7B and 14B variants are practical for most teams. The 7B model fits in 8GB of VRAM and is fast enough for real-time autocomplete.

DeepSeek Coder V2

DeepSeek Coder V2 uses a mixture-of-experts architecture. The Lite variant (16B active parameters out of 236B total) scores well on HumanEval and SWE-bench. It does better at:

• Multi-step reasoning about code architecture
• Generating longer, contextually coherent functions
• Code chat: explaining, refactoring, and debugging

The tradeoff is that it requires more inference infrastructure than Qwen3-Coder 7B. Managed inference APIs make this transparent, but if you're self-hosting, the Lite variant needs more VRAM than a 7B model.

Which to Use for Each Task

For autocomplete/FIM, use Qwen3-Coder 7B or 14B. The FIM format is well-supported and the model is fast.

For code chat, refactoring, and explanation, DeepSeek Coder V2 Lite or Qwen3-Coder 32B produce noticeably better output.

A practical approach is to use both: a smaller fast model for inline autocomplete and a larger one for the chat panel. With an API that supports multiple models per account, you can switch model per request without maintaining two inference servers.

Codebase Indexing

A coding assistant that only sees the current file is limited. The real value comes when the model has context about the broader codebase: what functions exist, how types are defined, what patterns are used elsewhere.

The problem is that a 500K-line codebase won't fit in any context window. The solution is retrieval: index the codebase upfront, find the relevant chunks at query time, and inject them into the prompt.

Step 1: Parse with Tree-sitter

Instead of splitting code at arbitrary line counts, use Tree-sitter to parse it into an AST and split at meaningful boundaries (functions, classes, methods).

import tree_sitter_python as tspython
from tree_sitter import Language, Parser

PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)

def extract_functions(source_code: str) -> list[dict]:
    tree = parser.parse(bytes(source_code, "utf8"))
    root = tree.root_node
    functions = []
    for node in root.children:
        if node.type == "function_definition":
            name_node = node.child_by_field_name("name")
            functions.append({
                "name": name_node.text.decode() if name_node else "",
                "body": source_code[node.start_byte:node.end_byte],
                "start_line": node.start_point[0],
            })
    return functions

Tree-sitter has grammar packages for TypeScript, Rust, Go, Java, and most other languages. The same parsing logic works across languages; you swap the grammar package.

Step 2: Embed the Chunks

Embed each function or class using a code-aware embedding model. Nomic Embed Code is a solid open-source option: 768-dimensional embeddings, fast, and it understands code semantics better than general-purpose text embedders.

from sentence_transformers import SentenceTransformer

embed_model = SentenceTransformer("nomic-ai/nomic-embed-code")

def embed_chunks(chunks: list[str]) -> list[list[float]]:
    return embed_model.encode(chunks, batch_size=32).tolist()

Store these in a vector database. For a local single-developer setup, SQLite with the sqlite-vec extension works well and avoids running a separate process.

import sqlite_vec
import sqlite3

db = sqlite3.connect("codebase.db")
db.enable_load_extension(True)
sqlite_vec.load(db)
db.enable_load_extension(False)

db.execute("""
    CREATE VIRTUAL TABLE IF NOT EXISTS code_chunks
    USING vec0(
        chunk_id INTEGER PRIMARY KEY,
        file_path TEXT,
        chunk_text TEXT,
        embedding float[768]
    )
""")

For teams where multiple developers share an index, Qdrant or Chroma work as standalone services.

Step 3: Retrieve at Query Time

When a user triggers autocomplete or asks a question, retrieve the top-k most relevant chunks and inject them into the prompt.

def get_relevant_context(query: str, top_k: int = 5) -> list[str]:
    query_embedding = embed_model.encode([query])[0].tolist()
    results = db.execute("""
        SELECT chunk_text
        FROM code_chunks
        ORDER BY vec_distance_cosine(embedding, ?)
        LIMIT ?
    """, (query_embedding, top_k)).fetchall()
    return [row[0] for row in results]

Keep the retrieval fast. A query to a local SQLite index should run in under 10ms. If you're hitting a remote vector DB, cache embedding lookups for repeated queries.

The Inference Layer

Both Qwen3-Coder and DeepSeek Coder expose an OpenAI-compatible API, so you can use the standard openai SDK with any compatible endpoint.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.generalcompute.com/v1",
    api_key="your-api-key"
)

FIM for Autocomplete

Fill-in-Middle is the completion format where you provide the code before and after the cursor, and the model fills what goes in between. Qwen3-Coder uses this special token format:

<|fim_prefix|>{code before cursor}<|fim_suffix|>{code after cursor}<|fim_middle|>

FIM requests go through the completions endpoint (not chat completions):

def get_fim_completion(prefix: str, suffix: str, model: str = "qwen3-coder-7b") -> str:
    prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
    response = client.completions.create(
        model=model,
        prompt=prompt,
        max_tokens=128,
        temperature=0.1,
        stop=["<|fim_pad|>", "<|endoftext|>"]
    )
    return response.choices[0].text

Keep temperature low for autocomplete (0.0 to 0.2) and cap max_tokens tightly. For single-line completion, 64 tokens is enough. For function-level completion, 256 is a reasonable ceiling. You want a fast response, not a complete file.

Chat Mode with Retrieval

For the chat panel, use the chat completions endpoint with retrieved context injected into the system prompt:

from typing import Generator

def ask_about_code(question: str, context_chunks: list[str]) -> Generator[str, None, None]:
    context = "\n\n".join([f"```\n{chunk}\n```" for chunk in context_chunks])

    stream = client.chat.completions.create(
        model="deepseek-coder-v2-lite",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a coding assistant. Use the following code context "
                    f"to answer questions accurately:\n\n{context}"
                )
            },
            {"role": "user", "content": question}
        ],
        stream=True
    )

    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            yield delta

Streaming matters here. If the chat panel waits for the full response before rendering anything, users perceive it as slow even when the actual generation time is reasonable. Yielding tokens as they arrive makes the interface feel much more responsive.

VS Code Integration

There are two ways to surface this in VS Code: using an existing framework that supports custom backends, or building a minimal extension directly.

Option 1: Continue (Fastest Path)

Continue is an open-source VS Code extension that handles all the UI: inline completion, diff viewer, chat panel, slash commands, and keyboard shortcuts. You configure it to point at any OpenAI-compatible API.

Add this to ~/.continue/config.json:

{
  "models": [
    {
      "title": "DeepSeek Coder (Chat)",
      "provider": "openai",
      "model": "deepseek-coder-v2-lite",
      "apiBase": "https://api.generalcompute.com/v1",
      "apiKey": "your-api-key"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen3-Coder (Autocomplete)",
    "provider": "openai",
    "model": "qwen3-coder-7b",
    "apiBase": "https://api.generalcompute.com/v1",
    "apiKey": "your-api-key"
  },
  "contextProviders": [
    { "name": "codebase", "params": {} },
    { "name": "diff", "params": {} },
    { "name": "open", "params": {} }
  ]
}

This gets you a working coding assistant in a few minutes. Continue handles codebase indexing internally using its own embedding pipeline, so you can skip the custom indexing step if you don't need fine-grained control over chunking.

Option 2: Custom VS Code Extension

If you need more control -- custom indexing logic, internal auth, specific keybindings, or your own chat UI -- a lightweight VS Code extension is straightforward to build.

Here's a minimal inline completion provider in TypeScript:

import * as vscode from 'vscode';
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.generalcompute.com/v1',
  apiKey: process.env.GC_API_KEY ?? '',
});

let debounceTimer: ReturnType<typeof setTimeout> | undefined;

class FIMCompletionProvider implements vscode.InlineCompletionItemProvider {
  provideInlineCompletionItems(
    document: vscode.TextDocument,
    position: vscode.Position,
  ): Promise<vscode.InlineCompletionItem[]> {
    return new Promise((resolve) => {
      clearTimeout(debounceTimer);
      debounceTimer = setTimeout(async () => {
        const prefix = document.getText(
          new vscode.Range(new vscode.Position(0, 0), position)
        );
        const suffix = document.getText(
          new vscode.Range(position, new vscode.Position(document.lineCount, 0))
        );

        try {
          const prompt = `<|fim_prefix|>${prefix}<|fim_suffix|>${suffix}<|fim_middle|>`;
          const response = await client.completions.create({
            model: 'qwen3-coder-7b',
            prompt,
            max_tokens: 128,
            temperature: 0.1,
            stop: ['<|fim_pad|>', '<|endoftext|>'],
          });
          const text = response.choices[0]?.text ?? '';
          resolve(text ? [new vscode.InlineCompletionItem(text)] : []);
        } catch {
          resolve([]);
        }
      }, 150);
    });
  }
}

export function activate(context: vscode.ExtensionContext) {
  context.subscriptions.push(
    vscode.languages.registerInlineCompletionItemProvider(
      { pattern: '**' },
      new FIMCompletionProvider()
    )
  );
}

The 150ms debounce fires the request only after the user has paused typing. Without it you'd send a request on every keystroke, which wastes API calls and often returns completions for a half-finished token.

Latency Targets and What Affects Them

Users notice autocomplete latency above roughly 200ms. If a suggestion appears quickly it feels like the editor is helping; at 500ms or more, users start ignoring it and the feature loses its value.

Things that affect latency:

• Model size: Qwen3-Coder 7B is meaningfully faster than 14B or 32B. Use the smallest model that produces acceptable quality for your use case.
• Max tokens: Cap it low. For single-line autocomplete, 64 tokens is enough. For function-level completion, 256 works. Higher ceilings don't improve quality for autocomplete -- they just slow the response.
• Streaming: For autocomplete, streaming doesn't help because you need the full completion before showing it. For chat, stream from the first token.
• Retrieval overhead: Your vector DB query needs to be fast. A local SQLite index runs in under 10ms. If you're querying a remote service, cache frequent lookups.

On a fast inference API, Qwen3-Coder 7B can produce a 64-token autocomplete in 50-150ms end to end. That's within the range where the latency is invisible to most users.

The Full Request Flow

For autocomplete:

1. User pauses typing for 150ms.
2. Extract the prefix (everything before the cursor) and suffix (everything after) from the current file.
3. Optionally retrieve relevant chunks from the codebase index to add as context.
4. Format the FIM prompt and send to the completions endpoint.
5. Return the completion text to VS Code's inline completion API.

For chat:

1. User sends a message in the chat panel.
2. Retrieve the top-k most relevant chunks from the codebase index.
3. Build a messages array with the retrieved context in the system prompt.
4. Stream the response back to the chat panel token by token.

Getting Started

The fastest path to a working coding assistant:

1. Install the Continue VS Code extension.
2. Get an API key at generalcompute.com.
3. Add the config above pointing at qwen3-coder-7b for autocomplete and deepseek-coder-v2-lite for chat.
4. Open any project, position your cursor, and press Tab or Cmd+I.

For teams that need custom indexing, the Tree-sitter parsing, sqlite-vec embeddings, and OpenAI client code above give you a solid foundation to build on. The components are all independent: you can swap in a different embedding model, a different vector store, or a different inference backend without rewriting the rest.