The local LLM serving layer: vLLM, SGLang, llama.cpp and Ollama

When an organization has picked the model it intends to run, what it holds is a file: weights, a vocabulary and the prompt format the model expects. Turning that file into a service is a separate piece of software. It queues incoming requests, shares out GPU memory, decides how many users can hold a conversation at once, and exposes an API surface.

This note separates four names — vLLM, SGLang, llama.cpp and Ollama — not by a feature table but by their architectural approach along four axes: scheduling, memory, reuse and packaging. It does not rank them for speed, and every number below is quoted with the date and baseline of its own source. The calculations on the model-file side — quantization levels, VRAM arithmetic and context window defaults — are covered in the note “What to check for Turkish when picking a local LLM”, and are not repeated here.

What the serving layer does, what the model file does

Two responsibilities come apart. The model file carries the weights, the vocabulary and the prompt format. The serving layer takes on scheduling, memory management, concurrency and the API surface. The split has a concrete consequence at deployment time: the same model file can run on all four layers, but how many users get an answer at once on the same hardware is decided by the serving layer, not by the file.

There is a sign the split is read the same way from the outside. In vLLM’s own feature list, memory management with PagedAttention stands as its own bullet, while continuous batching, chunked prefill and prefix caching are gathered into a single one.

In practice the layer usually takes the shape of an HTTP server: llama.cpp’s server documentation describes its own tool as a lightweight, pure C/C++ HTTP server built on httplib and nlohmann::json, offering a set of REST APIs and a web UI. The substrates overlap too — on 15 May 2025 Ollama announced its own engine on top of the GGML tensor library for multimodal models, saying in the same post that until then it had relied on the llama.cpp project for model support.

Batching approaches: static, dynamic and continuous

The first decision a serving layer makes is how to group requests that arrive at the same time. Three approaches can be laid out in order.

• A fixed batch: requests are gathered into a set, the set is executed as a whole, and the next batch does not start until every request in it has finished. This label is descriptive only; it is not a term from any source.
• Dynamic batching: the batch is formed by the server at run time. Requests wait briefly and are dispatched once the preferred size is reached or the delay budget is spent.
• Continuous batching: the contents of the batch are decided again at every iteration. A finished request leaves, and a waiting one joins as soon as the running iteration completes.

The documented form of the middle approach sits in NVIDIA Triton (see Triton Inference Server — Dynamic Batcher). Dynamic batching combines individual requests into a single batch at a moment the server chooses, and is recommended for stateless models. The wait is bounded by max_queue_delay_microseconds: if the preferred size is reached within the window the batch is dispatched immediately, and if the window expires it is dispatched at whatever size it has. The example configuration uses 100 microseconds.

The primary source for the third approach is Orca (USENIX OSDI ’22, pp. 521-538). The paper states the finding first: systems up to that point scheduled execution at request granularity, and the engine held a fixed set of requests until every request in the batch was done. That cuts both ways — a request that finishes early cannot return to the client, and a later arrival waits until the current batch is fully drained. Orca’s answer is to move scheduling down to iteration granularity: the scheduler selects the requests to run, calls the engine for a single iteration, and collects the results. Because control returns at every iteration, a finished request returns immediately and a new one can join once the running iteration ends. This is the source of what is now called continuous batching.

The paper defines a second technique as well: selective batching. The next iteration of two requests cannot always be merged — if both are in the initiation phase with different input token counts, if both are in the increment phase at different indices, or if one is in each phase, the attention tensor shapes do not line up. Batching is therefore applied not to all operations but to selected ones. Orca reports 36.9x throughput over NVIDIA FasterTransformer on GPT-3 175B at the same latency level: a 2022 measurement, against the baseline of that day.

The terms are not drawn the same way in every document. llama.cpp’s server documentation defines the --cont-batching flag as “continuous batching (a.k.a dynamic batching)” and enables it by default. The conceptual split above belongs to vLLM and SGLang terminology; when reading that document, the flag name should be taken as written.

PagedAttention: splitting the KV cache into pages

Once scheduling granularity drops to the iteration level, memory management becomes the deciding factor. The reason is the behaviour of the KV cache: it is large for every request and it grows and shrinks as generation proceeds. The PagedAttention paper says so directly — that behaviour determines the batch size (see Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180, SOSP ’23).

What the paper measures is the share: only 20.4% to 38.2% of a pre-allocated contiguous memory block holds actual token states, against 96.3% for vLLM in the same measurement. The remainder goes to slots reserved for future tokens, internal space produced by allocating against the maximum sequence length, and scattered space from the allocator.

The answer comes from operating systems. PagedAttention splits a request’s KV cache into blocks, each holding the key and value vectors of a fixed number of tokens, and does not require those blocks to be adjacent in physical memory. The paper’s analogy is the load-bearing sentence of this section: "one can think of blocks as pages, tokens as bytes, and requests as processes".

The bookkeeping follows from that. Each entry in the block table holds a physical block number and the number of filled slots in that block; a request’s cache is a sequence of logical blocks filled left to right, and a new physical block is allocated only once the previous ones are full. Every physical block carries a reference count and copy-on-write is applied at block granularity: in parallel sampling, two outputs sharing a prompt keep a single copy of the prompt state, and a write allocates a new block and copies the contents.

The payoff of sharing has been measured: in beam search, block sharing yields memory savings of up to 55%, and one experiment records the prompt portion accounting for 12% of total KV cache memory in parallel sampling. The paper reports 2-4x throughput against FasterTransformer and Orca at the same latency level — a 2023 measurement against the baselines of that day.

RadixAttention and prefix cache: reusing shared context

The third change of granularity is on the reuse side. PagedAttention made sharing within a single request cheap; SGLang targets sharing between requests (see SGLang: Efficient Execution of Structured Language Model Programs, arXiv:2312.07104). SGLang is not a model runner on its own but the co-design of a frontend language and a runtime.

What sets RadixAttention apart is simple: it does not discard the cache once generation ends, it keeps it in a radix tree. A radix tree is a space-efficient form of the prefix tree, and its edges can be labelled with variable-length sequences of elements rather than single elements. The tree manages the mapping between token sequences and their corresponding KV cache tensors, and both prompts and generations are cached.

This is a layer, not an alternative. The paper states plainly that RadixAttention is compatible with continuous batching, paged attention and tensor parallelism, and that each page holds a single token; when there is no cache hit, the memory and time overhead it introduces is negligible.

Reclaiming follows an LRU policy and starts from the leaves: the least recently used leaf is evicted first, and shared ancestors stay reusable until they become leaves themselves. Nodes used by the running batch stay in place until their reference counts fall to zero. No fixed-size cache pool is pre-allocated; cached tokens and running requests share the same memory pool.

Scheduling is cache-aware as well. The hit rate is the number of cached prompt tokens divided by the number of prompt tokens, and requests are sorted by matched prefix length. Theorem 3.1 states that when the cache size is not smaller than the maximum request length, longest-shared-prefix-first ordering is equivalent to a depth-first traversal of the radix tree and yields the optimal hit rate.

Where it pays off is concrete: few-shot prompts, benchmarks carrying a shared question prefix, agent templates together with prior calls, multi-turn chat history, and the shared context in a RAG pipeline. Across those benchmarks the hit rate ranges from 50% to 99%, and cache-aware scheduling approaches 96% of the optimal hit rate on average. In a month-long deployment on Chatbot Arena, hit rates of 52.4% for LLaVA-Next-34B and 74.1% for Vicuna-33B were observed, with average time to first token for Vicuna-33B reduced by 1.7x — a 2024 observation, specific to that workload.

vLLM offers the same function today; the difference is structural. In vLLM, block identity is hash-based: the block hash is formed from the parent block’s hash, the tokens in that block and extra values that make the block unique, only full blocks are cached, and a cache salt enters the hash to separate caches in multi-tenant environments. The default hash algorithm has been sha256 since v0.11, and prefix caching is on by default. On the SGLang side the same function is built from a radix tree with LRU leaf eviction. This is where the architectural difference is most concrete: the same function, a different data structure.

GGUF and the single binary: packaging and operating model

The fourth axis departs from scheduling and memory: packaging. GGUF is a file format for storing models for inference with GGML and GGML-based executors. Its design goals are listed in order in the specification: single-file deployment — files can be distributed and loaded easily and require no external files; extensibility — new features can be added without breaking compatibility; mmap compatibility; ease of use; and full information — everything needed to load a model is contained in the file.

That last goal is not abstract. The general metadata keys are defined as general.architecture, general.quantization_version, general.file_type and general.alignment. The tokenizer travels in the file in full: vocabulary, merge rules, token types, special token IDs, and a field that can embed an entire Hugging Face tokenizer.json when needed. The most striking key is tokenizer.chat_template — a Jinja template describing the input format the model expects, sitting in the same file as the weights.

The format is the successor to GGML, GGMF and GGJT. According to the history section, the positional parameter layout of its predecessors could not take on new hyperparameters while preserving compatibility; the key-value structure covers that.

In an on-premises deployment, two different “single files” should not be conflated. The first is the model’s single GGUF file. The second is the runner itself: llama.cpp’s first feature bullet is a plain C/C++ implementation without dependencies, and even the HTTP server is built on single-header libraries. The unit of distribution is not a container stack but a downloadable binary. The whole packaging pipeline sits in the same tree as well: convert_hf_to_gguf.py, quantize, gguf-split, imatrix and server.

Ollama’s unit of packaging is the Modelfile, described in the documentation as the blueprint for creating and sharing a model. Its instructions are FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE and MESSAGE; FROM accepts three sources — an existing model name, a Safetensors directory, or a path to a GGUF file. Quantization attaches to the creation step, in the documented form ollama create --quantize q4_K_M. The distribution behaviour is familiar: the model is copied into a username namespace, shared with ollama push, and run on the other side with ollama run. For what a quantization level means on the quality and memory side, the published note on choosing a model for Turkish is the place to look; the subject here is packaging itself.

Where the ecosystem converged: the TGI repository moving to archive

The freshest signal for why these four names are mentioned together comes from a project that closed. Hugging Face’s Text Generation Inference repository was archived on 21 March 2026; the date is written in the banner on the repository page. The README carries a maintenance-mode notice.

What matters is the second sentence of that same notice. It says TGI initiated the movement for optimized inference engines to rely on transformers model architectures, that the approach has since been adopted by downstream inference engines, and it points the reader by name to vLLM, SGLang, and locally-run engines with inter-compatibility such as llama.cpp or MLX. Three of the four layers in this note are named in the closing project’s own text.

The timeline is short: the last tagged release was v3.3.7 on 19 December 2025, with archiving roughly three months later. The repository’s feature list also shows what counted as standard at that date — tensor parallelism for faster inference on multiple GPUs, continuous batching of incoming requests for increased total throughput, a Messages API compatible with the OpenAI Chat Completion API, and optimized inference code using Flash Attention and Paged Attention.

The OpenAI-compatible API surface and how portable the choice is

The four layers share a surface, and it is not a compatibility shim bolted on afterwards. The implementation section of the PagedAttention paper describes vLLM as an end-to-end system consisting of a FastAPI frontend and a GPU-based inference engine, and notes that the frontend extends the OpenAI API interface to allow per-request customization of sampling parameters.

Today all four are entered through the same door. On the vLLM and llama.cpp side, /v1/completions, /v1/chat/completions, /v1/embeddings and /v1/responses are documented; llama.cpp also offers a surface compatible with the Anthropic Messages API, and a one-line start is available — llama serve -hf serves a model straight from a Hugging Face repository. Ollama states its own scope: compatibility with parts of the OpenAI API. On the SGLang side, chat/completions and completions are documented.

The limit of portability shows up here too: the surface travels, the settings do not. Structured output is the clearest example. vLLM offers five constraint types — choice, regular expression, JSON schema, context-free grammar and structural tag — and works with four separate backends. SGLang offers JSON schema, regular expression and EBNF constraints with XGrammar as the default. llama.cpp provides a schema-constrained JSON response format. The same request body carries over; the constraint definition and the default backend do not.

This surface standardizes access to the model; the layer that standardizes the model’s access to enterprise systems is a separate one, and it is covered in the note “What an MCP server is, and what it is not”.

Which layer fits which scenario in an on-premises setup

The most concrete pair of numbers separating these layers by operating posture sits in Ollama’s documentation. OLLAMA_NUM_PARALLEL is the number of concurrent requests per model and defaults to 1; OLLAMA_MAX_LOADED_MODELS is the number of models that can be loaded at once and defaults to 3 per GPU. The queue bound, OLLAMA_MAX_QUEUE, is 512. Those defaults describe a posture: low concurrency, many models.

The defaults in vLLM and SGLang describe the opposite: one model, high concurrency. vLLM schedules first-come-first-served; because the blocks of a sequence are accessed together, eviction is applied all-or-nothing, and recovery happens through swapping to CPU memory or recomputation. On the multi-GPU side, Megatron-LM style tensor parallelism is supported; because every model shard processes the same set of input tokens, a single KV cache manager and a single logical-to-physical block mapping are kept inside the central scheduler.

SGLang describes itself as a serving framework that scales from a single GPU to large distributed clusters. The defaults in its server arguments confirm the posture: radix cache on, lru as the eviction policy, page size 1, tensor parallel size 1.

llama.cpp’s posture is minimal setup: a backend list running from BLAS to CUDA and Metal to Vulkan, and CPU+GPU hybrid inference that partially accelerates models larger than total VRAM capacity. On the Ollama side there is a detail operations should know: model directories live in documented locations and move with the OLLAMA_MODELS environment variable. Where data locality is a written condition, that path belongs in the deployment document as well.

The architectural counterpart of the question of who may see which context in a shared deployment is worked through in the note “Access control in RAG: who gets to see what”.

Release cadence is quick. As of 8 August 2026 the tagged release of vLLM is v0.26.0 (27 July 2026) and of SGLang v0.5.17 (8 August 2026). The flag names and defaults in this note belong to that same date; compare them against the documentation for your own version before deploying.

Sources

• Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI ’22, pp. 521-538 (iteration-level scheduling, selective batching, the 36.9x measurement)
• Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv:2309.06180, SOSP ’23 (block table, reference counting and copy-on-write, token-state share)
• SGLang: Efficient Execution of Structured Language Model Programs — arXiv:2312.07104 (RadixAttention, LRU leaf-first reclaiming, cache-aware scheduling, Theorem 3.1)
• NVIDIA Triton Inference Server Documentation — Dynamic Batcher (dynamic batching, max_queue_delay_microseconds)
• vLLM — GitHub repository, README.md and vllm/config/cache.py (feature list, prefix caching default)
• vLLM Documentation — Automatic Prefix Caching, Structured Outputs, OpenAI-Compatible Server, Parallelism and Scaling
• SGLang — GitHub repository and Documentation, Server Arguments (radix cache default, lru, page size 1)
• Text Generation Inference — GitHub repository (archived on 21 March 2026; last release v3.3.7, 19 December 2025)
• GGUF Specification — ggml-org/ggml, docs/gguf.md (design goals, general metadata, tokenizer.chat_template)
• llama.cpp — GitHub repository and tools/server/README.md (dependency-free implementation, backend table, OpenAI-compatible routes)
• Ollama Documentation — Modelfile Reference, Import a model, OpenAI compatibility and FAQ (instructions, concurrency defaults, model directories)
• Ollama’s new engine for multimodal models — Ollama Blog, 15 May 2025
• GitHub Releases — vllm-project/vllm v0.26.0 and sgl-project/sglang v0.5.17