The local LLM serving layer: vLLM, SGLang, llama.cpp and Ollama

When an organization has picked the model it intends to run, what it holds is a file: weights, a vocabulary and the prompt format the model expects. Turning that file into a service is a separate piece of software. It queues incoming requests, shares out GPU memory, decides how many users can hold a conversation at once, and exposes an API surface.

This note separates four names — vLLM, SGLang, llama.cpp and Ollama — not by a feature table but by their architectural approach along four axes: scheduling, memory, reuse and packaging. It does not rank them for speed, and every number below is quoted with the date and baseline of its own source. The calculations on the model-file side — quantization levels, VRAM arithmetic and context window defaults — are covered in the note “What to check for Turkish when picking a local LLM”, and are not repeated here.

What the serving layer does, what the model file does

Two responsibilities come apart. The model file carries the weights, the vocabulary and the prompt format. The serving layer takes on scheduling, memory management, concurrency and the API surface. The split has a concrete consequence: the same model file can run on all four layers, but how many users get an answer at once on the same hardware is decided by the serving layer, not by the file.

In practice the layer usually takes the shape of an HTTP server: llama.cpp’s server documentation describes its own tool as a lightweight, pure C/C++ HTTP server built on httplib and nlohmann::json, offering REST APIs and a web UI. The substrates overlap too — on 15 May 2025 Ollama announced its own engine on top of the GGML tensor library for multimodal models, saying in the same post that until then it had relied on the llama.cpp project for model support.

Batching approaches: static, dynamic and continuous

The first decision a serving layer makes is how to group requests arriving at the same time.

• A fixed batch: requests are gathered into a set, executed as a whole, and the next batch does not start until every request in it has finished. This label is descriptive only; it is not a term from any source.
• Dynamic batching: the batch is formed by the server at run time. Requests wait briefly and are dispatched once the preferred size is reached or the delay budget is spent.
• Continuous batching: the contents of the batch are decided again at every iteration. A finished request leaves, and a waiting one joins as soon as the running iteration completes.

The documented form of the middle approach sits in NVIDIA Triton (see Triton Inference Server — Dynamic Batcher). Dynamic batching combines individual requests into a single batch at a moment the server chooses. The wait is bounded by max_queue_delay_microseconds, and the example configuration uses 100 microseconds.

The primary source for the third approach is Orca (USENIX OSDI ’22, pp. 521-538). The paper states the finding first: systems up to that point scheduled execution at request granularity, holding a fixed set of requests until every request in the batch was done, so a request that finished early could not return to the client and a later arrival waited until the current batch was fully drained. Orca’s answer is to move scheduling down to iteration granularity: the scheduler selects the requests to run, calls the engine for a single iteration, and collects the results. This is the source of what is now called continuous batching.

The paper defines a second technique as well: selective batching. The next iteration of two requests cannot always be merged — if both are in the initiation phase with different input token counts, or if one is in each phase, the attention tensor shapes do not line up. Batching is therefore applied not to all operations but to selected ones. Orca reports 36.9x throughput over NVIDIA FasterTransformer on GPT-3 175B at the same latency level: a 2022 measurement, against the baseline of that day.

The terms are not drawn the same way in every document. llama.cpp’s server documentation defines the --cont-batching flag as “continuous batching (a.k.a dynamic batching)” and enables it by default. The conceptual split above belongs to vLLM and SGLang terminology.

PagedAttention: splitting the KV cache into pages

Once scheduling drops to iteration granularity, memory management becomes the deciding factor. The reason is the behaviour of the KV cache: it is large for every request and grows and shrinks as generation proceeds. The PagedAttention paper says so directly — that behaviour determines the batch size (see Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180, SOSP ’23).

What the paper measures is the share: only 20.4% to 38.2% of a pre-allocated contiguous memory block holds actual token states, against 96.3% for vLLM in the same measurement. The remainder goes to slots reserved for future tokens and to space produced by allocating against the maximum sequence length.

The answer comes from operating systems. PagedAttention splits a request’s KV cache into blocks, each holding the key and value vectors of a fixed number of tokens, and does not require those blocks to be adjacent in physical memory. The paper’s analogy carries the section: "one can think of blocks as pages, tokens as bytes, and requests as processes".

The bookkeeping follows from that. Each entry in the block table holds a physical block number and the number of filled slots in that block; a request’s cache is a sequence of logical blocks filled left to right, and a new physical block is allocated only once the previous ones are full. Every physical block carries a reference count and copy-on-write is applied at block granularity: two outputs sharing a prompt keep a single copy of the prompt state, and a write allocates a new block.

The payoff of sharing has been measured: in beam search, block sharing yields memory savings of up to 55%. The paper reports 2-4x throughput against FasterTransformer and Orca at the same latency level — a 2023 measurement against the baselines of that day.

RadixAttention and prefix cache: reusing shared context

The third change of granularity is on the reuse side. PagedAttention made sharing within a single request cheap; SGLang targets sharing between requests (see SGLang: Efficient Execution of Structured Language Model Programs, arXiv:2312.07104).

What sets RadixAttention apart is simple: it keeps the cache in a radix tree once generation ends rather than discarding it. A radix tree is a space-efficient form of the prefix tree, whose edges can be labelled with variable-length sequences rather than single elements. The tree maps token sequences to their KV cache tensors, and both prompts and generations are cached.

This is a layer, not an alternative. The paper states plainly that RadixAttention is compatible with continuous batching, paged attention and tensor parallelism, and that when there is no cache hit, the memory and time overhead it introduces is negligible.

Reclaiming follows an LRU policy and starts from the leaves: the least recently used leaf goes first, and shared ancestors stay reusable until they become leaves themselves. No fixed-size cache pool is pre-allocated; cached tokens and running requests share the same memory pool.

Scheduling is cache-aware as well: requests are sorted by matched prefix length. Theorem 3.1 states that when the cache size is not smaller than the maximum request length, longest-shared-prefix-first ordering is equivalent to a depth-first traversal of the radix tree and yields the optimal hit rate.

Where it pays off is concrete: few-shot prompts, multi-turn chat history and the shared context in a RAG pipeline. Across those benchmarks the hit rate ranges from 50% to 99%, and cache-aware scheduling approaches 96% of the optimal hit rate on average. In a month-long deployment on Chatbot Arena, hit rates of 52.4% for LLaVA-Next-34B and 74.1% for Vicuna-33B were observed, with average time to first token for Vicuna-33B reduced by 1.7x — a 2024 observation, specific to that workload.

vLLM offers the same function today; the difference is structural. In vLLM, block identity is hash-based: the block hash is formed from the parent block’s hash and the tokens in that block, only full blocks are cached, and a cache salt enters the hash to separate caches in multi-tenant environments. The default hash algorithm has been sha256 since v0.11, and prefix caching is on by default. On the SGLang side the same function is built from a radix tree with LRU leaf eviction — the same function, a different data structure.

GGUF and the single binary: packaging and operating model

The fourth axis departs from scheduling and memory: packaging. GGUF is a file format for storing models for inference with GGML and GGML-based executors. Its design goals are listed in order in the specification: single-file deployment — files can be distributed and loaded easily and require no external files; extensibility; mmap compatibility; ease of use; and full information — everything needed to load a model is contained in the file.

That last goal is not abstract. The general metadata keys are defined as general.architecture, general.quantization_version, general.file_type and general.alignment. The tokenizer travels in the file in full: vocabulary, merge rules, token types and special token IDs. The most striking key is tokenizer.chat_template — a Jinja template describing the input format the model expects, sitting in the same file as the weights.

In an on-premises deployment, two different “single files” should be told apart. The first is the model’s single GGUF file. The second is the runner itself: llama.cpp’s first feature bullet is a plain C/C++ implementation without dependencies, and the unit of distribution is a downloadable binary rather than a container stack.

Ollama’s unit of packaging is the Modelfile, described in the documentation as the blueprint for creating and sharing a model. Its instructions are FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE and MESSAGE; FROM accepts an existing model name, a Safetensors directory or a path to a GGUF file. Quantization attaches to the creation step, in the documented form ollama create --quantize q4_K_M, and the model is then shared with ollama push and run on the other side with ollama run. For what a quantization level means on the quality and memory side, the published note on choosing a model for Turkish is the place to look; the subject here is packaging itself.

Where the ecosystem converged: the TGI repository moving to archive

The freshest signal for why these four names are mentioned together comes from a project that closed. Hugging Face’s Text Generation Inference repository was archived on 21 March 2026; the date is written in the banner on the repository page. The README carries a maintenance-mode notice.

What matters is the second sentence of that same notice. It says TGI initiated the movement for optimized inference engines to rely on transformers model architectures, and points the reader by name to vLLM, SGLang, and locally-run engines with inter-compatibility such as llama.cpp or MLX. Three of the four layers in this note are named in the closing project’s own text.

The timeline is short: the last tagged release was v3.3.7 on 19 December 2025. The repository’s feature list also shows what counted as standard at that date — tensor parallelism on multiple GPUs, continuous batching of incoming requests, a Messages API compatible with the OpenAI Chat Completion API, and inference code using Flash Attention and Paged Attention.

Which layer fits which scenario in an on-premises setup

The most concrete pair of numbers separating these layers by operating posture sits in Ollama’s documentation. OLLAMA_NUM_PARALLEL is the number of concurrent requests per model and defaults to 1; OLLAMA_MAX_LOADED_MODELS is the number of models that can be loaded at once and defaults to 3 per GPU. Those defaults describe a posture: low concurrency, many models.

The defaults in vLLM and SGLang describe the opposite: one model, high concurrency. vLLM schedules first-come-first-served; because the blocks of a sequence are accessed together, eviction is applied all-or-nothing. Megatron-LM style tensor parallelism is supported, and because every model shard processes the same set of input tokens, a single KV cache manager and a single logical-to-physical block mapping are kept inside the central scheduler.

SGLang describes itself as a serving framework that scales from a single GPU to large distributed clusters. The defaults in its server arguments confirm the posture: radix cache on, lru as the eviction policy, page size 1, tensor parallel size 1.

llama.cpp’s posture is minimal setup: a backend list running from BLAS to CUDA and Metal to Vulkan, and CPU+GPU hybrid inference that partially accelerates models larger than total VRAM capacity. On the Ollama side, model directories live in documented locations and move with the OLLAMA_MODELS environment variable; where data locality is a written condition, that path belongs in the deployment document as well.

The API surface standardizes access to the model; the layer that standardizes the model’s access to enterprise systems is a separate one, and it is covered in the note “What an MCP server is, and what it is not”.

The architectural counterpart of the question of who may see which context in a shared deployment is worked through in the note “Access control in RAG: who gets to see what”.

Release cadence is quick. As of 8 August 2026 the tagged release of vLLM is v0.26.0 (27 July 2026) and of SGLang v0.5.17 (8 August 2026). The flag names and defaults in this note belong to that same date; compare them against the documentation for your own version before deploying.

Sources

• Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI ’22, pp. 521-538 (iteration-level scheduling, selective batching, the 36.9x measurement)
• Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv:2309.06180, SOSP ’23 (block table, copy-on-write, token-state share)
• SGLang: Efficient Execution of Structured Language Model Programs — arXiv:2312.07104 (RadixAttention, cache-aware scheduling, Theorem 3.1)
• NVIDIA Triton Inference Server Documentation — Dynamic Batcher (max_queue_delay_microseconds)
• vLLM — GitHub repository, README.md and vllm/config/cache.py (feature list, prefix caching default)
• vLLM Documentation — Automatic Prefix Caching, Structured Outputs, OpenAI-Compatible Server, Parallelism and Scaling
• SGLang — GitHub repository and Documentation, Server Arguments (radix cache default, lru, page size 1)
• Text Generation Inference — GitHub repository (archived on 21 March 2026; last release v3.3.7)
• GGUF Specification — ggml-org/ggml, docs/gguf.md (design goals, tokenizer.chat_template)
• llama.cpp — GitHub repository and tools/server/README.md (dependency-free implementation, backend table)
• Ollama Documentation — Modelfile Reference, Import a model, OpenAI compatibility and FAQ (concurrency defaults, model directories)
• Ollama’s new engine for multimodal models — Ollama Blog, 15 May 2025
• GitHub Releases — vllm-project/vllm v0.26.0 and sgl-project/sglang v0.5.17