What to check for Turkish when picking a local LLM

When an organization picks an open-weight model to run on-premises, the reflex is to reach for a comparison table. Almost all of those tables are built on English tasks, and whatever they say about Turkish is indirect. Turkish is agglutinative: a surface word is typically a root plus a chain of suffixes. That structure propagates through every link of the stack — the tokenizer, the effective context window, the KV cache, and ultimately the VRAM budget. This note does not tell you how any particular model behaves in Turkish. It gives you the criteria to check and the method to measure each one on your own corpus. The result table is yours to produce, because it is only meaningful against your own documents and your own tasks.

1. Tokenizer: how many tokens does the same sentence cost

The tokenizer is the first point of contact between a model family and Turkish, and it is the easiest criterion to measure. The metric is called fertility: the average number of subword units per word. The value varies with the tokenizer family and with the text itself, so take your own measurement on your own corpus as the reference rather than importing a number from elsewhere.

Standard tokenizers such as BPE and WordPiece do not optimize for linguistically meaningful morphemes; they optimize for orthographic substrings that occur frequently in the training corpus. Because a Turkish surface word fuses a root and a suffix chain, the resulting split does not line up with morpheme boundaries (arXiv:2502.07057). The practical consequence is twofold: a Turkish text carrying the same meaning as its English counterpart may expand into more tokens, which shows up as cost anywhere you pay or allocate memory per token, and as less real content fitting inside a fixed context window.

A representative sample of your own text is enough for the measurement — contracts, support tickets, technical documentation, email bodies, whichever will actually flow through the system in production. Run the candidate tokenizers over the same file and compare the ratio.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("<model-repo>")

with open("sample-corpus-tr.txt", encoding="utf-8") as f:
    text = f.read()

words = len(text.split())
tokens = len(tok(text, add_special_tokens=False)["input_ids"])

print(f"words     : {words}")
print(f"tokens    : {tokens}")
print(f"fertility : {tokens / words:.2f}")

Vocabulary size is usually stated on the model card. The Gemma 3 technical report (arXiv:2503.19786), for instance, describes the SentencePiece tokenizer shared with Gemini 2.0 and a 262k-entry vocabulary, with split digits, preserved whitespace and byte-level encodings. Vocabulary size on its own is not a verdict: the same vocabulary size produces different Turkish behaviour on different training corpora. Note the vocabulary as a hint; let the fertility measurement drive the decision.

2. Turkish in the training mix: read the resolution of the claim

There is no direct way to learn how much Turkish a model saw. All you have are model card and technical report statements, and their resolution varies. Meta states for Llama 3 that over 5% of the pretraining set is high-quality non-English data covering more than 30 languages, and explicitly says it does not expect the same level of performance in those languages as in English. The Qwen3 technical report (arXiv:2505.09388) reports pretraining on 36 trillion tokens covering 119 languages and dialects. Google’s Gemma 3 model card mentions training data spanning over 140 languages, while the technical report itself gives no exact language count and only notes that the multilingual share was increased.

What these three have in common is that none of them break out Turkish as a percentage. What gets disclosed is either a language count or a total non-English share. That makes “how many languages does it support” not a discriminating criterion on its own. The more discriminating question is how specific the disclosure is, and whether Turkish appears by name among the languages the vendor says it evaluated. Treat this as an input to your evaluation plan rather than a conclusion: the vaguer the disclosure, the more weight your own benchmark has to carry.

3. Quantization and hardware: do the arithmetic first

Quantization is the dial between quality and VRAM, and the weight side of it is fully deterministic. FP16 costs two bytes per parameter. The llama.cpp quantize documentation makes the relationship concrete with measurements on Llama-3.1-8B: F16 at 14.96 GiB, Q8_0 (8.5 bits per weight) at 7.95 GiB, Q4_K_M (4.89 bits per weight) at 4.58 GiB. For your own model you can run the same arithmetic straight from the parameter count and the bits-per-weight figure. Q4_K_M is not a uniform 4-bit format; it belongs to the k-quant family and keeps part of the tensors at higher precision.

The quality side is not deterministic and cannot be summarized as a single percentage. Measure the quantization level against your own evaluation set: run the same question set through FP16, Q8 and Q4 builds and decide whether the delta matters for your task. One detail is directly useful for Turkish. The `llama-imatrix` tool in llama.cpp computes an importance matrix from calibration data, and `llama-quantize --imatrix` uses that matrix during quantization on the premise that some weights matter more than others. Because you choose the calibration data, quantizing with a Turkish-calibrated importance matrix is technically available to you.

Weights are only one component of the VRAM budget. The second is the KV cache, which grows linearly with sequence length and is usually the real constraint at long context. The formula falls straight out of the architecture parameters.

def kv_cache_bytes(layers, kv_heads, head_dim, seq, batch=1, elem_bytes=2):
    # elem_bytes: 2 for BF16/FP16, 1 for FP8
    # leading factor of 2 accounts for K and V
    return 2 * layers * kv_heads * head_dim * seq * batch * elem_bytes

# 80 layers, 8 KV heads, head_dim 128, BF16
print(kv_cache_bytes(80, 8, 128, 1) / 1e6)         # ~0.33 MB per token
print(kv_cache_bytes(80, 8, 128, 128_000) / 1e9)   # ~42 GB
print(kv_cache_bytes(80, 8, 128, 1_000_000) / 1e9) # ~328 GB

Grouped-Query Attention shrinks this by dropping the KV head count below the query head count; a 64-query / 8-KV configuration gives an eightfold reduction over full multi-head attention. Architecture matters too. The Gemma 3 technical report describes the published configuration as a pattern of five local layers per global layer with a 1024-token sliding window on the local ones, so only the global layers attend over the long context. The practical effect is that KV cache growth on most layers is bounded by the window size rather than by sequence length. The planning rule stays simple: total VRAM ≈ weights + KV cache + activations and working space. With your model and target context fixed, the first two terms are pencil-and-paper work.

Plan hardware around your target context length, not around parameter count. In a deployment serving 128K context, the KV cache can approach or exceed the size of the weights themselves.

4. Context window: advertised length versus usable length

The context length on a model card is a ceiling, not a guarantee. RULER, the synthetic long-context benchmark from NVIDIA researchers (arXiv:2404.06654), makes the distinction measurable. It spans four task categories: retrieval (extended needle-in-a-haystack variants), multi-hop tracing (variable tracking), aggregation, and question answering with distractor documents. It defines effective context length as the largest context size at which a model stays above a performance threshold. The headline finding is that only about half of the models advertising 32K or more sustain satisfactory performance at 32K, and that models scoring near-perfectly on the plain needle test degrade noticeably on RULER’s harder tasks as length grows.

The second thing to check is how the window was obtained. The Qwen3-8B model card states native support for 32,768 tokens and validation up to 131,072 tokens with YaRN. The same card carries a caveat: in the static YaRN implementation the scaling factor stays constant regardless of input length, which can affect performance on shorter inputs, so the extension should be enabled only when long context is genuinely required.

{
  "rope_scaling": {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 32768
  }
}

The third thing is the serving layer. Ollama’s official documentation states that the default context length scales with available VRAM: 4k below 24 GiB, 32k between 24 and 48 GiB, 256k at 48 GiB and above. A model may support 128K while the runtime quietly defaults to 4K. Before you start evaluating, confirm the context length the serving layer is actually running with — otherwise you are measuring a default configuration rather than the model.

5. Instruction following and the language of the system prompt

In enterprise use what you usually need from the model is form rather than knowledge: return valid JSON and nothing else, produce at most three bullets, stay inside the given field names, add nothing that is absent from the source text. The established way to measure this is IFEval (arXiv:2311.07911), which defines 25 types of verifiable instruction and designs each one so the response can be checked by a program rather than judged by another model.

For Turkish there is a concrete gap here. M-IFEval (arXiv:2502.04688, Findings of ACL: NAACL 2025), the multilingual extension of IFEval, covers French, Japanese and Spanish — it does not include Turkish. That same work reports that across eight models, performance varies widely across languages and instruction types. So extrapolating from an English instruction-following score to Turkish has no support, and no off-the-shelf standard measurement for Turkish exists. The design idea, however, transfers cleanly: build your own Turkish instruction set so that every instruction has a checker function behind it.

import json

def at_least_bullets(answer, n=5):
    return sum(1 for line in answer.splitlines()
               if line.strip().startswith("- ")) >= n

def is_valid_json(answer):
    try:
        json.loads(answer)
        return True
    except json.JSONDecodeError:
        return False

def keeps_turkish_diacritics(answer):
    return any(ch in answer for ch in "çğıöşüÇĞİÖŞÜ")

CHECKS = {
    "at_least_5_bullets": at_least_bullets,
    "valid_json": is_valid_json,
    "turkish_diacritics": keeps_turkish_diacritics,
}

The language of the system prompt is a separate variable. Run the same instruction set twice: once with a Turkish system prompt, once with an English system prompt and Turkish user input. The delta between the two directly shapes your prompt architecture, and the measurement itself is a few hours of work.

6. Building your own evaluation set

Every criterion above converges on one artifact: a question-and-answer set derived from your organization’s own documents. Existing Turkish infrastructure helps you shape it. Cetvel (arXiv:2508.16431, Koç University KUIS-AI) is a unified Turkish evaluation suite built on `lm-evaluation-harness`, covering 23 tasks across seven categories, and it was run across 33 open-weight models up to 70B. One methodological observation from that work is that grammatical error correction and extractive question answering turned out to be particularly discriminative between capability levels — a useful signal for deciding which task types to weight in your own set. TR-MMLU (arXiv:2501.00593) offers 6,200 multiple-choice questions across 62 sections, drawn from a pool of 280,000. The OpenLLM Turkish Leaderboard and a Turkish-adapted fork of `lm-evaluation-harness` are also available.

Those suites are reference points for general capability; they do not measure your workload. A practical frame for your own set: collect 100–200 examples from the question types that actually arrive in production, write the expected answer or the acceptance criterion for each with an internal subject-matter expert, tag every item by task type (extractive QA, summarization, format conversion, classification), and record the reference phrases that must appear in a correct answer. The Cetvel paper underlines that machine-translated benchmarks fall short on cultural content — so gather examples from Turkish sources rather than pushing an English set through automatic translation.

Writing the evaluation set down has a second payoff: once it exists, a model swap, a quantization change and a prompt change all become comparable on the same scale. Without it, every decision starts from scratch.

To pull it together: measure tokenizer fertility on your own text, note the resolution of the language disclosure on the model card, derive the VRAM side of quantization arithmetically and test the quality side on your own set, validate the advertised context window against RULER-style tasks and against the serving layer’s real configuration, measure instruction following in Turkish with programmatic checkers, and plan hardware from weights plus the KV cache formula. None of these steps produce a model ranking, and none of them need to. What they produce is a repeatable measurement floor that shows which configuration meets your requirements under your own load. Model preferences shift several times in a couple of years; the measurement floor stays.

Sources

• Meta — Introducing Meta Llama 3 (statement on non-English share of the pretraining data and language count)
• Qwen3 Technical Report — arXiv:2505.09388 (36 trillion tokens, 119 languages and dialects)
• Gemma 3 Technical Report — arXiv:2503.19786 (tokenizer, vocabulary size, local/global layer ratio, sliding window)
• Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark — arXiv:2502.07057
• llama.cpp — quantize tool documentation (bits-per-weight and file size table for Llama-3.1-8B)