← Insights

DE — Deutsche Version

AI Glossary

A plain-language reference for local language models, inference engines, model architectures, quantization, memory, multi-GPU operation and speculative decoding.

Last updated: 26 August 2026

How this glossary is organized

Instead of a flat alphabetical list, the terms sit under the broader concept
they belong to. That makes the relationships between them visible:

  • Local LLM inference
    • Model and architecture
      • Dense, MoE, attention, KV heads, DeltaNet
    • Number formats and quantization
      • BF16, FP8, INT4, AWQ, GPTQ, GGUF
    • Memory and hardware
      • VRAM, KV cache, bandwidth, PCIe, NVLink
    • Inference engines
      • vLLM, SGLang, llama.cpp, TensorSharp
    • Parallelism and serving
      • TP, PP, EP, batching, PagedAttention
    • Speculative decoding
      • MTP, EAGLE, DFlash, DSpark, n-gram

Many terms belong to more than one area. The KV cache, for example, is an
architecture topic, a memory topic and a vLLM topic at the same time.

Reading model names in ten seconds

A name like Qwen3.6-35B-A3B-FP8 is a very compact technical description:

Part Meaning
Qwen3.6 Model family and generation
35B Roughly 35 billion parameters in total
A3B Roughly 3 billion parameters are active per token
FP8 The model weights are stored in an 8-bit floating-point format

Other common notations:

Notation Short explanation
27B 27 billion parameters; B stands for billion
A10B About 10 billion parameters are active per token
262k Context limit of roughly 262,000 tokens
1M Context limit of roughly one million tokens
TP4 Tensor parallelism across four GPUs
TP2+PP3 Two GPUs per tensor group and three pipeline stages, six GPUs in total
Q4_K_M GGUF quantization with roughly four bits per weight
W4A16 4-bit weights, but 16-bit activations during the computation
it / Instruct Variant post-trained for instructions and chat
Base Base model with little or no chat/instruct post-training

Important: the name still says nothing about whether a model is good,
fast or compatible with a particular engine.

1. Basic terms

Term Plain-language explanation Related to
LLM Large Language Model. It computes which token is likely to come next. Model, token, inference
Local LLM A language model that runs on your own hardware instead of at a cloud provider. Privacy, GPU, inference engine
Model The trained network together with its learned numerical values. It is the actual “brain”. Architecture, weights
Checkpoint A saved state of a model. A checkpoint can be the original or a quantized variant. Safetensors, GGUF
Open Weights The model weights can be downloaded and run by yourself. That does not automatically mean training, data or license are fully open. Hugging Face, checkpoint
Parameter A learned number inside the model. Billions of such numbers determine its behavior. Weight, B, active parameters
Weight A parameter that gives inputs more or less influence inside the network. Quantization, VRAM
Active Parameters The part of an MoE model that is actually used for a single token. MoE, A3B, decode speed
Inference Using a fully trained model to produce answers. Prefill, decode
Training Learning the weights from very many examples. It is considerably more expensive than inference afterwards. Fine-tuning, calibration
Fine-Tuning Additional training of an existing model for particular tasks or behaviors. Instruct, drafter
Distillation A smaller model learns to imitate the outputs or internal signals of a larger one. Draft model, EAGLE, DFlash
Token A building block of text. A token can be a word, part of a word, punctuation or a code fragment. Tokens are not the same as characters or words. Tokenizer, context
Tokenizer Translates text into token IDs and later back again. Vocabulary, chat template
Token ID The number of a token in the vocabulary. Tokenizer, structure token
Vocabulary The complete list of tokens a model knows. Two models can have incompatible vocabularies. Tokenizer, draft model
Prompt The input to the model. It often includes more than just the visible user question. System prompt, context
System Prompt Hidden or prepended base instruction for the model’s role, rules and tools. Chat template, prefix cache
Completion The part of an answer that the model newly generates. Output tokens, decode
Context Everything the model can see in the current request: rules, history, tool output, files and the new question. Context window, KV cache
Context Window The maximum number of tokens that input and output may occupy together. 262k, 1M, YaRN
Output Budget Upper limit for the tokens still to be generated. A large context window does not automatically mean an equally large output budget. max_tokens, reasoning
Reasoning / Thinking Thinking steps the model produces before the actual answer. They can be visible, separated out or hidden, but they occupy context and compute time. Think tokens, reasoning parser
Think Tokens Tokens of the thinking process. Many thousands of think tokens can fill the context window even though the user only sees a short final answer. Reasoning, context
Tool Calling / Function Calling The model requests a tool in structured form, for example reading a file, a web search or a shell command. Tool parser, agent
Agent / agentic A model works in several steps, uses tools, checks results and carries the task forward. Tool loop, context, prefix caching
RAG Retrieval-Augmented Generation: matching document excerpts are retrieved before the answer and inserted into the prompt. Retrieval, context, embedding
Hallucination A plausible-sounding but invented or incorrect statement by the model. Grounding, verification

2. Model architecture

2.1 The building blocks of a model

Term Plain-language explanation Related to
Transformer The classic base architecture of modern language models. It processes tokens in many successive layers. Attention, layer
Layer / Block One computation layer of the model. A token passes through many layers one after another. Attention, MLP, PP
Embedding Vector of numbers that represents a token for the model. Similar meanings can have similar vectors. Token, hidden state
Hidden State Internal numerical representation of what the model has “understood” at a given point. EAGLE, DFlash, LM head
Logits Raw scores for all possible next tokens. Sampling turns them into a concrete choice. LM head, softmax, sampling
LM Head The model’s final projection. It translates the hidden state into logits over the vocabulary. Logits, vocabulary
FFN / MLP Computation block inside a layer that transforms the token representation. In an MoE there are many alternative experts of this kind. MoE, expert
Dense Model Every token uses all model parameters. Simpler and often robust, but hungry for memory and bandwidth in large models. MoE, active parameters
MoE Mixture of Experts. Many experts are stored, but only a small selection is activated per token. Router, A3B, expert parallelism
Expert A specialized FFN inside an MoE layer. The name does not mean that a single expert maps cleanly onto a topic such as mathematics. MoE, router
Routed Expert Expert that the router selects for each token. Top-k, router
Shared Expert Expert that always contributes, independently of the router’s selection. MoE, active parameters
Router / Gate Small network that picks the appropriate experts for every token and every MoE layer. MoE, quantization
Top-k Experts Number of experts selected per token, 8 out of 256 for example. Not to be confused with the sampling parameter top_k. Router, MoE
A3B / A10B Shorthand for roughly 3 or 10 billion active parameters per token. MoE, decode speed
Hybrid Architecture The model mixes different layer types, for example full attention and DeltaNet. DeltaNet, SWA, KV cache
Recurrent State Compressed memory that is passed on from step to step. In some layers it replaces a growing KV cache. DeltaNet, SSM
SSM State Space Model. Processes sequences through a compact running state instead of full attention over all earlier tokens. Recurrence, hybrid model

2.2 Attention, heads and KV memory

The simplest way to think about attention: a token puts a question to the
text so far and looks for matching keys and contents.

Term Plain-language explanation Related to
Attention Mechanism by which a token weights relevant earlier tokens. Query, key, value
Self-Attention Attention within the same token sequence. The text looks at its own preceding content. Transformer, KV cache
Query (Q) The current search question of an attention head: “which earlier information do I need right now?” Key, attention head
Key (K) Search key of an earlier token. Query and key together determine how relevant that token is. Query, KV cache
Value (V) The actual information content that is read when a key matches. Key, KV cache
Attention Head / Query Head One parallel viewpoint of attention. Different heads can capture different relationships in the text. MHA, GQA
KV Head A head that produces key and value data. Several query heads can share one KV head. Fewer KV heads usually mean a smaller KV cache. GQA, MQA, TP
MHA Multi-Head Attention. Every query head has its own key and value heads. Flexible, but the KV cache becomes large. Attention head, KV head
GQA Grouped-Query Attention. Several query heads share one KV head. This saves KV memory, usually with little loss of quality. MHA, MQA
MQA Multi-Query Attention. All query heads share a single KV head. Very economical with memory, but the sharing is correspondingly heavier. GQA, KV cache
KV Cache Store for the keys and values of tokens already processed. Without it, the model would have to recompute the entire history for every new token. Context, VRAM, PagedAttention
KV Memory Informal name for the space the KV cache occupies. With GPU inference it usually sits in VRAM. KV cache, KV dtype
KV Cache per Token The memory one additional context token occupies. It depends on the number of layers, the KV heads, the head dimension and the data type, among other things. Context window, BF16 KV
Head Dimension Size of the number vector of an attention head. KV cache, model architecture
Head Divisibility With TP, the number of attention and KV heads usually has to be divisible by the TP size. Eight KV heads fit TP2, TP4 and TP8, for example, but not TP6. Tensor parallelism
KV Cache DType Number format of the KV cache, for example BF16, FP8, INT8 or q8_0. It is independent of the format of the model weights. KV quantization
Cache Eviction An old KV cache is discarded to make room for another request. It has to be rebuilt on the next access. Multi-user, PagedAttention

2.3 Long contexts and alternative attention

Term Plain-language explanation Related to
Full / Dense Attention Every new token can in principle look at all previous tokens. Accurate, but expensive with long contexts. KV cache, quadratic cost
Quadratic Attention Cost Double the sequence length and, with full attention, roughly four times as many token pairs can become relevant. Prefill, long context
Sliding-Window Attention (SWA) A layer only looks at a limited window of the most recent tokens, 512 or 2048 for example. Hybrid model, local context
Sparse Attention Only selected parts of the context are considered. This saves computation with long contexts. MSA, indexer
MSA MiniMax Sparse Attention. An index selects the relevant context blocks instead of always reading everything. Sparse attention, GQA
MLA Multi-Head Latent Attention. Keys and values are kept in a compressed latent representation in order to save memory and bandwidth. Known from DeepSeek models. KV cache, attention
Linear Attention Processes the past through a compact state. Cost and memory grow more favorably than with full attention. DeltaNet, recurrence
DeltaNet Family of linear attention methods that carries relevant information forward in a fixed state. These layers need no classic, linearly growing KV cache. GatedDeltaNet, hybrid model
GatedDeltaNet DeltaNet with controlling gates that decide which information is kept, updated or forgotten. Recurrent state
RoPE Rotary Position Embedding. Encodes a token’s position mathematically into query and key. Context length, YaRN
YaRN Method that scales RoPE to longer contexts. A large configured value still guarantees nothing about unchanged model quality. RoPE, long context
Native Context Length The length that model and position mechanism were designed or trained for. YaRN, override
Context Override Runtime configuration that permits a larger window than the checkpoint declares by default. Something that starts technically is not automatically reliable in quality. YaRN, max_model_len

2.4 Multimodality

Term Plain-language explanation Related to
Multimodal The model can process inputs beyond text, such as images, audio or video. Encoder, vision
Vision Encoder Translates an image into number vectors that the language model understands. Multimodal, embedding
Audio Encoder Translates audio into model vectors. Multimodal
mmproj Separate projector file in the llama.cpp/GGUF world that adapts image features for the language model. Vision encoder, GGUF
Language-model-only Serving mode that does not load multimodal encoders. It saves VRAM but makes the endpoint text-only. vLLM, --language-model-only

3. Number formats and quantization

3.1 The basic idea

Quantization is like saving a large photo as a compressed image: it needs less
space and often loads faster, but details can be lost. In a model it is not
photos that are compressed but billions of numbers.

Term Plain-language explanation Related to
Precision How exactly a number is represented. More bits usually allow finer values but need more memory. FP16, INT4
Bit Smallest unit of digital information, 0 or 1. Eight bits make one byte. BPW, quantization
Quantization Weights or cache values are stored with fewer bits. The goals are lower memory use and often more speed. Dequantization, calibration
Dequantization Quantized numbers are translated back into a usable compute format during the computation. Marlin, W4A16
Weight Quantization Quantization of the model weights. It mainly determines model size, engine compatibility and weight throughput. AWQ, GPTQ, GGUF
KV Cache Quantization Separate quantization of the attention memory. It increases the possible context but can carry different quality risks than weight quantization. FP8 KV, q8_0 KV
Post-Training Quantization (PTQ) An already trained model is quantized afterwards, without retraining it completely. GPTQ, AWQ
Calibration Sample texts are sent through the model to find out which values are particularly sensitive. AWQ, GPTQ, outlier
Importance Matrix / imatrix For GGUF quants, measures which weights matter for typical data and should keep more accuracy. GGUF, IQ quant
BPW Bits Per Weight. Average number of bits per weight. Metadata and mixed precision mean that a “4-bit model” does not have to be exactly 4.00 BPW. Quantization
Mixed Precision Sensitive parts stay at high precision, robust parts are quantized more aggressively. PrismaQuant, smart quant
Outlier An unusually large numerical value. With too coarse a quantization, a few outliers can do disproportionate damage. Scale, mixed precision
Quantization Group Weights are scaled together in groups, of 32 or 128 values for example. g32, g128, scale
Group Size g32 / g128 Number of weights per shared quantization scale. Smaller groups are more flexible but often cost more memory or kernel time. GPTQ, AWQ, Marlin
Scale Conversion factor between the compressed number and its real magnitude. Wrong scales can wreck otherwise correct computations. FP8, INT4
Zero Point Reference value of an integer quantization. It defines which quantized value corresponds to a true zero. Asymmetric quantization
Smart Quant Not a fixed file format but the idea of protecting router, attention, norms and outlier layers more strongly than robust bulk weights. Mixed precision, MoE

3.2 Number formats

Format Short explanation Typical context
FP32 32-bit floating point. Very accurate, but usually unnecessarily large for local inference of big models. Reference, training
FP16 16-bit floating point. Half the size of FP32 and a classic model format. Safetensors, activations
BF16 Brain Float 16. Also 16 bits, but with a wide value range and less fractional precision than FP16. Often the master or compute format of modern models. Training, KV cache
B16 Usually a typo or an imprecise short form of BF16. The standardized format is called BF16. BF16
FP8 8-bit floating point. Saves about half the weight or cache memory compared with BF16. Not every GPU can compute FP8 natively. E4M3, E5M2
E4M3 FP8 variant with 4 exponent and 3 mantissa bits. More accuracy, smaller value range. FP8 KV, outlier
E5M2 FP8 variant with a larger value range but less accuracy. FP8
INT8 8-bit integer format. Often robust and about half the size of BF16. Weights, KV cache
INT4 4-bit integer format. Very compact and fast when model, calibration and kernel fit together. GPTQ, AWQ, W4A16
W4A16 Weights at 4 bits, activations at 16 bits during the computation. Marlin, INT4
FP4 Family of 4-bit floating-point formats. Extremely compact, but heavily dependent on the exact format and the hardware. NVFP4, MXFP4
NVFP4 NVIDIA-oriented block-scaled FP4 format. On hardware without native FP4, the work often goes through specialized dequantization kernels. vLLM, Marlin
MXFP4 Microscaling FP4. Small blocks of values share scales. Some models were designed directly for this format. GPT-oss, vLLM
Q8_0 Simple 8-bit GGUF quant. Large, but usually very close to high precision. GGUF, llama.cpp
Q4_K_S / Q4_K_M GGUF 4-bit quants. S is more compact, M typically keeps more parts at higher precision. llama.cpp, GGUF
Q4_K_XL Larger, quality-oriented dynamic 4-bit GGUF profile. The exact layout is quant-specific. Unsloth, GGUF
IQ4_XS / IQ3_KS Importance-aware GGUF quants. IQ denotes a family of sophisticated low-bit methods, not automatically a particular quality. imatrix, ik_llama
Ternary / 1.x-bit Weights use only very few states, often roughly -1, 0 and +1. Extremely compact, but strongly dependent on model and runtime. UD-TQ1_0

3.3 Quantization methods and file formats

Term Plain-language explanation Typical engine
GPTQ Post-training method that quantizes weights layer by layer and partly compensates for the resulting error. vLLM, SGLang, ExLlama
AWQ Activation-Aware Weight Quantization. Protects the weights that appear particularly important given real activations. vLLM, SGLang
AutoRound Quantization method that optimizes rounding decisions instead of always rounding to the nearest fixed value. vLLM-compatible quants
PrismaQuant Mixed-precision approach: different parts of the model get different formats such as NVFP4, MXFP8 or BF16. vLLM
GGUF File format of the llama.cpp ecosystem. It can hold the model, metadata and many flexible quantization types. llama.cpp, ik_llama
Safetensors Safe, fast tensor file format in the Hugging Face ecosystem. It is a container, not a particular precision. vLLM, SGLang, Transformers
Compressed Tensors Schema close to Hugging Face and vLLM that describes quantized tensors and their quantization rules. vLLM, Marlin
Shard One of several files a large checkpoint is split into. All shards together make up the model. Safetensors, GGUF
Marlin Fast GPU kernel for quantized weights. It combines reading and dequantizing so that INT4 and FP4 models run efficiently. GPTQ, AWQ, Ampere+
Weight-only Quantization Only the weights are compressed; activations are computed at higher precision. W4A16, Marlin

3.4 What the format names do not tell you

Misconception The accurate picture
FP8 in the model name and FP8-KV are the same thing The first usually describes the weights, the second the KV cache. Both can be configured independently.
Every 4-bit model is equally good Calibration, protected layers, group width and model architecture often matter more than the bare bit count.
BF16 is always faster than INT4 BF16 avoids dequantization but reads far more data. Which format is faster depends on kernel, GPU and workload.
FP8 runs at FP8 speed on any GPU Ampere GPUs such as the RTX 3090 have no native FP8 compute. The runtime can still store FP8 and process it through fallback kernels.
GGUF is a quantization level GGUF is the file format. Q4_K_M, Q8_0 or IQ4_XS are quantization types within it.

4. Memory and hardware

Term Plain-language explanation Related to
GPU Processor with very many parallel compute units. Good for the large matrix operations of an LLM. CUDA, VRAM
VRAM Fast memory directly on the GPU. Weights, KV cache, working buffers and often CUDA graphs all have to fit into it. OOM, KV cache
RAM / System RAM Main memory of the machine. Much larger and cheaper than VRAM, but considerably slower to reach over PCIe for GPU inference. Offloading
GB vs. GiB GB is usually decimal, GiB binary. A vendor’s 24 GB is not exactly 24 GiB of usable memory. Capacity planning
Memory Bandwidth Amount of data that can be read from a memory per second. At batch 1 it often determines decode speed more than compute power does. Memory-bound, tok/s
GDDR / HBM Types of GPU memory. HBM usually offers very high bandwidth and capacity but is common on expensive datacenter GPUs. VRAM, H100/H200
Compute Capability / SM NVIDIA generation identifier such as SM70, SM86 or SM90. It decides which kernels and data types can be used. CUDA, FP8, Marlin
CUDA NVIDIA’s platform for GPU programs. Engines use CUDA kernels for model computation. Kernel, driver
CUDA Kernel Small, specialized function that runs directly on the GPU. Good kernels are decisive for speed. Triton, Marlin
Tensor Core GPU compute unit for fast matrix operations in formats such as FP16, BF16, INT8 or, on newer hardware, FP8. Compute capability
AVX2 / AVX-512 CPU instruction sets that process many numbers at once. AVX-512 can speed up CPU offloading considerably when software and processor support it. CPU offloading
AMX Intel matrix units for fast INT8 and BF16 computation on newer server CPUs. Particularly relevant for MoE experts on the CPU side. Expert offloading
Memory-bound The computation mainly waits for data from memory. Typical for single-stream decode. Bandwidth, quantization
Compute-bound The compute units are the bottleneck. This happens more with prefill, large batches or compute-heavy kernels. TFLOPS, batching
PCIe Standard connection between CPU, RAM and GPUs. In multi-GPU setups on consumer hardware it often stands in for a fast dedicated interconnect. P2P, AllReduce
PCIe x4/x8/x16 Number of PCIe lanes in use. More lanes mean more possible transfer bandwidth. Multi-GPU, bifurcation
PCIe Bifurcation One large PCIe link is split into several smaller ones, x16 into two times x8 for example. Multi-GPU rig
P2P / Peer-to-Peer GPUs exchange data directly, without the detour through CPU RAM. Whether this really works depends on driver and topology. PCIe, NVLink
NVLink Fast direct NVIDIA link between GPUs. Consumer 3090 NVLink connects only limited pairs and is no replacement for a datacenter NVSwitch. AllReduce, TP
NVSwitch Switch that connects many datacenter GPUs with each other at very high bandwidth. NVLink, H100/H200
PCIe Root Complex The CPU and chipset domain that PCIe devices attach to. Transfers across different root complexes can be problematic or slower. P2P, NCCL
NCCL NVIDIA library for communication between GPUs, such as AllReduce, broadcast and send/receive. TP, PP
AllReduce All GPUs combine their partial results and then hold the overall result. With TP this happens very often. NCCL, tensor parallelism
OOM Out Of Memory. A memory allocation no longer fits into VRAM or RAM. KV cache, context window
Thermal Throttling The GPU lowers its clock because of excessive temperature. A model stays reachable but can suddenly become much slower. Sustained load, benchmark
Power Cap A set power limit for a GPU. Can reduce heat and load spikes without slowing memory-bound decode much. Thermals

5. Inference engines and serving

5.1 Engines

Term Plain-language explanation Typical strength
Inference Engine / Runtime Software that brings checkpoint, GPU kernels, memory and requests together. The same model can be very differently fast or stable depending on the engine. Serving, kernels
vLLM GPU-oriented open-source engine for fast API operation. Its strengths are PagedAttention, continuous batching, multi-GPU and OpenAI-compatible serving. Safetensors, TP, PP
SGLang Inference and serving stack with strong scheduling, prefix/radix caching, multi-GPU and modern speculative decoding support. vLLM, radix cache
llama.cpp Lean C/C++ engine, especially for GGUF, mixed CPU/GPU operation and consumer hardware. GGUF, layer split
ik_llama llama.cpp fork with additional MoE, offloading and multi-GPU experiments. Expert offloading, graph split
TensorSharp Specialized in-house serving and execution stack, used among other things for DeepSeek-V4-Flash and DSpark. GGUF, DSpark
Ollama User-friendly local model management built on llama.cpp. Good for quick first tests, not always for maximum server performance. GGUF, local API
Hugging Face Transformers Reference-oriented Python library for a very wide range of model architectures. Flexible and close to correct behavior, but not always the fastest server engine. Safetensors, tokenizer
ExLlama / TabbyAPI GPU-oriented stack for certain low-bit formats and OpenAI-like serving. EXL2/EXL3, quantization

5.2 API and model packaging

Term Plain-language explanation Related to
Serving Providing a model as a permanent service so that clients can send requests. API, endpoint
OpenAI-compatible API Endpoints and JSON structure resemble the OpenAI API, which lets many clients use local models without a special integration. /v1/chat/completions
Endpoint Network address of a model server, for example http://host:8000/v1. Port, API
Port Number of a network service on a host. Two services cannot occupy the same host port at the same time. Endpoint
Served Model Name The model name the API reports outwards and expects in a request. It does not have to match the file name. /v1/models
Provider Client configuration that bundles endpoint, model ID and capabilities. OpenCode, API
Server Limit What the runtime can actually load and process. max_model_len
Client Limit What the client sends at most. A 262k server does not help if the client already truncates at 16k. Context, compaction
Docker Image Ready-made software package with engine, libraries and CUDA dependencies. Container, version
Container Running instance of an image with model mounts, ports and start arguments. Docker, healthcheck
Healthcheck Checks whether a service is reachable. It does not prove that the model output is correct in content. Smoke test, garbling
Smoke Test Short functional test with a known result. Besides reachability, it should also check the content. Healthcheck, verification

5.3 Templates and parsers

Term Plain-language explanation Related to
Chat Template Formats roles, messages, tools and thinking into the token sequence the model knows from training. A wrong template can make a good model look useless. Jinja, tokenizer
Jinja Template Commonly used templating language for chat templates. llama.cpp, Hugging Face
Reasoning Parser Separates thinking text from the visible answer text. <think>, serving
Tool Call Parser Recognizes the model’s own tool syntax and converts it into structured API tool_calls. Tool calling
Auto Tool Choice The model may decide for itself whether and which tool to call. Tool parser
Structure Token / Special Token Token with protocol meaning, for example the start or end of a role. Treating the same text as ordinary payload can cause problems. Chat template, tokenizer
Garbling / Gibberish Character salad, foreign fragments or null bytes instead of meaningful output. The cause can be the model, the quant, a kernel, a parser, a KV scale or the runtime. Smoke test, corruption

6. Key vLLM terms

Term Plain-language explanation Why it matters
PagedAttention vLLM manages the KV cache in small blocks instead of one large contiguous memory region, similar to the memory pages of an operating system. Less waste, flexible requests
KV Block / Page Smallest managed unit of the paged KV cache. PagedAttention
Continuous Batching The server continuously mixes tokens from different requests into shared GPU steps. Finished requests leave, new ones come in. Higher overall throughput
Prefix Caching An identical prompt prefix that has already been computed is reused. Particularly effective with stable system prompts and append-only tool loops. Less prefill
Automatic Prefix Caching (APC) vLLM’s name for token-exact reuse of matching prefix blocks. Prefix caching
Radix Cache Tree-like prefix management, best known from SGLang. Shared prompt prefixes share cache branches. Prefix caching
CUDA Graph Pre-recorded sequence of GPU operations. It reduces CPU and kernel launch overhead during decode. More VRAM, less latency
Eager Mode Operations are launched individually as usual instead of through recorded CUDA graphs. Easier to debug, often slower. --enforce-eager
PyTorch Widely used machine learning library that vLLM and many model implementations build on. torch.compile, CUDA
torch.compile PyTorch compiles model operations into more optimized execution graphs. The first start can take a long time because of the compilation. CUDA graph, Inductor
TorchInductor / Inductor Compiler backend behind torch.compile that generates optimized CPU or GPU kernels. PyTorch, Triton
Triton Language and compiler for custom GPU kernels. Not to be confused with the NVIDIA Triton Inference Server. Attention kernels, JIT
JIT Compilation A kernel is compiled the first time it is needed. The first request is therefore often slower than later ones. Cold start, Triton
FlashAttention Memory-optimized attention kernel that avoids putting intermediate values into VRAM unnecessarily. Prefill, attention
FlashInfer Library of specialized GPU kernels for LLM inference, covering attention, sampling and distributed paths among others. Attention backend, CUDA
Attention Backend The concrete kernel implementation for attention, for example Triton, FlashAttention or FlashInfer. Compatibility, speed
gpu_memory_utilization Share of GPU memory that vLLM may plan for model, KV pool and working buffers. Set too high, it leaves too little safety margin. OOM, KV pool
max_model_len Maximum sequence length the server should accept. It has to fit both memory and model architecture. Context window
max_num_seqs Maximum number of simultaneously active sequences. Higher means more parallelism, but more cache and scheduling pressure. Concurrency
max_num_batched_tokens Upper limit for the tokens one scheduler step processes together. Prefill, batching
KV Pool VRAM region that vLLM reserves for the KV blocks of all active requests. PagedAttention, eviction
Scheduler Decides which requests and tokens are handled in the next GPU step. Continuous batching
Custom All-Reduce vLLM’s own optimized GPU exchange path. On an unsuitable P2P topology the regular NCCL path can be more stable. TP, P2P
Trust Remote Code Allows model-side Python code from a repository. Can enable new architectures but increases the security risk. Hugging Face, vLLM

7. Parallelism and offloading

7.1 Several GPUs

Term Plain-language explanation Typical drawback
Single GPU The whole model runs on one GPU. Simple and light on communication, but limited by that card’s VRAM. Little total memory
Tensor Parallelism (TP) Every layer is split across several GPUs; all of them work on the same token at once. Very frequent GPU communication
Pipeline Parallelism (PP) Different groups of layers sit on different GPU stages. Data travels through the pipeline one stage after another. Pipeline bubbles and extra latency
TP+PP Combination: within each pipeline stage, several GPUs work tensor-parallel. Complexity, memory balance
Expert Parallelism (EP) MoE experts are distributed across GPUs. The router sends tokens to whichever experts are needed. All-to-all communication
Data Parallelism (DP) Several model replicas handle different requests in parallel. Unlike TP, DP does not split a single layer between the replicas. Additional weight and cache memory
Layer Split llama.cpp distributes whole layers across GPUs. Flexible for mixed cards, but a single token passes through the cards largely one after another. Low aggregate bandwidth utilization
Tensor Split Weights and computation inside a layer are divided across several GPUs. Communication after many layers
Graph Split Experimental ik_llama path that distributes parts of a compute graph within layers across several GPUs. Not to be equated with ordinary vLLM TP or PP. ik_llama, tensor split
Shard / Sharding Splitting weights, cache or data across several devices. Distributed communication
Rank A running process, that is, a participant in a distributed GPU group. With TP4 there are typically four ranks. NCCL
Pipeline Stage One stage holding a group of layers. PP, bubble
Pipeline Bubble Time in which a pipeline stage waits because no work has arrived yet or another stage is slower. PP throughput
Homogeneous GPUs Identical or very similar GPUs. Usually the easiest case for TP and shared kernels. TP
Mixed GPU Different GPU models in one setup. Layer split often handles this better than real TP. Slowest GPU, kernel compatibility
Bottleneck The slowest component, which limits overall performance. A single slow GPU can hold back an entire TP group. PCIe, pipeline

7.2 Offloading memory

Term Plain-language explanation Related to
Offloading Parts of the model or cache are moved out of VRAM into RAM or onto other devices. That creates space but costs transfers. CPU offload, PCIe
CPU Offloading Weights or cache partly live in system RAM and are processed by CPU or GPU when needed. RAM bandwidth
Expert Offloading With MoE, some experts stay in RAM while attention and other parts sit on the GPU. MoE, expert cache
Static Offloading Which layers or experts live on CPU and GPU is decided before the start. llama.cpp, ik_llama
Dynamic Offloading Parts that are currently needed get loaded or cached at runtime. ktransformers, expert cache
Expert Cache Frequently needed “hot” experts stay in VRAM; rare “cold” experts live in RAM. LRU, MoE
Hot / Cold Expert An expert the router selects often or rarely. Profiling, expert cache
LRU Cache Least Recently Used. When space runs short, the entry unused for the longest time is removed. Expert cache, KV cache

8. Request flow, speed and metrics

8.1 The two main phases

  1. Send prompt
  2. Prefill: read the whole new prompt and build the cache
  3. First answer token
  4. Decode: continue the answer token by token
Term Plain-language explanation Typical metric
Prefill / Prompt Evaluation The input text is processed in parallel and the internal cache is built. Long cold prompts can cost a lot of time here. Prompt tok/s, TTFT
Decode / Generation The model produces the answer one token after another. Decode tok/s, TPOT
TTFT Time To First Token. The wait from the request to the first answer token. Perceived responsiveness
TPOT Time Per Output Token. Average time per further token after the first. Decode speed
ITL Inter-Token Latency. Time between emitted tokens. Similar to TPOT, often looked at in finer detail. Streaming
tok/s Tokens per second. Without stating phase, context, batch and measurement method, the number says little. TG, throughput
TG In this documentation, the pure token generation or decode rate. tok/s, TPOT
Prompt Throughput How many input tokens per second are processed during prefill. Prefill
Output Throughput How many output tokens the server delivers per second in total. With several users it can rise while every individual user gets slower. Batching
Latency How long a single user or request waits. TTFT, TPOT
Throughput How much work the whole system completes per unit of time. Total tok/s, QPS
QPS Queries Per Second. Number of requests per second. Serving load
Concurrency Number of simultaneously active requests or sequences. max_num_seqs, batching
Batch Several inputs or tokens are computed together. This uses the GPU better but can increase latency per user. Continuous batching
Batch Size Number of sequences or tokens processed together, depending on the context of the parameter. Throughput, VRAM
Microbatch / ubatch Small part of a larger batch that is processed in one computation step. Prefill, OOM
Sequence / Slot An active generation with its own state or cache. llama.cpp frequently calls these slots. Parallelism, KV cache
Streaming The answer is delivered piece by piece as it is produced. A stream chunk is not necessarily exactly one token. TTFT, SSE
SSE Server-Sent Events. Common HTTP format for streamed responses. Streaming
Cold Start First start or first request without warm kernels, graphs and caches. JIT, prefill
Warm Run A repeat run after initialization or cache build-up. Usually faster than a cold run. Benchmark
Cache Hit Required data was found in the cache and does not have to be recomputed. Prefix caching
Cache Miss Matching data is missing; the runtime has to compute it again. Prefill
Context Compaction The client summarizes or removes parts of the old history so that the session fits into the context window. Details can be lost in the process. Client limit
finish_reason API field stating why the output ended, for example stop, length or a tool call. Output budget, truncation

8.2 Benchmarks and quality metrics

Term Plain-language explanation Roughly measures
Perplexity / PPL How surprised a model is by real text. Lower is usually better, but small differences say little about agent quality. Language modeling loss
HumanEval Small programming tasks with tests. Code generation
MBPP Python programming tasks for basic problem solving. Code generation
SWE-bench Real software bugs from GitHub repositories have to be fixed. The result also depends heavily on the agent harness. Repo-level coding
BFCL Berkeley Function Calling Leaderboard. Tool/function calling
MMLU Knowledge and multiple-choice questions from many subject areas. General knowledge
RULER Test family for the actual use of long contexts. Retrieval and long context
Needle Test A hidden piece of information has to be found again in a long filler text. A simple single-needle test is easier than real repository understanding. Long context
Pass@1 Share of tasks passed on the first attempt. Benchmark success
E2E Time Total duration including prefill, decode, tools, network and waiting time. Real user experience

9. Sampling: how the next token is chosen

Term Plain-language explanation Effect
Softmax Turns logits into probabilities. Basis of sampling
Greedy Decoding Always take the most likely token. Reproducible, but not automatically the best. temperature=0
Temperature Controls randomness and how sharp the distribution is. Low is more conservative, high more varied and riskier. Sampling
Top-p Only consider the most likely tokens whose combined probability reaches the value p. Nucleus sampling
Top-k Only the k most likely tokens may be selected. Not to be confused with top-k experts. Sampling
Min-p Tokens below a relative minimum probability are excluded. Sampling
Repetition Penalty Makes recently used tokens less attractive in order to reduce repetition loops. Loops
Presence Penalty Penalizes tokens that have already appeared, regardless of how often. Topic variation
Frequency Penalty Penalizes tokens more strongly the more often they have already appeared. Repetition
Seed Starting value of the random number generator. The same seed helps reproducibility but does not always guarantee it across other kernels or parallel paths. Benchmark
Deterministic The same input produces the same output under the same conditions. Distributed GPU and sampling paths can introduce small deviations. Seed, greedy

10. Speculative decoding

10.1 The umbrella term

Speculative decoding accelerates token generation with a cheap
“first-draft writer”. It proposes several tokens. The full target model checks
the block in one shared step and adopts only the valid proposals.

  • Speculative decoding
    • No learned model
      • n-gram / suffix decoding
    • Model-internal drafter
      • MTP
    • Learned helper drafter
      • EAGLE / DFlash / DSpark
  1. Proposal
  2. Verification by the target model
  3. Accept or replace correctly

The basic idea is comparable to a fast assistant and a thorough editor. The
assistant may work ahead, but the editor has the last word.

10.2 Shared terms

Term Plain-language explanation Related to
Speculative Decoding / Spec Decode Umbrella term for proposing several tokens and then checking them. Drafter, target
Target Model The actual large model. Its distribution ultimately determines the output. Verify, drafter
Drafter / Draft Model Fast auxiliary path that proposes candidate tokens. It can be a small model, an extra head or a search procedure. MTP, EAGLE, DFlash
Draft Token Token proposed by the drafter that is not final yet. Acceptance
Draft Block Several tokens proposed at once. Block size, verify
Verification / Verify The target model evaluates the proposed tokens. Rejected proposals are not emitted blindly. Target model
Verify Step One target model step that checks an entire candidate block. The more tokens are accepted in it, the larger the possible gain. Acceptance length
Acceptance Rate Share of proposed tokens that the target model accepts. Drafter quality
Acceptance Length Average number of tokens adopted per verify step. For speed this is often more informative than the percentage alone. Tokens per step
Draft Depth / n_max Maximum number of proposed tokens. Going too deep can be slower despite more candidates, because drafting and verification become more expensive. Tuning
Block Size Number of positions a block drafter is trained or designed for. It is specific to the method and the checkpoint. DFlash, DSpark
p_min Confidence threshold for draft proposals. Higher can avoid bad proposals, but can also shorten the possible block. Acceptance, tuning
Rejection Sampling Mathematical procedure that keeps the target distribution correct for probabilistic drafts, even though the proposals come from another model. Lossless speculation
Lossless / Exact Speculation Means that the target distribution should remain unchanged by the method. It does not mean that every piece of hardware delivers bit-identical text, or that spec decode is always faster. Rejection sampling
Speculation Overhead Additional time and memory for drafter, candidates, cache and verification. With low acceptance the optimization can be slower than plain decoding. Break-even
Draft KV Cache The drafter’s own cache. It comes on top of the target model’s cache. VRAM, context window
Vocabulary Compatibility Drafter and target must understand candidate tokens unambiguously in common, or have an explicit mapping. Tokenizer, EAGLE
Co-located Drafter and target run on the same machine, or close enough for very fast verify steps. A LAN round trip per token block usually destroys the gain. Latency

10.3 The methods compared

Method What is the drafter? Simple explanation Requirement
Generic draft model Smaller separate language model The small model writes ahead, the large one checks. Matching tokenizer and a very fast draft path
n-gram / prompt lookup Search in the existing context When a known token sequence reappears, its earlier continuation is proposed. Repetition or copy-heavy output
Suffix decoding Search for matching suffixes Uses known suffixes and their continuations as candidates. Suitable history
MTP The model’s own extra heads or assistant layers The target model was trained to predict more than just the immediately next token. Native MTP support in the checkpoint and the engine
EAGLE Small trained feature-level draft model Uses internal model features and previous tokens to produce several plausible continuations cheaply. EAGLE checkpoint matching the exact model family
DFlash Trained block drafter Predicts a whole token block in parallel instead of token by token. Matching DFlash drafter and runtime support
DSpark Block-wise MTP drafter specific to DeepSeek4 Produces candidate blocks from selected target layers; the full DeepSeek target checks them. Matching DSpark sidecar and DeepSeek4 stack

10.4 MTP

Term Explanation
MTP Multi-Token Prediction. The model has native capabilities or additional weights for proposing several future tokens.
MTP Head / NextN Head Small additional head that sits on internal states of the target model and predicts further token positions.
Native MTP The MTP weights belong to the model release or to an officially matching assistant checkpoint. No arbitrary small foreign model is used.
MTP Sidecar Separate file with MTP or assistant weights. It still belongs to exactly one matching model family and is not freely interchangeable.
MTP Training The model learns not only token t+1 but further future tokens as well. During serving, this capability can be used for spec decode.

MTP is not automatically a speedup. On a single-GPU or otherwise
communication-light path it can help a great deal. With TP over slow PCIe, the
additional draft and verify exchange can eat up the gain.

10.5 EAGLE, EAGLE-2 and EAGLE-3

Term Explanation
EAGLE Extrapolation Algorithm for Greater Language-model Efficiency. A small trained drafter works on internal features of the target model and produces candidates for speculative decoding.
Feature-Level Drafting The drafter uses hidden states instead of only finished text tokens. That gives it more information than a completely independent small language model.
EAGLE-2 Further development with a more dynamic organization of the candidates. The runtime can check promising draft paths more selectively.
EAGLE-3 Newer EAGLE generation that can use information from several target model layers. It requires a checkpoint trained specifically for it and a matching engine path.
EAGLE Tree Instead of only one linear continuation, several possible candidate paths are proposed and checked together.

An EAGLE checkpoint for Qwen3 is not automatically compatible with Qwen3.6,
Llama or any other vocabulary or architecture version.

10.6 DFlash and DFlash2

Term Explanation
DFlash Learned block drafter for speculative decoding. It uses internal states from several target model layers and predicts several positions in one parallel, non-autoregressive pass.
Non-autoregressive draft The draft positions are computed together instead of each one having to wait for the previous one. This reduces draft latency.
DFlash Block Drafter Small additional network whose layer and block count depend on the model. In-house examples use five drafter layers among others; the block size is not the same for every checkpoint.
DFlash2 Further developed DFlash path. In the in-house Qwen3.8 work, a five-layer drafter predicts seven tokens in one pass and selects one contiguous path from several candidates.
Selector Part of the DFlash drafter that builds one consistent token path from several candidates per position.
Mask Token Placeholder for positions that a parallel block drafter is meant to predict simultaneously.

DFlash is the name of a method, not a general synonym for “fast decoding”. A
DFlash drafter has to match the model family, the expected hidden states, the
vocabulary and the runtime.

10.7 DSpark

In the in-house inventory, DSpark is the specialized block-wise drafter for
DeepSeek-V4-Flash. It belongs to the MTP and speculative decoding path of
the DeepSeek4 stack.

Property How it is used in-house
Role Propose candidate blocks
Authority The full Q8 target model verifies; DSpark never decides the final output on its own
In-house sidecar DeepSeek-V4-Flash-DSpark-support.gguf
Structure Three stages, block size 5, target layers 40 to 42
Engine path TensorSharp-DeepSeek4 with MTP spec support

DSpark must not be confused with another DeepSeek MTP sidecar. Files with a
similar purpose can have different architectures, target layers, block sizes and
runtime contracts.

10.8 n-gram and prompt lookup

Term Explanation
n-gram Sequence of n tokens. An 8-gram consists of eight consecutive tokens.
Prompt Lookup Searches the existing prompt for the token sequence just generated and proposes whatever followed it there.
Copy-heavy workload The output takes over a lot of known text, for example rewriting code, filling tables or quoting parts of documents. n-gram can be very strong here.
Novel Output Genuinely new text with no matching template in the context. n-gram is usually less useful here.

n-gram needs no learned auxiliary model and almost no additional weight memory.
It complements MTP or DFlash because it is strong on exact copy stretches, while
learned drafters can also propose new continuations.

10.9 Why spec decode is sometimes slower

Cause Simple explanation
Low acceptance The drafter does its work, but the target rejects many proposals.
Drafter too large The helper model costs almost as much time as the target model work it saves.
Draft too deep More proposals raise draft, KV and verify cost more than they raise the number of accepted tokens.
Short context The normal target model step is already cheap; the extra effort does not pay off.
Slow multi-GPU communication Draft and verify trigger additional synchronization over PCIe.
High sampling randomness Candidates are harder to predict, so acceptance can fall.
High QPS Continuous batching already keeps the target model busy; an additional drafter can cost overall throughput.
More cache demand Draft KV and working buffers shrink the possible context or the concurrency.

11. Long-context and cache terms

Term Plain-language explanation Related to
Long Context Very large context window, typically 100k tokens or more. KV cache, RoPE
Active Context Tokens that are really occupied in the current request and relevant during decode. A server with 1M max is not automatically slow on a 2k prompt. Decode speed
Cold Context The prompt is not in the cache yet and has to be prefilled completely. TTFT
Warm Context A large part of the prefix or state is already present. Prefix caching
Retained Cache A finished session state is kept for a later continuation. Multi-session, RAM/VRAM
Prompt Cache General term for stored prompt states. The exact semantics differ between engines. Prefix cache, KV cache
Context Checkpoint Saved intermediate state that a runtime can jump back to later. llama.cpp, RAM
Needle in a Haystack A small piece of information is embedded in a great deal of irrelevant text and queried later. Long-context test
Retrieval Finding relevant information again in the context or a document index. RAG, needle
Context Degradation The model can technically process a large window but makes poorer use of distant information, or drifts during long agent runs. YaRN, benchmarks

12. Quality and failure terms

Term Plain-language explanation Related to
Quantization Degradation Loss of quality from numbers that are too coarse or badly calibrated. Outlier, router
Router Degradation The MoE router picks less suitable experts because of quantization error. MoE, smart quant
Loop / Doom Loop The model repeats thinking, tools or text without ever coming to a sensible end. Sampling, template, long context
Instruction Loss Early rules or requirements are no longer reliably observed over a long session. Context, compaction
Reasoning Drift The thinking process gradually moves away from the task or from facts already settled. Long context
Token Corruption Wrong numbers in a kernel, cache or communication path lead to broken token decisions. KV scale, P2P, garbling
Silent Corruption The system runs and delivers text, but the content is wrong without anyone noticing. More dangerous than a clear crash. Smoke test, verification
Model Defect The fault lies in the model’s own behavior or vocabulary and reproduces across different serving stacks. Structure tokens
Runtime Bug The model weights are fine, but engine, kernel, parser or scheduler processes them incorrectly. Version, backend
Regression A newer version behaves worse than a previously validated state. A/B test, version pinning
A/B Test Only one factor is changed between two runs. This makes it easier to attribute a cause. Benchmark, control
Control Run Comparison run without the optimization under investigation. For MTP that would be the same build with speculation switched off. A/B test
Reproducibility Other runs can arrive at the same result under documented conditions. Seed, version, hardware

13. Software, build and operations terms

Term Plain-language explanation Related to
Upstream The official main project that local forks or patches derive from. vLLM, llama.cpp
Fork Your own branch of a project with additional or altered functions. ik_llama, patch
Patch A change to existing source code. Fork, backport
Backport A new function is transferred to an older, otherwise stable version. DFlash2, runtime version
Commit A specific saved state of the source code. For reproducible builds this is more precise than a version name alone. Git, build
Build An executable program or container image produced from source code. Compiler, CUDA
Nightly Automatically produced, very recent development state. Contains current features, but also more regression risk. vLLM, SGLang
Version Pinning An exactly validated version is held fixed instead of always taking the newest one automatically. Reproducibility
Driver / GPU Driver Operating system software between the CUDA application and the GPU. Version and open/proprietary variant can change low-level behavior. CUDA, P2P
JIT Cache Stored, already compiled kernels. After deleting them or after an update, the first start is slow again. Triton, torch.compile
Restart Policy Docker rule for whether a container starts again automatically after the process or the host restarts. Healthcheck
Watchdog External monitor that detects an unhealthy service and restarts it or raises an alert. Healthcheck
Runbook Step-by-step operating instructions with concrete commands and recovery procedures. Model documentation

14. The most important mix-ups at a glance

Do not confuse The difference
Model vs. engine Qwen, DeepSeek or Gemma are models; vLLM, SGLang and llama.cpp run them.
Checkpoint vs. architecture A checkpoint is a concrete weight file; the architecture describes how the network is built.
File format vs. quantization Safetensors and GGUF are containers; BF16, GPTQ-INT4 or Q4_K_M describe the numbers, that is, the quantization.
Weight FP8 vs. FP8 KV One compresses the model, the other the context memory.
Total parameters vs. active parameters An MoE has to store all weights but computes with only a part of them per token.
Attention heads vs. KV heads Query heads ask parallel questions; KV heads deliver the shared keys and values.
Context limit vs. occupied context max_model_len=1M is the upper bound; the current request can still contain only 2k tokens.
Technically startable vs. usable quality A model can allocate 1M tokens without understanding just as well at 1M.
TTFT vs. decode tok/s TTFT is the wait for the first token; decode tok/s is the writing speed after that.
Latency vs. throughput A single user wants low latency; a server operator often wants high overall throughput. The two can work against each other.
TP vs. layer split TP splits every layer and computes in parallel; layer split places whole layers on different GPUs.
MTP vs. generic draft model MTP is native to the model or close to it; a generic draft model is a separate, smaller LLM.
EAGLE vs. DFlash Both are learned drafters, but with different architecture and checkpoint semantics. They are not interchangeable.
DFlash vs. DSpark DFlash is a family of block drafters; DSpark is the DeepSeek4-specific block-wise drafter in the MTP path here.
Lossless spec decode vs. always identical text The target distribution is preserved mathematically; numerical details, seeds or parallel kernels can still decide close calls differently.
Healthcheck vs. correct output HTTP 200 proves reachability, not intelligence or uncorrupted tokens.
More GPUs vs. more speed More GPUs give more VRAM. Communication, few active parameters or pipeline bubbles can keep the speed gain small.

15. Quick decision aid

Question Look at these terms first
Does the model fit into memory? Weight format, VRAM, KV cache, context window, CUDA graphs
Why is an MoE fast despite having many parameters? Active parameters, router, memory bandwidth
Why does TP6 not work? Attention heads, KV heads, head divisibility
Why does a long context become slow? Prefill, active KV cache, full attention, TTFT, TPOT
Why does vLLM help with several users? Continuous batching, PagedAttention, scheduler
Why does prefix caching help so much? Stable prompt prefix, cache hit, less prefill
Why is the 4-bit model bad? Calibration, outliers, router, attention, mixed precision
Why is MTP not faster? Acceptance length, speculation overhead, TP communication
Which spec method belongs where? MTP inside the model; EAGLE/DFlash learned drafters; DSpark DeepSeek4-specific; n-gram context-based
Why does a healthy server answer with garbage? Chat template, parser, KV scales, runtime bug, quant, P2P

Sources and further reading

This glossary mainly summarizes knowledge gathered in-house. Public background
material: