vLLM ///
High-Throughput LLM Inference Engine
PagedAttention · Continuous Batching · Tensor Parallelism · KV Cache Management
Live Engine Simulator

Runs the educational Python engine (PagedAttention blocks + continuous batching). Connect API for step-by-step traces.

Click Run Sim — uses API at ...
● API Entrypoint
API Layer
OpenAI-compatible
HTTP
🌐
FastAPI Server
POST /v1/completions
POST /v1/chat/completions
OpenAI-compatible REST API. Accepts prompt, max_tokens, temperature, stream. Routes to AsyncEngine.
→
🔌
Streaming SSE
token-by-token output
AsyncGenerator
Server-sent events stream tokens as they're generated. Uses asyncio.Queue internally.
→
🔑
Request Parser
SamplingParams
prompt → token IDs
Tokenizes prompt using the model's tokenizer. Validates SamplingParams (temperature, top_p, etc.) and creates a SequenceGroup.
SequenceGroup → AsyncLLMEngine
● AsyncLLMEngine — Core Orchestration
Async Engine
event loop driven
Core
⚙️
LLMEngine
step() loop · add_request()
abort_request()
Main synchronous engine. Drives the per-step execution loop. Coordinates Scheduler, BlockManager, and Worker(s). Each step() call generates one new token per sequence.
→
🔑 Key
📋
Scheduler
FCFS · preemption
waiting → running → swap
The brain of continuous batching. Decides which sequences to run each step. Uses 3 queues: waiting (not yet scheduled), running (on GPU), swapped (evicted to CPU). Implements preemption when GPU memory is full.
→
KV
🧱
BlockSpaceManager
PagedAttention allocator
logical → physical blocks
Manages KV cache memory via paging. Maps logical blocks (sequence-visible) to physical blocks (GPU memory pages). Enables copy-on-write for beam search and prefix sharing across requests.
→
🔍
Prefix Cache
hash-based lookup
prompt sharing
Caches KV blocks for repeated prompt prefixes. If two requests share a system prompt, the KV blocks are reused without recomputation. Hit rate can reach 60-80% in chat applications.
SchedulerOutput → execute_model()
● Continuous Batching Engine
Batching
continuous / paged
Core
📦
Continuous Batching
vs static batching
per-step scheduling
Unlike static batching (wait for all requests to finish), continuous batching inserts new sequences into the batch the moment a slot frees up. This eliminates GPU idle time between requests and is the #1 throughput gain in vLLM.
→
🔢
Token Budget
max_num_seqs
max_num_batched_tokens
Controls how many sequences and total tokens can be processed in a single forward pass. Tuned based on GPU memory capacity and model size.
→
🔀
Chunked Prefill
split long prompts
prefill + decode mix
Splits long prompt prefill across multiple steps, allowing decode tokens from other requests to be processed concurrently. Reduces time-to-first-token for concurrent users.
→
🔄
Speculative Decoding
draft model → verify
2-4× decode speedup
Uses a small draft model (e.g. 68M params) to generate K candidate tokens, then the large model verifies all K in one forward pass. Achieves 2-4× speedup on decode-heavy workloads.
ModelInput → Worker.execute_model()
● Worker Layer — Tensor + Pipeline Parallelism
Workers
TP · PP · ray
🗂️
Worker (GPU 0)
TP rank 0 · NCCL
KV cache replica
Each Worker process manages one GPU. In tensor parallelism, attention heads and MLP layers are split across workers. Workers communicate via NCCL all-reduce during forward pass.
⟷
🗂️
Worker (GPU 1)
TP rank 1 · NCCL
KV cache replica
Mirror worker. Holds complementary attention head shards. Communicates via high-bandwidth NVLink or InfiniBand.
→
⟳
NCCL All-Reduce
NVLink 600GB/s
InfiniBand fallback
After each attention layer, partial results from all TP workers are summed via NCCL all-reduce. NVLink bandwidth is critical — bottleneck here limits TP scaling efficiency.
→
🎯
Ray Distributed
multi-node PP
actor model
For pipeline parallelism across nodes, vLLM uses Ray actors. Each pipeline stage runs on a different node. Ray handles scheduling, fault tolerance, and inter-node communication.
logits → sampler → next token
● Model Execution — Attention + Sampling
Model Exec
forward pass
CUDA
⚡
PagedAttention CUDA
custom kernel
non-contiguous KV blocks
The core innovation. Custom CUDA kernel that reads KV cache from non-contiguous physical memory blocks (pages), eliminating memory fragmentation. Inspired by OS virtual memory paging. Reduces memory waste from ~60% to ~4%.
→
⚡
FlashAttention v2
IO-aware · fused
O(N) HBM reads
Tiled attention computation that minimizes HBM (High Bandwidth Memory) reads. Fuses Q·K·V matmuls into a single CUDA kernel. Reduces memory bandwidth bottleneck by 5-8×.
→
Quant
🗜️
Quantization
AWQ · GPTQ · FP8
INT4 / INT8
Weight quantization reduces model memory footprint. AWQ (Activation-aware Weight Quantization) is near-lossless at INT4. FP8 is natively supported on H100 with minimal quality loss and 2× throughput gain.
→
🎲
Sampler
top-p · top-k · temp
beam search · greedy
Applies sampling strategy to logits. Supports temperature scaling, top-p nucleus sampling, top-k truncation, repetition penalty, and beam search (with copy-on-write KV blocks).
token IDs → detokenize → response
● Memory Hierarchy — GPU → CPU → Disk
Memory
HBM → DRAM → NVMe
HBM
🎮
GPU HBM
model weights + KV cache
80GB H100 / 24GB 4090
GPU High Bandwidth Memory. Shared between model weights (static, ~14GB for Llama-3 8B in fp16) and KV cache (dynamic, remainder). gpu_memory_utilization=0.9 leaves 10% for activations.
⟷ swap
CPU
🖥️
CPU DRAM Swap
preemption target
PCIe 64GB/s
When GPU memory is full, the Scheduler preempts lower-priority sequences and swaps their KV blocks to CPU RAM via PCIe. Swap adds latency (~2-5ms per block) but prevents OOM errors.
→
📊
KV Cache Budget
block_size=16 tokens
num_gpu_blocks calc
At startup, vLLM runs a dummy forward pass to measure peak memory usage, then computes num_gpu_blocks = (total_memory × gpu_memory_utilization - model_weights) / kv_block_size.
→
💾
Disk Offload
NVMe via llm-compressor
extreme memory saving
For extremely large models, weights can be offloaded to NVMe SSD via llm-compressor integration. Much slower but enables running 70B+ models on consumer GPUs with NVMe speed ~7GB/s.
● Optimization Stack
Optimize
speed + efficiency
🗜️
AWQ / GPTQ
INT4 weights
near-lossless
AWQ: Activation-aware Weight Quantization. Finds salient weights by analyzing activations, preserves them in higher precision. Achieves INT4 compression with <0.5% perplexity degradation.
+
⚡
FP8 / BF16
H100 native
2× vs FP16
H100 has native FP8 tensor cores. FP8 inference gives 2× throughput vs FP16 with minimal quality loss. BF16 is the default for most deployments — better dynamic range than FP16.
+
🔧
Torch Compile
torch.compile()
graph capture
PyTorch 2.0 compile captures the forward pass as a graph and applies XLA-style optimizations: operator fusion, dead code elimination, kernel selection. Gives 10-30% additional speedup.
+
🎯
CUDA Graphs
decode phase replay
reduce CPU overhead
Captures the decode-phase forward pass as a CUDA Graph. Replaying the graph eliminates Python overhead and CPU-GPU synchronization during decode, reducing per-token latency by 15-25%.
+
📡
Lora Serving
multi-adapter · hot-swap
shared base weights
Serves multiple LoRA adapters on a single base model simultaneously. Adapters are hot-swapped between requests. Base model weights stay on GPU; only small adapter deltas are loaded per request. Enables multi-tenant fine-tuned serving.
PagedAttention — Virtual Memory for KV Cache

Traditional attention requires contiguous GPU memory per sequence (pre-allocated to max_length). This wastes 60%+ of KV cache memory. PagedAttention borrows OS virtual paging: each sequence gets logical blocks mapped to non-contiguous physical blocks (pages) of fixed size (default: 16 tokens per block). Memory is allocated on demand, one block at a time.

Seq A (chat)
B0
K+V
B1
K+V
B2
K+V
B3↗
active
4 blocks × 16 tok = 64 tokens
Seq B (code)
B7
K+V
B9
K+V
B12↗
active
Non-contiguous! B7, B9, B12
Shared prefix
P0
shared
P1
shared
System prompt reused by 3 requests (CoW)
Evicted (swap)
E3
→CPU
E4
→CPU
E8
→CPU
Swapped to CPU RAM — preempted low-priority seqs
Free blocks
F5
F6
F10
F11
F13
F14
F15
F16
Available for new sequences
KV Block Lifecycle
StateTriggerLocation
AllocatedNew token appended to seqGPU HBM
SharedCoW prefix hit (beam/prefix)GPU (ref counted)
EvictedGPU full, low-priority seq preemptedCPU DRAM
FreedSequence completes / abortedFree pool
ReloadedPreempted seq rescheduledGPU (PCIe copy)
KV Cache Size Formula
# Per layer, per token:
kv_size = 2 × num_heads × head_dim × dtype_bytes
# Llama-3 8B (FP16):
# 2 × 32 × 128 × 2 = 16,384 bytes/tok/layer
# × 32 layers = 524,288 bytes = 0.5 MB/token

# Total GPU KV budget:
avail = gpu_mem × gpu_memory_utilization
avail -= model_weights_bytes
num_blocks = avail // (block_size × kv_size)

# Example H100 80GB, Llama-3 8B FP16:
# model: ~16GB → KV budget: ~56GB
# num_blocks ≈ 7,000 blocks × 16 tok = 112K tokens
Static Batching vs Continuous Batching
❌ Static Batching (old)
Waits for ALL requests in batch to finish. GPU idles while longest request runs.
Req A (short: 10 tok)
70% idle wait ↑
Req B (medium: 50 tok)
30% idle wait ↑
Req C (long: 200 tok) ← dictates batch
All wait for this one
GPU utilization: ~35% · Throughput: low
✓ Continuous Batching (vLLM)
New requests join the batch the instant a slot frees. No GPU idle time.
A✓
D ← joined when A finished
D✓
B (medium)
E ← B slot freed
C (long — runs full duration)
GPU utilization: ~92% · Throughput: 3-4× higher
Scheduler State Machine
Every request flows through 3 queues. The Scheduler makes decisions each step() call.
WAITING queue
New requests not yet scheduled. Moved to RUNNING when GPU blocks are available. FCFS ordering (can configure priority).
↓ blocks allocated
RUNNING queue
Active sequences generating tokens. Each step() adds one token. Completed seqs are removed, freeing blocks immediately (continuous batching!).
↓↑ GPU full → preempt
SWAPPED queue
Preempted seqs with KV blocks swapped to CPU RAM. Re-promoted to RUNNING when GPU memory frees up.
Batching Modes Comparison
ModeDescriptionThroughput
StaticFixed batch, wait all1×
DynamicVariable batch size per step1.5×
ContinuousPer-step slot reuse3-4×
Chunked prefillSplit prefill + decode mix4-5×
SpeculativeDraft model + verify5-8× decode
Key Config Knobs
max_num_seqs=256         # concurrent seqs
max_num_batched_tokens=8192# tokens per step
enable_chunked_prefill=True
speculative_model="ngram"
preemption_mode="swap" # or recompute
GPU Memory Layout — H100 80GB with Llama-3 70B
Model weights (BF16)
~70 GB
→ Need 4× H100 (tensor parallel) for 70B at BF16
Per-GPU with TP=4 (17.5GB weights/GPU):
Weights (TP shard)
17.5 GB
KV Cache (90% util.)
~58 GB
Activations (reserved)
4 GB
With INT4 AWQ quantization (÷4 weights):
Weights (AWQ INT4)
4.4 GB
KV Cache (remaining)
~70 GB
→ Single H100 can serve 70B with AWQ, 10× more KV budget
GPU Utilization — Prefill vs Decode Phase
Prefill phase (processing prompt) — compute-bound:
SM Utilization94%
matrix-heavy
HBM Bandwidth3.35 TB/s
Tensor Core Util88%
Decode phase (generating tokens) — memory-bandwidth-bound:
SM Utilization28%
low
HBM Bandwidth95% (bottleneck)
bandwidth wall
→ Decode is memory-bound: batching more seqs helps amortize weight reads
# Roofline insight:
Prefill: FLOPS-bound → add GPUs (TP)
Decode: BW-bound → batch more seqs
Speculative decode breaks the BW wall
Parallelism Strategies — When to Use Each
StrategySplitsUse WhenComm overheadvLLM config
Tensor Parallel (TP)Attention heads + MLP weights across GPUsSingle node, NVLink available, large modelsAll-reduce per layertensor_parallel_size=4
Pipeline Parallel (PP)Layers across nodes (stage 0: L0-15, stage 1: L16-31)Multi-node, large model that doesn't fit even TPP2P per microbatchpipeline_parallel_size=2
Data ParallelReplicate full model, split request loadMultiple identical model replicas, high trafficNone (independent)Multiple vLLM instances + LB
Expert Parallel (MoE)MoE expert layers across GPUsMixtral, Qwen-MoE, DeepSeek-V3All-to-all routingenable_expert_parallel=True
Direct Answer
Yes — but it depends heavily on which FDE role you're targeting.
vLLM internals (PagedAttention, continuous batching, KV cache management) are table-stakes for infrastructure/platform FDE roles at companies serving LLMs at scale. For application-layer FDE roles, a conceptual understanding is enough — you don't need to know CUDA kernel code. The clearest signal: if the JD mentions "LLM serving," "inference optimization," "GPU infrastructure," or "model deployment at scale" → deep vLLM knowledge is expected.
Tier 1 — Deep Required
ML Infra / AI Platform FDE
NVIDIA, Anyscale, Modal, Together AI, Fireworks AI, Baseten, Replicate, AWS Bedrock team
Expected to explain PagedAttention, continuous batching, KV cache eviction policy from first principles
Tune gpu_memory_utilization, max_num_seqs, quantization for customer workloads live in interviews
Diagnose OOM errors, latency spikes, throughput bottlenecks from metrics
Design multi-GPU serving topologies (TP + PP + data parallel)
Understand roofline model: when you're compute-bound vs bandwidth-bound
Depth required
Tier 2 — Conceptual + Applied
AI Solutions / Principal FDE
Anthropic, OpenAI, Google DeepMind, Cohere, Mistral AI, Distyl AI (like your opportunity)
Must explain why vLLM is faster (paging vs contiguous, continuous vs static batching)
Know when to use vLLM vs TGI vs TensorRT-LLM for a given customer use case
Size GPU clusters for given throughput/latency SLAs (back-of-envelope)
Explain quantization trade-offs (AWQ vs GPTQ vs FP8) to technical customers
Don't need CUDA kernel internals — architecture-level is sufficient
Depth required
Tier 3 — Awareness Enough
Enterprise AI / App-Layer FDE
Salesforce, SAP, ServiceNow, Lucid (your current role), enterprise LLM deployments
Know vLLM exists and why it's the standard choice for self-hosted LLM serving
Able to answer "why is our inference slow?" at a systems level (batching? GPU? quantization?)
Understand cost/performance trade-offs when advising on cloud vs self-hosted
Know the difference between TTFT and TPS and what affects each
vLLM deep dive is a differentiator, not a requirement, at this tier
Depth required
For YOUR specific job search — Principal AI Architect / Head of AI track
Must Know Cold (IC + Leadership track)
✓PagedAttention = OS virtual memory for KV cache (eliminates fragmentation)
✓Continuous batching = per-step slot reuse = 3-4× throughput vs static
✓KV cache size formula (can estimate GPU budget for any model)
✓Prefill is compute-bound, decode is bandwidth-bound — practical implications
✓AWQ/FP8 quantization — trade-offs vs quality
✓TP vs PP vs DP — when to use each topology
✓TTFT, TPS, P50/P99 latency — how to measure and what affects them
Competitive Edge — Differentiators
★Speculative decoding mechanics (draft model → verify in 1 pass)
★Chunked prefill — mixing prefill + decode tokens in same step
★Multi-LoRA serving — shared base weights, hot-swap adapters
★Prefix caching hit rates and cost savings in production
★vLLM vs TensorRT-LLM vs TGI decision framework for customer scenarios
★Designing GPU cluster sizing for given TPM/latency SLAs
★A Substack post on vLLM architecture puts you in top 5% of candidates
Bottom line for you, Venkat: At the Principal AI Architect and Head of AI level you're targeting, vLLM internals are expected interview material — not because you'll be writing CUDA kernels, but because you're the person who tells a CTO "here's why your inference costs are 4× too high and here's how to fix it." The ability to reason about KV cache budgets, batching strategies, and quantization trade-offs is a core part of the credibility signal in those conversations. This is also directly publishable as AegisAI + vLLM integration content on your Substack — high-value, low-competition niche.