The evidence table includes 29 pinned configurations: 21 with checkpoint-header evidence; the remaining configurations use configuration and implementation equations.

Sparse models distinguish retained cache from selected reads; recurrent and compressor state keep their implementation precision.

Llama 4 attention restarts at chunk boundaries.

Gemma 4 learned K/V sharing is distinct from materialized serving copies.

Text work includes resident auxiliary weights but excludes image/audio execution.

Training and supervised full fine-tuning use all backbone parameters.

LoRA counts rank × (input + output) for each selected attention or feed-forward matrix, including every targeted expert, excluding embeddings and LM head.

Base-model forward and activation-gradient work remain.

QLoRA includes NF4 double-quantization metadata, unquantized norms/embeddings and BF16 dequantization workspace.

The model list supplies trainable base variants; training precision follows the selected method.

The modeled GPU-only profile uses BF16 weights/adapters, FP32 gradients, FP32 master weights and two FP32 AdamW moments.

Frozen base weights have no optimizer or gradient state.

FSDP synchronizes gradients each microbatch and accounts for transient gathered layers in addition to sharded state; Megatron keeps full FP32 main-gradient buffers while sharding optimizer/master states over the dense and expert data groups (6 + 12/d bytes per BF16 parameter for the conventional dense profile).

FlashAttention avoids stored quadratic attention scores.

Checkpointing retains layer inputs plus recomputed-layer activations.

Workspace includes logits, a 2 GB runtime allowance and communication buffers.

Default memory reserve is 10%.

The minimum is the smallest supported active GPU configuration found for this GPU, method, sequence length and effective batch.

The planner searches GPU counts automatically.

The search retains the method and effective batch, varying supported sharding, microbatch and accumulation settings.

Whole physical hosts are reserved separately.

Architecture-specific components supply weight and active-work counts; adapter counts require actual matrix shapes; unsupported target shapes block adapter sizing.

When a whole expert layer cannot be gathered, nested FSDP wraps dense blocks and individual experts separately.

No CPU or NVMe offloading is included.

DDP replicates state; FSDP shards weights, gradients and optimizer state.

The full-weight Megatron profile has world size TP × PP × CP × DP.

Expert tensor parallelism equals TP in this profile.

EP divides the data group and expert optimizer states use DP/EP; EP is not another world-size multiplier.

TP remains inside an NVLink domain.

Tensor/sequence, pipeline, expert and context communication are included without assuming perfect overlap.

Balanced contiguous pipeline layouts account for uneven dense/expert stages, the busiest-stage compute, and in-flight microbatch activations.

Nondivisible layer counts require an explicit custom Megatron pipeline layout.

Adapter methods use DDP/FSDP; unsupported combinations are rejected.

One dataset pass processes the stated data once.

Training defaults to one pass; fine-tuning to three epochs.

Packing forms full sequences; without packing, examples are padded or split at the sequence length.

The final optimizer batch of each pass is padded to preserve its effective batch and pipeline schedule.

Both actual data tokens and padded token positions are reported.

Forward/backward matrix work uses active operators, with frozen-base activation gradients retained for adapters.

Attention work depends on sequence/window length.

Sequential projection, attention, feed-forward/expert and optimizer operations each use their own compute/HBM bound.

FSDP compute reads gathered matrices; two weight gathers and a gradient reduce-scatter per microbatch are modeled separately.

Hierarchical intra-domain and cross-domain transfers serialize.

Recomputed activations and pipeline bubbles are included.

Central dense BF16 compute efficiency defaults to 35%; scenarios use 0.6× and 1.4× that value.

HBM/link efficiency is 50/65/80%; other job overhead defaults to 10%.

Deadline sizing uses the slow scenario; TCO uses the central duration with its range retained.

The integrated comparison provisions AWS capacity for the full analysis period for every pricing model.

On-premises totals include purchases, implementation and operating costs.

A run allocation is the total for that side divided by analysis hours, multiplied by the central estimated run duration.

It is an allocation of the assessment, not incremental spend or an AWS job-only quote.

Checkpoint storage includes trainable weights, master weights and optimizer state; dataset storage is reviewed separately.

Training is excluded from inference token-API comparisons.

Pinned Safetensors headers identify shapes, data types, and payload offsets.

Packed values are not logical parameter counts.

Architecture counts are checked against metadata; checkpoint storage and runtime allocations remain separate.

The selected vLLM profile replicates MLA caches across tensor-parallel ranks and replicates GQA heads when TP exceeds KV heads.

Pipeline stages own their assigned layers.

Local windows, fixed recurrent state, allocator rounding, and replicated components are counted separately.

The device reserve defaults to 10%.

Prompt working memory scales with a bounded 2,048-token prefill chunk and declared activation dimensions.

The baseline uses eager execution and no prefix caching.

Prefix caching removes projection work and attention work for query positions already in the cache.

Reusable-prefix length applies on a successful resident hit; request-hit percentage separately mixes warm and cold service time.

Only complete runtime-aligned blocks before the final prompt position are eligible.

Remaining queries attend to the cached prefix.

Full context and memory are provisioned.

Hybrid models reserve an extra recurrent-state snapshot per request at the cached boundary.

System power uses GPU TDP plus platform host overhead, with a GPU-derived or 10 kW fallback when platform details are unavailable.

Training power follows scheduled active hours with a 30% GPU idle-power allowance.

Hosts and shared infrastructure remain powered on.

Tensor parallelism splits operators, pipeline parallelism assigns layers, and independent replicas serve additional requests.

For the PP=1 expert-sharing profile, EP=TP×attention-DP.

Attention workers own their requests and caches; experts are distributed across the pool.

Independent deployments replicate the whole pool.

Balanced routing uses whole token/expert assignments on the busiest rank.

EP is not multiplied into GPU count again.

TP stays within an NVLink domain; PP stages use separate domains.

EP pools follow local all-to-all or cross-domain DeepEP transport.

Whole multi-server replicas retain their reservations in TCO.

The search compares supported TP/PP layouts and PP=1 expert-sharing layouts.

Communication follows the actual NVLink domains and physical servers.

Capacity-only sizing minimizes purchased physical servers that fit the model and all concurrent requests at the configured maximum context, including memory reserve and any explicit replica admission cap.

With throughput or service-time targets enabled, it minimizes physical servers satisfying capacity and every enabled target together.

Automatic sizing evaluates at most 4,096 active requests per instance, a planning search limit rather than a measured runtime limit.

A batching-throughput preference never adds hardware.

Performance breaks ties between equal-size deployments; it remains an estimate rather than a guarantee.

Minimum-fit, active and purchased GPUs are separate.

Continuous batching admits available requests without waiting for the replica admission limit.

The estimate charges prompt compute, memory reads, GPU communication and execution overhead for the active requests assigned to a replica.

A cached prefix skips its reusable prompt work.

Active requests sharing the GPU can increase processing time; unused request slots add no batch-fill delay.

This is a service-time estimate for the active cohort, not a simulation of request arrivals or chunk-by-chunk scheduling.

Admission queueing and network transit are additional.

Each decode step charges the active requests’ model computation, memory reads and GPU communication.

Prompt-lookup speculation charges lookup, target verification and acceptance/rollback work.

Proposal availability mixes fallback and verification cycles; finite-response expectations reduce proposal depth near the output limit.

TPOT is amortized and speculative tokens may arrive in bursts, whereas Tokens Per Second counts output across all active requests and replicas.

Sequential projection, attention, feed-forward/expert, output and replicated operations each contribute their own maximum of compute time and HBM time.

Communication and scheduling overhead follow.

Pipeline stages are sequential for an individual request.

Time to first token includes prompt processing; response time adds the remaining generation steps.

Actual requests are divided into integer cohorts across replicas, and cohort completion rates are summed.

Idle replicas contribute no workload throughput.

Output tokens/sec equals completed requests/sec times output length.

Decode-only speed remains a diagnostic in Details.

Shared weights and routed expert weights are separate.

For E experts, k selected per token, and B independent tokens, the central scenario expects E × (1 − (1 − k/E)^B) distinct experts.

The central scenario uses uniform independent routing; conservative routing reads all experts and includes imbalance.

Using the configured compute and memory efficiencies, the optimistic scenario uses 1.2× efficiency, capped at peak; conservative uses 0.7× and serial communication with less expert reuse.

These are uncalibrated engineering scenarios, not statistical confidence intervals.

Recommendations use conservative service times and central sustainable throughput with workload headroom.

Vendor sparse throughput is not a dense compute limit.

Bidirectional link totals are converted to one-way bandwidth for transfers.

NIC bandwidth and application efficiency follow the selected deployment settings.

Performance sizing requires a complete hardware profile.

The Standard profile uses vLLM 0.29.0 and explicitly pinned newer architecture implementations for text generation, eager execution, chunked prefill, and GPU-resident weights and caches.

Prefix caching and prompt-lookup speculative decoding are optional.

Speculation verifies the proposed tokens plus one new token together; expected output per pass is 1 + acceptance + acceptance squared, through the proposal count.

Cache reserves include proposed tokens and architecture-specific recurrent rollback/snapshot envelopes.

Index rings retain their own allocation.

These buffer reserves and kernel efficiencies are analytical assumptions; no matching GPU measurements were fitted.

CUDA graph pools and disaggregated serving are not modeled.

Without maximum lengths, the calculation assumes the stated lengths for every active request.

A publisher-advertised extended context is used only with a verified runtime profile; changing positional limits alone does not establish supported behavior.

HNSW includes 2M base-layer links plus an expected upper-layer allowance, vector IDs, offsets, and levels.

Replicas occupy separate failure domains.

CPU count and query performance are unavailable without an engine-specific model.

Backup storage is separate from serving replicas.