Getting Started
Use one calculation framework to size model capacity, estimate serving performance, and compare the resulting physical deployment or API consumption over time.
Recommended workflow
1. Size capacity
Load a public Hugging Face model, provide a token for private or higher-quota access, or enter a manual configuration. Set precision, sequence length, peak active requests, GPU, and topology.
2. Review performance
Inspect the selected TP/EP/PP shape, data-parallel replicas, local serving load, latency, throughput, and low/base/high directional ranges.
3. Compare cost
Push the accepted deployment to TCO, enter sourced hardware and service prices, then compare on-premises against exact cloud quotes or a Token API.
GenAI sizing framework
Resident model and sequence state
- Weights: Total checkpoint parameters plus persistent auxiliary parameters are multiplied by the selected effective bytes per parameter. MoE experts remain resident even when only a subset is active per token.
- Token compute: Active parameters drive prefill and decode work. Multimodal towers can consume HBM without being counted as text-token compute.
- Sequence state: Capacity uses average input plus output length. GQA/MHA, MLA and DSA, mixed sliding/global attention, and Mamba/SSM state each use architecture-specific formulas.
- Usable HBM: A replica must fit weights and at least one complete request within 90% of aggregate GPU memory; the remaining 10% is reserved for activations and runtime overhead.
Parallel placement and replicas
Automatic placement searches legal tensor, expert, and pipeline parallel shapes. Data parallelism creates independent replicas. Rack-scale systems are modeled as physical topologies: for example, an NVL72 domain contains eighteen four-GPU compute nodes, not one server and not nine eight-GPU servers.
Peak active requests is cluster-wide demand and drives the minimum capacity replica count.
Replica admission is a hard runtime ceiling, further limited by available HBM. If global demand falls below a retained manual ceiling, the effective modeled load is clamped to demand without silently rewriting the saved policy.
Recommended operating load is the workload-mode batching knee. It can add replicas to avoid operating every replica at the hard memory ceiling.
Throughput target and latency SLA add replica constraints only when their toggles are enabled. Throughput cannot become the displayed dominant constraint when its target is disabled.
Supported architecture patterns
Dedicated estimators cover dense MHA/GQA, Mixtral-style and DeepSeek-style MoE, MLA/DSA, multimodal wrappers, Gemma mixed attention, Kimi text-plus-vision residency, and Nemotron-H hybrid Mamba/attention/MoE blocks. Curated context limits override misleading base-config defaults where required; NVIDIA Nemotron 3 Ultra supports a maximum context of 1,048,576 tokens.
Performance methodology and ranges
Performance is rerun after the final replica count is selected, using the resulting local concurrency. It is not calculated for one replica and multiplied afterward.
Prefill and TTFT
A roofline estimate combines active-parameter compute, attention work, tensor-parallel collectives, pipeline transfers, overlap, and pipeline efficiency.
Decode and TPOT
The estimate combines resident-weight and sequence-state traffic with tensor, expert, and pipeline communication across the selected NVLink and scale-out domains.
TTCR equals TTFT plus TPOT across the expected output length. Cluster token and request rates use the selected data-parallel replica count. The low/base/high bands are directional sensitivity ranges around provisional calibration seeds. They are intended for comparative planning, not benchmark certification or an SLA guarantee.
TCO methodology
On-premises
- CAPEX: Topology-aware compute nodes, control and storage servers, typed Ethernet switches, and optical pluggables use editable quantities and unit prices.
- Network BOM: Conventional HGX uses eight 400GbE rails per node; B300 HGX uses two 800GbE planes with eight links per node per plane. An optical link counts two pluggables, one at the host and one at the switch. The two 400G Ethernet cables used by an 800G pluggable are outside the estimate.
- Energy: IT kW × 8,760 hours/year × PUE × electricity rate × years.
- Racks: The larger of the 42U space requirement and configured kW-per-rack requirement determines rack count; monthly rack rent is then applied.
- Operations: Loaded labor uses salary plus 30%; maintenance is the configured annual percentage of hardware cost; customer-operated platform services are added from the selected service BOM.
Cloud IaaS
On-Demand compute uses the exact selected AWS region and instance SKU when a quote is available. Reserved and Spot require an exact manual quote rather than a synthetic discount. Storage, transfer, support, labor, optional services, and explicit discounts are added separately.
Token API
Cluster output tokens/second and each use case's input/output ratio determine token volume. Only the configured cache-hit share receives a cached-input rate; the default is 0%. Token API is an all-in SaaS service, so platform-service costs are always $0.
Financial comparison
Hardware and network CAPEX occur at month zero; recurring costs are modeled monthly. Break-even is the first month cumulative cloud cost avoided repays on-premises CAPEX after recurring on-premises costs. NPV uses monthly discounting derived from an 8% effective annual rate. Every cost-affecting selector, toggle, node count, quantity, and valid price edit triggers recalculation; stale asynchronous cloud quotes are ignored.
Vector DB and facility planning
Vector DB sizing
Vector count comes from corpus bytes and chunking or a direct count. Storage includes typed vector bytes, index-specific overhead, metadata, replication, and retention-based backup equivalents. RAM depends on all-in-memory, hybrid hot share, or DiskANN mode and is replication-aware. Node counts take the maximum of RAM, storage, CPU, and heuristic throughput needs.
Facility design
The rack designer imports the physical equipment profile from TCO and packs equipment into separate network, storage, management, and compute racks while enforcing U-space, power, and weight limits. It reports aggregate space and power utilization and flags racks approaching their limits.
Save, share, and export
The analysis is saved locally as you work. Share links serialize the complete versioned analysis, create a short code when the sharing service is configured, and restore the destination page before redirecting. PDF export renders the Results view with the same sizing, ranges, deployment, platform-service, and TCO state.
Review before procurement: Directional estimates establish a defensible comparison envelope. Replace quote-required items and provisional hardware assumptions with current vendor quotes and measured serving benchmarks before treating the output as a purchase plan.
Platform demo
Watch the walkthrough for a guided tour of the planner interface.