NVIDIA AIPerf Replaces GenAI-Perf for LLM Inference Benchmarking
NVIDIA has introduced AIPerf as the designated successor to GenAI-Perf for benchmarking generative-AI inference. The tool is a ground-up rewrite built around multiple worker processes, separate result-processing services and ZeroMQ coordination so the load generator can scale with high-concurrency inference servers.
AIPerf supports more than 15 endpoint types, including chat, responses, ranking and image-generation workloads. It can generate synthetic traffic, use public datasets such as ShareGPT, and replay traces in formats used by Mooncake, Baseten and WEKA AgentX. Load can be shaped with constant, Poisson or gamma arrival patterns, including configurable burstiness and gradual request-rate or concurrency ramps.
For LLM serving, the core measurements are time to first token (TTFT), inter-token latency (ITL), end-to-end request latency and output-token throughput. AIPerf reports percentile distributions including p25, p50, p75, p90, p95 and p99. With DCGM or pynvml available, the same run can include GPU power, utilization and memory telemetry.
AIPerf versus GenAI-Perf
NVIDIA designates AIPerf as the replacement for GenAI-Perf and describes it as a ground-up architectural rewrite. GenAI-Perf was built on NVIDIA Perf Analyzer and used a single-process architecture that could become constrained by Python's Global Interpreter Lock as request concurrency increased.
AIPerf distributes request generation across worker processes and handles results in separate services. This architecture is intended to keep the benchmarking client from limiting a server whose inference capacity exceeds what a single Python process can drive.
NVIDIA's migration documentation says AIPerf is designed as a drop-in replacement for currently supported GenAI-Perf features. Most command-line options map directly. Existing workflows using --max-threads should move to AIPerf's --workers-max control, while the old passthrough -- argument is no longer required. Some GenAI-Perf analyze functionality is still planned and currently unavailable.
| Area | AIPerf |
|---|---|
| Client architecture | Multiprocess workers with separate record processing |
| Endpoint coverage | 15+ endpoint types |
| Traffic patterns | Constant, Poisson and gamma; configurable burstiness and ramps |
| Workloads | Synthetic data, public datasets and trace replay |
| LLM latency | TTFT, ITL and request latency |
| Capacity metric | Output-token throughput |
| Distribution reporting | p25 through p99 plus averages, min/max and standard deviation |
| GPU telemetry | DCGM or local pynvml |
| Result export | Console summaries plus machine-readable run output |
A reproducible first benchmark
NVIDIA's introductory workflow uses Qwen3-0.6B served through vLLM. AIPerf can be installed with uv:
uv tool install aiperf
A fixed 128-input/128-output-token streaming profile can then be driven against an OpenAI-compatible chat endpoint:
aiperf profile \
--model Qwen/Qwen3-0.6B \
--endpoint-type chat \
--streaming \
--url localhost:8000 \
--synthetic-input-tokens-mean 128 \
--synthetic-input-tokens-stddev 0 \
--output-tokens-mean 128 \
--output-tokens-stddev 0 \
--extra-inputs min_tokens:128 \
--extra-inputs ignore_eos:true
The fixed token lengths matter when comparing serving configurations. Allowing a model to stop naturally can produce different output lengths between runs, which changes both the work performed and the resulting throughput. The min_tokens and ignore_eos inputs force the example server to emit the requested output length when that server supports those controls.
Streaming is required for TTFT and ITL measurement because those metrics depend on observing the first token and subsequent decode-token events individually.
Testing production-style request arrival
Static token-length tests are useful for controlled comparisons, while production capacity planning also needs variable prompts and realistic arrival timing. NVIDIA demonstrates a Poisson workload at an average 10 requests per second:
aiperf profile \
--model Qwen/Qwen3-0.6B \
--endpoint-type chat \
--streaming \
--url localhost:8000 \
--request-rate 10 \
--arrival-pattern poisson \
--synthetic-input-tokens-mean 512 \
--synthetic-input-tokens-stddev 128 \
--output-tokens-mean 128 \
--output-tokens-stddev 32 \
--random-seed 42 \
--request-count 200
Poisson arrivals introduce bursts and gaps around the requested average rate. The random seed makes the generated arrival sequence and token-length draws reproducible, which is useful when comparing inference engines, quantization settings, parallelism strategies or hardware configurations.
Read percentiles alongside throughput
A single tokens-per-second figure cannot describe interactive serving quality. TTFT measures how long a user waits before generation begins; ITL measures decode smoothness after the first token; request latency captures the complete response; output-token throughput measures aggregate serving capacity.
Percentiles expose tail behavior hidden by averages. A deployment can maintain a strong mean TTFT while p95 or p99 rises sharply as queues form. For interactive services, a useful capacity test therefore increases request rate or concurrency while tracking both aggregate throughput and a defined tail-latency objective.
GPU telemetry adds another diagnostic layer. AIPerf can correlate the benchmark window with power draw, utilization and memory consumption through DCGM or pynvml. That makes it easier to distinguish an inference server that has reached GPU capacity from one limited by scheduling, networking, CPU-side processing or an undersized load client.
Migration and deployment considerations
Teams already using GenAI-Perf can migrate existing commands using NVIDIA's option mapping and preserve their current benchmark suite as a baseline. Re-run the old baseline with equivalent model, prompt/output lengths, endpoint settings and concurrency before changing the workload shape; this preserves a useful comparison point during migration.
For new deployments, keep at least two profiles: a controlled synthetic test for repeatable software or hardware comparisons, and a traffic-shaped profile based on the application's expected prompt lengths, output lengths and request-arrival distribution. Trace replay is preferable when representative production traces are available and can be handled under the organization's data policy.
On Arm64 systems, NVIDIA notes that AIPerf's crick dependency is distributed as source and requires a C build toolchain. Server-side metrics and GPU telemetry also depend on the corresponding metrics endpoints or monitoring stack being available.
Bottom line
AIPerf gives inference teams a standardized load-generation and measurement path as serving systems move from single-GPU endpoints to high-concurrency and distributed deployments. Its main operational advantages are multiprocess load generation, configurable traffic distributions, percentile latency reporting, trace replay and integrated server/GPU telemetry.
The practical benchmark target remains workload-specific: establish the TTFT and ITL limits users can tolerate, reproduce the application's request shape, then measure how much throughput the deployment sustains inside those latency limits.
Sources
- NVIDIA Technical Blog, Benchmarking LLM Inference at Scale with AIPerf, September 18, 2026.
- NVIDIA AIPerf documentation, Migrating from GenAI-Perf.
- NVIDIA AIPerf documentation, GPU Telemetry with AIPerf and Server Metrics Collection.
- BPDATA AI News, NVIDIA releases AIPerf: a multiprocess, scalable LLM benchmarking client, September 18, 2026.