Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
NVIDIA's developer blog published the walkthrough on November 13, 2026, written by Fatima Fall. It shows how to serve Z.ai's GLM-5.3 NVFP4 model from one OpenAI-compatible endpoint hosted across two DGX Stations. The model is a mixture-of-experts design with 753 billion parameters, about 40 billion of them active for each token. The post is a step-by-step recipe that also measures how the setup behaves under concurrent load.
Memory sets the layout. The NVFP4 checkpoint is about 464 GB, more than the 252 GB of HBM3e on one GB300 GPU, so one station cannot hold it. The machines connect over ConnectX-8 RDMA, and vLLM applies tensor parallelism at TP=2, leaving each GPU with roughly 218 GiB of weights. Station A hosts the API while Station B runs a headless worker, and both compute every request. The setup also uses multi-token prediction that drafts four tokens per step, an FP8 KV cache and prefix caching. Served context is capped at 98,304 tokens, and vLLM may use 93% of GPU memory.
On the SPEED-Bench Coding dataset, aggregate output peaks at 607.8 tokens per second at 16 concurrent requests, with 482.90 ms time to first token and 136.46 seconds mean latency. At 32 concurrent requests, output falls to 385.0 tokens per second and mean latency rises to 453.5 seconds. Per-user throughput drops from 115.3 tokens per second at one request to 13.0 at 32. Each station needs Linux, Docker, the NVIDIA Container Toolkit and 464 GB of free disk space for the checkpoint.
Why it matters
For teams building on open-weight models, the result shows that a 753-billion-parameter model can run on two desk-side machines behind a standard API. That changes the build-or-rent math for workloads with strict data control requirements or steady, predictable request volumes. The hardware bill is the real gate: two GB300-class stations, each with hundreds of gigabytes of free storage for the checkpoint alone. Teams should not treat the 607.8 figure as a service guarantee, since it is a peak at one concurrency level on one benchmark.
For companies selling hosted inference, local deployments at this size are a competitive signal. A customer that can serve GLM-5.3 on its own hardware has less reason to pay per token for steady traffic, while bursty demand still favors hosted capacity. For investors, the open questions are how quickly these recipes move from vendor tutorials into routine practice, and whether the cost of this hardware falls far enough to make local serving common beyond large labs.
Key points
- 1The NVFP4 checkpoint is about 464 GB, more than the 252 GB of HBM3e on one GB300 GPU.
- 2Tensor parallelism at TP=2 leaves each GPU holding roughly 218 GiB of model weights.
- 3Station A hosts the OpenAI-compatible API while Station B runs as a headless worker.
- 4Peak throughput was 607.8 tokens per second at 16 concurrent requests, versus 385.0 at 32.
- 5
