Large models: several cards, several nodes#

A model too big for one card runs on several. llmgw chooses how from the model's own figures and the cluster as it is; this page is the rules it follows, the commands, and what to look at when a first one is brought up on new hardware.

How a model is split#

In order, the first that fits:

1. One card. Nothing to split.
2. Several cards of one node, tensor parallel. Every layer is cut across the cards, so they talk for every token: they must be on one NVLink domain. The agent reads nvidia-smi topo -m and gives a replica the cards with the most NVLink between them (install.md).
3. Several nodes, pipeline parallel between them. Each node holds a run of layers on all its cards (tensor parallel inside it), and only a layer's output crosses to the next node. The nodes must share one InfiniBand or RoCE fabric: a replica never spans two, and never a node with none. Any number of nodes, not only two or four.

The plan takes the fewest cards, then the fewest nodes, then the better precision up to eight bits.

Tensor width and the KV heads. The cards split the model's KV heads among them, or past their number each card holds a copy of one. So the width divides the KV heads or is a multiple of them: a model with 8 KV heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.

Mixture of experts. A model of experts (Qwen3-235B-A22B, DeepSeek-V3, Llama 4, Mixtral) on more than one card is served with vLLM's --enable-expert-parallel: each card holds whole experts rather than a slice of every one, which is less traffic per token. It is read from config.json (num_experts, n_routed_experts, num_local_experts).

When it does not fit. A smaller precision first: an FP8 or AWQ repository of the same model, or a GGUF quantisation on llama.cpp if one node's cards hold it. Then more nodes. llmgw ctl model plan REPO --precision bf16,fp8 shows each precision's fit from the model's config.

What a way needs#

Each way the plan offers says what a replica needs of each card, part by part, and of all its cards together:

$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
FILE      SIZE    BEST WAY        A CARD   OF     ALL CARDS  NOW
original  131 GB  4 × h100-80g    37.4 GB  72 GB  150 GB     free

Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).

• weights: the files, divided over the cards.
• context: the KV cache for the context planned, divided likewise.
• engine: the engine's own: 2 GB and a fiftieth of the weights for vLLM.
• usable: 90% of a card; the rest is the driver's and fragmentation's.

The console's Serve a model shows the same for every way a file runs. A context longer than planned is checked against it before anything is fetched.

Serving one#

Without flags the plan's best way is taken:

llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768

Or a way given by hand, for instance two nodes of eight:

llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
    --gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO

--gpus is the cards on each node and --nodes the nodes. The pipeline is the nodes and the tensor width the cards of each.

--env KEY=VALUE goes into the engine's environment on every node: NCCL_DEBUG=INFO while bringing a cluster up, NCCL_IB_* to steer it. Only the variables engines are tuned with are taken. llmgw already sets these:

• NCCL_IB_HCA to the fabric's adapters;
• NCCL_IB_GID_INDEX on RoCE;
• NCCL_SOCKET_IFNAME to the management network;
• for an NVLink domain spanning nodes, NCCL_MNNVL_ENABLE.

The job's first step says what it needs of each card. The weights then come to the mirror once and from it to each node. The startup timeout (20 minutes unless the profile's startup_timeout says otherwise) runs from when a node's download last moved. So a model of hundreds of gigabytes is not failed while it is still arriving, and one whose download stops is failed that long after.

Watching it come up#

With NCCL_DEBUG=INFO, NCCL says which transport it took between nodes: NET/IB is the fabric. NET/Socket means it fell back to TCP, and the model will run, slowly.

Bringing it up on new hardware#

A first run on a fabric, in order. Each step is worth its output if something goes wrong.

1. The nodes. Add each with its fabric: llmgw node add host --fabric ib-a:mlx5_0,mlx5_1. Then llmgw ctl nodes shows every node ready, on ib-a.
2. The fabric. llmgw ctl fabric-test node1 node2: each rail at its rated speed (a 400 Gb/s rail near 45 GB/s).
3. One node, several cards. A dense 70B at bf16 on four or eight cards of one node, by the plan: llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors. Ready, then a question answered.
4. Two nodes. The same model forced over two: --gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO, under another --name. Ready, answering, and NET/IB in both shards' logs.
5. Experts. A mixture-of-experts model on one node's eight cards, for instance Qwen/Qwen3-235B-A22B-Instruct-2507-FP8: its profile carries --enable-expert-parallel (llmgw ctl models, or its definition from the admin API).
6. A node lost. With the two-node replica serving, stop one node's agent (systemctl stop llmgw-agent): the replica fails and is placed again once the node is back or another has room. Nothing is left holding cards.

For each, worth keeping:

• the time from serve to ready;
• tokens a second from llmgw ctl bench;
• the plan's per-card figure against what nvidia-smi shows once loaded;
• anything it said that was wrong.