InfiniBand fabrics and NVLink#

For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both — several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what llmgw does with them, what it needs from you before it can, how to check a node in one command, and where to look when a replica spanning nodes will not run.

Read install.md first; this is the part of it that only matters once there is a fabric.

Where this stands. Every piece here was written against the real formats of sminfo, saquery, ibnetdiscover, nvidia-smi topo -m and nvidia-smi -q, and tested against them — but none of it has yet run on a site with the hardware. Run llmgw doctor --json on two or three nodes first and keep the reports: if a parser meets a format it does not know, that report is what fixes it, and the fix is a control plane upgrade, not one per node.

What llmgw does with them#

Two kinds of interconnect. InfiniBand or RoCE joins nodes (scale-out). NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes is placed, in order of preference:

1. inside one multi-node NVLink domain, if any has enough usable nodes: tensor parallelism between its nodes then runs over NVLink;
2. on one physical InfiniBand or RoCE fabric, preferring nodes under one leaf switch and putting nodes with a degraded cable last.

It is never split across two fabrics or two NVLink domains. Within a node, the GPUs a shard gets are the set with the most NVLink pairs, then the nearest to the fabric's adapters, then on one NUMA node.

A fabric has a name and an identity. The name is what agent.yaml says; the identity is what the adapters report — on InfiniBand the subnet managers every port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name is how you address a fabric; the identity is how llmgw catches the name being wrong: two independent fabrics given one name are placed as two, and the cluster page says so.

Before a node joins#

On every GPU node:

On an NVL72 or NVLink Switch system, also:

On each InfiniBand fabric:

• A subnet manager, and a standby. Without one, ports come up to INIT and stay there; without a standby, the fabric keeps its routes if the master stops but cannot recover from any change after. llmgw watches this but does not run one.
• Its own name in agent.yaml on every node attached to it. Optionally, a distinct subnet_prefix in each subnet manager's configuration, which makes GIDs unique across fabrics; llmgw does not need it.

Declaring the fabrics#

In /etc/llmgw/agent.yaml, the adapters on each fabric, by name:

fabrics:
  - name: ib-compute
    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
  - name: ib-storage
    hcas: [mlx5_4]

or --fabric ib-compute:mlx5_0,mlx5_1,… when onboarding (node-onboarding.md). An adapter belongs to one fabric; a node may be on several. A replica on ib-compute is given only ib-compute's adapters (NCCL_IB_HCA).

The rate is measured, not taken from the file: a fabric's rate on a node is the slowest of its active adapters, because a collective runs at the speed of its slowest link.

The agent checks the declaration against the adapters on every heartbeat. An adapter that is missing, or whose port is down, takes that fabric off the node within seconds — the node stops being offered multi-node work on it — and returns it when the port comes back.

Checking a node: llmgw doctor#

sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
sudo llmgw doctor --json > $(hostname)-doctor.json

It reads the node and says what is wrong and how to fix it, most serious first, and exits non-zero when anything fails:

FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
              → modprobe nvidia_peermem
OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
OK    agent   NCCL bootstraps over ens3 (10.1.0.21)

followed by each GPU's NVLink peers and nearest adapter, each port's state, rate and errors, and the NCCL environment a shard on each fabric would be given. --json writes every fact it read as well as what it concluded — send that file when the report is not enough.

Checking two nodes: llmgw ctl fabric-test#

Every check on one node can pass while two nodes still cannot reach each other — a cable into the wrong switch, a partition key, a RoCE network with no route — and a replica spanning them then hangs on its first all-reduce. Before placing one, on the control plane:

llmgw ctl fabric-test gpu01 gpu02

gpu01 → gpu02 on ib-compute, 41s

RAIL              LINK      MEASURED    VERDICT  DETAIL
mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517

It runs ib_write_bw (perftest, installed on a node the first time) from each of the first node's adapters on the fabric to the same adapter on the second — mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the agents, so no SSH between nodes is needed. Each rail is graded against its link's rate: ok at 80% or more, slow at 50%, fail below that or when nothing got through. --fabric picks one when the nodes share several, --seconds the length of each rail's run. The traffic is real: run it when the fabric is quiet.

It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says the fabric carries traffic between these two nodes at its rate, which is what a first all-reduce needs and what nothing on one node can show.

Watching them#

• Cluster page, Fabrics. Each declared fabric and what it physically is; a table of subnet managers per fabric — master, priority, standbys, the last change of master — and every disagreement: one name on two fabrics, two names on one, an adapter up but undeclared, ports up with no subnet manager, a fabric with no standby, a recent failover.
• /admin/fabrics. Each fabric's map, from ibnetdiscover run by one of its nodes every five minutes: switches in their tiers, adapters under their leaves, cables coloured by rate, degraded ones in red, rail alignment, and the cluster's own adapters live with their traffic. Click anything for its detail.
• /admin/nvlink. Each multi-node NVLink domain as a rack of trays: which GPUs are joined, which node's IMEX is down, which link runs degraded.
• A node's page. Its ports with state, rate, errors and what they are physically on; its GPU-by-GPU NVLink matrix; its load.
• Metrics, for alerts:

What an engine is given#

For a shard on an InfiniBand or RoCE fabric:

A profile's env overrides any of them. vLLM launches a multi-node replica with its own multiprocessing (engines.vllm.launch_mode: mp, the default): every node runs vllm serve with its rank and the leader's address, the workers headless. launch_mode: ray is there for a vLLM too old for that.

When a replica spanning nodes will not run#