Self-hosted LLM gateway · v0.16.3-29-g748f777-dev
llmgw runs vLLM, SGLang and llama.cpp across your servers and serves them behind one OpenAI-compatible API. It knows which GPUs share an NVLink domain or an InfiniBand fabric, and a model split across machines never straddles two.
$ sudo llmgw server init --admin you@example.com
One static binary. It runs as the control plane, as the agent on every GPU node, and as the weights mirror.
Point any OpenAI client at it. Only the base URL changes. Streaming, tool calls, /v1/models and /v1/responses work as your client expects.
Agents report GPU class, memory, NVLink peers and InfiniBand ports. Replicas go where they fit, including ones larger than one machine, on the GPUs with the most NVLink pairs.
Keep more models in the catalog than your GPUs hold. The first request starts one, the chat page shows it loading, and it stops again after the idle timeout.
Organisations, teams and projects, each with limits. Priority bands decide who waits when the cluster is full. Managers see their organisation's spend; people see their own.
Continue, Codex or Cline from the account page: a key, and the configuration filled in or a one-line command that writes it. Claude Code and OpenCode work with the same key, over the Anthropic and OpenAI APIs.
Markdown and maths rendered, stop mid-reply, tokens per second and time to first token, a system prompt and a temperature. The gateway keeps no conversations.
Explore what is trending on Hugging Face, read a model card under its row, and download once for the whole cluster. Only what your nodes can run is shown by default.
llama.cpp on every node; vLLM on every GPU node that can run it, from PyPI with every file hash-checked. A node that cannot run it says why.
Upgrade the control plane, then the agents. A restarted agent adopts the engines it was running instead of reloading them, so replicas stay up throughout.
/admin shows capacity per GPU class, CPU and memory per node, who is using what, each node's page and logs. It fits a phone. Prometheus is optional.
People add PDF, Office and mail from their own machines with llmgw-kb; only text and vectors are sent. Each organisation, team and project sees only its own collections, down to the database row.
Personal data, secrets, prompt injection and deny patterns, with a judge model on CPUs. Policies observe or enforce, per organisation, team or project; an answer nobody could check is refused.
Entra ID, Active Directory or any OIDC provider; users and groups by SCIM; staff roles, and an audit log of every change.
Three machines: each can run the gateway, PostgreSQL under Patroni keeps the store on two, etcd votes on all three. Setup builds it; losing any one machine loses nothing.
ComfyUI on the same cards: pictures and meshes as jobs, queued, followed and returned as files, each carrying the licence it was made under.
A site may have independent InfiniBand fabrics, eight-GPU HGX boards and NVL72 racks side by side. llmgw tells them apart by what the adapters report, not by what a config file calls them.
NVL72 or NVLink Switch. Every card joined, IMEX up. Shards get NCCL_MNNVL_ENABLE=1.
Nodes under one leaf first, nodes with a degraded cable last.
Two fabrics given one name are placed as two, and the cluster page says so.
$ sudo llmgw doctor FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive → systemctl enable --now nvidia-fabricmanager WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off → modprobe nvidia_peermem OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3 OK nvlink NVLink groups: [0 1 2 3 4 5 6 7] OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 OK fabric subnet managers: 0x…f0c0 (master, priority 15), 0x…f1d4 (standby, priority 14) OK agent NCCL bootstraps over ens3 (10.1.0.21)
/admin/fabrics draws switches in tiers, cables coloured by rate, degraded ones in red, and live traffic on your adapters./admin/nvlink shows which trays are joined, whose IMEX is down, which link runs degraded.Three machines for high availability, or one. Ubuntu, Debian or RHEL 10; llmgw installs its engines, Python and CUDA itself.
The install guide shows every step on the screens of a real install, recorded.
Static, no dependencies, the same file everywhere.
VERSION=vX.Y.Z # from the releases page
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgwWith nothing installed, sudo llmgw opens the setup: one machine or three, what to install, the other machines' logins. It checks every machine first, shows the plan, and installs. From a script, llmgw server init does the same.
sudo llmgwllmgw setup llm-1 · 10.77.0.11 · v0.14.9 ╭─ Steps ────────────────╮╭─ Checks ───────────────────────────────────────────────────────────────────────────────────╮ │✓ Start ││ │ │✓ Choices ││ ✓ All three are ready. │ │✓ Machines ││ │ │› Checks ││ llm-1 (this machine) │ │ Plan ││ ✓ login logged in │ │ Install ││ ✓ system Ubuntu 24.04.5 LTS │ │ ││ ✓ architecture x86_64 │ │ ││ ✓ cpus 4 CPUs (needs 2) │ │ ││ ✓ memory 11961 MB (needs 3500) │ │ ││ ✓ disk 36 GB free (needs 15) │ │ ││ ✓ clock within a second of this machine │ │ ││ ✓ ports free │ │ ││ ✓ internet reaches the package mirrors │ │ ││ ✓ network reaches every other machine │ │ ││ │ │ ││ 10.77.0.12 │ │ ││ ✓ login logged in │ │ ││ ✓ system Ubuntu 24.04.5 LTS │ │ ││ ✓ architecture x86_64 │ │ ││ ✓ sudo without a password │ │ ││ ✓ cpus 4 CPUs (needs 2) │ │ ││ ✓ memory 11961 MB (needs 3500) │ │ ││ ✓ disk 36 GB free (needs 30) │ │ ││ ✓ clock within a second of this machine │ │ ││ ✓ ports free │ │ ││ ✓ internet reaches the package mirrors │ │ ││ ✓ network reaches every other machine │ │ ││ │ │ ││ 10.77.0.13 │ │ ││ ↓ more — PgDn │ ╰────────────────────────╯╰────────────────────────────────────────────────────────────────────────────────────────────╯ Enter see the plan · R check again · Esc back · PgUp/PgDn scroll
From the control plane, over SSH. The node gets llmgw, is checked by llmgw doctor, receives its certificate and starts its agent; it installs its engines by itself.
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1$ llmgw ctl nodes
NODE STATE GPUS MEM CLASSES FABRICS REPLICAS LAST SEEN
gpu01 ready 8/8 free 503G h100-80g ib-a 0 2sFrom /admin/mirror, what your nodes can run, downloaded once for the cluster; or by name. If it does not place, llmgw ctl explain-placement says why.
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "coder", "messages": [{"role": "user", "content": "hello"}]}'Port 8080 is plain HTTP. Put it behind Caddy or nginx with a public certificate. Only 8080 goes through the proxy: agents reach 7443 directly, because it is mutual TLS with the cluster's own authority.
Bind llmgw to loopback when the proxy is on the same machine: listen: "127.0.0.1:8080".
# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
# certificate Caddy obtains and renews by itself.
#
# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
# run: caddy run --config /etc/caddy/Caddyfile
#
# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
# own authority and must reach llmgw directly; 9090 and 9091 stay private.
llm.example.com {
# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
# more here gives the client a 413 before the upload, not after it.
request_body {
max_size 32MB
}
# One control plane. With more than one, see the end of this file.
reverse_proxy 127.0.0.1:8080 {
# Tokens are written the moment they are generated. Without this a
# streamed reply can arrive in bursts, and a client measuring time to
# first token measures the proxy instead.
flush_interval -1
# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
# no trusted_proxies, discards whatever the client sent. llmgw relies
# on that: the proto decides whether session cookies are Secure, and
# the host is the base URL the landing page tells people to use.
# Behind a load balancer of your own, name it in trusted_proxies.
transport http {
# A long generation can be silent for a while before the first
# token. llmgw has its own limits (timeouts.*); this must not be
# the shorter one.
response_header_timeout 15m
}
}
# No `encode` on purpose: compressing a stream buffers it.
# Access log to stderr, which under systemd is the journal.
log
}
# More than one control plane (server init --standbys, server add): one of
# them leads, the others stand by. Name them all and send to the one that
# leads — /leader is 200 there and 503 on a standby. Not /ready: that is 503
# on a leader with no model running yet, and would take the portal away with
# it. A standby would pass a request on to the leader anyway, but not while
# there is none, and the hop is for nothing.
#
# With a floating address (--vip) the address already follows the leader:
# give reverse_proxy that one address instead, and keep the rest as above.
#
# llm.example.com {
# request_body {
# max_size 32MB
# }
# reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
# # The first that is healthy: the leader, whichever it is.
# lb_policy first
# health_uri /leader
# health_interval 2s
# health_timeout 2s
# # While leadership moves — a few seconds when a leader is lost —
# # none is healthy; a request waits that long for one rather than
# # failing at once.
# lb_try_duration 15s
# lb_try_interval 250ms
# flush_interval -1
# transport http {
# response_header_timeout 15m
# }
# }
# log
# }# llmgw behind nginx: the client-facing port (8080) on a public name.
#
# Put this in /etc/nginx/sites-available/llmgw (or conf.d/llmgw.conf), replace
# llm.example.com, and get a certificate — certbot --nginx -d llm.example.com
# fills in the ssl_ lines below. Then: nginx -t && systemctl reload nginx
#
# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
# own authority and must reach llmgw directly; 9090 and 9091 stay private.
upstream llmgw {
server 127.0.0.1:8080;
keepalive 32;
}
server {
listen 80;
listen [::]:80;
server_name llm.example.com;
return 301 https://$host$request_uri;
}
server {
# `listen ... http2` rather than `http2 on;`: the latter needs nginx
# 1.25.1, and Ubuntu 22.04 and Debian 12 ship older ones. Newer nginx
# accepts this form with a deprecation warning.
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name llm.example.com;
ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
# The router reads at most 32 MiB of body (router.max_body_bytes).
client_max_body_size 32m;
location / {
proxy_pass http://llmgw;
proxy_http_version 1.1;
proxy_set_header Connection "";
# Set, never appended: llmgw decides from X-Forwarded-Proto whether
# session cookies are Secure, and builds the base URL the landing page
# shows from X-Forwarded-Host. A value passed through from the client
# would be the client's choice.
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Streamed replies: tokens go out as they are generated. Buffering
# turns a stream into one late burst, and gzip buffers too.
proxy_buffering off;
proxy_cache off;
gzip off;
# nginx's default is 60 seconds between two reads from upstream. A
# large model can think longer than that before its first token, and
# the client then sees a 504 for a request that was about to succeed.
# llmgw has its own limits (timeouts.*); this must not be the shorter.
proxy_read_timeout 15m;
proxy_send_timeout 15m;
}
}X-Forwarded-Proto and shows X-Forwarded-Host as the base URL.Built by CI from the tag and checked against checksums.txt. Every release is on the releases page with its changelog.
This is a development build (v0.16.3-29-g748f777-dev). Take a release from the releases page, where every file is listed with its checksum.
Pre-1.0: configuration formats may change between minor versions. This is what has run on real hardware, and what has not yet.
sminfo, ibnetdiscover and nvidia-smi, but not yet run on the hardware. Run llmgw doctor --json on a few nodes first.