llmgw docs

The admin API#

Written for whoever automates a cluster rather than typing at it: how to reach the control plane's management API, what it asks of a request, and where its description is.

llmgw ctl and the console are clients of this API and nothing more, so anything they do can be done with curl, a generated client or the Terraform provider. It adds and removes models, scales them, cordons and upgrades nodes, sets capacity quotas, attaches guardrail policies, adds organizations, teams and projects, issues and revokes API keys, puts people in the tree, and reads the fleet, usage and the audit log.

Reaching it#

The API is under /api/v1 on the admin listener, admin_listen in /etc/llmgw/server.yaml: 127.0.0.1:9090 unless changed. It is on loopback on purpose. The same listener serves pprof when it is turned on, and the server refuses to start with it on any other address unless admin_remote: true — the token would cross the network in clear.

From another machine, tunnel to it rather than moving it:

ssh -N -L 9090:127.0.0.1:9090 admin@control-plane &
curl -s -H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" http://127.0.0.1:9090/api/v1/summary

With admin_remote: true and admin_listen: 10.0.0.5:9090, call that address directly — on a network you trust, or behind a TLS proxy that keeps the Host header an address.

The token#

Every path under /api needs the admin token as a bearer token:

Authorization: Bearer 6f1c…

The server writes it the first time it starts, 0600, to /var/lib/llmgw/admin-token under its systemd unit (admin_token_file moves it; a server run by hand keeps it in secrets/admin-token beside its configuration). llmgw ctl reads it from there, which is why it runs with sudo, or from LLMGW_ADMIN_TOKEN. To give it to automation:

sudo cat /var/lib/llmgw/admin-token

The token is the whole of the API's authorisation: whoever holds it can do everything the API does. Keep it where you keep other root credentials.

What a request must look like#

Besides the token, the listener refuses three things before a handler sees them, as defences against a browser being turned against it:

  • A name in Host. The listener answers to an address or localhost, not a DNS name, so a page on some other site cannot reach it by rebinding a name to 127.0.0.1. Call it as http://127.0.0.1:9090, not http://cp.example.com:9090; behind a proxy, have the proxy send an address.
  • A cross-site request. One whose Origin is another host, or whose Sec-Fetch-Site says another site sent it.
  • A change that is not JSON. Every method but GET and HEAD must carry Content-Type: application/json, even a DELETE or a POST with no body. A form post from a page cannot set that header.

A change is recorded in the audit log with who made it, as the request says in X-Llmgw-Actor (letters, digits and ._-@, at most 64; unknown when absent). llmgw ctl sends the name of whoever ran it; automation should send its own, terraform or ci@deploy, so llmgw ctl audit can tell them apart.

A refusal is a 4xx or 5xx with {"error": "…"} saying why. 501 means this control plane lacks what the endpoint needs — a store, a mirror, guard policies — and says which.

Some calls#

H=(-H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" -H "Content-Type: application/json" -H "X-Llmgw-Actor: $USER")
API=http://127.0.0.1:9090/api/v1

# The fleet in one answer, and every model.
curl -s "${H[@]}" $API/summary
curl -s "${H[@]}" $API/models

# Add a model, read back its definition with the defaults filled in, scale it.
curl -s "${H[@]}" -X POST $API/models -d '{
  "name": "qwen-small",
  "source": "/srv/models/qwen2.5-1.5b-instruct-q4_k_m.gguf",
  "profiles": [{"name": "cpu", "engine": "llamacpp", "min_memory_gb": 4, "cpu_cores": 4}],
  "scaling": {"min": 1, "max": 2}
}'
curl -s "${H[@]}" $API/models/qwen-small
curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2, "idle_timeout": 1800000000000}'

# A team's capacity quota.
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'

# An organization, a team in it and a project in that; then a key for the
# project. The answer holds the key once: only its hash is kept.
curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'

# Remove the model; a DELETE carries the content type too.
curl -s "${H[@]}" -X DELETE $API/models/qwen-small

Durations in bodies are integer nanoseconds, as Go encodes them: half an hour is 1800000000000.

The description#

The API is described in OpenAPI 3.1:

  • served by every control plane at GET /api/v1/openapi.json (with the token, like the rest), as built into that binary, so it describes the server you are talking to;
  • committed as admin-api.openapi.json for reading without a cluster.
curl -s "${H[@]}" $API/openapi.json > llmgw-admin.json

It lists every endpoint, its path and query parameters, and the schema of each request and answer, with the meaning of each field. Load it into any OpenAPI tool — Swagger UI, Redoc, an editor — or generate a client from it:

docker run --rm -v "$PWD:/w" openapitools/openapi-generator-cli generate \
  -i /w/llmgw-admin.json -g python -o /w/llmgw-admin-client

How it is kept true#

The document is generated, not written. internal/control/adminapi/routes.go holds a table of every endpoint: its method and path, a summary, its query parameters, and Go values of the types it reads and answers with. The generator in internal/docgen/openapi reads those types by reflection, the way encoding/json encodes them, and takes each field's description from its comment in the source.

Two tests keep it from drifting. One registers the API on a recording mux and fails when an endpoint is served and not in the table, or in the table and not served. The other regenerates the document and fails when it differs from the committed one. After changing an endpoint or a type it carries:

go generate ./internal/docgen

What it does not cover#

  • Making people's accounts. People sign up, are invited, or come from the identity provider (OIDC, SCIM); the API lists them and puts them in the tree, and llmgw users approves, disables and deletes them. Key lifetime limits and spending limits are llmgw keys ttl and llmgw keys limits.
  • The client-facing API (/v1/chat/completions and the rest), which is OpenAI's and Anthropic's, on listen, with API keys.
  • The rest of the admin listener: /metrics, /health, the dashboard pages and /debug/pprof, which are not under /api.