Guardrails#

Written for the operator deciding what the gateway lets through: what a policy can check, what it can do, and how one is attached to a team.

llmgw server init writes a baseline policy that applies to every request, with Qwen3Guard-Gen 0.6B as its judge on CPUs; see install.md for what it checks. The policies are part of the cluster's settings, kept in the store: llmgw ctl config edit (or the console's Settings) changes them, and the change is in force on every control plane within seconds, without a restart. A cluster whose settings were brought in from a server.yaml without a guard: section gets the baseline policy, without a judge. With no policies, nothing is checked and nothing costs anything. With one, every request is judged once it is authorised and within its limits, with its stored conversation spliced in, and before it is sent to a model; and the answer is judged before the client sees it.

Turning them off and on#

llmgw ctl guard off --note "why"   # no request is checked, whatever the policies say
llmgw ctl guard on                 # every policy checks again, as it was

That is guard.enabled in the cluster's settings, in force on every control plane within seconds without a restart. The policies and where they are attached are kept. While off, llmgw ctl status, llmgw ctl guard list and the guardrails page say so. To stop one policy rather than all, give it mode: off, or mode: shadow to have it record what it would do.

A policy#

guard:
  # Applied to every request.
  default: [baseline]
  policies:
    baseline:
      input:
        pii: {secret: block, card: redact}
        injection: flag
      output:
        pii: {secret: block}
    finance:
      mode: enforce          # enforce (default), shadow, off
      fail: closed           # open (default) or closed, for checks that can fail
      no_store: true         # no conversations, notes, prefix routing or external models
      max_output_tokens: 2048
      system_prompt: "You answer questions about Acme's products only."
      drop_client_system: true
      stream: sentence       # sentence (default) or buffer_all
      require_capabilities: true   # no tools/images to models whose catalog entry lacks them
      validate_structured: true    # answers must fit response_format's json_schema
      input:
        pii: {email: redact, phone: redact, iban: block, tckn: redact}
        injection: block
        max_chars: 40000
        deny:
          - {name: codename, regex: "(?i)project falcon", action: block}
      output:
        pii: {iban: redact, tckn: redact, secret: block}
      tools:
        allow: [search, fetch]
        deny: [shell]
        on_denied: strip     # strip (default) or block
        max_calls_per_turn: 4
        validate_arguments: true

What is found.

• Personal data: email, phone, iban, card, tckn, ip. Each is checked before it counts: IBANs by their mod-97 check, cards by Luhn and issuer prefix, TCKNs by their two check digits, phone numbers only when written to be dialled (+90…, 00…, 0532…).
• Secrets (secret): OpenAI- and Anthropic-style keys, AWS access key ids, GitHub, Slack and Google tokens, JWTs, PEM private keys and llmgw's own keys.
• Prompt injection: instructions to drop instructions (English and Turkish), chat-template control tokens in user text, invisible Unicode, long encoded blobs. These are found through disguises: fullwidth letters, Cyrillic look-alikes, zero-width characters, and base64, hex and URL encoding two levels deep.
• deny patterns: matched against the text as written, and as a model reads it once those disguises are removed.

What is done.

• flag lets the request through and records it.
• redact rewrites the value as a placeholder ([EMAIL_1]; the same value is the same placeholder throughout a request).
• block stops it.

Putting them back. With restore: true in a policy's input, the values a request had redacted are put back in its answer: the model reads "write to [PERSON_1] at [EMAIL_1]", the client reads the name and address it sent. Streamed answers too — a placeholder split across two events is held until it is whole. Only that request's own values, only into its answer; the map is kept in memory for the request and nowhere else. The output rules then read the answer as the client will, so a policy that redacts personal data in answers still does.

input:
  pii: {email: redact, phone: redact}
  restore: true

Shapes.

• system_prompt is put first.
• drop_client_system removes the caller's own system and developer messages.
• max_output_tokens is set where a request asks for none, and lowers one that asks for more.
• A tool deny/allow list strips the tool from the request, or refuses it.
• In the answer, a call to a tool that is not allowed, more calls than max_calls_per_turn, or arguments that do not fit the parameters the request declared (validate_arguments) stop it.
• require_capabilities refuses tools offered to a model whose catalog entry does not say capabilities.tools, and images to one without capabilities.vision. It is opt-in because an entry that says nothing is read as "cannot".
• validate_structured checks an answer against the json_schema the request asked for in response_format. Buffered, or streamed with buffer_all, an answer that does not fit is stopped; streamed a sentence at a time it has gone out already, and is flagged.

A judge model#

Patterns find what has a shape. For what does not — violence, crime, sexual content, self-harm, jailbreaks, whatever a moderation model was trained on — a policy can have a judge model on the cluster read the text.

The judge must be in the catalog under the name the policy gives, as a judge (role: guard), with a replica ready. When it is not, what the policy judges is refused with guard_unavailable (fail: closed) or served unjudged (fail: open), and llmgw ctl status and the console's Overview say which judge and why.

Which judge#

Measured on the reference cluster, as the judge's probability of "unsafe" for ordinary coding requests against harmful ones, Llama Guard 3 1B could not tell them apart: "continue" 0.51, "continue the code above" 0.89, "how do I kill all child processes in bash" 0.92, against 0.97 to 1.00 for harmful requests, at any quantisation. Qwen3Guard-Gen 0.6B gave the ordinary ones 0.15 at the most and the harmful ones 1.00. Its verdicts are safe, unsafe and controversial; controversial counts as safe unless the policy says controversial: unsafe.

A policy that names one of these is enough: the leader fetches its weights into the mirror and serves it as a judge, and serves it again if it is removed from the catalog. Another judge is served by hand with --repo.

llmgw ctl guard judge status              # each judge the policies ask, and what is wrong with it
llmgw ctl guard judge enable              # Qwen3Guard-Gen 0.6B on CPUs, or the judge the policies ask that is missing
llmgw ctl guard judge enable --gpu        # Qwen3Guard-Gen 4B on a GPU
llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6b   # the order every policy asks them in
llmgw ctl guard judge enable --replace    # when a model of that name is there, but not as a judge

or Do → Serve the guardrails' judge in the console.

A GPU and CPUs together#

judge.models is a list asked in turn: the first judge that answers gives the verdict. The next is asked when one has no replica serving, fails, or does not answer within latency_budget; a judge with no replica at all is passed over at once. A judge on a GPU first and one on CPUs after it is the GPU's speed while it has its card, and the CPUs' when it does not.

judge:
  models: [qwen3guard-4b, qwen3guard-0.6b]
  input: block
  output: block
  latency_budget: 5s           # each judge's wait

A judge on a GPU is served as a standard model, on one card: it takes a card only from what may give one up, and waits while every card is busy, the judge on CPUs standing in. A judge on CPUs is critical, with two replicas, because every judged request waits on one. llmgw ctl guard judge status says, for each judge, which is asked when it gives no verdict.

When it refuses ordinary requests#

A refusal by a judge names the hazard it saw — the client is told refused by the gateway's policy (judge: Violent) — and the same is kept with the request: llmgw ctl request <id>, the server's log (guard_rule) and the guardrails page show it. To see what the judge says of any text, without anything being kept:

llmgw ctl guard judge ask 'How do I sort a list in Python?'
# qwen3guard-0.6b: safe, 1900 ms
# it answered: "Safety: Safe\nCategories: None"

A judge that says unsafe to ordinary text is usually Llama Guard 3 1B (llmgw ctl guard judge use qwen3guard-0.6b), the wrong weights (another repository's, without the judge's template: judge enable --replace serves the right ones), or a policy counting every category: give its judge categories. The baseline counts Violent, Non-violent Illegal Acts, Sexual Content or Sexual Acts, Suicide & Self-Harm and Jailbreak; PII is the patterns', and copyright, unethical acts and politically sensitive topics refuse too much ordinary work. While finding out, judge.input: flag records instead of refusing.

A judge's file served from the mirror page — Llama Guard, Qwen3Guard, ShieldGemma — is served as a judge too, under the name the policies ask. By hand, the catalog entry is:

# a catalog model (server.yaml models:, or ctl model add)
- name: qwen3guard-0.6b
  source: http://mirror:8081/qwen3guard-0.6b/…/Qwen3Guard-Gen-0.6B.Q8_0.gguf
  role: guard
  guard:
    format: qwen3guard         # qwen3guard, llamaguard3, shieldgemma or json
  scaling: {min: 2, max: 2}
  profiles: [{name: cpu, engine: llamacpp}]

# in a guard policy
judge:
  models: [qwen3guard-0.6b]    # or model: one judge
  input: block                 # flag or block; empty does not ask about input
  output: flag
  categories: [Violent, Jailbreak]   # only these count; empty: any unsafe verdict
  controversial: safe          # or unsafe
  latency_budget: 250ms        # each judge's wait; after the last, fail: open serves (flagged), closed refuses

A model with role: guard is the guardrails' alone:

• it is not in /v1/models or the chat page, and is refused (404) to every key, as a name and as a fallback;
• it must always run — on demand is refused. One on CPUs is critical and should have two replicas on different nodes, because every judged request waits on it.

The gateway's calls to it go past admission, accounting and the guardrails themselves.

• Formats:
  • qwen3guard asks in Qwen3Guard's own words and reads "Safety: Safe | Unsafe | Controversial" and its "Categories:".
  • llamaguard3 reads Meta's "safe" / "unsafe\nS1,S10"; give the catalog entry categories to have S1 recorded as its name.
  • shieldgemma asks Google's model whether the text breaks the listed principles (categories, each a sentence) and reads Yes or No.
  • json expects {"safe": false, "categories": [...]}.
• What is read:
  • Input: the request's last user turn, after the patterns.
  • Output: the answer, together with that user turn. A streamed answer is read every 600 characters and at the end, so text between two readings has gone out already; stream: buffer_all holds it until it is read.
  • A long text is read by its start and its end (about 6000 characters), so a message with a file pasted into it fits the judge's context rather than overflowing it and going unjudged.
• Cache: the same text is not asked about twice within five minutes.

An outside service#

A site required to use a particular guardrail — Azure Content Safety, Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a policy's services. It is asked over HTTP beside the patterns and before the judge; its "unsafe" blocks or flags like theirs, and a service that does not answer is what the policy's fail says.

policies:
  strict:
    fail: closed
    services:
      - name: lakera
        url: https://api.lakera.ai/v2/guard
        header_files: {Authorization: /etc/llmgw/secrets/lakera}   # "Bearer …", read each call
        body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
        blocked: flagged                 # where the answer says unsafe
        reason: breakdown.0.detector_type
        on: both                         # input, output or both
        action: block                    # or flag
        timeout: 2s

body is a template given .Text ({{json .Text}} writes it as a JSON string; empty sends {"input": …}). blocked and reason are dotted paths into the JSON answer, numbers for list items; true, a number above zero, a non-empty list or a string other than "false" is unsafe. On input the service reads the latest user turn as the model will, with what redact took out still out; on output the whole answer, once, at its end, as the client reads it — with restore, the values put back. Being at the end, a service can stop a streamed answer only where it is held back (stream: buffer_all): released a sentence at a time, the client has read it by then, and what the service finds ends the stream with content_filter, flags it and is recorded. The rule recorded is service:<name>:<reason>. The service is told the text only, never who sent it. Under fail: open, the default, a service that cannot be asked lets the text through — and a caller can make it fail, sending what it refuses to read; give fail: closed where the service must decide. Of an answer asked for with n above 1, the service reads the first.

Where a policy applies#

guard.default applies everywhere. Others are attached to a part of the tenancy tree, and apply to everything below it:

llmgw ctl guard attach project:Acme/finance/payroll finance
llmgw ctl guard attach key:key-7f3a baseline
llmgw ctl guard list
llmgw ctl guard detach project:Acme/finance/payroll finance

Policies narrow, as limits do. A policy attached lower can tighten what applies above it, never loosen it. Combined, they take:

• the strictest mode and failure;
• for each kind, the strictest action;
• every deny pattern and denied tool;
• only the tools every allow list allows;
• the smallest limits;
• no_store if any of them asks;
• every pinned prompt, widest first.

An attachment applies at once.

What a client sees#

A refused request is answered with 400:

{"error": {"type": "invalid_request_error", "code": "content_policy_violation",
           "message": "this request was refused by the gateway's policy (secret)"}}

It names the category, never the text, and costs nothing.

A stopped answer ends with finish_reason: "content_filter". A stream ends with that chunk and [DONE], and the engine is cancelled. The Responses API reports it as incomplete with incomplete_details.reason: "content_filter".

Headers: x-llmgw-guard (flagged, redacted, blocked) and x-llmgw-guard-policy say what was done.

Streams. A streamed answer is judged a sentence at a time. Text is held until a sentence, a line or 200 characters ends, released up to its last whitespace — so a value still arriving is never sent in two halves — and rewritten if needed. What was released cannot be taken back: a policy that must never let a word through uses stream: buffer_all, which holds the whole answer until it is judged.

Shadow mode#

mode: shadow judges and records — the usage row says shadow-blocked, shadow-redacted — and changes nothing. Run a new policy in shadow first, and read what it would have done before enforcing it.

What is recorded#

Each request's usage row has the action, policy, category and the time the checks took. The access log has the action, stage and policy. Neither has the text, a matched value, or the placeholder map: the map exists only in the gateway's memory while the request is served.

In the console#

sudo llmgw has a Guardrails view: each policy with its mode, what it does when it cannot judge, and what it checks; whether each judge model it leans on has replicas ready — a policy that fails closed refuses everything while its judge has none; where policies are attached, with A to attach one and Enter on a row to detach it; and what they did in the last day (W for an hour or a week), totals first and the newest requests after, each with the request id llmgw ctl request reads. The same is at GET /api/v1/guard and GET /api/v1/guard/events?since=24h, and on the portal's Guardrails page.