> ## Documentation Index
> Fetch the complete documentation index at: https://docs.the40thieves.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference gateway

> Stand up the optional generative tier — reflect synthesis, the challenge red-team, the sleep-time plane — via a self-hosted LiteLLM (or any OpenAI-compatible endpoint).

The engine's retrieval never needs an LLM — embeddings, BM25, graph expansion, and
`vault_context` all run locally. The **inference gateway** is the one optional seam
to a *text-generating* model, and everything that writes or judges prose routes
through it by **role**:

| Role | Used by |
| - | - |
| `synthesize` | `reflect` (grounded answers with provenance), the sleep-time `plane` consolidation |
| `judge` | citation-inference stage 2, preference extraction |
| `extract` | reserved for extraction pipelines |

**Absence is a supported state.** With no gateway configured, `reflect` returns
recall with `available: false`, `knowledge_challenge` reports unavailable, the
evaluator runs its deterministic rules without the judge layer, and the `plane`
scheduler idles. Nothing else degrades.

## How the engine finds it

Two environment variables — there is deliberately no config-file block, so the
gateway can differ per launcher:

```
RELIQUARY_TC_GATEWAY_URL=http://127.0.0.1:4000
RELIQUARY_TC_GATEWAY_TOKEN=<optional bearer, e.g. a LiteLLM virtual key>
```

The engine speaks the OpenAI-compatible chat API to that base URL and requests
**model names equal to the role names** (`extract`, `synthesize`, `judge`). Your
gateway's only job is to answer those three aliases and route them to real models.
The resolved `provider:model` comes back with every response and is stamped into
the attestation of anything the gateway produces (e.g. a persisted reflection's
`source_model`), so model swaps stay auditable.

## Recommended setup: LiteLLM in Docker

One container plus a config file mapping the role aliases. All-local example
(Ollama backend — zero keys, nothing leaves the machine):

```yaml theme={null}
# litellm-config.yaml
model_list:
  - model_name: extract
    litellm_params: { model: "ollama/qwen2.5:7b-instruct", api_base: "http://host.docker.internal:11434" }
  - model_name: judge
    litellm_params: { model: "ollama/qwen2.5:7b-instruct", api_base: "http://host.docker.internal:11434" }
  - model_name: synthesize
    litellm_params: { model: "ollama/qwen2.5:7b-instruct", api_base: "http://host.docker.internal:11434" }
litellm_settings:
  drop_params: true
```

```sh theme={null}
docker run -d --name litellm-gateway --restart unless-stopped \
  -p 127.0.0.1:4000:4000 \
  -v /absolute/path/litellm-config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000
```

Then set `RELIQUARY_TC_GATEWAY_URL=http://127.0.0.1:4000` in your MCP client's
server entry (the `env` block) and restart the client.

A plain `http://` gateway URL on loopback needs nothing else. A non-loopback `http://` URL (for example a
LiteLLM container at `http://litellm:4000`) must resolve to a private address and should be named in
`network.plainHttpHosts`; a public address is refused. See
[Plain-http provider endpoints](https://docs.the40thieves.dev/reliquary-tc/configuration/config-yaml.md#plain-http-provider-endpoints-networkplainhttphosts).

Deployment notes, learned the hard way:

* **Pin the image.** LiteLLM's PyPI channel was compromised in early 2026; the
  container channel is the safe distribution, but pin a digest
  (`ghcr.io/berriai/litellm@sha256:…` from `docker image inspect`) rather than a
  floating tag, and keep the container free of credentials it doesn't need — an
  all-local config holds **zero keys**, which makes the supply-chain blast radius
  zero.
* **Bind loopback-only** (`-p 127.0.0.1:4000:4000`) unless you add a master key.
* On Windows git-bash, prefix `docker run` with `MSYS_NO_PATHCONV=1` or the
  container-side `/app/...` paths get mangled into Windows paths.

### Verify

```sh theme={null}
curl -s http://127.0.0.1:4000/v1/models        # → extract, judge, synthesize
```

Then call the `reflect` tool with any query: a working gateway returns a grounded
`synthesis` with chunk provenance; a missing/broken one returns sources with
`available: false` (that distinction is your health check).

## Choosing models per role

The role→model map is a policy decision, and the yaml is the whole switch —
reliquary-tc never changes:

* **All-local** (above): private, free, offline. Trade-off: a 7B-class
  `synthesize` produces grounded but imperfect prose — treat syntheses as drafts
  and verify against the returned sources. The engine is designed for weak
  judges: the evaluator's judge can only lower eligibility verdicts, and a parse
  failure disables that layer rather than corrupting anything.
* **Hybrid**: point `synthesize` at a hosted model
  (`model: "anthropic/claude-sonnet-5-5"` + the API key in the **container's** env)
  and keep `judge`/`extract` local. Best quality where it shows most — but
  retrieved vault content leaves the machine on synthesis calls. That is a privacy
  boundary; cross it deliberately.
* Any OpenAI-compatible endpoint works in place of LiteLLM if it can serve the
  three aliases (including Ollama directly, using model aliases created via
  Modelfiles).

## What wakes up

With the gateway live: `reflect` synthesis and `mode: "challenge"`,
`knowledge_challenge` (red-team against decision history), gateway-gated
preference extraction, and citation-inference stage 2.

**The 4-hour `plane` sleep-time consolidation is the one exception — it is
opt-in, separate from the gateway itself.** A gateway alone is not
consent to unattended, whole-vault LLM calls, so `plane.enabled` also defaults
to `false` and must be set explicitly:

```json theme={null}
"plane": { "enabled": true }
```

A deployment that has a gateway configured and never set this key gets a
one-line notice on the server's stderr at boot, naming exactly this fix. The
sleep-time half also runs offline via `reliquary-tc reflect` (CLI), which gates
on the same env vars but not on `plane.enabled` — that flag only gates the
scheduled/ambient path.

The **episode-eligibility evaluator is deliberately not on this list.** It used
to consult the `judge` role, and no longer does: measurement showed the judge's
entire effect was reproducing a `status = "error"` comparison, so it was removed
and the pass is now purely deterministic. It therefore runs identically with or
without a gateway, and acquires no network dependency at all.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.