The Agent Files

Field Report // NO. 007

Stand up a LiteLLM gateway on a VPS: the install, the chain API, and the env-var trap

25 July 2026· 9 min read· Case № 007
hermeslitellminfrastructuremodels

How to install LiteLLM on a VPS with PostgreSQL, wire provider credentials, add a four-leg model chain through the REST API, and avoid the STORE_MODEL_IN_DB environment-variable trap that made my models vanish on restart.

The first real request I sent through my new gateway came back with an error. HTTP 422, a context-length problem on the primary model. Then the gateway did the thing I had built it for: it stepped to the next model in the chain and answered in 1.2 seconds. Fallback routing under a real failure, not a simulation.

That is the outcome this article reproduces. By the end you will be able to stand up a LiteLLM gateway on your own VPS — install it with PostgreSQL as the backing store, wire provider credentials, add a four-leg model chain through the REST API, and avoid the environment-variable trap that made my models look saved and then vanish on restart.

Why a gateway at all

Before LiteLLM, my agent’s model routing lived in a models.yaml file with declarative fallback chains: try MiniMax, then OpenCode Go, then OpenRouter. That solved config consolidation, but not operations. I had no single place that showed what I was spending across providers. The fallback chain was static — if a provider was slow or erroring, the agent hammered the same endpoint before moving on. LiteLLM’s router handles fallback at the transport layer with health awareness. That is what I wanted.

Step 1 — Install

LiteLLM runs as a raw Python venv on the VPS, no Docker. PostgreSQL comes from apt. This is the path I verified on a Hetzner VPS (Debian, 15 GB RAM):

# System packages: PostgreSQL + Python 3.11
sudo apt update
sudo apt install -y postgresql python3.11 python3.11-venv

# Service directory and venv
mkdir -p ~/services/litellm && cd ~/services/litellm
chmod 0755 ~/services/litellm        # prisma (first start) needs a traversable parent
python3.11 -m venv .venv
.venv/bin/pip install 'litellm[proxy]'   # litellm 1.93.0 in this build

The official quick start uses uv tool install 'litellm[proxy]' (Python 3.10+). A project venv with pip install 'litellm[proxy]' is equivalent and is what this build runs. One gotcha: pip install 'litellm[proxy]' does not pull the Prisma client that the proxy needs for a database-backed install. The first start generates it — if the parent directory is not 0755, that step fails with Permission denied. Make the directory traversable before the venv exists.

Create the database role and database (PostgreSQL, loopback-only auth):

sudo -u postgres createuser --login --pwprompt litellm
sudo -u postgres createdb -O litellm litellm

Then the systemd user unit so the gateway survives reboots. This is the unit this build runs under:

[Unit]
Description=LiteLLM gateway (Tailscale pilot)
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/<your-user>/services/litellm
ExecStart=/home/<your-user>/services/litellm/start.sh
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/<your-user>/services/litellm/logs /home/<your-user>/services/litellm/backups
StandardOutput=journal
StandardError=journal
SyslogIdentifier=litellm

[Install]
WantedBy=default.target

The start script sources .env, binds the gateway to your Tailscale interface address (so the admin UI is reachable from your Tailnet, not just localhost), and launches the proxy:

#!/usr/bin/env bash
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
cd "$HERE"

if [[ -f .env ]]; then
  set -a
  . ./.env
  set +a
else
  echo "FATAL: $HERE/.env missing" >&2
  exit 1
fi

export HOST="${LITELLM_BIND:-your-tailscale-ip}"
export PORT=${PORT:-4000}
export LITELLM_LOG="${LITELLM_LOG:-INFO}"
export DATABASE_URL="$LITELLM_DATABASE_URL"

exec ./.venv/bin/litellm \
  --config "$HERE/config/config.yaml" \
  --host "$HOST" \
  --port "$PORT" \
  --num_workers 1

Enable and start:

systemctl --user daemon-reload
systemctl --user enable --now litellm.service

Step 2 — The config file

The gateway config lives at config/config.yaml. This is the working version from the build — the parts that matter are general_settings:

router_settings:
  num_retries: 0
  timeout: 60
  enable_pre_call_checks: false

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/LITELLM_DATABASE_URL
  store_model_in_db: True
  disable_spend_logs: False
  telemetry: False
  alerting: []
  proxy_listen: your-tailscale-ip:4000
  database_connection_pool_limit: 10

litellm_settings:
  drop_params: true
  set_verbose: false
  redact_messages_in_exceptions: true
  request_timeout: 60

Three keys matter at install time:

  • master_key: os.environ/LITELLM_MASTER_KEY — the admin key, read from .env, never written in the file.
  • database_url: os.environ/LITELLM_DATABASE_URL — the PostgreSQL connection string, also from .env.
  • store_model_in_db: True — persist models to PostgreSQL instead of keeping them in memory only. This is the flag the whole rest of the article is about.

The .env file (mode 0600, gitignored) holds the secrets:

LITELLM_MASTER_KEY=<your-master-key>
LITELLM_DATABASE_URL=postgresql://litellm:<db-password>@127.0.0.1:5432/litellm
LITELLM_LOG=INFO

Provider credentials (MiniMax, OpenCode Go, Alibaba, OpenRouter API keys) go in through the LiteLLM admin UI at https://your-tailscale-ip:4000/ui. The UI encrypts them at rest and they are not readable back out of the database, which is the right behavior.

Step 3 — The env-var trap that ate my models

This cost me the most time of the whole setup. Here is the trap, so you do not repeat it.

I had set store_model_in_db: true in config.yaml. The gateway was also loading a .env that contained this line:

STORE_MODEL_IN_DB=False

Environment variables take precedence over config files in LiteLLM. The line in .env silently flipped my config-file true back to False. The gateway ran in memory-only mode.

The symptoms were confusing:

  • POST /model/new returned HTTP 200, but the response said db_model: false
  • LiteLLM_ModelTable stayed at 0 rows after every API call
  • After every restart, every manually added model was gone

I also misread db_model: false as “not persisted.” It actually means “the in-memory router does not know this came from the database.” The model had been written to LiteLLM_ProxyModelTable — I was checking the wrong table, which made it look like nothing was being saved.

The fix is one line: remove STORE_MODEL_IN_DB=False from .env (and do not export it in start.sh). Let the config file’s store_model_in_db: True be the only source of truth. After a restart, persistence works: db_model: true on every reloaded row confirms it survived.

There are two database tables, and the difference matters for debugging:

  • LiteLLM_ProxyModelTable — the active table. POST /model/new writes here.
  • LiteLLM_ModelTable — legacy, not updated by the API. Checking this after a create makes it look like nothing saved.

Step 4 — Chains through the REST API

There is no “chain object” in LiteLLM. A chain alias is N rows in the database that share the same model_name, and the row order is the fallback priority. The first row is primary; each subsequent row is a fallback tried in order.

So the general-heavy alias — the tier my heavy reasoning profiles use — is four rows, all with model_name: general-heavy:

  1. minimax/minimax-m3 — MiniMax Token Plan credential (inline key)
  2. opencodego/deepseek-v4-pro — OpenCode-Go credential
  3. alibaba/qwen3.7-max — Alibaba Cloud credential
  4. openrouter/deepseek/deepseek-v4-pro — OpenRouter credential

Order is set by the order field on litellm_params (1, 2, 3, 4). The build script that created all the chains passed it in the POST body, one leg at a time:

MK=$(grep '^LITELLM_MASTER_KEY=' ~/services/litellm/.env | cut -d= -f2)
GW="http://your-tailscale-ip:4000"

# One POST per leg, in priority order. The order field makes the chain.
curl -s -X POST \
  -H "Authorization: Bearer $MK" \
  -H "Content-Type: application/json" \
  -d '{"model_name":"general-heavy","litellm_params":{"model":"minimax/minimax-m3","custom_llm_provider":"minimax","order":1}}' \
  "$GW/model/new"

curl -s -X POST \
  -H "Authorization: Bearer $MK" \
  -H "Content-Type: application/json" \
  -d '{"model_name":"general-heavy","litellm_params":{"model":"opencodego/deepseek-v4-pro","custom_llm_provider":"custom_openai","litellm_credential_name":"OpenCode-Go","order":2}}' \
  "$GW/model/new"

curl -s -X POST \
  -H "Authorization: Bearer $MK" \
  -H "Content-Type: application/json" \
  -d '{"model_name":"general-heavy","litellm_params":{"model":"alibaba/qwen3.7-max","custom_llm_provider":"custom_openai","litellm_credential_name":"Alibaba Cloud","order":3}}' \
  "$GW/model/new"

curl -s -X POST \
  -H "Authorization: Bearer $MK" \
  -H "Content-Type: application/json" \
  -d '{"model_name":"general-heavy","litellm_params":{"model":"openrouter/deepseek/deepseek-v4-pro","custom_llm_provider":"openrouter","litellm_credential_name":"OpenRouter","order":4}}' \
  "$GW/model/new"

To reorder an existing chain later, PATCH /model/<model_info.id>/update with the new order on each row — do not delete and re-POST; that churns the cache rows unnecessarily.

Two notes from this build:

  • Use the API, not the database. I tried direct PostgreSQL edits early on and created phantom rows the in-memory router never knew about. LiteLLM keeps an in-memory router alongside the database; SQL bypasses it. POST /model/new and POST /model/delete are the control surface.
  • Some upstreams reject the provider prefix. On this build, OpenCode Go and Alibaba rows were later stripped from opencodego/deepseek-v4-pro to deepseek-v4-pro (and alibaba/qwen3.7-max to qwen3.7-max) because the upstream APIs rejected the prefixed names. If a row 422s or 404s at the upstream, strip the provider prefix. The payloads above are the canonical documented shapes; the strip is the known fix if yours misbehaves.

Step 5 — Verify

Reads are how you confirm state. The master key sees everything:

# Health
curl -s http://your-tailscale-ip:4000/health/liveliness

# What is loaded right now
curl -s -H "Authorization: Bearer $MK" http://your-tailscale-ip:4000/v1/models

# The alias's legs, with db_model state and order
curl -s -H "Authorization: Bearer $MK" "http://your-tailscale-ip:4000/model/info" | \
  python3 -c "
import json, sys
d = json.load(sys.stdin)
legs = [m for m in d['data'] if m['model_name'] == 'general-heavy']
for m in legs:
    p = m['litellm_params']
    print(f\"  order={p.get('order')} model={p.get('model')} db_model={m['model_info']['db_model']}\")
"

# Delete a row — takes the id, not the model_name
curl -s -X POST -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{"id":"<model_info.id from /model/info>"}' \
  "$GW/model/delete"

The live request test — and the reveal. After the chain was in place, I sent one completion to the alias:

curl -s -X POST -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{"model":"general-heavy","messages":[{"role":"user","content":"Reply with just PONG"}],"max_tokens":10}' \
  "$GW/v1/chat/completions"

The primary leg (minimax/minimax-m3) returned HTTP 422 — a context-length error, because MiniMax’s 200k context tier requires a different model name than the one in the primary slot. The router stepped to leg two (opencodego/deepseek-v4-pro), which returned 200 in 1.2 seconds. The chain behaved correctly under a real failure condition, not a simulated one. To see which leg served, check the gateway journal:

journalctl --user -u litellm.service --since '10 seconds ago' | grep 'completion() model='

The recipe

Everything above, in one place:

  1. Install: python3.11 -m venv .venv + .venv/bin/pip install 'litellm[proxy]'; apt install postgresql; create role and database litellm; systemd user unit running start.sh.
  2. Configure: config/config.yaml with general_settings.store_model_in_db: True, master_key and database_url from .env; proxy_listen: your-tailscale-ip:4000.
  3. Avoid the trap: no STORE_MODEL_IN_DB line anywhere in .env or start.sh. Env vars beat config files; the config file must be the only source of truth for store_model_in_db.
  4. Credentials: through the admin UI at https://your-tailscale-ip:4000/ui — encrypted at rest.
  5. Chains: POST /model/new once per leg, same model_name, litellm_params.order 1..N; delete with POST /model/delete + {"id": ...}; inspect with GET /v1/models and GET /model/info.
  6. Verify: PONG completion against the alias, then journalctl --user -u litellm.service | grep 'completion() model=' to confirm which leg served.

The takeaway

LiteLLM’s chains are rows, not objects: the same model_name across N rows, priority by order. The REST API is the control surface — never the database. And the one flag that decides whether your work survives a restart is store_model_in_db, which the config file sets and the environment can silently override. Remove the env var. Let the file win.

What is still open in my setup: provider token plans (MiniMax, OpenCode Go, Alibaba) meter on 5-hour rolling windows, and LiteLLM’s budget model is a flat monthly cap per key. The two do not map cleanly. I run a conservative monthly cap plus provider-dashboard monitoring, and budget enforcement against rolling windows remains unsolved. That is the next article.

Want the other half of the story first? See Choosing a Hermes Model for how the tier names on the left side of those payloads — heavy, medium, simple — get assigned to profiles in models.yaml, and how to change a profile’s tier in one line.

← All transmissions