ollama-proxy/README.md

ollama-proxy

Multi-account reverse proxy for Ollama Cloud. Round-robin load balancing across multiple API keys with automatic failover on 429 (rate limit) and 5xx errors. Streaming (SSE + NDJSON) is passed through without buffering.

Listens on 127.0.0.1:11435 by default; exposes both the native Ollama API (/api/*) and the OpenAI-compatible API (/v1/*) that ollama.com serves natively. Designed to sit in front of OpenCode, ollama CLI, or any OpenAI-compatible client.

Build

make build

Produces ./ollama-proxy. Requires Go 1.21+ (uses log/slog).

Install

sudo make install
sudo systemctl enable --now ollama-proxy

This installs the binary to /usr/local/bin/ollama-proxy and the systemd unit to /etc/systemd/system/ollama-proxy.service, then reloads systemd.

Configure accounts

Accounts are stored in ~/.config/ollama-proxy/accounts.json (chmod 0600). Manage them via the CLI:

# Add an account (prompts for the key with echo disabled if not given on the CLI)
ollama-proxy accounts add --name work
ollama-proxy accounts add sk-xxxx-... --name personal

# List accounts (keys are masked)
ollama-proxy accounts list

# Remove by name or id prefix
ollama-proxy accounts remove work
ollama-proxy accounts remove 3e94140f

# Override the default upstream (rarely needed; default is https://ollama.com)
ollama-proxy accounts set-base-url https://ollama.com

API keys are created at https://ollama.com/settings/keys.

Run as a service

The systemd unit runs the proxy on 127.0.0.1:11435 and restarts on crash. Logs go to journald:

journalctl -u ollama-proxy -f

Configuration flags can be overridden via environment variables in the unit (or in a drop-in):

Variable	Default	Description
OLLAMA_PROXY_ADDR	127.0.0.1:11435	listen address
OLLAMA_PROXY_BASE_URL	https://ollama.com	upstream root
OLLAMA_PROXY_COOLDOWN	60s	per-account 429 cooldown window
OLLAMA_PROXY_RETRIES	3	max failover attempts
OLLAMA_PROXY_LOG_LEVEL	info	debug | info | warn | error

Use with OpenCode

The pool is wired into OpenCode via a separate config so that opencode and openlama expose different providers:

Command	Providers visible
opencode	marg, fireworks (your defaults)
openlama	ocp only (Ollama Cloud pool)

How it works

1. The global config at ~/.config/opencode/opencode.json contains marg + fireworks — no ocp.
2. A dedicated config at ~/.config/opencode/openlama.json contains ocp + "enabled_providers": ["ocp"]. OPENCODE_CONFIG merges with the global config, but enabled_providers overrides it so marg/fireworks are hidden.
3. The openlama launcher (at ~/.local/bin/openlama) sets OPENCODE_CONFIG, NO_PROXY, and execs opencode.

openlama.json

{
  "$schema": "https://opencode.ai/config.json",
  "enabled_providers": ["ocp"],
  "provider": {
    "ocp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Cloud (pool)",
      "options": {
        "baseURL": "http://127.0.0.1:11435/v1",
        "apiKey": "proxy"
      },
      "models": {
        "gpt-oss:20b": { "name": "gpt-oss:20b" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
      }
    }
  }
}

apiKey: "proxy" is a placeholder — the proxy ignores the client's Authorization header and substitutes the selected account's key. You must list models explicitly — opencode does not auto-fetch /v1/models for @ai-sdk/openai-compatible providers. To get the full list:

curl -sS http://127.0.0.1:11435/v1/models | jq -r '.data[].id'

NO_PROXY (important)

If your environment has HTTP_PROXY/HTTPS_PROXY set (e.g. http://127.0.0.1:2080), opencode's Bun runtime will route requests to the proxy through the HTTP proxy, resulting in 502 Bad Gateway. The openlama launcher sets this for you; for manual launches:

NO_PROXY=127.0.0.1,localhost,::1 OPENCODE_CONFIG=~/.config/opencode/openlama.json opencode

openlama launcher

A convenience wrapper is installed at ~/.local/bin/openlama. It sets NO_PROXY/no_proxy for localhost, sets OPENCODE_CONFIG to the ocp-only config, tries to start ollama-proxy if it's down, and execs opencode with all forwarded args:

openlama                                   # TUI with only ocp available
openlama run -m ocp/gpt-oss:120b "hello"   # non-interactive, specific model

Run /models in the OpenCode TUI to pick a model from the pool, then chat as usual.

Use with the ollama CLI

OLLAMA_HOST=127.0.0.1:11435 ollama run gpt-oss:20b

Endpoints

The proxy forwards (passthrough) any of these paths to ollama.com:

• Native Ollama: /api/chat, /api/generate, /api/tags, /api/show, /api/ps, /api/version, /api/delete
• OpenAI-compatible: /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings, /v1/files/*

Anything else returns 404.

How failover works

1. For each request, the balancer picks the next account round-robin.
2. If the upstream returns 429 before any bytes are sent to the client, the account is put in cooldown (default 60s) and the next account is tried, up to --retries attempts.
3. If the upstream returns 5xx, the next account is tried without cooldown (5xx may be transient).
4. Once a 2xx response has started streaming to the client, the request is committed: mid-stream upstream errors are surfaced as-is (a truncated response). The proxy never restarts a streamed request on a different account, because the client would see duplicated output.
5. If every account is in cooldown, the client receives 429 {"error":{"message":"all accounts rate-limited"}}.
6. If every account fails for non-429 reasons (network, 5xx), the client receives 502 {"error":{"message":"all accounts failed: …"}}.

Tests

go test ./...      # unit + integration
go vet ./...
gofmt -l .

The integration tests build the binary and exercise the full HTTP stack against an httptest upstream.

License

MIT.