ollama-proxy/README.md

# ollama-proxy

Multi-account reverse proxy for [Ollama Cloud](https://ollama.com). Round-robin load balancing across multiple API keys with automatic failover on 429 (rate limit) and 5xx errors. Streaming (SSE + NDJSON) is passed through without buffering.

Listens on `127.0.0.1:11435` by default; exposes both the native Ollama API (`/api/*`) and the OpenAI-compatible API (`/v1/*`) that `ollama.com` serves natively. Designed to sit in front of OpenCode, `ollama` CLI, or any OpenAI-compatible client.

## Build

```sh
make build
```

Produces `./ollama-proxy`. Requires Go 1.21+ (uses `log/slog`).

## Install

```sh
sudo make install
sudo systemctl enable --now ollama-proxy
```

This installs the binary to `/usr/local/bin/ollama-proxy` and the systemd unit to `/etc/systemd/system/ollama-proxy.service`, then reloads systemd.

## Configure accounts

Accounts are stored in `~/.config/ollama-proxy/accounts.json` (chmod 0600). Manage them via the CLI:

```sh
# Add an account (prompts for the key with echo disabled if not given on the CLI)
ollama-proxy accounts add --name work
ollama-proxy accounts add sk-xxxx-... --name personal

# List accounts (keys are masked)
ollama-proxy accounts list

# Remove by name or id prefix
ollama-proxy accounts remove work
ollama-proxy accounts remove 3e94140f

# Override the default upstream (rarely needed; default is https://ollama.com)
ollama-proxy accounts set-base-url https://ollama.com
```

API keys are created at <https://ollama.com/settings/keys>.

## Run as a service

The systemd unit runs the proxy on `127.0.0.1:11435` and restarts on crash. Logs go to journald:

```sh
journalctl -u ollama-proxy -f
```

Configuration flags can be overridden via environment variables in the unit (or in a drop-in):

| Variable                  | Default             | Description                       |
| ------------------------- | ------------------- | --------------------------------- |
| `OLLAMA_PROXY_ADDR`       | `127.0.0.1:11435`   | listen address                    |
| `OLLAMA_PROXY_BASE_URL`   | `https://ollama.com`| upstream root                      |
| `OLLAMA_PROXY_COOLDOWN`   | `60s`               | per-account 429 cooldown window    |
| `OLLAMA_PROXY_RETRIES`    | `3`                 | max failover attempts              |
| `OLLAMA_PROXY_LOG_LEVEL` | `info`              | `debug` \| `info` \| `warn` \| `error` |

## Use with OpenCode

Add a custom provider to `~/.config/opencode/opencode.json`:

```json
"ocp": {
  "npm": "@ai-sdk/openai-compatible",
  "name": "Ollama Cloud (pool)",
  "options": {
    "baseURL": "http://127.0.0.1:11435/v1",
    "apiKey": "proxy"
  },
  "models": {
    "gpt-oss:20b": { "name": "gpt-oss:20b" },
    "gpt-oss:120b": { "name": "gpt-oss:120b" },
    "kimi-k2.6": { "name": "kimi-k2.6" }
  }
}
```

`apiKey: "proxy"` is a placeholder — the proxy ignores the client's `Authorization` header and substitutes the selected account's key. **You must list models explicitly** — opencode does not auto-fetch `/v1/models` for `@ai-sdk/openai-compatible` providers. To get the full list:

```sh
curl -sS http://127.0.0.1:11435/v1/models | jq -r '.data[].id'
```

### NO_PROXY (important)

If your environment has `HTTP_PROXY`/`HTTPS_PROXY` set (e.g. `http://127.0.0.1:2080`), opencode's Bun runtime will route requests to the proxy *through* the HTTP proxy, resulting in `502 Bad Gateway`. Exclude localhost from proxying:

```sh
# ~/.config/environment.d/no-proxy-local.conf
NO_PROXY=127.0.0.1,localhost,::1
no_proxy=127.0.0.1,localhost,::1
```

Or launch opencode with `NO_PROXY=127.0.0.1,localhost opencode`.

Run `/models` in the OpenCode TUI to pick a model from the pool, then chat as usual.

## Use with the `ollama` CLI

```sh
OLLAMA_HOST=127.0.0.1:11435 ollama run gpt-oss:20b
```

## Endpoints

The proxy forwards (passthrough) any of these paths to `ollama.com`:

- Native Ollama: `/api/chat`, `/api/generate`, `/api/tags`, `/api/show`, `/api/ps`, `/api/version`, `/api/delete`
- OpenAI-compatible: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings`, `/v1/files/*`

Anything else returns 404.

## How failover works

1. For each request, the balancer picks the next account round-robin.
2. If the upstream returns `429` before any bytes are sent to the client, the account is put in cooldown (default 60s) and the next account is tried, up to `--retries` attempts.
3. If the upstream returns `5xx`, the next account is tried **without** cooldown (5xx may be transient).
4. Once a 2xx response has started streaming to the client, the request is committed: mid-stream upstream errors are surfaced as-is (a truncated response). The proxy never restarts a streamed request on a different account, because the client would see duplicated output.
5. If every account is in cooldown, the client receives `429 {"error":{"message":"all accounts rate-limited"}}`.
6. If every account fails for non-429 reasons (network, 5xx), the client receives `502 {"error":{"message":"all accounts failed: …"}}`.

## Tests

```sh
go test ./...      # unit + integration
go vet ./...
gofmt -l .
```

The integration tests build the binary and exercise the full HTTP stack against an `httptest` upstream.

## License

MIT.