ollama-proxy/README.md

# ollama-proxy

Multi-account reverse proxy for [Ollama Cloud](https://ollama.com). Round-robin load balancing across multiple API keys with automatic failover on 429 (rate limit) and 5xx errors. Streaming (SSE + NDJSON) is passed through without buffering.

Listens on `127.0.0.1:11435` by default; exposes both the native Ollama API (`/api/*`) and the OpenAI-compatible API (`/v1/*`) that `ollama.com` serves natively. Designed to sit in front of OpenCode, `ollama` CLI, or any OpenAI-compatible client.

## Build

```sh
make build
```

Produces `./ollama-proxy`. Requires Go 1.21+ (uses `log/slog`).

## Install

```sh
sudo make install
sudo systemctl enable --now ollama-proxy
```

This installs the binary to `/usr/local/bin/ollama-proxy` and the systemd unit to `/etc/systemd/system/ollama-proxy.service`, then reloads systemd.

## Configure accounts

Accounts are stored in `~/.config/ollama-proxy/accounts.json` (chmod 0600). Manage them via the CLI:

```sh
# Add an account (prompts for the key with echo disabled if not given on the CLI)
ollama-proxy accounts add --name work
ollama-proxy accounts add sk-xxxx-... --name personal

# List accounts (keys are masked)
ollama-proxy accounts list

# Remove by name or id prefix
ollama-proxy accounts remove work
ollama-proxy accounts remove 3e94140f

# Override the default upstream (rarely needed; default is https://ollama.com)
ollama-proxy accounts set-base-url https://ollama.com
```

API keys are created at <https://ollama.com/settings/keys>.

## Run as a service

The systemd unit runs the proxy on `127.0.0.1:11435` and restarts on crash. Logs go to journald:

```sh
journalctl -u ollama-proxy -f
```

Configuration flags can be overridden via environment variables in the unit (or in a drop-in):

| Variable                  | Default             | Description                       |
| ------------------------- | ------------------- | --------------------------------- |
| `OLLAMA_PROXY_ADDR`       | `127.0.0.1:11435`   | listen address                    |
| `OLLAMA_PROXY_BASE_URL`   | `https://ollama.com`| upstream root                      |
| `OLLAMA_PROXY_COOLDOWN`   | `60s`               | per-account 429 cooldown window    |
| `OLLAMA_PROXY_RETRIES`    | `3`                 | max failover attempts              |
| `OLLAMA_PROXY_LOG_LEVEL` | `info`              | `debug` \| `info` \| `warn` \| `error` |

## Use with OpenCode

The pool is wired into OpenCode via a **separate config** so that `opencode` and `openlama` expose different providers:

| Command | Providers visible |
|---------|-------------------|
| `opencode` | `marg`, `fireworks` (your defaults) |
| `openlama` | `ocp` only (Ollama Cloud pool) |

### How it works

1. The global config at `~/.config/opencode/opencode.json` contains `marg` + `fireworks` — **no `ocp`**.
2. A dedicated config at `~/.config/opencode/openlama.json` contains `ocp` + `"enabled_providers": ["ocp"]`. `OPENCODE_CONFIG` merges with the global config, but `enabled_providers` overrides it so `marg`/`fireworks` are hidden.
3. The `openlama` launcher (at `~/.local/bin/openlama`) sets `OPENCODE_CONFIG`, `NO_PROXY`, and execs `opencode`.

### openlama.json

```json
{
  "$schema": "https://opencode.ai/config.json",
  "enabled_providers": ["ocp"],
  "provider": {
    "ocp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Cloud (pool)",
      "options": {
        "baseURL": "http://127.0.0.1:11435/v1",
        "apiKey": "proxy"
      },
      "models": {
        "gpt-oss:20b": { "name": "gpt-oss:20b" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
      }
    }
  }
}
```

`apiKey: "proxy"` is a placeholder — the proxy ignores the client's `Authorization` header and substitutes the selected account's key. **You must list models explicitly** — opencode does not auto-fetch `/v1/models` for `@ai-sdk/openai-compatible` providers. To get the full list:

```sh
curl -sS http://127.0.0.1:11435/v1/models | jq -r '.data[].id'
```

### NO_PROXY (important)

If your environment has `HTTP_PROXY`/`HTTPS_PROXY` set (e.g. `http://127.0.0.1:2080`), opencode's Bun runtime will route requests to the proxy *through* the HTTP proxy, resulting in `502 Bad Gateway`. The `openlama` launcher sets this for you; for manual launches:

```sh
NO_PROXY=127.0.0.1,localhost,::1 OPENCODE_CONFIG=~/.config/opencode/openlama.json opencode
```

### `openlama` launcher

A convenience wrapper is installed at `~/.local/bin/openlama`. It sets `NO_PROXY`/`no_proxy` for localhost, sets `OPENCODE_CONFIG` to the ocp-only config, tries to start `ollama-proxy` if it's down, and execs `opencode` with all forwarded args:

```sh
openlama                                   # TUI with only ocp available
openlama run -m ocp/gpt-oss:120b "hello"   # non-interactive, specific model
```

Run `/models` in the OpenCode TUI to pick a model from the pool, then chat as usual.

## Use with the `ollama` CLI

```sh
OLLAMA_HOST=127.0.0.1:11435 ollama run gpt-oss:20b
```

## Endpoints

The proxy forwards (passthrough) any of these paths to `ollama.com`:

- Native Ollama: `/api/chat`, `/api/generate`, `/api/tags`, `/api/show`, `/api/ps`, `/api/version`, `/api/delete`
- OpenAI-compatible: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings`, `/v1/files/*`

Anything else returns 404.

## How failover works

1. For each request, the balancer picks the next account round-robin.
2. If the upstream returns `429` before any bytes are sent to the client, the account is put in cooldown (default 60s) and the next account is tried, up to `--retries` attempts.
3. If the upstream returns `5xx`, the next account is tried **without** cooldown (5xx may be transient).
4. Once a 2xx response has started streaming to the client, the request is committed: mid-stream upstream errors are surfaced as-is (a truncated response). The proxy never restarts a streamed request on a different account, because the client would see duplicated output.
5. If every account is in cooldown, the client receives `429 {"error":{"message":"all accounts rate-limited"}}`.
6. If every account fails for non-429 reasons (network, 5xx), the client receives `502 {"error":{"message":"all accounts failed: …"}}`.

## Tests

```sh
go test ./...      # unit + integration
go vet ./...
gofmt -l .
```

The integration tests build the binary and exercise the full HTTP stack against an `httptest` upstream.

## License

MIT.