3963eede70
Go reverse proxy for ollama.com that balances requests across multiple API keys with round-robin and failover on 429/5xx. Exposes both native Ollama API (/api/*) and OpenAI-compatible (/v1/*) passthrough. - Round-robin balancer with per-account cooldown (60s default) - Pre-stream failover: 429 → cooldown + next account; 5xx → next account - Streaming invariant: once 2xx starts streaming, no account switch - SSE (text/event-stream) and NDJSON passthrough with http.Flusher - CLI: accounts add/list/remove/set-base-url, serve, version - Accounts stored in ~/.config/ollama-proxy/accounts.json (chmod 0600) - systemd unit (User=dueattendant149, 127.0.0.1:11435, Restart=always) - 43 tests (unit + integration with httptest upstream) - opencode integration: custom provider 'ocp' with explicit model list - Requires NO_PROXY=127.0.0.1,localhost when HTTP_PROXY is set
28 lines
1.4 KiB
Go
28 lines
1.4 KiB
Go
package proxy
|
|
|
|
// retry.go holds the failover helpers used by handler.go. The retry loop itself
|
|
// lives inside Handler.proxyWithFailover (see handler.go) because it needs
|
|
// tight control over the moment a response starts streaming vs. is rejected
|
|
// pre-stream. This file documents the invariants the loop must maintain.
|
|
|
|
// Streaming invariant
|
|
// ===================
|
|
// Once an upstream account has returned a 2xx status AND the proxy has started
|
|
// writing the response body to the client (a single byte flushed), the request
|
|
// is committed: we MUST NOT switch accounts for that request. Any mid-stream
|
|
// upstream error is surfaced to the client as-is (truncated response); we never
|
|
// attempt to "restart" a streamed request on a different account, because the
|
|
// client has already received partial output and a retry would duplicate it.
|
|
//
|
|
// Pre-stream failover
|
|
// -------------------
|
|
// The window in which we CAN retry on another account is exactly:
|
|
// 1. The upstream HTTP request returned an error (network, timeout, EOF
|
|
// before any response).
|
|
// 2. The upstream returned 429 (Too Many Requests) — we mark the account
|
|
// cooldown and try the next.
|
|
// 3. The upstream returned 5xx — we try the next account WITHOUT marking
|
|
// cooldown (5xx may be transient and is not necessarily a rate limit).
|
|
//
|
|
// Once copyResponse has called WriteHeader, no further retries are possible
|
|
// for this request.
|