I wanted to use my local models (Ollama, vLLM) with tools that only speak OpenAI or Anthropic API formats — Claude Code, OpenCode, Cursor, and custom agents. Running LiteLLM felt overkill. So I built Relay: a single Go binary (~12 MB, static) that bridges any local backend to the API formats your tools already speak.
Relay is a local AI gateway that exposes endpoints compatible with OpenAI Chat Completions and Anthropic Messages while connecting to your local models.
Architecture
┌──────────────────────┐
curl ─┐ │ │
opencode ─┤ │ relay ├──→ Ollama
Claude Code ─┤ │ (local AI gateway) │──→ OpenAI-compat (vLLM/LiteLLM)
Cursor ─┘ │ │──→ CLI agent (claude/opencode)
└──────────────────────┘The idea is simple: a network-level adapter that sits between your tools and your models. No config files needed to get started, no Python runtime, no containers.
Key Features
Dual-Protocol API
Single binary serves both API formats:
| Protocol | Endpoint | Example |
|---|---|---|
| OpenAI | POST /api/v1/chat/completions | Works with any OpenAI SDK |
| OpenAI (legacy) | POST /api/v1/generate | Simplified generation |
| Anthropic | POST /v1/messages | Anthropic Messages API |
| Models | GET /api/v1/models | List available models |
| Health | GET /api/v1/health | No auth required |
Interchangeable Backends
| Backend | Use Case |
|---|---|
| ollama | Local models via Ollama API |
| openai-compat | vLLM, LiteLLM, or any OpenAI-compatible server |
| claude | Use your claude CLI subscription as a backend |
| opencode | Use OpenCode CLI as a backend |
| echo | Testing backend — always replies with ECHO: <message> |
MCP Server Mode
Relay also exposes 5 MCP tools via stdio:
generate_key,revoke_key,list_keys— API key managementlist_models— query available backendshealth— check server status
Perfect for agents that need to spin up a local inference endpoint dynamically.
Streaming SSE
Real-time streaming for both OpenAI SSE and Anthropic SSE event formats.
Tech Stack
| Component | Choice | Why |
|---|---|---|
| Language | Go 1.26+ | Single static binary, no runtime deps |
| API Server | net/http + gorilla/mux | Standard, well-tested |
| Config | YAML + CLI flags | Quick overrides without editing files |
| Auth | Auto-generated API keys | Zero-config security |
| Persistence | ~/.localmcp/keys.json | Keys survive restarts |
| Size | ~12 MB static binary | No Python, no Node, no containers |
| Tests | Go test suite | Core API and backend tests |
Quick Start
git clone https://github.com/fxckcode/relay.git
cd relay
make build
./bin/localmcpThat’s it. The API key auto-generates on first run. Default listen is 127.0.0.1:8080.
# Test it
KEY=$(./bin/localmcp --show-key)
curl -X POST http://127.0.0.1:8080/api/v1/chat/completions \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"model":"echo","messages":[{"role":"user","content":"Hello"}]}'The echo backend replies immediately — perfect for testing. With Ollama running, switch to "model":"llama3.2".
Why Not LiteLLM / Ollama Built-in?
LiteLLM is a full Python project with heavy dependencies. Relay is a single Go binary — no pip install, no virtualenv, no Python version management. It also adds the CLI agent backend (use your claude CLI subscription as an API), which neither LiteLLM nor Ollama can do.
Ollama’s built-in OpenAI compatibility is solid but limited to Ollama models. Relay chains to any backend.
The project is MIT licensed and open on GitHub. Contributions welcome.