Relay — Local AI Gateway as an MCP Server

I wanted to use my local models (Ollama, vLLM) with tools that only speak OpenAI or Anthropic API formats — Claude Code, OpenCode, Cursor, and custom agents. Running LiteLLM felt overkill. So I built Relay: a single Go binary (~12 MB, static) that bridges any local backend to the API formats your tools already speak.

Relay is a local AI gateway that exposes endpoints compatible with OpenAI Chat Completions and Anthropic Messages while connecting to your local models.

---

Architecture

                       ┌──────────────────────┐
  curl ─┐              │                      │
  opencode ─┤          │        relay         ├──→ Ollama
  Claude Code ─┤       │   (local AI gateway) │──→ OpenAI-compat (vLLM/LiteLLM)
  Cursor ─┘            │                      │──→ CLI agent (claude/opencode)
                       └──────────────────────┘

The idea is simple: a network-level adapter that sits between your tools and your models. No config files needed to get started, no Python runtime, no containers.

---

Key Features

Dual-Protocol API

Single binary serves both API formats:

Protocol	Endpoint	Example
OpenAI	POST /api/v1/chat/completions	Works with any OpenAI SDK
OpenAI (legacy)	POST /api/v1/generate	Simplified generation
Anthropic	POST /v1/messages	Anthropic Messages API
Models	GET /api/v1/models	List available models
Health	GET /api/v1/health	No auth required

Interchangeable Backends

Backend	Use Case
ollama	Local models via Ollama API
openai-compat	vLLM, LiteLLM, or any OpenAI-compatible server
claude	Use your claude CLI subscription as a backend
opencode	Use OpenCode CLI as a backend
echo	Testing backend — always replies with ECHO: <message>

MCP Server Mode

Relay also exposes 5 MCP tools via stdio:

• generate_key, revoke_key, list_keys — API key management
• list_models — query available backends
• health — check server status

Perfect for agents that need to spin up a local inference endpoint dynamically.

Streaming SSE

Real-time streaming for both OpenAI SSE and Anthropic SSE event formats.

---

Tech Stack

Component	Choice	Why
Language	Go 1.26+	Single static binary, no runtime deps
API Server	net/http + gorilla/mux	Standard, well-tested
Config	YAML + CLI flags	Quick overrides without editing files
Auth	Auto-generated API keys	Zero-config security
Persistence	~/.localmcp/keys.json	Keys survive restarts
Size	~12 MB static binary	No Python, no Node, no containers
Tests	Go test suite	Core API and backend tests

Quick Start

git clone https://github.com/fxckcode/relay.git
cd relay
make build
./bin/localmcp

That’s it. The API key auto-generates on first run. Default listen is 127.0.0.1:8080.

# Test it
KEY=$(./bin/localmcp --show-key)
curl -X POST http://127.0.0.1:8080/api/v1/chat/completions \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"echo","messages":[{"role":"user","content":"Hello"}]}'

The echo backend replies immediately — perfect for testing. With Ollama running, switch to "model":"llama3.2".

Why Not LiteLLM / Ollama Built-in?

LiteLLM is a full Python project with heavy dependencies. Relay is a single Go binary — no pip install, no virtualenv, no Python version management. It also adds the CLI agent backend (use your claude CLI subscription as an API), which neither LiteLLM nor Ollama can do.

Ollama’s built-in OpenAI compatibility is solid but limited to Ollama models. Relay chains to any backend.

---

The project is MIT licensed and open on GitHub. Contributions welcome.