Baton

A multi-provider gateway concept for AI tools, with one interface for model routing, fallback, caching, and proxy chaining.

October 15, 2026 Rust / LiteLLM / API Routing / AI Infrastructure

What this project is

Baton is a project I'm designing around a problem I keep running into while working with coding agents: model access is spread across different providers, proxies, formats, and capabilities. I want one place to manage those differences instead of solving the same integration problem in every tool.

The idea is a metaprovider. An agent connects to Baton, and Baton decides how to reach an appropriate upstream model or proxy. It is meant to work alongside existing tools, not replace the coding agent or the model provider.

The proposed architecture

coding agent / application
           |
           v
       Baton gateway
           |
   routing and policy
     /     |      \
 provider  proxy  fallback
     \     |      /
      upstream models

LiteLLM is the starting point for provider connectivity. I'm exploring a Rust gateway around that foundation to handle the policy and reliability concerns that a single provider connection does not solve.

What I want it to handle

Provider-aware routing. Choose a usable route based on model capabilities, availability, and configuration.
Proxy chaining. Support an upstream proxy as well as direct provider connections.
Fallbacks. Make the response to an unavailable route explicit instead of forcing each client to reinvent it.
Caching and cost control. Explore where repeated requests can be handled more efficiently without changing their meaning.
A consistent client surface. Keep the integration small for the tools that use it.

The hard part is preserving what a client actually asked for. Different providers do not support the same features, and a fallback that quietly discards an important capability is not a successful fallback.

Current status

Baton is in the design and prototyping stage. The architecture and provider boundaries are being worked out; this page is not a claim that a production gateway, benchmark suite, or publicly hosted service is complete.

The next milestone is a narrow end-to-end prototype: one client, more than one upstream route, visible capability checks, and a tested failure path. I'd rather prove those basics before adding another layer of configuration.