The AI Gateway
Posted on

TLDR;

Developers write code. They call models. They build agents and give them tools. They do this without a proxy. The keys leak. The bills run high. The data goes out and does not come back. Old gateways do not understand tokens. They do not know prompts. You need an AI gateway.

The Chaos

It is the wild west. Developers hardcode API keys. They send customer data to external servers. They give autonomous agents direct access to databases and shell environments.

  graph TD
    App1[App] -->|Raw Prompts & Secrets| OpenAI[OpenAI API]
    App2[Agent] -->|Direct Raw SQL & Shell| DB[(Company Database)]
    App2 -->|Raw Prompts| Gemini[Gemini API]
    App3[App] -->|Hardcoded Keys| Anthropic[Anthropic API]

A standard gateway does not help. It knows HTTP and REST. It does not know tokens. It cannot read prompts. It does not understand tool calls.

The Gateway

An AI gateway is a clean proxy. It sits in the middle. It sees all the communication.

  graph TD
    classDef gw fill:#fff9c4,stroke:#fbc02d,stroke-width:2px;
    
    App[AI App] --> GW[AI Gateway]
    Agent[Agent] --> GW
    
    subgraph Target ["Target Systems"]
        Models[LLM APIs 
Northbound]
        Tools[Databases & APIs 
Southbound]
    end
    
    GW -->|Redacted & Cached Request| Models
    GW -->|Validated Parameters| Tools

Governance requires three distinct proxies inside the gateway.

1. The Model Proxy (Northbound)

This proxy faces the model providers. The client talks to the gateway. The gateway talks to Gemini, Anthropic, or OpenAI.

  sequenceDiagram
    App->>Gateway: User Prompt (includes raw email & secret key)
    Note over Gateway: Scans content for secrets & email
    Note over Gateway: Redacts sensitive data
    Note over Gateway: Injects real model key from vault
    Gateway->>LLM: Clean Prompt + Secure Key
    LLM->>Gateway: Response
    Note over Gateway: Logs token usage
    Gateway->>App: Response
  • Secrets: The gateway holds the keys. The developer gets an internal token. The real API key never leaves the vault.
  • Privacy: The gateway scans the prompt. It redacts emails, passwords, and credit cards before they go to the cloud.
  • Cost: It caches responses. If two users ask the same question, the second user gets the cached answer. It is fast, and it costs nothing.

2. The Tool Proxy (Southbound)

Agents use tools. Tools are functions that query databases, read files, or call internal APIs. An agent with unmonitored tools is dangerous.

  sequenceDiagram
    Agent->>Gateway: Execute Tool: delete_user(id=99)
    Note over Gateway: Gateway checks Client Identity
    Note over Gateway: Verifies authorization for 'delete_user'
    Gateway--xTools: Blocked (Unauthorized)
    Gateway-->>Agent: Error: Tool execution rejected

The tool proxy stops the damage:

  • Validation: The gateway checks the parameters. It ensures the model is not passing malicious SQL or shell commands.
  • Authentication: The gateway verifies if this specific client is allowed to run this tool.
  • Human Gate: If a tool changes state—like deleting a user or sending money—the gateway pauses. It asks a human for approval. The agent waits.

3. The Agent Proxy (East-West)

Agents talk to other agents. They delegate tasks. This proxy coordinates the mesh:

  • Integrity: It signs messages. An unauthorized agent cannot inject instructions.
  • Loop Control: Agents can get stuck in infinite discussions. The proxy counts the messages. It stops the loop before the tokens burn your budget.

A Simple Gateway in Code

This is how you start. Here is a simple middleware written in Python. It redacts data, logs tokens, and proxies the call.

import os
import re
import httpx
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
import logging

app = FastAPI()
logging.basicConfig(level=logging.INFO)

# Hemingway-style: Simple, concrete rules.
SENSITIVE_PATTERNS = [
    r"(?i)password\s*=\s*['\"][^'\"]+['\"]",
    r"(?i)api[-_]key\s*=\s*['\"][^'\"]+['\"]",
    r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"
]

def clean_prompt(prompt: str) -> str:
    cleaned = prompt
    for pattern in SENSITIVE_PATTERNS:
        cleaned = re.sub(pattern, "[REDACTED]", cleaned)
    return cleaned

@app.post("/v1/chat/completions")
async def proxy(request: Request):
    body = await request.json()
    
    # Clean the input
    messages = body.get("messages", [])
    for msg in messages:
        if "content" in msg and isinstance(msg["content"], str):
            msg["content"] = clean_prompt(msg["content"])
            
    headers = {
        "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",
        "Content-Type": "application/json"
    }
    
    async with httpx.AsyncClient() as client:
        try:
            response = await client.post(
                "https://api.openai.com/v1/chat/completions", 
                json=body, 
                headers=headers, 
                timeout=30.0
            )
            response.raise_for_status()
            data = response.json()
            
            # Log the cost
            usage = data.get("usage", {})
            logging.info(
                f"Client: {request.client.host} | "
                f"Tokens: {usage.get('total_tokens', 0)}"
            )
            
            return JSONResponse(content=data)
            
        except Exception as e:
            logging.error(f"Failed: {str(e)}")
            raise HTTPException(status_code=500, detail="Gateway Error")

Deployment

Deploy the gateway close to your code. Use a sidecar. Store your cache in Redis. It must be fast. Enforce schemas on your tools.

If you build with AI, stop calling models directly. Use a gateway. It is the only safe way.

Table of Contents