Extending AI gateways with Rust: Custom transformations in agentgateway and kgateway

Posted on May 15, 2026 by Michael Uzukwu, kgateway and agentgateway Contributor

CNCF projects highlighted in this post

Every gateway ships with a set of built-in policies. Authentication. Rate limiting. Request routing. Prompt guards. These cover most use cases. But what about the ones they don’t cover?

What if you need to add a custom header based on a database lookup? What if you need to transform a request body in a way no existing filter supports? What if your business has unique logic that no off-the-shelf gateway can anticipate?

You build your own extension.

This article walks through exactly how to do that using agentgateway, Envoy, and Rust. In this tutorial, you’ll learn how to:

Build a custom Envoy dynamic module in Rust
Package it into a production-ready Docker image
Deploy it to Kubernetes with kgateway and agentgateway
Test the entire stack with a mock LLM endpoint

What you’ll need: Basic familiarity with Kubernetes, Docker, and command-line tools. No prior Rust experience required — I’ll explain the key parts as we go.

Time to complete: About 30-45 minutes.

Cost: Zero. Everything runs locally.

Architecture overview

Before diving into code, let’s understand what we’re building.

The lab routes a request through four layers:

A curl client sends a POST request
agentgateway-proxy (Envoy) receives it
A custom Rust module transforms the request
httpbun (a mock LLM) returns a fake response

curl → agentgateway-proxy → Rust Module (.so) → httpbun (mock LLM) → response

Here’s the complete architecture:

This flow chart shows requests that flow through the AI Gateway. The client sends a request to the proxy, which loads and executes the Rust module. The module returns a modified request, the proxy forwards it to httpbun, and the mock response flows back to the client.

Everything runs locally on your laptop using kind (Kubernetes in Docker). No cloud costs. No API keys. The Rust module can be replaced with any transformation logic you need — the lab just shows the mechanism.

The stack

Here’s what each tool does:

Tool	Purpose
kind	Creates a local Kubernetes cluster on your laptop
kgateway + agentgateway	Control plane that manages Envoy and handles Gateway API resources
Envoy	The proxy that sits between your client and backend, processing every request
Rust	Your custom transformation code, compiled into a shared library that Envoy loads at runtime
httpbun	A mock LLM that returns fake responses (no API key required)

Everything is open source. Everything runs locally. You don’t need to spend a dime to follow along.

Before you start

Make sure you have these tools installed:

Docker (latest) – Runs containers, including your Kubernetes cluster and the Envoy proxy
kind (v0.20+) – Creates a local Kubernetes cluster
kubectl (v1.27+) – Talks to your Kubernetes cluster
Helm (v3.10+) – Installs kgateway and agentgateway packages
Rust (1.85+) – Builds the Rust module (optional; you can build inside Docker)

Create your cluster:

kind create cluster --name ai-gateway-lab

This command spins up a local Kubernetes cluster. All your gateway components will run inside it, isolated from your main system.

Part 1: The Rust module

The Rust code is split into two crates. Think of crates as folders that each contain a small library:

rustformations – The main Envoy filter that contains your transformation logic
transformations – A helper library that provides Jinja templating and shared transformation traits

Project Structure
rust/

├── rustformations/

│ ├── Cargo.toml

│ └── src/

│ ├── lib.rs # Registers the filter with Envoy

│ └── http_simple_mutations.rs # Your actual transformation logic

└── transformations/

├── Cargo.toml

└── src/

├── lib.rs # Defines transformation traits

└── jinja.rs # Jinja templating for dynamic transformations

The Cargo.toml file

Every Rust project has a Cargo.toml file. It lists dependencies and build instructions. Here’s what ours looks like:

[package]
name = "rustformations"
version = "0.1.0"
edition = "2021"

[dependencies]
# The Envoy SDK – tells Rust how to talk to Envoy's C ABI
envoy-proxy-dynamic-modules-rust-sdk = { path = "../patched-envoy-sdk/..." }

# Serialization – for parsing JSON requests and responses
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

# Templating – for dynamic prompt transformations
minijinja = { version = "2.12.0", features = ["loader"] }

# Our helper library
transformations = { path = "../transformations" }

# Error handling and shared state
anyhow = "1.0.100"
once_cell = "1.21.3"

[lib]
name = "rust_module"
path = "src/lib.rs"
crate-type = ["cdylib"]   # Creates a .so file that Envoy can load

Key dependencies explained:

crate-type = ["cdylib"] – This is the most important line. It tells Rust to compile your code into a C-compatible shared library (.so file). Envoy can load this file at runtime without restarting.
envoy-proxy-dynamic-modules-rust-sdk – The official SDK that provides bindings between Rust and Envoy’s C API.
minijinja – A templating engine that lets you dynamically transform prompts using templates.
serde – Converts JSON requests into Rust structs and back.

The transformation trait

A “trait” in Rust is like a contract. It says “If you want to be a transformation filter, you must implement these functions.”

pub trait TransformationOps {
    // Add a new header to the request (appends if header exists)
    fn add_request_header(&mut self, key: &str, value: &[u8]) -> bool;
    
    // Set a header (overwrites if it exists)
    fn set_request_header(&mut self, key: &str, value: &[u8]) -> bool;
    
    // Remove a header entirely
    fn remove_request_header(&mut self, key: &str) -> bool;
    
    // Same as above but for responses
    fn add_response_header(&mut self, key: &str, value: &[u8]) -> bool;
    fn set_response_header(&mut self, key: &str, value: &[u8]) -> bool;
    fn remove_response_header(&mut self, key: &str) -> bool;
    
    // Parse the request body as JSON so you can read and modify it
    fn parse_request_json_body(&mut self) -> Result<JsonValue>;
    
    // Get the raw request body as bytes
    fn get_request_body(&mut self) -> Vec<u8>;
    // ... more methods for streaming bodies, responses, etc.
}

What this means: When Envoy calls your Rust module, it gives you access to the request headers, request body, response headers, and response body at different points in the request lifecycle. You can read, modify, or replace anything you need.

You don’t need to implement all of them for a simple filter. Start with the headers you want to change and grow from there.

Part 2: The Docker image

We need to package Envoy with our Rust module into a single Docker image. This Dockerfile uses a multi-stage build to keep the final image small.

# Stage 1: Build the Rust module
FROM rust:1.85 AS builder
WORKDIR /build


# Install clang – needed to compile C bindings for the Envoy SDK
RUN apt-get update && apt-get install -y clang


# Copy all Rust source code into the container
COPY rustformations/ ./rustformations/
COPY transformations/ ./transformations/
COPY patched-envoy-sdk/ ./patched-envoy-sdk/


# Build the Rust module in release mode (optimized, no debug symbols)
WORKDIR /build/rustformations
RUN cargo build --release


# Stage 2: Final Envoy image
FROM envoyproxy/envoy:v1.36.4


# Install CA certificates – Envoy needs these to validate HTTPS backends
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*


# Copy the envoyinit wrapper binary (handles Envoy startup)
COPY envoyinit-linux-amd64 /usr/local/bin/envoyinit
RUN chmod +x /usr/local/bin/envoyinit


# Copy the compiled Rust module from the builder stage
COPY --from=builder /build/rustformations/target/release/librust_module.so /usr/local/lib/


# Copy the entrypoint script (decides how to start Envoy)
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh


# Tell Envoy where to find dynamic modules
ENV ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib


# Run as non-root for security
USER 10101


ENTRYPOINT ["/docker-entrypoint.sh"]

http://docker-entrypoint.sh

What each stage does:

Stage 1 (builder) – Compiles your Rust code. Uses a larger image with Rust and build tools. Creates the .so file.
Stage 2 (runtime) – Only contains Envoy and your compiled .so file. Keeps the final image small (319MB).

Build the image:

docker build -f Dockerfile.rust85 -t envoy-wrapper:test .

This creates a Docker image named envoy-wrapper:test that contains Envoy plus your custom Rust module. You can run this image anywhere Docker runs.

Part 3: Deploying to Kubernetes

Now we deploy everything to your local Kubernetes cluster.

Install Gateway API CRDs

kubectl apply -f

Install

What this does: Installs the Custom Resource Definitions (CRDs) for Gateway API. These let you define Gateways, HTTPRoutes, and other routing resources in Kubernetes.

2. Install kgateway (Control Plane)

helm upgrade -i kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace --namespace kgateway-system \
  --version v2.2.1

helm upgrade -i kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace kgateway-system \
  --version v2.2.1

What this does: Installs kgateway, the control plane, into your cluster. It runs in the kgateway-system namespace and manages Envoy instances.

3. Install agentgateway (AI Data Plane)

helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --create-namespace --namespace agentgateway-system \
  --version v1.1.0

helm upgrade -i agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
  --namespace agentgateway-system \
  --version v1.1.0

What this does: Installs agentgateway, the AI-focused data plane that works alongside kgateway. This component actually handles AI traffic.

4. Deploy httpbun (the Mock LLM)

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbun
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbun
  template:
    metadata:
      labels:
        app: httpbun
    spec:
      containers:
      - name: httpbun
        image: sharat87/httpbun
        env:
        - name: HTTPBUN_BIND
          value: "0.0.0.0:3090"
        ports:
        - containerPort: 3090
---
apiVersion: v1
kind: Service
metadata:
  name: httpbun
  namespace: default
spec:
  selector:
    app: httpbun
  ports:
    - protocol: TCP
      port: 3090
      targetPort: 3090
EOF

What this does: Deploys httpbun – a fake OpenAI-compatible LLM. It listens on port 3090 and returns mock responses. No API key needed.

5. Create the AgentgatewayBackend

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: httpbun-llm
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: gpt-4
      host: httpbun.default.svc.cluster.local
      port: 3090
      path: "/llm/chat/completions"
EOF

What this does: Tell agentgateway that there’s an LLM backend at that address speaking the OpenAI API format.

6. Create the Gateway and HTTPRoute

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbun-llm
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/chat/completions
      backendRefs:
        - name: httpbun-llm
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
EOF

What this does:

Gateway – Creates an entry point for traffic. Listens on port 80.
HTTPRoute – Routes requests matching /v1/chat/completions to the httpbun backend.

Part 4: Testing it all works

Port-forward the gateway

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8082:80

What this does: Forwards traffic from your laptop’s port 8082 to the gateway pod running in Kubernetes. This lets you test locally as if you were outside the cluster.

2. Send a test request

Open a new terminal and run:
curl -X POST http://localhost:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}]}'

3. Expected response

{
  "choices": [{
    "message": {
      "content": "This is a mock chat response from httpbun."
    }
  }]
}

If you see this, everything works:

The Rust module loaded successfully
The gateway routed the request correctly
The mock LLM responded

Troubleshooting common issues

Problem 1: Rust version mismatch

Error:

text
error: feature `edition2024` is required

Cause: Some Rust crates require newer compiler features. Your Rust version is too old.

Fix: Upgrade Rust in your Dockerfile from 1.75 to 1.85 or newer.

Problem 2: Missing ABI Symbol

Error:

text
undefined symbol: envoy_dynamic_module_callback_http_add_response_header

Cause: Your SDK doesn’t match your Envoy version. Envoy v1.36.4 expects certain functions that older SDKs don’t provide.

Fix: Copy the official SDK directly from the Envoy source:

bash
cp -r envoy/source/extensions/dynamic_modules/sdk/rust patched-envoy-sdk/

Problem 3: filter_config format

Error:

text
error parsing filter config: EOF while parsing a value

Cause: Envoy expects configuration to be wrapped in a protobuf Any type. Without the wrapper, it passes an empty object that your Rust code can’t parse.

Fix: Use the protobuf wrapper in your Envoy config:

yaml
filter_config:
  "@type": type.googleapis.com/google.protobuf.StringValue
  value: "{}"

Next steps: Production and real LLMs

This lab uses httpbun as a mock. To use a real LLM:

Get an API key from OpenAI, Anthropic, or Gemini
Create a Kubernetes secret with your key
Update the AgentgatewayBackend to use the real host and authentication

yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:
        model: gpt-4
    host: api.openai.com
    port: 443
  policies:
    auth:
      secretRef:
        name: openai-secret

For production, also add:

Authentication (API keys, JWT, or mTLS)
Rate limiting to control costs
Observability (metrics, logs, tracing)
Deploy to a real Kubernetes cluster (EKS, GKE, or AKS)

agentgateway supports all of these through its policy CRDs.

Complete code

Everything is on GitHub: github.com/Mike-4-prog/ai-gateway-lab

The repo includes:

All Kubernetes manifests
Complete Rust source code
Multi-stage Dockerfile
Quick start README

You can clone it and run the entire lab in about 10 minutes.

Final thoughts

Building this lab taught me three things:

Extending agentgateway with Rust is powerful but strict. The SDK must match Envoy exactly. The Rust version must support your dependencies. One version mismatch and everything breaks.
The filter_config format is not obvious. The protobuf wrapper is documented, but easy to miss. I spent hours on this error before finding the solution in the docs.
Starting with a mock LLM saves time and money. httpbun let me focus on the gateway, not the AI provider. I could test everything locally without worrying about API keys or costs.

If you’re building on agentgateway and need a capability that doesn’t exist yet, you now know how to build it yourself.

Questions? Find me on GitHub.

Special thanks to Art Berger and the kgateway team for their guidance and encouragement.

Mumbai, India

Architecture overview

The stack

Before you start

Part 1: The Rust module

├── rustformations/

│ ├── Cargo.toml

│ └── src/

│ ├── lib.rs # Registers the filter with Envoy

│ └── http_simple_mutations.rs # Your actual transformation logic

└── transformations/

├── Cargo.toml

└── src/

├── lib.rs # Defines transformation traits

└── jinja.rs # Jinja templating for dynamic transformations

The Cargo.toml file

The transformation trait

Part 2: The Docker image

Part 3: Deploying to Kubernetes

Part 4: Testing it all works

Troubleshooting common issues

Problem 1: Rust version mismatch

Problem 2: Missing ABI Symbol

Problem 3: filter_config format

Next steps: Production and real LLMs

Complete code

Final thoughts