Conference SessionArtificial Intelligence and Machine LearningIntermediate

Building an AI Inference Gateway on Kubernetes: Model Routing, Rate Limiting & Cost Control

Oct 29, 2026, 2:30 PM · 60 min · Room A

Running multiple AI models on Kubernetes without a unified gateway means duplicated auth, no rate limits, and opaque costs. This talk builds an inference gateway from scratch using Envoy, Kubernetes CRDs, and Prometheus. We cover model routing with A/B traffic splitting, token-based rate limiting per team, fallback chains, and per-team cost attribution. Includes a live demo deploying the gateway and routing traffic across models in real time.

Speaker

Rohit Mishra

Cloud Solutions Architect, Google Cloud

I work as a Cloud Solutions Architect and Engineer for Google Cloud. Over the years I have helped some of the top brands in retail, logistics, food and restaurant industry to transform their digital presence through cloud adoption. I have driven and led successful migrations, transformation and modernization projects of strategic significance. A builder at heart, I have been deeply enaged with developers and architects on either side of the fence (Google & it's customers) to explore the art of possible with cloud native systems powered by open source and community powered software. Lately I have expanded my fields into AI platforms and infrastructure and have been helping Google customers with AI adoption, running large scale AI workloads, AI security and threat prevention, inferencing at scale.

Speaker profile →

Want to be in the room?

Register through Rohit’s link and the speaker code takes 10% off the Conference + Workshop Bundle, applied automatically at checkout. Codes expire Sept 30.

Building an AI Inference Gateway on Kubernetes: Model Routing, Rate Limiting & Cost Control

Session description