A Practical Guide to Running LLMs on Kubernetes

Session description

The path from “I have a model” to “it’s serving production traffic on my cluster” is full of sharp edges. This talk is an end-to-end walk-through: GPU device plugins and node scheduling, model weight distribution strategies for 10–140 GB artifacts, health checks that survive multi-minute model loads, and GPU-aware auto-scaling. Includes a live demo deploying an open-weight LLM on Kubernetes and serving inference traffic with real-time GPU metrics.