Sooner or later every platform team gets the same ticket: “We need to run a model in production.” The first instinct is to add a GPU node group, slap a nvidia.com/gpu: 1 limit on a Deployment and call it done. That works for a demo. In production it leads to idle accelerators, ten-minute cold starts and a cloud bill that makes finance call you directly.
LLM inference behaves differently from the stateless web services Kubernetes was optimised for. Pods are huge, images are huge, model weights are huger, and the scarce resource is not CPU but GPU memory. This post covers the patterns that matter when you treat inference as a first-class platform workload.
1. Isolate GPU nodes properly
GPU nodes should never accidentally host a log shipper’s sidecar-heavy batch job. Taint them and make inference workloads opt in explicitly:
kubectl taint nodes -l accelerator=nvidia-l4 nvidia.com/gpu=present:NoScheduletolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
nodeSelector:
accelerator: nvidia-l4Use the NVIDIA GPU Operator (or your cloud’s managed device plugin) so drivers, the container toolkit and DCGM exporters are managed declaratively rather than baked into AMIs by hand.
2. Right-size the accelerator, not just the pod
The question is not “how many GPUs” but “how much GPU memory does this model need at the batch size and context length we serve?” A rough rule: weights in FP16 take about 2 bytes per parameter, and the KV cache grows with concurrent requests and context length. A 7–8B parameter model fits comfortably on a 24 GB card with room for cache; a 70B model needs quantisation or multiple GPUs with tensor parallelism.
Before you buy H100 capacity, benchmark on smaller cards with a quantised model. Many internal use cases — summarisation, classification, RAG over internal docs — are perfectly served by small models on L4 or A10G class GPUs.
3. Share GPUs where it makes sense
A single small model rarely saturates a modern GPU. Kubernetes has three main sharing options:
- Time-slicing — the device plugin advertises one physical GPU as several. Simple, but no memory isolation; one noisy pod can OOM its neighbours.
- MIG (Multi-Instance GPU) — on A100/H100 class hardware, partition the card into isolated slices with dedicated memory. Best for multi-tenant clusters.
- Model multiplexing in the server — let one inference server (vLLM, Triton, TGI) host several models or LoRA adapters and route by request.
For dev and staging, time-slicing is usually fine. For production multi-tenant platforms, prefer MIG or one model server per GPU with clear ownership.
4. Fix cold starts before they fix you
Scaling an inference pod from zero means pulling a multi-gigabyte image and downloading tens of gigabytes of weights. Mitigations that actually work:
- Keep model weights out of the container image. Store them in object storage and cache them on node-local NVMe or a shared read-only volume.
- Pre-pull images on GPU nodes with a DaemonSet, or use lazy-loading snapshotters such as SOCI or stargz.
- Keep a minimum replica count during business hours instead of true scale-to-zero for latency-sensitive APIs.
5. Autoscale on the right signal
CPU utilisation is meaningless for GPU inference. Scale on what users feel: queue depth, in-flight requests or tokens per second. vLLM and Triton both expose Prometheus metrics; feed those to KEDA or the HPA via a custom metrics adapter:
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
query: sum(vllm:num_requests_waiting{model="support-assistant"})
threshold: "10"Pair pod autoscaling with a node provisioner (Karpenter or Cluster Autoscaler) that knows about your GPU instance families, and set sensible consolidation so expensive nodes do not linger empty.
6. Observe the GPU, not just the pod
Deploy the DCGM exporter and build a dashboard with GPU utilisation, memory used, SM occupancy and power draw per node and per workload. If utilisation sits below 30% for a week, you are paying for headroom you do not need — consolidate models or move to smaller instances.
The platform takeaway
Treat GPU inference as a product your platform offers, not a snowflake deployment. Provide a paved road: a Helm chart or template with tolerations, model caching, metrics-based autoscaling and dashboards baked in. Teams get models into production faster, and you keep the most expensive nodes in your fleet busy doing useful work.
