Lesson · ⏱ 8 min read

☸️ Kubernetes: running containers at scale

Q: Traffic triples during a sale. What adds more pods automatically?

A HorizontalPodAutoscaler. The autoscaler watches load (e.g. CPU or requests) and changes the replica count.

🏠 Real-life example

A big wedding caterer doesn't run one stove. A manager watches many cooking stations, adds stations when guests pour in, replaces a cook who falls sick, and makes sure food keeps flowing. Kubernetes is that manager for containers: it keeps the right number running, restarts failures and scales up with traffic.

💡 In simple words

You tell Kubernetes the desired state ('run 3 copies of my model API, each with 2 GB of memory and 1 GPU') and it makes reality match. Key objects: a Pod is one or more running containers, a Deployment keeps N pods healthy and handles rolling updates, a Service gives pods a stable address, and a HorizontalPodAutoscaler adds pods when load rises. For AI, you also schedule GPU nodes and set resource limits so one model can't starve another. Helm packages all this YAML into reusable charts. Start locally with kind or minikube.

🎬 See how it works

As traffic rises, the autoscaler adds pods. When a pod crashes, Kubernetes replaces it automatically, so the desired state is always kept.

🪜 Step by step

A Pod runs one or more containers. A Deployment keeps a set number of identical Pods running and replaces failed ones.
A Service gives the Pods one stable address and spreads traffic across them.
Set resource requests and limits (CPU, memory, GPU) so the scheduler places Pods correctly.
Add readiness and liveness probes so traffic only reaches healthy Pods.
Use a Horizontal Pod Autoscaler to add Pods when load rises, and roll out new versions gradually.

🔑 Key terms

Pod: The smallest unit Kubernetes runs: one or more containers.
Deployment: Keeps a set number of pods running and updates them safely.
Autoscaling: Automatically adding or removing pods based on load.

👀 See it in action

yaml

apiVersion: apps/v1
kind: Deployment
metadata: { name: price-api }
spec:
  replicas: 3
  selector: { matchLabels: { app: price-api } }
  template:
    metadata: { labels: { app: price-api } }
    spec:
      containers:
        - name: api
          image: registry.example.com/price-api:1.4
          resources: { limits: { memory: "2Gi", cpu: "1" } }

'Keep 3 copies of version 1.4 running, each limited to 2 GB.' Kubernetes does the rest.

⚠️ Common mistakes to avoid

No resource limits: one Pod eats all memory and takes the node down.
No readiness probe: traffic hits Pods still loading a large model and requests fail.

✅ Quick check

Traffic triples during a sale. What adds more pods automatically?

A HorizontalPodAutoscaler
A Dockerfile
A Service

Show answer

A. A HorizontalPodAutoscaler: The autoscaler watches load (e.g. CPU or requests) and changes the replica count.

🙋 Questions learners ask

Do I need Kubernetes for my first model?

Usually not. A managed service (Cloud Run, App Runner, a PaaS) is simpler. Kubernetes pays off when you run many services at scale.

What is Helm?

A package manager for Kubernetes: it templates the YAML for an app so you can install and upgrade it with one command.

📌 Remember

Declare the desired state, and Kubernetes keeps it true: restarts, scaling, updates.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

MLOps / AI Platform Engineer

Continue in the course