Kubernetes High Availability: Core Concepts
A single-node Kubernetes cluster is convenient for development but is a single point of failure in production. This article walks through the key concepts for building an HA cluster.
Control plane redundancy
The control plane consists of the API server, etcd, the scheduler, and the controller manager. For HA, you need at least three control-plane nodes so that etcd — which uses the Raft consensus algorithm — can tolerate the loss of one node.
┌──────────────────────────────────────────────┐
│ Load balancer (e.g. HAProxy / kube-vip) │
└───────────┬──────────────┬───────────────────┘
│ │
┌─────────▼──────┐ ┌────▼───────────┐ ┌────────────────┐
│ Control [1] │ │ Control [2] │ │ Control [3] │
│ API / etcd │ │ API / etcd │ │ API / etcd │
└────────────────┘ └────────────────┘ └────────────────┘
Use kubeadm with --control-plane-endpoint pointing to the load-balancer VIP:
kubeadm init \
--control-plane-endpoint "k8s-api.example.internal:6443" \
--upload-certs \
--pod-network-cidr 10.244.0.0/16
Additional control-plane nodes join with:
kubeadm join k8s-api.example.internal:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <cert-key>
Pod disruption budgets
Prevent all replicas of a deployment from being evicted simultaneously during node maintenance or rolling updates.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 2 # Always keep at least 2 pods running
selector:
matchLabels:
app: api
Liveness and readiness probes
Kubernetes relies on probes to decide whether to route traffic to a pod and whether to restart it.
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 15
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
- Liveness: if this fails, the pod is restarted.
- Readiness: if this fails, the pod is removed from the Service endpoints (no traffic).
Node affinity and topology spread
Spread replicas across availability zones to survive zone failures:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api