EMK - High available workloads
Estimated time to read: 2 minutes
Your Kubernetes cluster control plane can be high available (HA). This is possible by setting your cluster HA setting.
Source: Gardener documentation
In maintenance windows or during performed maintenance it can happen that a node will be drained from your Kubernetes cluster. When this happens there are systems in place within Kubernetes that take care of your workload. In case of a node drain to keep your applications to be highly available Kubernetes will look at your applications and it's PodDisruptionBudget
.
Below we go a bit more in-depth about restrictions with PodDisruptionBudget
, although there are more best practices. You can find some at the Gardener documentation.
PodDisruptionBudget
Kubernetes offers a feature called PodDisruptionBudget
(PDB) for each application. A PDB limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions.
The most common use case is when you want to protect an application specified by one of the built-in Kubernetes controllers:
- Deployment
- ReplicationController
- ReplicaSet
- StatefulSet
A PodDisruptionBudget
has three fields:
- A label selector
.spec.selector
to specify the set of pods to which it applies. .spec.minAvailable
which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod.minAvailable
can be either an absolute number or a percentage..spec.maxUnavailable
which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage.
Cluster failures with PDB
Misconfiguration of the PDB could block the cluster upgrade or node deletion processes. There are two main cases that can cause a misconfiguration. Those are:
- The replica of Kubernetes controllers is 1
- HPA configuration violates PDB
The replica of Kubernetes controllers is 1
- Only 1 replica is running: there is no
replicaCount
setup orreplicaCount
for the Kubernetes controllers is set to 1 - PDB configuration
- To fix this PDB misconfiguration, you need to change the value of
replicaCount
for the Kubernetes controllers to a number greater than 1
HPA configuration violates PDB
In Kubernetes, a HorizontalPodAutoscaler
automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand. The HorizontalPodAutoscaler
manages the replicas field of the Kubernetes controllers.
- There is no
replicaCount
setup orreplicaCount
for the Kubernetes controllers is set to 1 - PDB configuration
- HPA configuration
- To fix this PDB misconfiguration, you need to change the value of HPA
minReplicas
to be greater than 1