What is an Evicted Pod?
When a node’s disk or memory limit is reached, a flag on the Kubernetes node is set to signal that it is under load. This flag also prevents fresh allocations on this node, triggering an eviction procedure to free up some resources.
This is the Kubelet for the under-pressure node, which will handle the eviction process. This one will handle failing Pods until the node’s spent resources fall below the eviction threshold, at which point the Kubelet will terminate all Pod containers and set the PodPhase to Failed.
If a Deployment is in charge of the evicted Pod, the Deployment creates a new Pod for Kubernetes to schedule.
When Will the Pods Be Evicted?
This distinction has to do with how Kubernetes handles resource pressure. If node resources become scarce, Kubernetes must evict pods, a process is known as node-pressure eviction. The node scheduler can accommodate a CPU that is fully occupied; therefore, eviction is not necessary.
It must evict pods from the node and try to place them in another node if memory is insufficient. This is known as eviction due to memory restrictions. Node-pressure eviction can also be caused by a lack of disk space.
What Factors Does Kubernetes Consider When Deciding Which Pods to Evict?
The node pressure is caused by pods being evicted based on resource constraints, like memory or disk space as we have mentioned above. Pods in a failed state are the first to be evicted, as they are not running but may still be consuming resources. Kubernetes then examines the running pods.
Evicting the pod that uses the most memory won’t work because it is most likely an active pod that will be difficult to deploy. Instead, Kubernetes makes this selection based on two different classes: QoS (Quality of Service) and Priority.
What Follows When a Pod Is Evicted?
When a critical condition is detected, Kubernetes performs some housekeeping (yes, housekeeping-interval is the official term for the eviction monitoring interval) and evicts pods. This procedure “evicts” (terminates) a pod and the containers that built it, but not the pod itself. By default, Kubernetes does not distinguish between highly critical and not at all important pods; it only has one duty, which is to reclaim resources.
Workloads managed by a workload resource or Deployment will automatically produce new pods and remove evicted pods.
Despite the fact that the evicted pods have been terminated and should no longer consume any resources from the node, they remain – and Kubernetes continues to manage them. With an increasing number of evicted pods, this might lead to a rise in the number of resources required inside the Kubernetes administration.
We will now try to figure out how to find an evicted pod and delete it successfully. There are a few requirements that must be met initially.
Prerequisite
To run commands for deleting evicted pods, first, we will need to set up a Minikube cluster on our PC. Ubuntu 20.04 was used to implement the commands of deleting evicted pods in this topic. We can also utilize our preferred operating system for this since we already have kubectl installed. It must be installed before the commands can be run.
Now, it is time to start a terminal. One option is to use our operating system’s application bar to reach the terminal. Another way is to use the keyboard shortcut “Ctrl + Alt + T.” To start a terminal, select one of these options. First, we must boot a Minikube cluster that was previously deployed on Ubuntu 20.04. Now, we will launch Minikube by typing the following command into the terminal. The relevant output for the below shown instruction is attached here.
How to Detect an Evicted Pod?
In order to find out the exact number of pods that have been evicted, execute the following command. This command displays a list of the pods that have been evicted from your system. The relevant output for the below shown instruction is attached here.
Now, let’s try a different form of the same command. We have added -c to the end of the command, as you can see. The rest of the instruction is undistinguishable from the one above. It will tell you how many pods have been evicted in total. There are no evicted pods in our scenario. As a result, as you can see in the final line of the attached screenshot, the command returns 0. The relevant output for the below shown instruction is attached here.
If your system has any evicted pods, the above command will output the number, such as 10, 3, 9, and so on.
How Can the Evicted Pods Be Manually Cleaned?
If you don’t want to wait for the garbage collection threshold to be met, you can destroy the pods manually with kubectl. In a while loop, each identified evicted pod can be eliminated using the command given above.
This command can assist you in manually cleaning up the evicted pods. The relevant output for the below shown instruction is attached here.
Conclusion
Eviction is the procedure of deliberately failing one or even more than one Pods on any of the resource-starved Nodes. We communicated about evicted pods in this tutorial. We have also discussed how to delete an evicted pod in simple steps. Now, you can implement the same guide in your system using kubectl and see how many evicted pods exist in your system. You can now list all of them and delete the selected ones.
