How to Fix Kubernetes Pods Stuck in Terminating Status

The Kubernetes pod stuck terminating error takes place when a pod remains in the “Terminating” state for a prolonged period. This could be caused by several distinctive issues and may be pretty irritating for developers. A pod’s ability to terminate may become trapped for a variety of reasons. Resource shortages and pod-related issues are both possible. No matter the issue, the Kubernetes pod stuck terminating issue may be a major headache for developers and may cause extreme delays in deploying applications. This article deals with the causes of this issue and their step-by-step solution. You may learn about the problem and how to solve it in full detail right here.

What Causes This Problem?

Recognizing the root cause of this problem is a critical step to fixing this issue. Some reasons why pods may get stuck at a “terminating state” includes:

Reason # 1: Lack of Resources

Kubernetes pods require proper amounts of resources to function without any issues. If there is an insufficient number of resources, multiple pods may start competing with each other for resources, which as a result may cause one of the pods to be stuck in a terminating state.

Reason # 2: Problems with the Pod Itself

An issue with the configuration or the code of the pod may result in it being stuck in a terminating state. If there are finalizers in the pod, the root problem may be that the finalizers are not completed. It may also be the case that the pod is not responding to the termination signal.

Reason # 3: An Underlying Node May Be Broken

Whenever Kubernetes pods will not exit the terminating condition, the underlying node is likely malfunctioning. When this takes place, apps may additionally fail to schedule causing unavailability. This could become a financial drain for your organization due to the fact that this problem can cause pointless scaling. It can be challenging for many teams to diagnose this problem because Kubernetes pods frequently terminate, making it difficult to tell which ones lingered for too long. Solving this problem is complex because Node draining in Kubernetes has to be configured in a manner to work for your environment.

If you see from the configuration file that all of the pods on a single node are in the state of “terminating,” then this might be the problem.

How to Fix This Issue?

The following ways can help you fix the issue easily.

Deleting the Pod

First, you will need to try to manually delete the pod by doing the following steps:

kubectl delete –wait=false pod
kubectl delete –grace-period=1 pod
kubectl delete –grace-period=0 –force pod

However, there is little chance that manual removal of the pod from the namespace will help resolve the issue even if you give the exact name of the pod you wish to delete.

If so, the problem might be that the pod is not ending because a certain process is not reacting to a signal. So, you will need to command the pod to be removed forcefully by using the following command:

> kubectl delete pod [Name_of_The_Pod] --grace-period=0 --force kubectl delete pod [Name_of_The_Pod] --graceperiod=0 --force -n [Name_of_the_namespace]

Make sure to add the name of your pod in the command if it is in a dedicated namespace.

Removing the Finalizers

If removing the pod forcefully does not work, then the main issue may be with the pod itself. A common issue with the pod is the inability of the finalizers in it to be completed, which may be the main issue that is causing the pod to be stuck in a terminating state. So, you will need first to check for the finalizers in the pod by getting the pod’s configuration:

> kubectl get pod -n [NAMESPACE] -p [Name_of_the_pod] -o yaml > /tmp/config_file.txt

Then, search under metadata for the finalizers section. If any finalizers are found, you will need to remove them by doing the following:

> kubectl patch pod [Name_of_the_pod] -p '{"metadata":{"finalizers":null}}'

Restart the Kubelet

If the mentioned solutions do not resolve this issue, then you should try to restart the kubelet. However, you may need to get an administrator involved if you do not have permission. If you do have access, you should restart the kubelet process by SSHing into the node.

How to Avoid Pods Being Stuck in the Future?

These are some steps you can take to make sure this problem doesn’t occur in the first place:

Thoroughly check your pods first to see if they are functioning properly before deploying them.
Make sure you have sufficient resources. A lack of resources may cause the pods to start competing with each other for resources, which as a result, may cause one of the pods to be stuck in a terminating state.
Make sure your pods don’t consume too many resources.
Make sure to keep your Kubernetes cluster up to date to avoid any problems in the future.
Constantly check to see if there are any issues with the configuration or the code of your pods.

Conclusion

The issues that may arise as a result of a pod being stuck in the terminating state make it worthwhile to take extra steps to ensure, before deploying, that there aren’t any issues with the pod itself, for example, there may be an issue with the configuration of the pod which will most likely cause the pod to be stuck in the terminating state. You should also be extra careful to avoid things that may result in this issue, such as a lack of resources or the Kubernetes cluster not being up to date. If this issue still occurs despite taking the necessary steps to avoid it, the first thing that will need to be done will be to pinpoint the root cause of this issue and use a solution accordingly.