Terminating

This week I had to compute some stats on how much time pods take to shut down for a few specific workloads and that required knowing what “Terminating” means for Kubernetes pods. If you have used kubectl to interact with a Kubernetes cluster, you have surely noticed that immediately after a pod is deleted and is shutting down, it is indeed shown as “Terminating”.

What not everybody knows is that “Terminating” is not a status for pods, but only something that is shown (for convenience?) by kubectl. So what is kubectl doing? This¹. This means that the definition of “pod terminating” is a pod in Running status with the DeletionTimestamp set.

This is not extremely surprising: a Terminating pod is in fact still “running” while it is shutting down. If you are talking with Kubernetes directly and are interested in determining the state of pods, for example to figure out the pods Terminating, you have to keep this in mind as there is no such state as Terminating. In the same way, while you get events for several things that happen during the startup of a pod, there is no event emitted for pods that will track the time needed to shutdown, missing a relatively important observability metric.

To come back at original task, I had to measure the time to shut down pods and to do that I used a Kubernetes PodInformer to get notified of the event of deletion of a pod and computed the total shutdown time as (time of the event - DeletionTimestamp). Kubernetes is always a great system in my opinion: no matter how often I bump into a “magic” behavior in kubectl or other things that are not great, the API driven approach always allows me to work around things and to extract the information that I’m looking for relatively easily.