How DaemonSet’s Status field works

02 Nov 2023

This week I embarked on an unexpected journey to figure out how the Status on DaemonSets work. I was trying to debug a problem in a piece of software that I maintain at work, which involves computing the Status for DaemonSet objects in Kubernetes, and I naively assumed that DaemonSets were just a “regular resource” in Kubernetes, but unfortunately I should have remembered that there are only guidelines for resources to implement Statuses and that controllers are pretty much free to do whatever they want. So, with a bug in front of me, I took my dear friend kubectl and started to build a mental model of how things work.

The DaemonSet Status Section

If we take a look at the DaemonSetStatus spec, we see there are a lot of fields and some are more interesting than others:

• CurrentNumberScheduled
• DesiredNumberScheduled
• NumberAvailable
• NumberUnavailable
• NumberMisscheduled
• NumberReady
• ObservedGeneration
• UpdatedNumberScheduled

And more. You can refer to the linked spec documentation for what they mean. I’m a bit stubborn and not really a quick learner so even if I had a description at hand, it took a while for me to understand how those values actually change.

Note that the DaemonSetStatus specification also talks about Conditions that should define the status. The field is in fact there, but it is actually not populated… So how does it work? Well, it’s complicated.

How the DaemonSet controller updates the status

Thankfully we have the code for the DaemonSet controller¹ to read and it’s quite well written and easy to follow albeit long. The code really helped me understand what happens to the various fields over time and to understand how they change in different situations that may arise. To explain how things work, let’s make two cases: the first in which we’re analyzing a newly created DaemonSet and the second an update of an existing DaemonSet. The two cases aren’t much different, but details of the behavior are important to how the Status is computed and what it means from the point of view of the system.

When a new DaemonSet is created, the controller will immediately create the Pod objects based on the Nodes. Once those are created, the ObservedGeneration field will go to 1, but numberReady and numberAvailable will be 0 and slowly growing. Reasonable, as Generation: 0 is never a thing, but it also means that there’s never a “0 generation” to compare with at the beginning…

In the case in which the DaemonSet was already existing and it was updated, we will see the following:

• The spec is updated, but the status isn’t. At this point in time, the Generation in the spec is updated (let’s say to 165) but the ObservedGeneration is still the old one, 164.
• The Pod objects are created and the ObservedGeneration is bumped to the new value 165. The pods aren’t necessarily running or ready.
• As the pods start and become ready, the UpdatedNumberScheduled will be increased accordingly.
• The NumberReady, NumberAvailabile and NumberUnavailable will change during the rollout.

Note an important thing: as the process is incremental, the NumberReady and NumberAvailable will fluctuate and will be the result of a sum across the generation of the pods. Those values will also depend on if the PodSpec part of the DaemonSet will contain a ReadinessProbe or not. I found this to be slightly confusing when dealing with such a complicated state as it mixes global cross generation counters (i.e. NumberReady) with generation specific ones (all the ones starting with Updated).

The simplified process can be represented with the following picture:

Conclusions

The way the status is computed for DaemonSet is significantly different from other “core” Kubernetes resources. Another good reference for how the Status actually is computed is the kstatus library which has a specific piece of code exactly to deal with the quirks of DaemonSets.

I hope that was an interesting read and maybe I will be writing more about all the weird things about Kubernetes Statuses that I discovered over the years. Until next time, have a wonderful day.