Kubernetes Service Accounts weirdness

Ah Kubernetes, my joy, it’s so full of quirks and it’s sometimes maddening. Today, service accounts. Service accounts are a resource that has basically always been there in Kubernetes, definitely since 1.0 which is the first I have ever tried. As a concept, they are quite easy to grasp: they provide identity to workloads, for example to authenticate with the Kubernetes API or other things. If you don’t have specific needs but you just need to run a workload, you most likely don’t define one, but you are given one by default. And here the fun starts.

The “default” service account

In every namespace there is a service account named “default”. This is what gets assigned to your pods in a namespace if you don’t define a custom service account. It makes totally sense: the Kubernetes authors wanted to reduce the burden to define workloads for users, so if you don’t need anything special you get a default, low privilege service account. It’s all nice and cool, but if in our workload definition we didn’t define one, it implies that the field is "" (empty string), but not (the string) “default”. Let me recap: not setting it or setting it to empty string gives you a service account named “default”. Weird, but somewhat in line with the mess that Kubernetes has always done with things named “default”. To simplify the setup of new workloads, it added logic around defaults that feels sometimes magic.. And magic always comes at a cost. Anyhow, that’s not the end of the world. There is more fun coming.

serviceAccount vs serviceAccountName

At some point 11 years ago, there was a decision to rename the serviceAccount field to serviceAccountName.

The serviceAccount field, though, was never removed (I think due to being in a stable API, but unclear to me). It’s still there and it has weird behaviors… like the impossibility to roll back a switch to a new service account.

Rolling back a service account change

So let’s say you have a deployment that doesn’t set any service account explicitly. To grant additional permissions you create a new service account, let’s call it “example-sa”, and you reference it in the deployment using the field serviceAccountName. You then proceed to deploy it and for whatever reason something goes wrong and you decide to restore the previous version of the deployment in which the service account was not referenced. At this point, you’re stuck. Kubernetes will keep referencing the old service account. Counterintuitive, but you are not alone: https://github.com/Kubernetes/Kubernetes/issues/72519

That means that applying a previous version of the file, does not cause a logical roll back. I think it’s bad as stuff like this can greatly confuse users. Anyway, works as intended according to the core Kubernetes maintainers.

You probably need a workaround

The issue with this is that you may want to explicitly set both the two fields to “default” when you are using the default service account, to guarantee and simplify rollbacks. I can’t understand why this isn’t what Kubernetes does, but so it is. You want this workaround and you want the semantic to be clear and consistent with the operational practices of your team. My biggest critic in this is that you have to build a hack for what is frankly just a usability bug. Still better than writing the whole Kubernetes yourself!