One reason why YAML is bad for configuration

YAML is a data serialization language that is widely used for application configuration. YAML is relatively readable, flexible and, compared to JSON, it allows for adding comments.

I don’t think that YAML is generally terrible for configuration, but the abuse of YAML when dealing with complex systems like Kubernetes makes all of its problems more evident: wrong indentations, the fact that you can cut a YAML in two and it’s likely still valid YAML, that problem with Norway and so on.

But today I’d like to talk about a more specific example that can seem surprising and that I found in a codebase I’m working on these days.

A simple Kubernetes deployment

I found myself facing a file that, for the sake of this blogpost, is equivalent to the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  annotations:
    foo: "bar"
    foo: "bar"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Is this a valid YAML? If you are unsure, you can use your preferred way for validating YAML, I will use Ruby’s irb:

% irb
2.6.0 :001 > require 'yaml'
 => true
2.6.0 :002 > a = YAML.load_file("deployment.yaml")
 => {"apiVersion"=>"apps/v1", "kind"=>"Deployment", "metadata"=>{"name"=>"nginx-deployment", "labels"=>{"app"=>"nginx"}, "annotations"=>{"foo"=>"bar"}}, "spec"=>{"replicas"=>3, "selector"=>{"matchLabels"=>{"app"=>"nginx"}}, "template"=>{"metadata"=>{"labels"=>{"app"=>"nginx"}}, "spec"=>{"containers"=>[{"name"=>"nginx", "image"=>"nginx", "ports"=>[{"containerPort"=>80}]}]}}}}

Valid, cool. Now look at the annotations.

2.6.0 :003 > a["metadata"]["annotations"]
 => {"foo"=>"bar"}
2.6.0 :004 >

The original deployment.yaml file had a duplicate annotation which is perfectly valid in YAML. You are basically saying “the key foo has value bar” and repeating it twice. Not too bad, except that the duplicate is not duplicate once parsed.

Things can be a little bit more fun though:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  annotations:
    foo: "bar"
    foo: "baz"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Now we have the same key with two different values. Let’s apply this to Kubernetes with kubectl and see what we have in the cluster:

kubectl get deployments nginx-deployment -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    foo: baz

[CUT]

Fun, there’s no sign of the value “bar”. This means that if for any reason you have two duplicate keys, you will not have an invalid YAML and just overwrite things.

This case seems rare and again not too terrible, but there are more similar cases:

% cat deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
  annotations:
    foo: "bar"
    foo: "baz"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        env:
          - name: foo
            value: bar
        env:
          - name: foo
            value: baz

The YAML file above has a duplicate env. Let’s kubectl apply it and look at the env:

 spec:
      containers:
      - env:
        - name: foo
          value: baz

Fun, isn’t it? Now imagine this “problem” over thousands of templated lines…

Please validate your YAML a lot

What is allowed in YAML is not always what you want to do. Config changes are still the reason for outages, issues in production and generally unexpected behaviors. I’m not going to say that YAML was a bad idea for Kubernetes resources because it would require a much more complicated and detailed discussion, but for sure if you want to deal with YAML files to configure your applications and infrastructure, there is a lot that you should be doing.

Validate your files. If you render them with a tool, validate the rendered files. If YAML is your source of truth, take care of it. Don’t generate and apply on the fly. And maybe try to not abuse YAML too much… I’m liking cue these days, but that’s a topic for another time.

Staging

“Death to staging”. “Staging lies”. “Staging is worst than it works on my machine”. “WTF!”

I’ve heard those a lot. And, in a way, they are all true. A snowflake staging environment is bad. The reason? Because production has its unique characteristics in terms of topology, data and a million other details that go from uptime of the machines, to pretty much anything else. Put like that, no other environment other than production makes sense. I believe this way of seeing things is missing a key point.

On staging, the word

The use of the word “staging” triggers bad feelings and for that reason it should not be used in a discussion lightly if we know that not all participants are on the same page. Staging triggers all the feelings described in the introduction of this blogpost. Its definition however, is not as bad:

2 a stage or set of stages or temporary platforms arranged as a support for performers or between different levels of scaffolding

And the definition of “staging area” is even better:

a stopping place or assembly point en route to a destination

Staging, the word, is like “legacy”: supposed to have a relatively positive meaning, left with a lot of negative ones, mostly because of the scars we all have operating systems. But it’s really just a stopping place en route to a destination.

A stopping place en route to a destination

Staging is not the final destination. Staging is a stop in the journey to build confidence on the quality of a change. That’s the key here: confidence on a change is built incrementally. We start when we write unit tests: we learn if a few units, in isolation, work on our machine. Then we push those and we learn if those units work in CI. Then we write integration tests, we do manual end to end testing and so on. We keep moving changes forwards until we reach production. How we reach production is completely up to us: slowly over the course of days or all at once. By testing changes in a different environment first or not. By feature flagging or not. There are a lot of combinations and different needs, but confidence in a change is an incremental process that tends to 100% only when the change is 100% enabled in production and serving traffic. How we combine those strategies depends on the needs of the application, the cost of testing and it’s ultimately a matter of tradeoffs that vary case by case.

Staging is dead…

The “staging environment” as the one environment alternative to production in its own slowflake configuration is almost useless and should be avoided. It will likely give a false sense of confidence in changes, especially if not approached with the right care. As time and money investment is probably the wrong one: I’d rather invest in getting changes safely to production and testing actively in production with things like feature flags. But it can’t be denied that using non-snowflake pre-production environments, we gain the possibility to test changes in an integrated environment before hitting production. And that’s good, as long as we understand that testing in staging will not catch all the possible bugs. Note that there are environments that are extremely complex for which is close to impossible to build a comprehensive staging environment. In those cases, maybe a staging environment is not the right thing. And maybe it’s worth questioning too if we really need all of that complexity.

… long live to dynamic pre-production environments.

Stigmatizing words is something that happens pretty frequently in the tech world. When this happens, we adapt our language to not be misunderstood and to be sure to get the best out of the conversations we have. Assuming we are not misunderstood, what we want is clear: to gradually increase our confidence in changes removing all the possible unknowns. Testing and conducting experiments in production is key. Using environments that are not production to test those changes is important as well and those tests are unlikely to backfire if the environments are not snowflakes and if we approach them with the assumption that there is no environment identical to production other than production.

Kubernetes sidecars

We often hear about sidecars in the context of Kubernetes pods. Kubernetes pods can contain multiple containers that will be guaranteed to run on the same machine, sharing the local network.

A popular pattern is the “sidecar pattern”. A main container is the application that we intend to run, but more containers are run together with it as part of the same pod. Those other containers are called “sidecars” because they provide additional functionalities that complement the main application. Some examples I’ve seen in the wild are:

  • TLS termination
  • Embedded logging or monitor agent

Those use cases seem perfect: sidecars allow to add functionalities to the application without modifying the application itself and to scale those naturally, which seems perfect for the agent use case.

But there are a number of downsides that must be considered.

There is no such thing as a sidecar

Sidecar containers were mentioned already in a blogpost on the Kubernetes blog from 2015. As it was said already, they have been a widespread pattern for years, but Kubernetes knows nothing about them. In fact, for Kubernetes, all containers in a pod are considered equal and there is no concept of a main container and a sidecar.

The Kubernetes community is working on a feature to enhance sidecar containers that is still being discussed and that will be available not earlier than Kubernetes 1.20, scheduled for a late 2020 release.

The proposal as it is today will allow to:

  • have containers in a pod that will start before the main container
  • handle sidecars for kubernetes jobs

Those are great additions that will solve problems for jobs and race conditions. When those features will ship, it will basically mean that Kubernetes will start to have real sidecars, but it won’t solve all the problems with sidecars.

The problems with sidecars

If you care about reliability (and I bet you do), you should consider how having more containers in a single pod can affect the reliability of the whole pod. In fact, all containers in a pod share the same failure domain: given the way the readiness is computed, if a single container in a pod is not ready then the whole pod is not ready.

Container readiness is a combination of the container being running and a successful readiness probe. If, for any reason, a sidecar container can’t run, then the whole pod is not ready. The same applies to the readiness probe, if you have any: if the readiness probe of a pod is not successful, then the whole pod is not ready.

What does this mean for us? It means that the sidecars that are “just adding functionality” could directly impact the readiness (and ultimately the availability, from a user perspective) of the application.

Also, in case of dynamic injection of containers, sidecars will likely lead to duplicating the same container everywhere, increasing overhead and reducing the possibility to optimize consumption centrally (without a redeploy).

Those points above are not everything. My friend Sandor has a one tweet opinion about sidecars that you should read.

With those considerations in mind, in my opinion, sidecars are less attractive than they look like when looked naively.

So… are sidecars considered harmful?

If you read “considered harmful” in an article title, it’s probably clickbait. And no, sidecars are not harmful or evil. You should consider the tradeoffs of what a multi container pod setup can offer and think if this is something good or if you can run without it. And remember that nothing is free: more containers, more problems.

Kpt, packages, YAMLs

A few days ago, Google announced Kpt, a tool “for Kubernetes packaging that uses a standard format to bundle, publish, customize, update, and apply configuration manifests”. I felt the urge to write a few words about the problem space, with no goal of being exhaustive… so here I am.

Kubernetes packaging

The whole Kubernetes ecosystem seems to be obsessed with the “packaging” problem. At first, Helm came out, providing a “Homebrew like” functionality. CNAB is a spec for packaging distributed applications on Kubernetes. And there’s probably more. What matters is that there have been multiple attempts at defining how to package an application or multiple applications. While it is important to have a single way to deploy an application and while reuse across different repositories is definitely useful, often an application is a fluid concept. It grows. People want to reuse parts of the configuration in other apps, but change a million things at the same time.

Well, I think that often a “package” is a cage. The analogy with Homebrew is especially wrong, in my opinion: installing an application on a Desktop is a story, running something on a production system is another one. I have no SLA on how vim runs on my machine. There are normally no customization flags to Homebrew.

On the other side, Helm, CNAB and others work in a totally different space. I have to confess that I was heavily biased on Helm and others exactly for this reason: they make it look like a helm install is enough to have core components running on your production system. The reality is much more complicated and depends mostly on where you are deploying, what your availability requirements are, what you are using charts/packages for.

The issue I have with Helm charts is that they hide the generated Kubernetes YAMLs but not completely. Helm install will talk directly to the cluster, but if you have problems with the cluster you will have to kubectl your way through it. This to say that Helm doesn’t build an abstraction and as such exposes the entire complexity of Kubernetes while providing the false sense of “it’s easy to get X up and running”. No it’s not and for a good reason.

Of course there are more shortcomings of using something like Helm: we complicate the ecosystem. Every time a tutorial starts with helm install something, it requires every user to install, learn and understand Helm. I see this as a problem, because instead of simplifying the procedures to get something up and running, we introduce additional tools which are complexity per se. If we believe we need those tools because Kubernetes doesn’t do certain things we should probably try to understand why Kubernetes doesn’t support those features and if there is anything that we can do to contribute them in the core project. Or to build something completely different on top of it. After all, Kubernetes is a platform for building platforms, isn’t it?

Manifests and Kustomize

I’m obsessed about making things as simple as they can be. Or maybe I’m just obsessed by exactly the contrary: complexity. Any additional tool introduces something that the users need to learn and consider when they are operating a system. That is cognitive load on the path to understand what will happen when they do X. Complexity, by definition.

In that regard, I often praised Kustomize. It allows us to start with non templated resources that are valid and can be independently used, customize them and render them back as modified resources. While the tool has a lot of features and it’s by no means the definition of simplicity, it has clear inputs and outputs: Kubernetes resources go in, Kubernetes resources go out. No weird templated things, nothing new. Moreover, it keeps the contract that the users have with Kubernetes still valid: the Kubernetes API (and its resources), nothing more.

Back to Kpt

It’s unclear if we are going to benefit from having yet another tool. The space is crowded, very crowded: there are at least 122 other tools for application management on Kubernetes and this does not even count all of the internal tools that companies have developed and that are closed source.

As Brian Grant says, at this point, it couldn’t hurt and I agree. It could teach us things and inspire others. But I still believe that those kinds of tools are very tied to the internal structure of the organization adopting them and the state in which an organization is with the development of home grown workflows to operate applications on Kubernetes.

I can’t help but see this as a missing opportunity to improve kubectl apply and kubectl wait. Those two basic commands that everyone uses lack basic functionalities and we keep building them somewhere else, many many times.

What’s helpful then?

I’m happy to see that with kpt we are talking about building blocks and workflows. There is still no one tool fits all and there will probably never be. Even committing to a tool or another makes very little sense. What’s important, IMO, is that:

  • tools should be composable: you don’t want to bet everything on a single tool as tools come and go.
  • the steps that make your workflow matters: you want to have clear building blocks, for example “rendering manifest”, “rollout”, “rollback” and so on. How those are implemented, is really left to your creativity.
  • identifying what you mean by an application and its dependencies and how they are mapped to your system is very important: the “application” is a concept that does not exist in Kubernetes.
  • raw resources are still the way to go: there is no one abstraction out there and as such hiding the complexity of Kubernetes’ resources can complicate things rather than simplifying them.
  • building sugar/helpers/whatever for you and/or for your organization is a good idea. Even if it’s the third time you write the same code.

That’s it, I have many other things to write, but it’s quarantine times and my tech energies are low :-)

I lost my Monday to Kubernetes

So I made a few changes to an app and I redeployed it once more to Kubernetes. I figured that it stopped working in the cluster that I am using but I don’t know why. It works locally after all. Works on my machine, the most classic statement a developer can make, still true once more.

I know that there were a bunch of things I touched: code and config (yeah I know, mistakes were made).

I tried to replicate the issue locally, to no success, and I can’t really figure out why things broke. All my changes looked legit and, worst of all, I had a few changes from the previous week (I told you, mistakes were made) that I didn’t fully remember but that seemed legit as well. I then decided to go deeper and figure out what was going on and it looked like my app was stuck doing nothing on some synchronization primitive.

Being extremely puzzled by this very unexpected behaviour, I did the following steps:

  • get the debugging symbols back in the binary (my Makefile was stripping them).
  • add a pprof handler to add profiling informations.
  • redeploy.
  • install gdb on the container to go step by step in the code.
  • install links (gotta thank Linux Nvidia drivers in the early 2000 for knowing that) to access the remote pprof data easily from the container.

I understood that the app was stuck in a particular part of the Kubernetes source code called WaitForCacheSync. Nothing seemed to have generated this new failure and after I tried my app locally, I figured that it was working perfectly with my local Kubernetes cluster. What was different? It looked like nothing was… Well the OS was different, wasn’t it (linux/darwin)?

From there i started a race to find an issue inside the vendored libraries, trying as well a local Linux environment running in docker with the same code/environment. I was able to install delve there and debug it even more deeply confirming what I found out on the remote cluster: there was an issue with the initial sync of the caches. After some debugging with the help of a colleague (thank you D, I appreciate it more than I can say), thinking of why those caches were never populated, I asked myself: why can’t we get that data?

That made me immediately think of the change that I made in config: an RBAC rule! And I was able to jump to the conclusion that the RBAC rule wasn’t working anymore as the namespace name was changed in a previous step! A few hours after the initial change, I finally unraveled the mistery: what seemed to be black magic, something wrong with libraries or even dark OS internals stuff, it was just a wrong RBAC rule that was implicitly denying access to my application.

As much as that wasn’t easy to debug, there are a bunch of lessons learned:

  • don’t touch code and config at the same time. If you are messing with YAMLs, do it in a separate step. The smaller the steps, the easier it is.
  • always assume that something simple rather than something complex and obscure is broken unless really proven otherwise. And most important, don’t try to prove it (this is where I got stuck) if not much later. Discard complicated assumptions by default before having first checked all the possible obvious things.
  • check for the obvious things: what are the dependencies of your app? How does the communication work? Is there a firewall rule or anything similar in between (be it a policy of any kind or anything else)?
  • don’t assume “this is kubernetes so I’m replicating the behavior”: kubernetes has a huge set of configurations and this means there is a lot that can be wrong other than just the deployment.
  • build tools for humans, even if what you are building in a library/SDK: Kubernetes had problems fetching data from the cached but there was no error returned of any kind. No RBAC access, no data, no error. Good luck debugging that.
  • use languages and frameworks that you know how to debug: no matter how much time I lost, it was incredibly easy to add debugging symbols and use gdb, delve, pprof to figure what was happening. I didn’t have all the automation to do that with one command (but maybe I’m building a tool for that, time will tell) but it wasn’t too out of reach which was a relief.
  • don’t do things that require thinking with jet lag.

Lots of learnings from a simple mistake, which is what I keep finding over and over again in my career. I need to periodically relearn my lessons, lose time and restart.

I am definitely not a 10x engineer :-P