The day ExternalDNS broke Kubernetes test-infra

On March 19th 2022, @dims sent me a message on Slack that reads like this:

dims: Folks, we have a problem with external-dns images. It looks like we are shipping a GPL dependency! dims: moreover, the dependency has gone AWOL too.

This issue in the ExternalDNS repository describes what happened. It unfortunately impacted the Kubernetes test-infra repository, breaking some of their tests.

The follow up message from @thockin was not what I wanted to read either:

thockin: Oh cool. Drama. Just what I needed this weekend

It’s never great to find the creator of Kubernetes in your DMs on a weekend, expecially if it’s not really the right weekend:

Raffo: thanks for reporting folks, I haven’t checked anything as I was (and still am) fighting with covid.

Yikes.

As I started to feel a bit better on that day, I decided on a quick plan of action, those are my words from later that day:

• We merged the removal of the hetzner provider which removes the library affected (https://github.com/kubernetes-sigs/external-dns/pull/2635).
• I checked that the version that we included 2 years ago did not have the GPL licensing. The library was relicensed after.
• I will conduct an audit of the go.mod to see if there are other libraries that could be affected in any way by change of licenses or weird things due to the current world situation.
• I’ll open an issue to track that the Hetzner provider was removed and move forward with the community with the possible next steps if the community wants the provider to be reintroduced.

And from there on we were basically cool. This is also the episode that made me think and design the webhook provider, with the ultimate goal to regain control of the project and remove a lot of its dependencies.

Lessons learned, or why I still remember that day

March 2022 other than being the first time I got COVID, was also the beginning of the war in Ukraine. I was just a humble developer and while I knew where I was standing, I didn’t know I would have had to take a public position on it. Specifically, what happened then is that a dependency that we had accepted in the project was hosted outside of GitHub. At the time when we accepted it, we were aware of that, but we also didn’t want to take a position like “all the software needs to be hosted on github” and instead be inclusive of dependencies hosted somewhere else. Turns out this is not such a good idea, especially if one of your dependencies is hosted in Russia, Russia decides to go at war with Ukraine, it gets a unilateral support from all NATO countries and the dependency decides to geo block access based on IP address, which in the case of Kubernetes test-infra are all in the United States.

So what’s the lesson behind this? Well, several things:

• Software is political, no matter how neutral you want to be and how much you want to avoid it.
• Don’t accept dependencies that are not carefully vetted.
• Don’t add dependencies that you really don’t need to add.
• Be ready to take a public position on things that matter to you and your community.
• Be ready to do swift changes to protect the project.
• Your code has more impact on the whole ecosystem than you think.

Conclusion

In the end this was just a brief scare and nothing with too major consequences, but I will remember it for a long time. I also hope that I will not have to experience anything similar in the future and sincerely hope that all the wars will end soon.