Keeping ExternalDNS Secure

Navigating to this blog, I realized that I haven’t written a blogpost in a year. It seems quite a long time, but 2022 has been a huge mess in my personal life. I had different health issues that affected my life and my mood and I didn’t really find a lot of time to write anything tech related. I’m gonna do this now: my topic for today is “keeping ExternalDNS secure”.

The status quo during summer 2021

During the summer of 2021, I realized that ExternalDNS’s official Docker image had several vulnerabilities. That sucked because I was not aware of it, the project had no scanning configured and I only noticed it by chance. I couldn’t sit and wait on that and I immediately thought I would inform the community of the findings and commit to ship improvements in this issue.

What I did to fix the problem

The strategy was relatively simple: add automation to scan the image, do it periodically and have more automation to keep the dependencies up to date.

To do so, I added trivy as an image scanner. This gave us a pretty good view of what was happening inside the image so that we could work on making sure to bump all the dependencies that were showing up as vulnerable. This brought us to zero vulnerabilities relatively quickly, but the challenge was keeping the image up to date with ExternalDNS’ somewhat infrequent, unscheduled releases. We needed a process and more automation.

Dependabot to the rescue

Keeping dependencies up to date is a time intensive task. You have to run things locally, make sure the project still compiles, the tests pass… It’s a lot of work and I didn’t want to do it. The obvious solution on GitHub is to use dependabot: with an easy configuration you can make sure to keep your dependencies up to date, including the ones in GitHub actions. It looks like we already closed 200+ dependabot PRs which shows how much Dependabot is helping.

More automation

Dependabot has a problem though: you end up with a lot of open PRs. To automate the process of merging them, I wrote a little script to “pack” PRs together. The script replicates the go get operation for each dependabot PR and builds the project with the changed dependency. If everything builds correctly, my script continues to the next PR, if not, it stops so that I can investigate what changed in the dependency and work on a fix. More often than I would wish, things do break, but the fixes that need to be done are minimal. This process is relatively quick and whenever everything passes, I create a grouped PR like this one.

Continuous scanning

Another problem that I am regularly facing is the fact that the image being built without vulnerabilities, means nothing over time. A perfect image with zero vulnerabilities today, will definitely have a few vulnerabilities in a year. This means that we have a need of continuously scanning at least the last release of the project so that we know when we have vulnerabilities, their criticality and when to act. It’s also a given that being at zero vulnerabilities all the time is hard and often doesn’t justify the amount of work that it requires. For this reason, we need to know what the vulnerabilities actually are and how they affect ExternalDNS. To do so, I configured a separate GitHub repository that uses GitHub actions to continuously scan the latest Docker image for vulnerabilities and build a report that I periodically check. When there are vulnerabilities that can’t be ignored or are above a certain criticality, we start the release process.

Conclusion

If you found all of this boring, chances are that you are already doing good with your project. If not, I hope this inspired you to take a look and to follow similar steps to keep your project and images (if you have any) up to date.

I can now say that at any point in time, I know how many vulnerabilities there are in ExternalDNS’ Docker image and how we need to influence our release schedule to address those. If users stay up to date with the latest development, they can make sure that they always run a relatively free from vulnerabilities image, which is obviously not executed as root by default. While ExternalDNS, by nature, is unlikely to be the most sensitive workload you will run on your Kubernetes cluster, I am proud to be able to play even a small part in keeping everyone’s Kubernetes infrastructure a bit more secure.