Things I've been doing recently

This blogpost is not the normal writeup about how I got into a new job, about the things that have been awesome or that suck. It’s not about love for my employer or previous ones and there is definitely no hate at all. So what’s left? This one is really about a bunch of things that I’ve been doing recently and in a somewhat different fashion:

• I switched my default working mode to “pair by default”. This was needed for me to get used to work with new colleagues, know them better and build a connection with remote members of the team. It turned out to be much better than that. It increases my concentration, improves the quality of the things I do and reduce risks. Everybody says that, but very few are using this as default way of working and I was definitely not doing it in my previous job, not because of any blocker, just because that was the way we were working (and it was fine). Well, I am enjoying this so much I’m myself very surprised about it.

• I take time in isolation every day. Seems like a contradiction, right? You cannot be all the time pairing. It is tiring and I need some time for myself as well. To solve that, I seek every single day some time to work in complete isolation, not long, maybe one hour. We have some nice little rooms in our office that are definitely not good if you suffer claustrophobia, but they work beautifully to regain full focus to crash a single specific task alone. It feels good in there.

• I learn new things. We have mandatory (!) personal development days, which means time dedicated to learn new things during working hours. During those days, I do only things I enjoy. If I haven’t written much code during the week (it happens, YAMLs are my life), I do coding. I can watch a long video or learn a new technology. What is important though is that it has to be something I enjoy, because I feel I learn much more when things are fun.

• I draw a lot of technical stuff. I do that because I realized that if I don’t draw things, I have problems visualizing problems and understanding how to solve them. Sometimes I even draw YAMLs! Where I work right now I’m not surrounded by whiteboards as I used to be for a long time and I really miss that… but guess what? You can just take some paper and make nice drawings on that. I’ve been using this technique for slides lately, instead of spending hours with digital graph tools, I just grab a bunch of sharpies, paper and start drawing, then I snap pictures… and the slides are made!

• I don’t allow myself to keep working hard when tired. Would you drink alcohol and drive at the same time? I bet you wouldn’t, cause it is irresponsible. The same applies to most of the tasks at work and being tired. Now, if you read my tweets you know I’ve been through some serious emotional storm lately due to the tragic loss of a dear friend. Well, sometimes my mood is not perfect or I feel I get tired easily due to some stress that is sticking around… guess what I do when this happen? I go home and relax! Of course this doesn’t mean that I work only few hours, but I try to apply reasonable approaches in which I try not to push myself too much over my limits.

• I trust (and I feel trusted). I mostly never need to know what my colleagues are doing and they don’t ask too much either. This doesn’t mean lacking care, more letting control go. We can work async. I can write this blogpost. They can take their time to do things. None of us need to know, we only make sure we enjoy working together and that we are more or less on track with what matters which is to keep doing the best for the company while becoming better at it.

• I take time to enjoy life. Taking a long lunch break to meet a friend from time to time isn’t a bad idea, it’s actually really fun! I’ve been meeting friends and building better relationships and if I can do this during the day, well that’s great.

End of my random selection of things. That was not an exhaustive list and it don’t know if there is any benefit in this but I hope nonetheless that you will find this useful!

EKS "Review"

NOTE: those are just early impressions on a very promising product that I am very happy to see available and I am sure it will get better over time compared to what it is today.

Today I spent some time playing with EKS (Elastic Container Service for Kubernetes), the newly announced Kubernetes service from AWS and I decided to write some notes down, not only for my team, but for whoever is interested in knowing more details on EKS.

I started by playing with the official tutorial that can be found here. I was right from the beginning pretty surprised by the lack of automation of the whole process: the UI or CLI only help you getting a very basic control plane up and running and nothing more while leaving a lot of steps to be executed manually.

During the time I was waiting for the cluster to be ready, I didn’t get so much feedback on what was going on. In general, I’d love to see AWS give a shot at a more user friendly experience: what I mean is, for example, to put a link to the documentation or tutorial directly in the UI such that users can jump to interesting material while the cluster is still creating. In the end, EKS was not only built for people that already know Kubernetes and how to operate it, but for the other AWS users out there that are interested in containers and container orchestration and that don’t want to operate a complex system like Kubernetes.

The getting started guide is unfortunately not super friendly from that point of view, but there are many other tutorials on the web that can get you up and running with the basics of Kubernetes.

Something that surprised me as well is that there is no button to download the kubeconfig. This is the configuration file needed for kubectl (which i pronounce koob-cee-tee-el because I can’t unlearn this way of pronouncing it) to connect to the cluster, which contains the name of the clusters and other information.

This file would be pretty easy to generate based on the information that are available in the UI and it would really be trivial to make it available for download such that we could easily get started. In a similar way, it is not exactly super easy to get started given that we have to download kubectl and Heptio’s authenticator. As a comparison, GKE on Google Cloud Platform, has a way to easily get both kubectl and the relevant configuration via the official gcloud tool which is indeed very handy.

In my case, the cluster creation took quite a while, around 15 minutes. That’s much more than I was used to on AWS with tools like kops which takes around 5 minutes to get a cluster up and running with very similar characteristics. It’s true though that this is not an operation that we are going to be doing really often, so it doesn’t really make a huge difference.

After the cluster creation, what we really get is a Highly Available control plane that we don’t see and can’t modify. This is in fact hosted somewhere in the AWS infrastructure, but we have no access to it. What is needed to be done to finish the cluster setup is to create worker nodes to attach to the cluster to actually run some applications. In order to do that, AWS provides a CloudFormation template that allows to spin up nodes easily. The output of the CloudFormation template is an AWS IAM Role that needs to be put in a ConfigMap. Why do we need that?

Well, the nodes need to somehow join the cluster and EKS is using the IAM identity and Heptio’s AWS authenticator to allow the nodes to do that. That’s very AWS native and, in my opinion, quite nice compared to other setups that use shared tokens or just certificates to do that.

Following all those steps, we finally have a working cluster. Was this painful? A bit, especially given that there are plenty of tools out there that made the experience of creating Kubernetes clusters really easy.

In that regards, I tried as well eksctl from the folks at WeaveWorks and the whole experience was so much better!

You can easily create a cluster with:

eksctl create cluster

While the whole process is still not super fast, you will get a cluster in one simple command and this includes the worker nodes. In my case, the command failed to setup a kubeconfig because it tried to use my local kubectl which has a weird version (because it’s a version I compiled by hand, but that’s a story for another time). The rest though, worked pretty smoothly.

The setup we get once we have an EKS cluster running is really simple. Here a list of pods that were running in my cluster (all in the kube-system namespace):

NAME                       READY     STATUS    RESTARTS   AGE
aws-node-6vrp2             1/1       Running   1          6h
aws-node-k2dc2             1/1       Running   1          6h
kube-dns-7cc87d595-cj9js   3/3       Running   0          6h
kube-proxy-8nccp           1/1       Running   0          6h
kube-proxy-lshj8           1/1       Running   0          6h

This is very minimal and has very little overhead, but it is also not a really complete setup. This is in fact lacking support for Ingress and does not have anything in place to setup IAM roles for the pods. This is a bit surprising as AWS talked multiple time about AWS IAM in Kubernetes and I assumed that they would start with Kube2IAM and then improve the setup with something different.

The decision for AWS IAM was probably to let the user roll their own solution from the ones that are available as opensource projects while leaving the door open to develop a better solution together with the community. AWS is already working on it and some details of a possible future implementation are available in this Google document that was discussed in the last sig-aws meeting.

Apart from some rough edges, I’m happy to finally be able to use EKS and I’m looking forward to see it available in other regions. From my experience using Kubernetes since version 1.0, it is not super trivial to operate the control plane, especially at scale and with an HA setup and that’s why such a proposition from AWS was so requested from AWS customers and hopefully we will go towards a more managed solution in which managing the cluster will just be a detail that we will not have to care about.

Junior Engineers

Yesterday night I went to see Erlend Øye live in Berlin. He did a great show, during which he played with some young musicians from the south of Italy where he is living (Syracuse). Some of them were really new to playing live: Erlend told us that one of them did not play ever live before the start of the tour and another one literally said “I wrote this song when I decided to start writing songs… last December”. Erlend left the stage, sat of the floor and listened to this not so tall guy play a very beautiful song of his.

When he finished, he got a long applause from the audience which he indeed deserved. Erlend came back on stage and said “a star is born” and “Prince was 1,5 meter tall” meaning that being short, Italian or having started only 6 months ago writing songs is not an impediment to do great stuff and it shouldn’t be.

What does this have to do with the title of this post? Well, if it’s not clear, this has many analogies with the work environment. Think about giving opportunities to junior members of your team, about hiring juniors to give a breath of fresh air to your songs (ehm codebase)… doesn’t it all make sense?

And if you are a senior engineer, you could think about mentoring those juniors, give them the opportunity to go on stage and demonstrate their skills (and of course allow for mistakes) like Erlend did. And don’t be the only one under the main lights on the stage, let them take ownership of some topics if they feel ready for it.

And go see Erlend Øye, it’s worth it.

Marco

Marco era un pezzo di me. Marco mi ha introdotto a Linux, prima che fosse davvero cool, quando Ubuntu non esisteva. Ricordo ancora i miei primi sguardi a Red Hat con lui. Se non fosse per lui, probabilmente starei ancora usando Windows 98 o qualcosa del genere.

Con Marco ho condiviso una casa e dei momenti unici quando vivevamo entrambi a Milano. Con Marco, ho giocato a fare l’adolescente, il maturo, il nerd, il birraio. Ho condiviso passioni e alcuni dei momenti più belli della mia vita. Abbiamo perfino comprato due magliette identiche con il famoso Blue Screen of Death!

Di Marco mi rimarranno sempre i suoi CD con scritto “Mack”, il sentirsi su Soulseek, le giornate passate su IRC, il Solero Smoover, la Pazzeria, Via Koch e il ricordo di tanti istanti indimenticabili. Con Marco ho condiviso anche alcuni momenti davvero stupidi, come in ogni buona amicizia che si rispetti. Stupidi come quando l’ho riportato dai genitori piuttosto ubriaco o come quando ha riportato a me a casa piuttosto ubriaco. Ci volevamo bene e ci aiutavamo.

La scorsa notte Marco se n’è andato e con lui se ne va un pezzo di me. Non avrei mai pensato quale sarebbe stata l’ultima cosa che ci saremmo detti. Io gli ho mandato una foto di una tazzina con scritto Danesi, trovata a Berlino. Scherzavamo spesso sul “Danesi”, che usava(mo) come sinonimo di caffè. Lui ieri mi ha risposto con il suo solito “Buahahahahaha”. Apparentemente felice, positivo.

Tre ore dopo, sarebbe morto, ma io lo ricorderò per sempre così: felice, che ride.

Quasi paradossale quel che ci dicemmo qualche giorno fa, parlando di linguaggi di programmazione, di Go e di Rust. Marco mi disse “Avrei bisogno di tempo infinito, Rafè.”.

Ora ce l’hai Marco. Non ti dimenticherò mai.

My Great OSS Sin

At my previous job, I was part of the original team that tried several of the existing tools to setup a vanilla Kubernetes cluster on AWS. We tried kube-aws, kops and looked at others and decided that none of them was perfect for us.

Many had shortcomings, mostly because of some limitations or bugs which were perfectly natural given the state of the Kubernetes project itself. We decided then to start from kube-aws, customized the configuration and never really looked back from there. We released our configuration as open source (and we even got some GitHub stars, yay!) but the configuration itself was never a project designed to be usable right out of the box for anyone else, for a lot of reasons including that it had a bunch of optimizations that were designed to work with our tool to update the clusters. The aforementioned tool was also not open because it relied on more assumptions and homemade tooling that wouldn’t have been valid or good for a wider audience.

I was myself opposed at releasing all of those things in the open because it provided, according to me, little to no benefit to other people outside of the organization. The real pity is that we/I never had the time to really look back and understand if we could just merge or integrate the project with other ones or just collaborate with others and I see the same happening in other opensource tools in the Kubernetes ecosystem today (again, kube-aws, kops). A colleague (hi Martin), hinted at possible collaborations and merges but I never really listened to him carefully and he eventually desisted. He was right and I had no ears to really pay attention to his reasoning. And that’s my mistake.

At my new job, I had to setup a Kubernetes cluster. The situation, from the tooling point of view, is a lot better than it was when I started looking in 2016, but still lots of tools are not complete nor perfectly fitting my wishes. And the former Kubernetes configuration which I helped develop is there, essentially unusable for me now. This is what it represents for me today: a good configuration for a best in class Kubernetes setup and at the same time my failure as a developer who firmly believes in open source and in building usable projects and communities around projects.

Looking back is always tough and I often need a special point of view to be able to fully do it. I am not sure what I would be doing differently if I could go back: probably rush the setup a little bit less and don’t aim at the moon from day one. Probably design for one cluster before designing for 30, probably deploying a full production one before deploying 30. And surely I would take less shortcuts that will get technical debt in with very little time to pay it back.