On measuring developer productivity

For the past decade I’ve been hearing about measuring developers’ productivity in tech organizations, I’ve read books about it, I participated in countless user surveys and I have even organized user surveys. Over the years, I’ve grown more and more skeptical about the measuring part of this practice: I think it can serve as a starting point to get a sense of how an organization is doing when a company starts from a point of not having any understanding of what affects productivity and what really is going on and that’s about it. With this I am actually arguing for looking into improving engineers’ lives and in general developer productivity, but, over the years, I grew more and more skeptical about the measuring part.

Surveys as a discovery tool

If you want to get a sense of your organization, you necessarily need to start somewhere and surveys are a great tool to do that. A well run survey can be invaluable in collecting information from employees and they are especially suited for larger organizations where it’s not feasible to go talk with everybody. At first, it seems easy to do a survey, but it’s actually hard and an art to master. Especially hard is deriving data out of a survey: even in the presence of measurable things and scores, the data will necessarily be affected by the internal dynamics of a company, recency bias and so on. Scores and any data that is represented as a number can also be “weaponized”: you can take the numbers and make them say whatever you want, a practice that I’ve seen used by people in different positions to justify a project or motivate not funding something. And then there are weird correlations that don’t always mean anything:

In conclusion, surveys are great for discovering things and learning about one organization, but not necessarily amazing for deriving measurable metrics.

There is no substitute to talking with people

Have you tried? Your peers know what they miss and what they want. We want surveys to reach out to people that won’t or can’t talk with us, but we want to be able to have conversations with our peers. I think a mistake I often see is treating colleagues in the same ways we treat external people. We use surveys when talking with people is not feasible, when we’re dealing with an extremely etherogenous population, when we are asking things to people that we can’t reach out to (i.e. population of a country). At work, we can talk to people! My extremely empirical experience is that this is overall a better use of time than diving deep into the numbers of surveys and that roughly most organizations, after having dealt with the initial discovery phase, they know what needs to be fixed, the struggle is not getting it prioritized and the numbers in the surveys won’t necessarily help with that.