Developer experience: What actually drives productivity?

Abi Noda, Margaret-Anne Storey, Nicole Forsgren, Michaela Greiler | 2023 | Paper

Summary

Devloper Experience or DX focuses on the lived experience of developers and the points of friction they encounter in their everyday work. In addition to improving productivity, Developer experience can drive business performance through increased efficiency, product quality and employee retention.

Quite a few recent approaches in this space have focussed solely on engineering metrics to measure whether an engineering organisation can be classed as “elite”. Think DORA. The insight that the DX team have driven is that qual matters as much as quant, and the non-technical (clear goals for work, psychological safety, focus time) matters as much as the technical (great tooling, fast releases to production, platform stability). They’ve shown that if we increase developer experience, we also increase productivity, satisfaction, engagement and employee retention - all desirable outcomes for any engineering team. The paper sets out a framework for how to measure DX, and I read it recently so I could quickly put together a survey to measure the developer experience of my team.

Three dimensions of developer experience

The framework breaks down developer experience into three dimensions: feedback loops, cognitive load and flow state. The three dimensions cross cut 25 factors of developer experience that the research team identified. This means you can ask developers about how they feel about each factor and then combine there responses to build up an overall picture of how your organisation is performing on one of the three dimensions. Once you can measure the dimension effectively, you can then drive the improvements to developer experience.

Feedback loops

Shortening feedback loops (the speed and quality of actions performed) allows for increased rates of learning about what’s being built which in turn allows for rapid course correction. Studies have consistently shown that organizations deploying more frequently and maintaining shorter lead times are twice as likely to exceed performance goals as their competitors. Feedback loops are also important to achieving and maintaining a flow state, think about why when programming languages with a REPL and JIT compilers are popular versus those with long compile times. Feedback loops apply everywhere in our work - test run times, deploy times into test and production environments, code review times or the ability to run experiments with users. Organizational structures should also be optimized to promote streamlined team interactions that minimize delay.

Cognitive load

Cognitive load is just the amount of brainpower it takes to do something. It increases if we’re doing unfamiliar work or grappling with complexity, and decreases if we’re working with strong patterns and well defined problems. Cognitive load has a direct relationship to the productivity - they move in tandem. You can reduce cognitive load by using well defined patterns when developing, documenting your work and designing software that has a strong a natural fit to the domain you’re working in. Doing these things reduces the amount of context you need to carry with you as you go about your day. Task switching (the rate at which the context your working in changes, for example going from meeting to developing and back again) also increases impacts cognitive load.

Flow state

Flow is optimal experience. The markers of flow experiences are that time whizzes by, you’re immersed in your challenge, you’re at ease, enjoying yourself and you have a level of total engagement that leads to heightened performance and creativity. When you experience flow you want to experience it again. Work, particularly coding, can lead to flow experiences. Enabling engineers to have flow experiences regularly makes them happier and enjoy their work more, but it also boosts productivity and quality. Interruptions and delays (think cognitive load) inhibit flow. Other factors include maintaining autonomy over work structure, having a clear goals, fast feedback loops, varied and (just the right level of) challenging work. We can create the optimal conditions for flow state: avoid unplanned work, batch support requests and reviews to avoid interruptions, cluster meetings and make sure work is engaging and interesting.

How to measure developer experience

The switch that the paper advocates for is that rather than focusing on quality metrics about the software or the development process itself (e.g. deployment frequency, lead time for changes, reliability targets, change failure rate and time to restore in DORA) we should instead focus on the lived experience of developing software. The DORA metrics are correlated with strong development practices and high produtivity and engagement because they are the output metrics of a top performing team. However, we all know correlation does not imply causation. The approach of the paper is closer to that proposed in SPACE, which makes sense, given that a few of the authors are the same. I think what the authors regretted about the SPACE paper is that we as an industry immediatly lifted the example measures given in SPACE and just started managing to those metrics. However, given we’re dealing with a human system here it’s clear that context and environment is important so there are no one size approach, just like there’s no one killer metric that describes everything. It’s a mosaic, and the pattern of that mosaic will be different for each organisation - even if we can reduce all the tile colours to RBG values (i.e. the three dimensions we discussed earlier, but perhaps I’m stretching the metaphor too far now).

To do this, the paper gives a mix of example measures that we should be using, proposing that we pick datapoints from three buckets: perceptions, workflows and overall KPIS. Perceptions are human attitudes and opinions. Workflows are datapoints that describe the health of the software and infrastructure that forms part of the development system and KPIs are the high level summary statistics that organisations tend to measure and optimise for already - things like employee engagement. All three of these are important in building up an accurate perception of what’s going on because we as developers (yes our thoughts and feelings) are a part of the development system, just as much as your CI or Kube cluster is. The KPIs get added to the picture as we want to look bottom up AND top down to make sure we’re getting a sense of what’s going on from every angle.

Using this mix means we need to capture data from people and systems. So we should grab our at-hand uptime metrics but also start to survey your developers about their day to day work experience. The survey data is often the lower hanging fruit. System data is generally heterogenous in source and complex to extract, interpret and store centrally. Survey data can be done quickly with Google forms. To this end, the paper gives some recomendations for running surveys effectively:

• Design surveys carefully
• Make sure you can break down results by team and persona (role, tenure, seniority)
• Compare results against benchmarks
• Mix in transactional surveys (capture feedback during the course of doing the job itself, not just as a one off exercise)
• Avoid survey fatigue (respect the surveying bandwidth that you have, make sure you follow up on actions)