The Engineering Maturity Review — a team self assessment approach to improvement
Even if engineering teams have a good understanding about their problem space, experience shows that many of them still need some ideas where to start improvements related to engineering practices.
This is specifically true for teams in rather traditional IT organisations that are faced with the need to move to a more adaptive way of software engineering.
Transformation processes are complex and need a lot of conversation around what can be improved and how that could be done. From our experience it is useful to have this conversation in a structured way — with and within the team.
The team of Engineering Coaches here at our company discussed how this can be done in a useful way and we came up with our Agile Engineering Maturity Review (AEMM).
This review is done based on a (rather) simple questionnaire that describes engineering capabilities in the areas of CI/CD, DevOps, Test and Test Automation in various degrees of maturity. In a joint session with engineering coaches, the team selects the answer that describes their current way of working best und thus puts itself into a maturity bucket.
It is strictly a self-evaluation process by the team, the framework just provides some guidance and structure and assures that all of the relevant areas are covered. This review process gives a baseline, it helps the teams to create common understanding about their way of working, and it also helps to figure out where to start any improvement activities.
Obviously, the self assessment is in no way calibrated and consequently can’t be used to compare and rank teams. This would be a futile exercise and besides this, will immediately make sure that teams won’t feel safe to answer honestly!
After the review, the coaches discuss the results and propose five top potential improvements — these proposals are then discussed with the team in a second session.
It is important to understand that the AEMM is just one tool to help teams improve. It can start conversations around important engineering topics but this is obviously not sufficient for any improvement.
Change is not happening linearly and there is no single path. Team context and culture, business needs and boundary conditions all need to be observed — but the most important thing is to start with something, collect feedback, and start all over!
Improvement is always possible — considering this it is clear that there is no end-state of an improvement journey. Even if a team scores itself top on all dimensions of the review it doesn’t mean that it has achieved an optimal state of engineering capabilities. This state is simply not existing.
Conversations must be focused on capabilities and not on maturity scores if change should happen.
Our experiences with the AEMM
We have really good experiences with this guided collaborative approach to figure out potential improvements.
But there are a couple of things that we learned in the practical execution:
- Doing the review with a team requires a lot of explanation upfront. It is crucial that teams really understand that this review process is about them, and this is not some kind of audit.
- Best results were with teams on site in a room. Difficult and hard to do in times of pandemic but whenever we had the opportunity to do it on site it was a rather joyful and engaging experience. Of course, it also works in a remote setting.
- Engineering team only works better. In some cases, if Product Owners and/or management is part of the meeting we experienced the tendency that they jump into the discussion and try to explain why things are done in a specific way and why it can’t be changed. We also observed defensive behaviour in some cases (rarely).
- Two engineering coaches are good to keep the discussion ongoing while still being able to take some notes.
- Doing the review after while again (maybe just partly) is a great way to track progress and keep focus.
Final words
The AEMM proved to be quite a useful tool in our toolbox to enable teams to improve in their engineering practices. It is especially useful for teams that don’t have a lot of experience and need and appreciate some guidance where to start. Providing context and explaining the usage of the results to the teams is very important! Teams don’t want to be measured and compared and they are right.
Our team has decided to open source the Agile Engineering Maturity Review questionnaire a while ago. So if you are interested check it out on GitHub here: https://github.com/raiffeisenbankinternational/AEMM
And by the way, we are hiring: https://jobs.rbinternational.com/job-offers.html