Why Scrum prevents and causes Technical Debt
4 Scrum anti-patterns that lead to technical debt and what you should be doing instead
Let me set one thing straight right off the bat: Scrum by itself doesn’t cause technical debt. However, few companies do Scrum right. As a result, many Scrum implementations do end up creating significant technical debt.
All four technical debt producing anti-patterns described in this article are widespread, based on my personal experience working at and advising many companies.
Why do some Scrum implementations inadvertently generate massive technical debt while others don’t? What can we do to prevent it from happening?
Before we dive into this further, let’s first start by defining technical debt. Then I will elaborate on why certain Scrum implementations produce technical debt and what we can do about it.
Let’s circle back to technical debt.
What is Technical Debt?
The best explanation of technical debt I’ve come across is by Henrik Kniberg in his article Good and Bad Technical Debt (and how TDD helps.
“Think of technical debt as anything about your code that slows you down over the long term. Hard-to-read code, lack of test automation, duplication, tangled dependencies, etc.“— Henrik Kniberg
However, he also makes the case that aiming for zero technical debt will slow you down too. Surprising and counter-intuitive, right?
Here’s how Kniberg explains it:
Think of your computer and desk when you are in the middle of creating something. You probably have stuff all over the place, old coffee cups, pens and notes, and your computer has dozens of windows open. It’s a mess isn’t it? Imagine if you had to keep your workspace clean all the time — every time you slice a vegetable, you have to clean and replace the knife. After each painting stroke, you have to clean and replace the brush. After each music take, you have to unplug the instrument and put it back in it’s case. This would slow you down and totally kill your creativity, flow, and productivity. In fact, the “mess” is what allows you to maintain your state of flow — you have all your work materials right at your fingertips.”
Whatever we do, we will always make a mess. It’s as inevitable as trying to escape the 2nd law of thermodynamics. ‘Fresh’ mess is inevitable to preserve flow and work effectively. It’s the ‘old’ mess that comes back to haunt us as technical debt.
The critical thing to remember: we will always produce a mess. That is both a good and a bad thing, depending on when we decide to address it.
Now that we’ve discussed what technical debt is, let’s discuss the different Scrum anti-patterns that produce technical debt and what we should be doing instead.
Anti-Pattern #1: Developer estimates exclude fixing tech debt from estimates
What usually happens:
You’re in refinement and the Scrum Team gives off an estimate. Except they don’t take into account the time to need to fix the technical debt. As a result, the technical debt in your product increases over time because they never grant themselves the time to fix the technical debt.
Technical debt is something the team handles later by visualizing it on the Product Backlog. Then the team needs to convince the Product Owner not to remove it and get it prioritized. Under constant pressure from stakeholders, the Product Owner deprioritizes or even closes it, as there are more important things to work on.
Is this really how things should be?
What we should do instead:
As a Product Owner, I expect developers to give off estimates that include fixing the technical debt. Including technical debt is something I explicitly ask from them. I literally tell the Scrum Team the following during refinement:
“When you cook, cleaning the kitchen is part of the work. A cook never asks for permission to clean the kitchen. It comes with the job. If you don’t clean the kitchen at some point, you won’t be able to cook anymore with all the mess piling up. You don’t need my permission to fix technical debt. Give off your estimate that includes fixing of any applicable technical debt. I don’t even need to be aware.
I usually don’t ask questions. Sometimes I do because something I expect to be easy turns out to be difficult. Then I want to understand why, because maybe we can reduce the scope get something out the door quicker. There could also be significant technical debt I need to know about. Perhaps our technical implementation is too complicated, while in the initial version, we can get away with being able to handle less traffic.
Technical debt is not something we prioritize separately on the Product Backlog unless we’re in such a dire situation that we have to.
The essence of Scrum is being able to deliver value regularly. If technical debt prevents you from delivering a Product Increment every Sprint, then you’re not capable of doing Scrum. Therefore, controlling your technical debt is vital to succeeding with Scrum.
Anti-Pattern #2: We must finish everything in our Sprint
What usually happens:
At the end of the Sprint, the whole team gets stressed. Everyone is worried we won’t be able to finish everything on our plate. Our stakeholders and Product Owner will be disappointed and angry. What should we show at the Sprint Review? Everybody puts in extra work to finish as much as possible. Yet, we still fail.
This happens because our estimates will always be off. No matter what we do, estimates are based on limited information: the knowledge of beforehand. Therefore it is inevitable your estimates will be wrong. This is the nature of complex work, you don’t know what you don’t know.
As a result, we often will have Sprints where we are doomed to fail from the start. Except we only realize this somewhere during the Sprint, often near the end when our degrees of freedom are limited. Regardless of how hard we work, we won’t complete all Sprint Backlog Items in the Sprint. Period.
Except there are teams that are still able to pull off finishing everything in the Sprint. How do they do this?
They achieve this by cutting corners and creating technical debt. Our stakeholders and Product Owner will notice uncompleted Backlog Items, but they will only notice the technical debt and lower quality when it’s too late. Like a frog that slowly gets heated until the water boils.
What we should do instead:
An Agile way of working requires flexibility. We need to be able to adjust our plans and actions on the fly, also depending on whether they produce the expected results.
A prerequisite for this flexibility is the presence of a clear overarching objective. What are we trying to achieve and why? What is the one thing that matters most now?
When you have limited time, you are forced to make choices. When completing everything in the Sprint is the goal, then there is no freedom to make decisions. Everything is equally important.
When you do Scrum, you need at least three things:
A clear and overarching objective for each Sprint called the Sprint Goal. The Sprint Goal should explain what we are trying to achieve and why.
Flexibility on the plan of the Sprint.
Flexibility on the scope of the Sprint.
Flexibility on the scope of individual Sprint Backlog Items.
Remember, after the Sprint starts, the following is fixed:
Time — duration of the Sprint.
Quality — Definition of Done.
Resources — adding people during Sprint often isn’t possible and generally will slow you down.
When those estimates are off, your plans are inaccurate, flexibility in scope is necessary to respond without sacrificing quality.
Anti-Pattern #3: We must increase our Velocity
What usually happens:
Many Scrum Teams are judged on their Velocity. The underpinning belief is the following:
Higher Velocity = More effort spent on delivering features = More value
As a result, the team cuts corners or artificially inflates numbers to increase Velocity. The result is a delusion that keeps stakeholders and management happy while achieving nothing but undercutting your efforts to deliver as much value as possible.
What we should do instead:
Your customer is oblivious to the effort you spent. They couldn’t care less how much work you put in. They only care to the extent the result of your work helps them to achieve their goals.
This article is ~ 2300 words long. Can you judge its quality by the word count? Imagine I write 8000 words worth of articles in one month and 3000 in another month. Which month did I do a better job?
Words only matter to the extent they resonate with your reader. More words does not mean more value. By the same token, completing a higher amount of Sprint Backlog Items in terms of effort is unimportant.
Your effort only shows to the extent it makes a difference for your customer.
Anti-Pattern #4: Weak, absent, or unaligned Definition of Done
What usually happens:
Many Scrum Teams have a weak, absent, or unaligned Definition of Done. The Definition of Done is a quality checklist for the Scrum Team to follow. It makes transparent that when something is ‘Done’, what this means.
When a Scrum Team has a weak Definition of Done, e.g., excluding a decent level of testing, this means you will be producing technical debt every time work is completed. Some teams don’t even define a Definition of Done, or others write it down but never use it as a checklist before putting an item to Done. Both cases produce the same result: technical debt.
But by far, the most common problem is the belief that each Scrum Team is allowed to set their own Definition of Done. This perceived freedom is completely false. If multiple Scrum Teams work on the same product, or the organization has set a quality standard for the teams, then all Scrum Teams should at least follow the same Definition of Done. The purpose of this shared Definition of Done is to make their combined work potentially releasable.
What we should do instead:
If the organization sets a quality standard, then all Scrum Team must follow this as a minimum. Individual Scrum Teams may adjust their Definition of Done to be more stringent, as long as the organizational standard is followed as a baseline.
If one or more Scrum Teams work on the same Product, then all teams working on that identical product should define a Definition of Done they all should follow as a minimum. The same applies to the Definition of Done set on an organizational level — individual teams are free to apply more stringent criteria.
Even when you don’t fall victim to these anti-patterns, technical debt is inevitable
Technical debt is perceived as something dirty. However, have you ever worked on a system without any technical debt? Have you ever worked on a system where the developers said, “Wow, isn’t this code amazing and pleasant to work with!”.
In my 10 years of building software products, I never have heard a single developer spontaneously utter these words out loud. Complaints about the technical quality of a system seem to be universal, no matter the state of the system. However, there is a difference between occasional complaints and incessant “We can’t work on this product” type of complaints.
Technical debt is not something to be prevented at all costs. It’s something you can never prevent entirely. The main problem is that software is built in a particular context. Whoever inspects the code, doesn’t have the same context and might have different (and better) information at their disposal. When viewing the code through their lens, the choices that were made seem suboptimal while they might have been considered optimal at the time.
The opposite can also happen. Some choices are later considered suboptimal, but they are actually optimal. Those who judge it now lack sufficient context to understand why those choices made sense at the time. Sometimes Developers can actually even make things worse by reverting those choices because they don’t have enough understanding of the domain or what the product is doing. The people who do have unfortunately often have left the company already.
Scrum purists will claim these technical debt creating teams are not doing Scrum. That’s just an excuse to not look yourself in the mirror and see what we can do better. Misunderstanding of Scrum is prevalent. It is too easy to point fingers without seeing what we can learn from it.
Scrum gives self-managing team massive power to control their technical debt. The developers are responsible for how the work is performed and also the estimates (if applicable). If technical debt fixing is part of the work, then it falls under their jurisdiction. They decide how the work is done. If the unforeseen rears its ugly head and increases the complexity, then the self-managing Scrum Team has flexibility on the scope to meet the Sprint Goal.
There is one caveat to the above: the Product Owner is responsible for maximizing the value that is delivered. Over-engineering, or fixing too much technical debt in the short term, may result in lower value being delivered. Therefore a healthy tension is necessary between the Product Owner and Developers to make the best trade-off decisions.
The Product Owner should talk to their team about the technical approach to understand what trade-off decisions were made. Cost of maintenance and time-to-market are important factors that affect the delivery of value.
Excessive Technical Debt in Scrum Teams should be uncommon. However, many Scrum teams fall victim to these anti-patterns and therefore it is quite common. Technical debt is never something you can eliminate completely, but you need to control it to be at acceptable levels.
Being flexible, predictable, and responding to changes becomes impossible with sufficient technical debt. You can’t be Agile when working in such conditions. When you use a disciplined Agile framework like Scrum, managing technical debt is a package deal. Scrum, and the regular delivery of value, won't happen without it.