Technical debt is quite a big topic and not equally understood, especially in planning and business discussions , this blog post is an attempt to tease out some strands of truth of technical debt. Having had the privilege of working with a DevOps team it has currently been top of mind for a while.
The term “Technical Debt” is commonly attributed to Ward Cunningham (https://en.wikipedia.org/wiki/Ward_Cunningham) and it is a metaphor for a financial debt:
You need some money, you borrow it, you pay it back with interest.
or is it? These days borrowing money is not so simple.
You need some money, you then get credit rated, you get investigated based on your ability to repay that debt, and then the funds are released, if you don’t pay back the debt then trouble happens, oh and there is interest, which means the financial importance of what you do now must exceed the importance of what you might want to be doing later.
In the same way.. technical debt.. is not so simple. Most people agree its something you incur at the short term, for a short term gain, at the expense of a longer term cost.
Lets take the example above and metaphorically break it down.
You need some money (you have a deadline), you then get credit rated (the technical architect is alarmed), you get investigated based on your ability to repay that debt, (you know the code is dirty but we have a deadline) and then the funds are released (the project manager forces it through with much protesting from everyone else), if you don’t pay back the debt then trouble happens (the next release is harder), oh and there is interest (we need bug fixes), which means the financial importance of what you do now must exceed the importance of what you might want to be doing later.
Some people have suggested technical debt is a calculated choice not as a result of messy code.. ie Uncle Bob (https://sites.google.com/site/unclebobconsultingllc/a-mess-is-not-a-technical-debt ). I think its a good thing to read this to distinguish between intentional tactical direction and sloppy code, however unless they are throwaway bits of code, they both amount up to pain in dealing with it later, and if someone has to pay it, its debt.
Another way of viewing it is by looking at different types of debt:
- A mortgage, is often perceived as a sensible debt, its still a debt though, and people work on repaying it. These are often paid off over 20 even 30 years.
For me technical debt of this kind is an unavoidable by-product of being a software developer, its simply keeping your processes clean, writing clean code, with sensible documentation, and refactoring regularly.
- A loan for a car, is a less sensible debt, but a car could facilitate transport to a place or work, it could be needed, but either way a car will cost you a lot more via a loan, than if you paid up front on it. Car loans are usually restricted to 3 years.
This is your tactical decision, where you chose a faster delivery vehicle over a sensible one for a short term gain, pun intended.
- A credit card debt, is not perceived as a sensible debt, unless you pay it off in full each month, a credit card will be very expensive and work towards making you bankrupt. This is your messy code, its still debt, its ungainly, horrible but needs dealing with because it happens.
My current view is this:
Technical debt, is the negative effect of producing a deliverable which results in:
- anything from a slight slow down to a complete halt in future work
- anything from rework of the existing deliverable just to keep the status quo to a complete rewrite of the existing deliverable
Its that big.
What are its causes:
- Poorly tested code – Either by a lack of testing or poor code testing coverage, maybe the developers are not writing automated unit tests… This results in rework, and will appear in all test environments as well as production, this means all teams will be experiencing it and having to deal with it at some point, as well as the developers who will need to fix the issue, the testers to test it, and the DevOps to promote and and the Production support team to support it.
- Poorly configured code – This hurts everyone, especially DevOps teams who need to take the configuration and apply it to as many environments that are required. Any mistakes or inefficiencies here are multiplied and have to be paid back in multiples. One developer on one machine has one problem to fix, but that becomes placed into multiple environments.
- Poorly documented code – What does this do? How does this work? I don’t understand? This is complex, this is too confusing. Why…what the? eH? Oh its broken, how do I fix this?
- Poorly written working code – Anything that is overly complex, and doesn’t follow existing design conventions, even well documented its going to take extra effort to either refactor or understand it.
- Unconventional Legacy code – Specifically code written that doesn’t follow current conventions or perhaps is in another language due to local optimisation or a lack of skills liquidity. This is something that would often require the hire a specialist skill to solve, or extra learning from existing team members.
- Lack of continuous integration – More related to scaling delivery, but manual actions repeated can oft get multiplied and really hinder growing projects with large numbers of people working in them.
- High staff turnover / Bus Factor – No matter how well documented, technical knowledge, competency and proficiency is ALWAYS held in people. If you have people always needing training every 6 months, this is a problem. If you have a person with all the keys leave, the project will slow and halt. If you invest time in training and increasing the Bus Factor then you are paying more now, so you don’t pay big later. Often these people are too busy doing the work to train anyone. In effect they have traded the long term for the short term, and the cost will come back like a bad debt. Remember in many job adverts there is a requirement for developers to pick up a steep learning curve? That’s because of the existing complexities of the code. You need to start paying the debt of learning that stuff before you can even begin. A creative designer however may only need to learn the simple curve of a company’s branding standards before actively working.
- People unwilling to repay technical debt – This one is a bit of a “brain bender”, but a reluctance to repay technical debt, is going to cause a “compound interest” effect to happen and accelerate and exacerbate the process. If you are unwilling to repay it, then you need investing in.
- Code that accumulates in production – Its in prod, better not take it out. You think it does nothing, but you dare not take it out of prod. Everyone wants a prod-like environment so now its everywhere. As it is everywhere, it needs to be maintained, because no-one knows what it does anymore its not documented, and as its not documented … we probably shouldn’t take it out of prod. This is extra overhead and messy, and hides genuine issues because instead of looking at a problem you are looking for a problem in the middle of false negatives. This is an equivalent of “junk DNA” and it takes a structured deliberate testing approach to remove it.
How do you fix this
Given my descriptions above, technical debt is actually an unavoidable byproduct of software development, where there are real people with real deadlines. Just as humans have natural (waste) byproducts, it doesn’t mean we need to stink of them. We simply need a system to deal with them, a regular cleaning schedule perhaps.
Moving back to the metaphor of paying off Debt, the answers are simple.
- You need to identify it: – Technical debt appears in different teams in different amounts with different effects. A certain amount of debt is unavoidable. (Try buying a house without a mortgage). The whole team needs to understand it, else you will end up with one side generating the debt and the other paying it off.
- If you can stop generating more debt. Make a change in the way you do things, stop spending.
- Pay of the debt with the highest rates of interest first - dont spend time gold plating something that’s never going to be looked at again, hit the targets that are going to be hitting you hardest.
- Discuss it in your next retrospective, everyone needs to know about the “slow elephant of code doom” in the room. If the business owners don’t understand it you can explain it will have a drag effect on your velocity. But usually business owners don’t understand it because they don’t get to hear about it.
- Set aside time each planning session to prioritise a piece of technical debt.
- Don’t fill your entire backlog with software deliverables – leave space for improving code quality and processes.
- Wiki – Most teams have this, living documentation, make sure you have regularly updated guides for team processes, delete old pages, do your wiki gardening.
- Invest in your people, encourage learning, pair programming (or at least working together).
- Invest in a collaborative environment, encourage problems to be discussed and solved earlier on, instead of hidden away in poorly documented or written code which is like a ticking time-bomb for anyone later on.
- Don’t try to hit all of your technical debt at once, you still need to deliver for your buisiness, you need to eat and clothe yourself before making the overpayment on that mortgage.
Notes on Scaling Agile with technical debt
- Technical Debt is non-linear, even more so in scaled teams. Just as the benefits will scale, the negatives will do also scale.
- Vendor technical debt can get you by “the proverbials” and is your problem as much as theirs
- It will take a lot more coordination to deal with technical debt in scaled agile, simply because there are more people, with different deadlines, different motivations and under different pressure.