Q 427. Agile software development focuses on shorter turn around time, however, it sometimes result in sub-optimal code also known as Technical Debt. Are there any guidelines for Agile leaders to decide between longer turn around time and technical debt? What is an acceptable level of technical debt?

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Technical Debt is – additional costs or incremental costs incurred in maintaining a software code, as part of its life cycle management so that it continues deliver intended objectives to customer, DUE TO compromise made in the quality against timelines at the design stage. This metaphor of Technical Debt is conceptualized by an American Software Professional Ward Cunningham in year 1992.

 

Like financial debt. Technical debt it compounds and increases exponentially if it is not paid back. There is no exact trade off standard on how much technical debt is acceptable, but the general guideline is to keep as low as possible at the design stage, even if it requires some 5 to 10% of additional time to market. High technical debt forces allocation of more resources in code refactoring during its life cycle and often reduces employee morale and affects new software developments or adding new customers. Time to Market, Technical Dept, Refactoring costs – relationship cab be seen on this below graph.

 

 

 

What is the solution

Agile methodology as opposed to water-fall method would significantly reduce technical dept by improving code quality at design stage. By understanding customer user story well, adopting INVEST guidelines, software code can be made better by addressing all three components like deliverable / scope, quality, timelines. Technical debt can be explained with “UNDONE “, “DONE “ works at design stage. UNDONE work results due to lack of good understanding of user requirements, not collaborating time to time with product owner etc.  Instead of living with Technical debt with Undone work, a DONE work can significantly ease out timelines pressure. By clearly articulating final test requirements, involving cross functional teams, dividing project into distinct sprints can significantly improve software quality code, in turn Technical debt. There are other technical guidelines in improving quality of software code. One such guidelines is Halstead complexity metrics which focuses on quantitative complexity of codes with resept to volume, difficulty, Efforts. From Halstead metrics, one can arrive at Time required for program, acceptable bugs in code.

 

Summary: Technical debt may not be avoidable ( up to 5-10%), but certainly can be minimized to large extent by adopting agile methodology , scrum tools at the software design stage in order to meet to customer deliverable in time and cost with quality.

Technical Debt - Is also called as "Tech Debt" or "Code Debt". This describes the output after a development team, taken action to accelerate the delivery of a piece of funtionality in which later needs to be refactored. In simple terms, it’s the result of stressing on faster delivery over perfect coding.

 

Guidelines for Technical Debt

 

It is the Agile Leader and/or Scrum Master's judgement wheter to decide of taking the longer TAT vs. having Technical Debt. However, to help them choose the better decision they may consider the following:

 

1. Factor Tech Debt into "release-level" stages

2. Alignment between organization and technical issues

3. Working Agreement

4. Shot-Term Investments for Long-Term Improvements

5. Iterations

6. Experimentations

7. Iteration-Level Activities

• Code Coverage
• Cost
• Overhead
• Velocity Reduction
• Developer's Satisfaction
• Refactoring
• Slack
• Mentoring
• Flow
• Static Code Analysis
• Retrospectives

The Amount of Acceptable Technical Debt (If Any)

 

The only evaluation of this is “snowball programming” methodology, that this accomplishment leads to more effectiveness in the least crucial parts of the project.

 

Get an estimate of the percentage for the bulk of what can be allowed into critical functions and less important functions.

 

Show and explain this benchmark to your developers from the startof the project, so they will have a quantifiable number to work with. As nearing to the project end, they will inherently include this productivity benchmark together with their other KPIs instead of trying to include efficiency as a last minute addition to the process.

Technical debt or code debt is about developing a solution with no so perfect code or low-quality code to speed up the delivery of the project which needs to be refactored. This has usually been seen in the software development.

Technical Debt can lead to many consequences like poor productivity, financial burden, high maintenance, etc.

Technical debt can be managed through below points –

• To reduce the overall cost, the development team needs to put consistent efforts.
• To reduce the technical debt, refactoring to be taken in account if the system is performing smoothly and as expected. 
• Technical debt to be reduce periodically and a plan to made to ensure that technical debt is reduced.
• Maintaining a knowledge base is extremely useful at any stage of the development. It helps the development team to access the required information and get the effective solution quickly.

Technical debt can have multiple challenges including financial and hence, lower the better. The technical debt ratio should not be more than 5%.

Technical debt which is also known as code debt results when the development teams take actions to expedite the delivery of a project which needs refactorization. It is the result of prioritizing the delivery speed over perfection in code.

 

The three main types of technical debts are as follows:

1.       Deliberate Debt

Engineers often know what the right way is; for doing something and what is the fastest way of doing something. Many a times, the quick way is the right way but sometimes the team intentionally does the wrong way because they need to quickly deliver the product in the market.

 

How to address it: The agile team will not agree, but it is ideally sensible to track this type of technical debt in the backlog specifically when the deferring work needs completion. It it’s not tracked; it will turn into accidental design debt over a period. Products owners are accountable for this kind of debt as it is incurred because of business decisions.

 

2.       Accidental Design Technical Debt:

The team tries to balance forward thinking and future-proofing their designs with simplicity and quick delivery while designing the software systems. This is a very difficult balance to get in a single time as systems regularly evolves and requirements change continuously. You will realise the design made is flawed and the new functionality is slow to implement and difficult. A newer design would be much easier to refactor but sometimes you may have to do more significant refactoring.

 

How to address it: Refactoring a system itself is a huge topic, but this needs to happen every now and then, otherwise the system gets over engineered and slows down all along. Product owners and team leaders are accountable to resolve this type of technical debt as it is incurred because of design decisions and frequent change in requirements.

 

3.       Bit Rot Technical Debt:

This debt happens over time, the system slowly passes onto unnecessary complexities through lot of incremental changes when worked by several people who may not have full understanding of the original design.

 

How to address it: This is the only type of technical debt which you should avoid consistently. The teams should take time to understand the design of system even if they have not originally designed it and they work to improve the design incrementally and work on always improving the code.  The development team is accountable to avoiding this kind of technical dent as it is incurred because of the developers involved in the development team.

 

Below are various common causes of technical debt:

 

• Insufficient definition where requirements are still getting defined during development which causes rework.
• Enhancements of projects over a very long period.
• Business Pressures - when the business considers getting something released sooner before the necessary changes are complete technical debt is involved.
• Decisions made due to lack of process understanding.
• Lack of proper documentation where the code is created
• In scenarios where the software functions are not modular and is not flexible to adapt the changes as per business needs
• Lack of knowledge sharing around the organisation and proper mentorship of developers around the organisation.
• Delay in refactoring, the longer it takes for refactoring the codes are added more and more and the technical debt becomes bigger.
• Development of codes parallelly which creates debt because of the work required to merge into a single source code.
• Lack of poor technological leadership and last minute specification changes.

 

While writing codes, for any software development, there are several best practices the developer needs to follow. These best practices help the codes to be more readable so that in future anyone can understand the code and perform modification. This obviously needs some additional comments and standards which will increase the coding time. In Agile we will have a strict timeline to deliver the required minimum viable product and therefore there is time constraint. The developer tends to ignore and violates the best practices with a hope that they will be modifying it later. This results into Technical Debt.

 

We need to understand if the technical debt is accepted and if yes to what extent. The below picture clearly explains the implication of the technical debt.

 

 

 

Debt in financial term means the company must repay this debt with interest. The longer the time taken to repay, more the interest is added to it. Still the company borrows loan to increase the business and similarly, technical debt is also acceptable to a certain limit.

 

Result of Technical Debt can also be understood with the below diagram. It clearly demonstrates that if we do not plan the sprints properly, technical debt is bound to happen and will result in low morale of the developer.

 

 

 We can also derive a mathematical formula which will help us to understand to what percentage of Technical Debt is acceptable.

 

Technical Debt Ratio = (Cost to fix the software coding) / (Cost to develop it) x 100%

 

This means till the cost of fixing the software coding is less than the cost of developing it, we can accommodate Technical Debt. As a product owner we must try to plan the Sprint in a way that we will not end up in Technical Debt.

 

Technical debt also known as code debt refers to the practice in software development where an inherent additional re-work cost is created by choosing an easy solution for now, instead of using a better or correct method that would take longer to finish the work. Technical debt usually occurs when one ignores the best practices and code requirements to meet the stringent project deadlines.

 

Technical debt can also be mathematically represented as ratio called Technical Debt Ratio [TDR], which is given by.

 

Technical Debt Ratio = (Remediation Cost / Development Cost) x 100%

 

where Remediation Cost is the cost to fix a software code or solution and Development Cost is the cost of developing the complete solution. As a best practice, software projects should have a TDR value that is equal to or less than 5%. Higher TDR scores means the software is in a poor state of quality.

 

In an Agile project cycle, introspection can significantly reduce the occurrence of technical debt. It can also be reduced during sprint reviews by incorporating the feedback from stakeholders or customers. Other methods to tackle technical debt includes:

·       Being transparent to the team about technical debt. The more aware team members are, the less likely it will be that they will pursue low-quality solutions.

·       Create a dashboard/ system to view the existing technical debt so that everyone is constantly reminded about it. Use metrics such as complexity, code coverage, and rule violations to track technical debt.

·       Have proper operational definition of “Done” to meet the understanding of product quality to keep the technical debt at a manageable level. Test and review all deliverables to identify gaps and as early as possible.

·       Continuously reduce the technical debt on every single sprint. Allocate fixed percent of the Development Team’s capacity each Sprint to handle refactoring and bug fixing.

·       Make sure that all tasks related to dealing with technical debt are part of the Product Backlog

The Model of Technical Debt was first coined and defined by Ward Cunningham in the Year 1992.To understand the technical Debt, let us dissect the process of Agile software development in a simple way.

 

Non-Acceptance is a clear raise of Technical Debit. Some examples are :-

1)      Unfit or bad design

2)      Defects in Software

3)      Insufficient test coverage

4)      Excessive manual Testing

5)      Poor Integration and Release Management

6)      Lack of platform Experience

These are 3 types of Technical Debt :- 

 

Type of Technical Debt	Causes	Methods to avoid Refactoring in Technical debts
Naive Technical Debt	Unclear Methods of execution, Accidental, Ignorance, non-Vigilant	Providing best design pattern, documentation and guidelines
Unavoidable Technical Debt	Upgrades, Newer versions availability in the market, Change requests or features addition in Scope during the mid of development	Future Proofing the Original Designs, Codes should be repeatable & often useful to avoid the huge cost incurrence due to scope change
Strategic Technical Debt	Missing proper quality code practices, Either refactoring too often than usual, Or refactoring entirely is missing	Including PR & Design Reviews, Monitoring Standards, Sign Off's at each developmental stage

 

There are also 4 causes of Technical Debt due to poor Management :-

1)      Pressure to meet the Deadline

2)      Trying to falsely Accelerate Velocity

3)      Fales Myth is that, to Test Less or Avoid Accelerating Velocity

4)      Debts on Debts – Raise of future Debts due to unsolved present Technical Debts

 

(Here, Velocity refers to the time taken to develop & deliver an acceptable code/software product with minimum user stories and shortest team size)

Most of the times in real cases the technical debts cannot be avoided, but if we manage then the technical debt will be always low enough to manage all future product development.

 

 

Method	Guidelines
Manage Accrual of Technical Debt	Use the best Technical Practices
	Use SCRUM framework
	Simple Design
	Test Driven Deployment
	Continuous Integration
	Automated Testing
	Carefully Managed Refactoring
	Strongly defining "done"
	Understanding the Technical Debt Economics
Visibility of Technical Debt	Make it Visible at Business Level
	Make it Visible at Technical Level
	Assign SCRUM Product Owner to balance between Client & Development Team
Servicing or Repaying the Technical Debt	Clearly Define Servicing & non-Servicing portions of Technical Debt
	Clean Up the Servicing portion immediately
	If the portion of non-Servicing belongs to: -
	(1) product Nearing end life
	(2) Throw away Prototype
	(3) Product built for a Short Life
	Then, Negotiate and balance/zero the repay methods with Clients
	For genuine non servicing portion repay the technical Debt incrementally

 

Guidelines for Agile leaders to decide between longer turnaround time and technical debt

Importantly, along with Agile, DevOps is another methodology playing its crucial role in balancing the longer lead times while managing minimum Technical Debt. It can be managed in 3 ways :-

1)      DevOps aims to reduce the time between the application of change in system to change in production

2)      End to End automation of deployment process while connecting Development, Testing & Operations

3)      Maintaining High Software Quality and support Automation (i.e., Continuous integration – to detect the defects in early stage and act to fix it)

 

 

What is an acceptable level of technical debt?

Most of the times technical debt is not accepted by the development teams however in general the acceptable presence of technical debt is at 5 % or less than 5 %.

 

Technical Debt is also known as Code or Design Debt. It is the additional cost that can be attributed to rework that could result due to hastening up a project as in Agile software development. Changes that are required and not made accumulate as technical debt. Hence, technical debt can accumulate as interest, implying that with accumulated technical debt, the ease with which change can be implemented becomes more and more difficult. Technical Debt, in itself, is not bad and helps in prioritizing various tasks of the project.

Technical debt could arise due to various reasons. Some of them are the starting the development before the completion of the design, lack of knowledge, lack of ownership, competitive business pressure, lack of understanding of technical debt, tightly coupled components, inadequate testing leading to sub-optimal quick-fix solutions, inadequate documentation, collaboration, parallel development, delayed refactoring, non-alignment to frameworks and standards, bad technical leadership and last-minute changes to the requirements.

Technical debt has been categorized by Kenny Rubin into Target, Known, and Happened upon Technical debt. Targeted and Know Technical Debt is known in advance and planned by the design and development team, however, happened-upon technical debt is the debt that is hidden and the team is totally unaware of this debt until revealed during the functioning of the product.

Acceptable Level of Technical Debt

Since debt is a necessary evil in the financial world, technical debt is a necessary evil in the project development world. Hence a balance between the technical debt and the Turn Around Time needs to be drawn. It is a choice of speed over perfection and where to draw the line in-between. Technical debt accumulates with the developments in the upstream project, increasing the debt in the future. This causes projects to miss their deadlines. Each change incorporated into the project adds to the technical debt, i.e., uncompleted work in the project. Deadlines are missed when the uncompleted work (technical debt) requires more time than the time available to finish the project.

An acceptable level of technical debt is limiting the amount of work in progress. This will lead to the amount of uncompleted work (debt) to the minimum. At times Projects are released along with a large amount of technical debt. If any further refactors are implemented, the technical debt can increase exponentially. Modifying the code that is already in production comes with a large risk especially if the contract has strict service level agreements.

Some projects accumulate technical debt by rushing out the software in order to get experience and then refactor the software and pay back the technical debt. The experience gained would lead to better software. It can be seen as a tool for getting ahead of the rest by test driving the software.

Since software companies are generally under competitive market pressure to develop and ship quickly, technical debt is a necessary evil in order to beat the market. Teams need to make a trade-off between taking on technical debt and being the first out of the blocks or launching a perfect product later. It is thus a consensus amongst software production teams that technical debt is necessary.

Consequences of Technical Debt

The consequence of technical debt according to Shaun McCormick is that it decreases agility as the project matures. Since it decreases the agility as the project matures, it should be intentional and not accidental. Technical debt leads to uglier and harder to maintain code.

References

https://en.wikipedia.org/wiki/Technical_debt

https://www.productplan.com/glossary/technical-debt/

 

 

 

Excellent attempt by all respondents at a slightly tricky question. Technical Debt though not very good for a software project, it is something that is inevitable. Every project will have some technical debt. However, lesser is always better as it means less refactoring and rework.

Chaitanya has provided the best answer to the question.