Q 216. According to a survey, around 16% of Agile projects are a complete failure amounting to a loss of around $50 Billion annually to the tech companies. What are the most common reasons for failure of Agile Sprints? What are some of the best practices to avoid such failures?

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time.  
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

Before we take a look at the potential causes for the failures and look at the ways/best practices to avoid these failures, let us take a look at what Agile and Sprint means.

 

Agile: It is a way of doing an iterative and incremental approach to deliver a project (product/service).  Agile projects needs a cultural mindset change and has certain set of principles to be followed as prescribed by Agilemanifesto.org

 

While there are many implementation techniques for doing in an agile way, one of the most popular techniques is Scrum, which is a framework and is used for developing, delivering and sustaining complex products. 

 

As per the Scrum Org (parental body of Scrum framework) definition, the heart of Scrum framework is the Sprint which is a time period of one-month or less(can be 1 or 2 or 3 weeks as well) during which a "Done" , usable and a potentially releasable product increment is created.

 

Now let me try to answer why and how failures could have happened for these organisations from my perspective.

 

Why Failures happen in Agile Sprint based projects?

Before we take a look at this, we need to understand the fact that the world is moving away from the traditional models such as waterfall to the Agile world because traditional methodologies had much higher percentage of failures. As per the Chaos report of 2011 by Standish Group  proclaims that Agile Projects are 3 times successful with respect to software development when compared with delivering through traditional methods (as referred in Scrum - A pocket guide book by Gunther Verheyen). 

 

Having said that , now even as we make use of Agile for any sort of work (and not only for IT related), the fact that 16% of the projects which run in a popular technique - Agile Sprint(read Scrum) fail is a serious sign that something fundamentally could be wrong somewhere in my view, as an Agile Practitioner.

 

Two cases which i want to highlight:

Case 1:

For organisations, which move newly(transformation) to Agile, the issues come around how they adapt quickly to Agile mindset and practices. Failure happens when the team lingers on the traditional approach(Command and Control mode and not following servant-leader approach and not doing collaborative work and not having self-organised approach) despite putting the Agile processes in place.

 

Case 2:

Organisations which are supposedly mature in Agile Transformation(because they are into Agile for some years and executed many Agile projects) can have different set of problems. They may have some complex projects which might involve multiple vendors and coordination and cooperation required amongst the different teams for a product has to be precise

 

In general Organisations (applicable to case 1 and case 2) are averse to Risk-taking , in terms of failure, then there is more chance that their projects will fail.  Agile Scrum works on Empiricism. You get to know more about the product as you progress. Therefore the mantra is Fail Fast if at all you are going to fail. This is why many organisations try to have lesser length for a Sprint (1-2 weeks rather than having the maximum 1 month period, so that you minimize your risk of not making the right product to a shortest possible period. Thats the whole concept of the timebox. Unlike the traditional model in which you finally find that the project is not meeting the customer needs, Agile projects get to know the feedback from the stakeholders at a very early stage and get a continuous feedback.  Sometimes even in case 2 organisations, this will not happen properly due to multiple reasons , which is again a recipe for disaster.

 

Classification of Failures:

With respect to the Failures (in the attached document),  the top two quadrants, 'Agile Maturity' and 'People' can happen in Case1 organisations and 3rd quadrant 'Agile Scrum Maturity and Scrum Process Compliance' can happen both in Case 1 and Case 2 organisations. The 4th quadrant 'Product Vision and Volatile Market' can happen in Case 2 organisations.

 

How to prevent such Failures in Agile Sprint based projects?

Most of the Failures would be addressed straight-away if organisations understand and practice Agile and Scrum as they are meant to be. For each of the failure classification, corresponding best practices are also mentioned below. 

 

Here are probable reasons (in my purview) for failure and some of the best practices (in my opinion) to overcome such failures.  

 

1. Agile Maturity

 

• Agile transformation could not have happened        

       properly (for organisations moving newly to

       Agile) :

       

Eg:  Proper vision would not have been put in

       place (as how to adopt Agile transformation

       across the organisation)

 

• Agile will expose the existing dysfunctions in the

       organisation/within teams in the organisation. So

       that could have been the reasons for failure 

   

   Eg: Inattention to Results, Avoidance of  

       Accountability, Lack of Commitment, Fear of

       Conflict, Absence of Trust

    

• Organisations wait endlessly to make the actual change to Agile. They would plan for Agile movement but neither they would be in waterfall nor in Agile and end up in between

 

Eg: They would have Dev Teams and QA teams working in Silo but calling the project as "Agile"
 

Agile Maturity Best Practices

 

For organisations moving newly to Agile

• Have a proper vision . Identify the need for the change , reflect upon your current organisational state (where you are) and what are the Key Progress Indicators (KPI) which can track the progress that you make in this journey.
• Ensure proper coaching happens to Key Stakeholders/leadership team on Agile nuances and mindset needed for that
• From leadership team till the last cadre in the position hierarchy this information should pass on
• Alleviate the concerns/fears and perceptions that people might have due to moving of Agile and its philosophy

 

Once the perception about Agile philosophy changes (for good) then dysfunctions will be minimized to a very negligible/minimal level

 

Teams would be truly cross functional which means they would be self-reliant and do their work without dependent on anyone.

 

2. People:

 

• People(key persons/leadership level) might have been reluctant to change to Agile way of working(applicable to Organisation transforming to Agile).  Agile requires shift in mindset from the tradition "Industrial" paradigm (Command and Control mode) to 'Servant-Leader' mode.  Reasons are many

·       People could have felt that their position power and influence was marginalised

·       They would have thought that adapting to a newer way of working was difficult to imbibe into their working DNA

·       They would have thought that Agile can expose their shortcomings and hence felt  therefore that their personal reputation  could get damaged which could hurt their ego.

 

• Management/Leadership team probably did not do enough groundwork to convince the organisation members on the need for Agile(applicable to Organisation transforming to Agile).  Effectiveness of a solution is determined by the Solution Quality and its acceptance. The acceptance factor(by the organisation employees) therefore might not have been there !!

 

Best Practices for People Management

 

• When Management team/Senior Leadership team is convinced and embraces the agile value and stresses the importance of Agile, then Agile Acceptance can happen
• When Leadership stresses the importance of Agile with statistical data or with proper justification or if there is an incentive (in terms of opportunity for employees while moving to Agile), then employees take upon Agile diligently

 

3. Agile Scrum Maturity and Scrum Process compliance

 

• Project Teams might not  have followed Scrum rules, principles  properly (ScrumBut)

 Eg: Scrum Events might not have been properly  adhered.

They are key to Inspecting the work done and provides a chance for teams to correct things(Adaptation)

 

• 'Definition of Done'(DoD) might not have been strong enough to provide a strong product in a short period of time duration(1 week -1 month timeline). DoD defines the completion of "DONE" which indicates that an item picked for that particular Sprint is completed.         

 

• Teams within the organisations, might not be either able to scale up to the dynamic scope changing  nature of the work within the sprints frequently  or might have seen some sprints cancelled due to the goal becoming obsolete which can demoralize them (the teams).

 

Agile Scrum Maturity and Scrum Process compliance - Best Practices

 

If Project Teams are educated on Scrum importance by Scrum Master then it helps the whole scrum team

 

4. Product Vision and Volatile Market

 

• Product owner might not have got his/her priorities right  while ordering the product backlog items. Reasons could be          

·       Stakeholder Engagement  might have been poor

·       Probably Prioritization was not given importance.

·       Too many stakeholders might have tried to influence their needs

·       Product Owner might not have been assertive and was not in a position to influence and shape the destiny of the product.

 

• The product was not probably good enough to survive in the market as the Industry(to which the product belongs) is itself volatile and/or highly competitive.

 

Product Vision and Volatile Market -Best Practices

• Product owner should have constant interaction with stakeholder.

 

Conclusion:

Agile is not just a methodology to be practised, but requires a mindset change. Scrum is a framework which has certain rules and boundaries that need to be respected. There are certain principles of Agile (developed by quite a few intellectuals) which needs to diligently followed and a set of Scrum values complementing these principles and set of Scrum guidelines that need to be applied on the project.

 

Doing these will ensure that organisation teams can have the ability to respond to ever changing needs of their customers. Failures when expectations are not met.

 

Gunther Verheyen , a popular Agile Scrum practitioner in his reputed book (aforementioned) says that Agile has 5 characteristics: People driven, Facilitation, Iterative-Incremental, Measuring success, Change.

 

In a typical Agile, People are respected for their creativity , intelligence and self-organising capabilities. Collaboration happens amongst them. Teams are facilitated by servant -leadership. Products are created piece-by-piece and agile processes are defined . Project progress and success is measured in terms of working product by frequently inspecting it and the actual value it can provide to the people  who will use it.

 

Even when requirements and implementations are predicted upfront, they would be susceptible to change due to volatile nature of the market and may be due to high competition as well.

 

So when organisations follow these Agile characteristics as part of their working DNA, the failure rates for them is going to be very minimal and their success rate would be high.

I wasn’t able to get the survey report to analyze the details behind the survey. The optimistic way of looking at it is that 84% of Agile projects are successful in some forms, which is in many ways around twice the success rates of traditional waterfall approach. Reference (https://vitalitychicago.com/blog/agile-projects-are-more-successful-traditional-projects/)

 

"Agile Software Development is a lightweight software engineering framework that promotes iterative development throughout the life-cycle of the project, close collaboration between the development team and business side, constant communication, and tightly-knit teams. "

 

A key aspect of Agile projects is fail fast. Sometimes there is a blur on what is genuine continuous improvement vs genuine failure

 

I feel it is also got to do with the type of organizations adopting this. We have the new age technology companies (Google, Facebook etc.) vs the traditional Enterprises adopting Agile.

For New Age companies, speed is of utmost importance. They constantly are looking at bringing new products to market. The mindset is towards faster rollout, fail fast and improve faster, focus on end to end straight through processes with customer focus as paramount importance. The people who they employ are also tech geeks suited for agile.

 

For Enterprise companies, they are saddled with ageing technologies, often a huge complex myraid set of technologies. They try to leverage existing IT ecosystem to bring in change needed to compete with nimble new competitors. They also train existing business and IT SMEs on Agile and expect them to adopt the new ways of working.  The same is the case with many of the legacy IT Services companies, where the large portion of the employee base is used to the traditional Waterfall methodologies.

 

The biggest reason for failure of Agile projects is due to Planning. Most companies don’t have two strong critical roles - Product Owners and Architects.

A badly designed and thought through project is only going to iteratively fail. Product Owners need to have a strong handle on the roadmap for the applications and value it delivers. Agile also requires the business SMEs to work in a lot more collaborative manner than before.

 

With myriad of new technologies available, and short durations of Sprints, its critical to have Architects who have a strong vision on the stack to take care of performance and security.  The role of the Architect is also to keep it simple, that in turns reduces the chances of failure.

 

Key aspect of Agile is strong collaboration and constant communication. A good collaboration tool is equally important to track the progress and have a strong control on the various aspects of the project. Teams also tend to be geographically distributed. Organizations need to invest in such collaboration tools, including video conference to make face to face interactions.

 

Since Agile involves a small team, its equally important that there is good synergy within the team members. Cannot have a hierarchical approach, discouraging members to speak up. In traditional approach, it became very compartmentalized with specialists for design, documentation, development, testing, deployment etc., but in Agile there is expectation of each member to own end to end, and not looking at passing things to another team. Retrospect and Learning needs to be done in all earnest.

 

More often teams tend to focus too much on process metrics like velocity, backlog, burn down charts etc. and tend to forget the goal - the business value that is to be delivered. This results in teams focusing on getting things done mechanically, being rigid on addressing changes required so as not to break the sprint. Ultimately working software is the key to successful projects.

 

Agile itself is a very simple approach, having some 10-12 key principles. As the saying goes "The devil is in the detail", making Agile successful or not depends on the details behind the planning and execution.

Agile methodology, a most prominent way to manage the projects in the Virtual domain (mostly in the Software development) has gained significant popularity in the recent times. It is based on the principle of breaking down the larger projects into smaller and manageable tasks called Iterations thereby creating something of a Value. Once an Iteration is produced, it is then made available to the users/stakeholders and seek feedback.

This methodology has proven to be highly successful and many organisations have been able to score high on Innovation, customer satisfaction, quicker resolutions, better products and thereby high revenues. On the other hand, there are many projects which failed to deliver leading to huge financial loses to the technology companies. Hence, we could assume that these powerful methodologies comes with many challenges which play a key role for the overall success of the projects.

 

Let's discuss each of these reasons and the measures that can be taken to reduce the risks of failure.

Lack of Management Support: Agile Projects require a great support from the management right from the Planning till implementation. However, in many cases it is found that there is not enough support the management provides to the team due to many factors. This might be due to the pressure from the external customers, competition or the need to deliver the products at a quicker pace. Due to this, the actual focus shifts from the long term benefits to short term gains (or to prevent loses) which adversely impact the overall purpose of applying the Agile methodologies.

 

To prevent these, the management must ensure that they do not deviate from their plan and need to stay focused on their objectives. Further, the management should motivate and actively support the functional managers in implementing the projects. The management should also make sure that all the parties are engaged and actively participating in their respective activities or tasks.

 

Resistance to change: Management of change is another major hurdle in the overall success of the project. The resistance to change can be observed across the functions and has a greater impact on the mobility of various projects. This can be due to organisational culture, lack of collaboration and poor communication.

The Management must take effective Change Management steps to prevent this. Effective communication, periodic Stake holder reviews and fostering engagement are some of the ways to ensure that the organization is moving in a right direction. Also, it should take into account the key issues for resistance and take necessary actions throughout the project implementation.

 

Competency and Skill Issues: It is also observed in many cases that lack of necessary skills and expertise is a major cause for the failure of Agile projects. Unavailability of expert resources who fully understand the methodology will lead to confusion among the functional teams and desired results will not be achieved.

To mitigate this, the organisation must employ skilled and trained resources to lead and work on the projects. Continuous training should be provided to the people on the Agile practices to develop the required capabilities and competencies thereby contributing to the successful implementation of the projects. Further, the resources who are leading the projects should be capable of mentoring the team members and troubleshoot wherever necessary.

 

Lack of Proper Communication: This is one of the most common concerns not only limited to the Agile projects. As common it may sounds, lack of proper communication is still a major contributor for the failure of a project. Key things not communicated at the right time through the right channels causes delays or errors in the entire process and leads to a major gap across the verticals.

 

The project leaders and the management should take effective measures to ensure a proper communication at regular intervals. This can be done through periodic reviews, regular Stake holder meetings or project meetings. The main objective is to confirm that there are no differences in the present and future actions on the project. Proper communication with the team also fosters the collaboration and boosts the morale of the project team members.

 

Improper Project Management: A vast majority of the projects do see the failure at the implementation phase due to incorrect project management practices. A project leader should be able to manage the ongoing activities in a transparent and acceptable way to negate any wrong doings or deviations from the plan. Many times, incompetent or inexperienced project leads who lack the vision and domain knowledge fail in delivering the desired results.

 

It is of utmost importance to delegate the right work to the right people. The management should deploy a project leader who has both technical expertise, business knowledge and the people management capabilities. The product vision should be clear and the team should ensure that the tasks are completed and should be periodically reviewed. If needed, a project management expert should be consulted.

 

 

Complex Requirements: Many a times, the projects are too complex and time consuming. This leads to many complex problems arising out from time to time and the team lacks the capabilities to battle these issues leading to failure of the projects. Also, in many cases, the requirements of the end product keeps changing from time to time due to customer or competitor pressure which also derails the progress made by the team and this creates issues too difficult to resolve.

The project leader should invest fair amount of time in issue resolving, planning and anticipate the kind of issues that may arise during the course of time and its troubleshooting. This is not always possible owing to the stringent timelines and delivery specifications. However, the project manager should effectively deal with these issues and take the buy-in from management if necessary to ensure the issues are resolved before becoming complex. Also, the management should ensure that the scope of the project doesn’t deviate much and should set realistic deadlines for the deliveries.

 

Above stated are some of the key factors responsible for the failure of the Agile projects and the necessary steps to prevent the failures.

Congrats to Ram, Pavan, and Rajesh for excellent answers. The winer is R Rajesh because of his comprehensive coverage of failures and best practices.