Q 697. What is Survivorship Bias? How can it negatively impact rational decision making in a project or a company? Provide some measures by which this bias can be avoided.

 

Note for website visitors -

• This platform hosts two weekly questions, one on Tuesday and the other on Friday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be approved. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error prone as our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://crossplag.com/ai-content-detector/. Only answers with less than 15-20% AI-generated content will be approved.

Survivorship Bias is an error that has caused due to focus only on one or few significant aspects of the subject or process while ignoring or overlooking other important aspects those could cause incorrect conclusions or output to the process  or subject undertaken.

Would like to share one of the projects that we had worked for Collections department caused due to the negative impact of this concept- 

Here the collections team is analyzing the past debt recovery with the help of the collections data, 
they realized that that by sending the timely reminders and phone calls technique they were able to 
achieve the highest recovery, due to this the focus was shifted mainly to these customer to double down this strategy for future collection, however while performing this activity they did not realize
that the debts are highly recovered from the specific customers only, while this technique did 
not work in case of other customers debt recovery  who did not respond to either calls 
or email leading to lower recovery rate from other segment of customers causing increased outstanding debts and potential financial instability to the department and company.

Steps taken to overcome the above issue - 

1. Customer Segment Analysis - Segment wise data breakdown helped to understand the demographics and different 
locations and types of customer to customize the strategy and work out more effective approach to recover
2. Tested Alternative methods for recovery - Other than timely emails and telephonic calls, checked on other methods based on the historical data to understand the nature of the customer which they may best respond to. This included personally meeting up the client who believes in traditional method of working.
3. Spotted failure patterns - Looked at the commonalities and differentiated factors which worked best for the specific customer and why, this helped us understand and categorize the customer which pattern which may work best for recovery
4. Feedback from Customers - By putting up feedback mechanism from the customers helped understand which technique would work best for the specific customer

 

With this holistic approach and targeted strategies, it helped the department to improvise on the overall debt 
recovery rates.
 

It’s a general human nature to give prominence and importance to events, tasks or issues which resonates with a particular thought process or hypothesis and are very widely published for consumption. This generic human thinking phenomenon often leads to mis - calculated decisions, as they are made, basis information which is readily available but often negates those bits and pieces of information thread, which gets worn out during the process and thus are non-existent before an outcome is delivered. Thus, leading to decisions based on the information which 'survived' the process, and their existence becomes prominent. Hence the term 'Survivorship Bias'.

 

Business Processes comprises of various resources which can be broadly categorized as Man, Machine, Material, Methods and Measurements. 'Survivorship Bias' phenomenon can easily impact decisions based around these resources or process as a whole, if expert governance is not present.

 

For Example - If a 'Selection Process' is followed to hire a Project Team with specific skillset, with a target of 10% Hiring Conversion rate (People targeted to be hired for Project Team from the number of applications received). Once Selection process is over and results are analyzed, its identified that only 8% applicants have successfully passed it and can be considered for the role.

 

Obvious inference which can be drawn from this, -  'APPLICANTS WERE NOT OF REQUIRED QUALITY WHICH LEAD TO 2% LESS HIRING'. But something which is overlooked is 'WERE THE STANDARD OF THE SELECTIONS PROCESS TOO TOUGH TO BE FULLFILLED BY EVEN THE WELL SKILLED APPLICANTS'. This Survivorship Bias phenomenon played its role!!!

 

Whenever process is being governed on various tangents, every effort should be made to make conscious and well-informed decisions by erecting necessary guardrails against Survivorship Bias.

 

Some basis pre checks which can be performed to avoid survivorship bias are 
- Try to collect every possible data to be subjected to analysis
- Stringent controls should be placed to check the data for any bias's
- Brainstorm to identify any data which could have been missed in the process of collection, if yes, collect it
- Deploy Data Experts and tenured resources from the process for Data collection as they have more insights on the possible data seepage

- If possible and applicable, put bots in place to collect datasets for study

- Use Generative AI platforms for more well-informed decisions

Survivorship bias is a logical error in decision making process, when we only concentrate on the Pass cases & ignore the fail cases. This can also consider as “Sampling Error”.

This could lead to Overestimating the success of a project as we are ignoring the fail cases. Hence we will set a unrealistic goal for the project or business. Any decision taken based on Survivorship Bias may lead to failure of the project or company.

To avoid this Bias we need to be careful in selecting the sample. We should be very careful and look for proper representative sample. Proper failure mode analysis should be done before considering any data to be considered for decision making.

What is Survivorship Bias:

There may be cases when only the surviving cases or events are considered for decision making. This practice of not considering the full dataset while taking a decision or making an analysis is called survivorship bias.

 

Example:

1.      Considering the financial performance of any industry based on running entities within the industries and not considering the entities which already collapsed.

2.      Considering cost of distribution based on sales volume & not considering depot damage

3.      Considering average grade of students as a performance index in certain class based on only the students who graduated to next class & not considering the students who failed

4.      Measuring efficacy of a medication based on feedback of the surviving patient & not considering the deceased ones

 

Negative impact:

Survivorship Bias will result in drawing wrong conclusions since the conclusions are drawn based on incomplete dataset.

 

Measures to avoid Survivorship Bias:

In order to ensure Survivorship Bias the dataset must be comprehensive and representative of the entire population of specific problem being analyzed. The sample must have representation of all relevant data, including failed efforts or assets, in the testing process

Survivorship Bias – A logical Error 

 

Success and failure both go together. When one only focusses on success and do not understand where and why they failed, wrong conclusions are taken. For example, if we study only successful companies with out considering which failed, we may wrongly attribute success in certain Strategies and treats because of which other companies are failed. 

 

This bias conclusion with considering only successful factors only called Survivorship Bias.  

 

 

 

 

Negative impact on rational decision : -  

1. There can be cases where overly optimistic conclusions are taken. 

2. Misguided strategy can be taken due to bais and portion data only. 

3. Cause analysis can be wrong as failure reason are not analyzed.  

 

 

 

Measures to Avoid Survivorship Bias 

1. Include all data in any analysis for a project or company. Data structure should be collectively exhaustive. 

2. Critical validation and evaluation must be done is all elements of data are added. 

3. Final decision must be taken after considering multiple points of view only. 

4. Historical data of both success and failure must be studied and taken in consideration. 

 

Happy learning...

Survivorship bias refers to the situation wherein the outcomes considered within the project or any such activity are only from the successful outcomes or the outcomes which met the success criteria. It results in an over optimistic belies as well as a biased outcome as the factors behind the failed outcomes are not considered resulting in a biased approach.

 

It can have very adverse impact on the project/company as the root causes behind the failure are ignored which in the long run will have an adverse impact on the success of the project/company.

 

This can be avoided by ensuring that:

• The data is considered for the failures along with successful outcomes while analysis is done
• Ensuring that approach is inclusive and unbiased based on the outcome
• In case sample size approach is being used then it should be Large
• Wide range of perspectives are considered 

 

 

What is Survivorship Bias?

It is a type of cognitive sample selection bias where the focus is concentrated on data points or entities that passed a selection process, while overlooking others that did not. It leads us to draw incorrect conclusions since we are only focusing on the winners.

For Example; an organization might study only successful businesses to identify winning strategies, but may ignore many other companies that implemented similar strategies, but failed. By focusing only on the survivors, incorrect conclusions may be drawn about the success of a strategy instead of also looking at other companies who might have failed just as often.

Negative impact on decision-making

 

Survivorship bias in an organization may lead to several issues,

1. Overestimating the success of certain strategies: Excluding companies from performance studies because they no longer exist causing the result of these studies to skew higher because only companies that were successful enough were included in the study.
2. Ignoring Key lessons from failures: Failures often contain lessons about what does not work. If we only analyse the survivors, crucial insights might be missed that could prevent repetitive mistakes.

3. Misallocation of resources: Allocation of resources in a seemingly promising technology, may not have a similar result as it did previously. This may occur if we study only those companies who were successful in the use and implementation of that technology. This can also lead to wasted time, effort and money on initiatives that are unlikely to succeed.

4. Creating a false sense of security: Leaders and decision makers might believe that their choices are fool proof, giving them a false sense of confidence, because these choices are modelled after successful examples.

How to avoid Survivorship Bias?
 

Avoiding survivorship bias involves consideration of both success and failures by ensuring that decisions are based on a complete picture.

1. Study failures as well as successes to understand what went wrong and why, which in turn will provide a balanced perspective.
2. Gather Comprehensive Data from multiple sources looking at both failed and successful ones. Involve Diverse Perspectives by brining in people with different backgrounds and experiences.
3. Promote rational decision making by encouraging a culture of critical thinking and challenging the assumptions.
4. Welcome both positive and negative feedback to highlight potential risks.
5. Conduct experiments through pilot projects along with randomized trials to compare outcomes.
6. Document and Review all outcomes to identify patterns, thereby, engaging in balanced decision-making.

 

Priyanka has provided the best answer to this question. Well done!

 

P.S. - There is a very interesting story about how Britons and Americans during the World War II were initially misled by survivorship bias while they were researching and strengthening their air planes. Do read up on it.

But then isn't history all about survivorship bias