While baseline is the performance level before improvement is made in a process, the utility of the baseline itself many a times becomes questionable. Provide examples where performance after improvement is not comparable with performance before improvement and one of the two needs to be adjusted for a fair comparison. 

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time.  
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

Primary influencing factor for baselines before and after improvement to become non-comparable is probability of process itself undergoing significant change. Which means that some of the key parameters might undergo major change, which calls for process to be re-baselined again before it can be compared. All possible influencing factors are to be kept under check as we make improvement is the key. This helps us to get at logical conclusion that whatever changes observed in process purely due to the improvements made. Just statistical analysis may show that there is significant change in process before and after improvement, but logical analysis to make sure that the improvement done on targeted parameter caused the same is very important.

DOE – Design Of Experiment if it fits in to a scenario then it would help in a situation where, process before and after have undergone change. Here some of the process factors that may have undergone changes due to influence of something outside the intended change then those factors can be made constant and then analyze the impact of factors under consideration and then may be baselines of process before and after improvement will become comparable.

It is true to say that the utility of the baseline itself many a times becomes questionable when the objectives and performance indicators are not defined. The term “performance improvement” is used to compare similar approaches, such as “quality improvement,” “continuous quality improvement,” “quality assurance,” “total quality management,” and “human performance technology.” Missing of any of the elements makes it difficult to compare the pre and post performance improvement., 

 

 

About Baseline

One of the requirements of the Measure Phase in Six Sigma DMAIC cycle is the Baseline measurement, sometimes expressed as Baseline Sigma. In fact it is hard to tell whether the baseline data is required as part of the Define phase or Measure phase.

Ideally, if we need to give the problem statement, which is expected to cover What, When, Magnitude and Impact. The ‘When’ portion is expected to show the metrics related to the problem for a time period as a trend chart, so that we can see the magnitude of the problem and the variation over a period of time – and acts as a baseline.

Baseline certainly helps to act as reference to compare and assess the extent of improvement. Baseline is important to get a good measure of the quantum of improvement and in turn to quantify the benefits in tangible terms.

However, the following discussion brings out certain practical challenges related to Baseline.

 

1.    Baseline metric did not exist, but is it worth post-creating it?

Suppose we are trying to improve an electronic product, based on certain customer complaints, our project objective will be to ensure that the incidents of customer complaints should be reduced or eliminated. Upon subjecting the product to a special lab evaluation, we could simulate the failure. However, a reasonable baseline metric will be possible only if we subject a set of sample units for a certain period of time. This could prove quite costly and time consuming. On the other hand the solution to the problem is known and we may proceed with the actions. Since our goal is to ensure zero failure, under the given conditions and duration, comparison with a baseline is not important here.

Many a time, when the company is anxious to implement the improvement to get the desired benefits, be It cost or Quality, it may not make much sense to build up a baseline data, unless, it is readily available.

 

2.    New measurement methodology evolved as part of improvement

Let’s take an example of Insurance Claims processing, where the payment / denial decisions are taken based on a set of rules and associated calculations. The improvement being sought is to reduce the rate of processing errors. However it was only as part of the improvement actions that an appropriate assessment tool was evolved to identify and quantify the errors by the processors. By this time, the improvement has already begun and it is not practically possible to trace backwards to use this tool and get a baseline measurement.

 

3.    When improvement is for ‘Delight factors’

Often we introduce enhancement features on product, for example, new models / variants of smart phones. In such cases, the emphasis is more on the delight factors for customers, for features that they haven’t experienced earlier and any baseline comparison may not have much relevance.

 

4.    Integrated set of modifications

Let’s examine another scenario where a series of modifications were implemented on a software application and was released together as a new version. Here, the set of actions taken influenced multiple factors, including performance improvement, elimination of bugs and inclusion of new innovative features. In such situations, any comparison with a baseline performance to the current will be very difficult and would have overlapping impacts. If we still need to do a comparison before vs after, we may have to do so after factoring and adjusting for such interaction effects on the pre / post improvement outcomes.

 

To conclude, in general, a baseline metric is an important information that we require to compare the post improvement results – However, it has to be borne in mind that certain situations challenge the feasibility and relevance of using a baseline measurement.

It’s a rare case where the baseline and performance after are not comparable. This may happen due to some special causes which occurs during the execution of project. Some examples are

 

1.    Consider our project is TAT reduction of call resolution time taken. We have calculated base line by considering the data available for last 3 months. After implementing the changes proposed and calculating the performance at the end shows drastic increase in the TAT instead of decrease in the time.

Reason: Base line was calculated during the festive season and when the volume is less. Performance after improvement was calculated when the volume of calls is very high nearly double the numbers. For getting better results the data collected for baseline and after the performance results should at least match.

 

2.    For” A” product no. of complaints are 45 in 2016, we want to decrease the number of complaints and by six sigma tools DMAIC project was executed and after implementing the changes the tendency of no. of complaints receipt increased to 60. There are some special causes to be back verified in the improved process where the product is doing wrong.

 

 

Measuring differences in calculating the baseline of the product also lead to performance after improvement is not comparable with performance before improvement.

Base line performance after improvement is not comparable with performance before improvement.  It is possible sometimes when we are estimating savings. For example a project that streamlined a production control system was aimed at improving morale by reducing unpaid overtime worked by exempt employees.  However no measure of employees more was obtained ahead of time.  Nor was the unpaid overtime documented anywhere.  Consequently,  project was not able to sustainable  it's claims of improvement.  

Process baselines are a critical part of any process improvement  effort as they provide the reference point for assertion of benefits attained.  In the absence of proper baseline estimate there can no credible evidence of sustainable improvements.  

Another example of average time.  For example an improvement team uses lean techniques to reduce the time to process an order.  A random sample of 25orders before the change had an average time to process of 3.5 hours.  A random sample of 25 orders after the change has an average time to process of 2 hours.  The team asserts they have decreased the order processing time by more than 40%. This is not credible because an improvement can not be asserted without showing that the new process is significantly different from a statistical point of view.  There is no evidence that the estimate of 3.5 hours for the first 25 samples is a valid estimate of the process,  since we have not shown Ruth at the process is stable or not. 

Use of a baseline is somewhat dubious in the following situations.

 

1	Project to design a process where there is none	There is no process at all to baseline against.
2	Upgrade of technology	The technical environment has so completely changed either through the project or otherwise, so much so that comparison cannot make any sense

Baseline:

this means a line from where the measurements are considered. This is the basic line of measurement from which the measurements are based upon.

 

the utility of the baseline itself can be questionable. Because the baseline itself could be difficult to achieve as this could be the line based on ideal situation or scenarios. 

 

In cars, the mileage claimed could never be achieved. Hence the baseline parameters could not be set up to that level where it is only possible to achieve in ideal situations. Practically the ideal situations cannot be see. 

 

 

 

Baseline can be defined as the current level or point from which we intend to improve the performance of the process or project. It becomes necessity to evaluate if the changes implemented in the process has made them move closer to the intended goal or not. It is helpful to understand and define the current level of the process and is also necessary to measure the impact of the improvement plans deployed to improve the performance of the process or system.

 

Baselining is of 3 types. They are 1) Schedule baselining 2) Cost Baselining and 3)Scope baseline, which we are as listed below.

 

Why baseline is important?

It is important so that we can

·         Be able to assess the current performance

·         Calculate the earned value of the process

·         Estimate the accuracy of the improvement process.

While baseline is set to compare the state of the process before and after process improvement, we have to accept the reality that not always this comparison is valid or relevant given the change the process undergoes while the improvement activities are made. Below are few examples which helps us understand how performance comparison of an improved process with the baseline can mislead or becomes irrelevant.

·         When there is change in the resources based on which the baseline was set, the impact of the new resource on the baseline will not be known as the resource has worked only in the updated process. For example, if baseline is set based on manual process which is automated, and a new resource joins the team who is trained directly in the automated process, comparing the performance of the team and resource with baseline is inaccurate as there is a change in the team composition before and after process change.

·         When there is a change in project scope which was accommodated during process improvement, the influence of the updated scope on the baseline is unknown and hence comparison of performance becomes debatable. In hospital industry, duration for effect of the medicine on patient is baselined inline with historical data. But if there is unexpected factors which leads to variation in the expected duration the baseline will have to be adjusted considering the changes observed.

          Also when the goal of the project selected is wrong, the baselining will have to be adjusted. For Eg. When we try to work on improving the Voice of the customer collected throught survey from the client, (as it is a long term goal), we tend to miss on the short term goals. In case of serving a food to the customer, he says, food taste is not good. instead of correcting this immediately, if a person too much involves in the customer' s other preferences like appearance of hte food, he losses his short term goal of pleasing he customer.

·         When different skill set is required to work on the improved process, the expertise of the resources in the new skills is unknown and hence comparison of performance can be questioned as the impact of the newly acquired skill on the other components in the process performance metric has to be considered. For Eg. For a radiology coder, the baseline done basis historical data is 350. But when the coder is assigned a same radiology process with different platform, the baseline needs to be adjusted in line with after improvement numbers. Though the process reengineering is successful, the baseline is meaningless when we will have to compare two sets of skills.

 

Conclusion:

Baselining depends on the process we chose to improve. Baseline will have to be adjusted when the after improvement numbers drastically changes.

 

Thanks

Kavitha

 

Why Baselining is important

Baseline is the first step for anyone to understand how well a process is working or should work and then how much further we can achieve or take it to the next level.  In order to calculate the process baseline sigma, we need to have the following information at hand:

 

1. No of units that a process can produce
2. Total no. of defects opportunities per unit
3. Total no. of defects

 

Let us take an example:

 In a software development project for an online site (a java language web based application),   30 files (codes) are there. Customer is moving to Agile framework for the first time and is ok to have 10% defects for the first quarter.

 

There are 5 opportunities to produce defects per file.  

 

S.No	Opportunity
1	Usage of console - Print statements for debugging purpose which will clog memory
2	Code Not structured properly
3	Not handling Exception properly
4	Improper Relational Database handling
5	User Interface guidelines not properly followed

 

Here is the defects count for each file

Files	Defects Count
File 1	3
File 2	2
File 3	3
File 4	4
File 5	5
File 6	5
File 7	5
File 8	2
File 9	4
File 10	1
File 11	0
File 12	0
File 13	2
File 14	3
File 15	4
File 16	2
File 17	2
File 18	3
File 19	1
File 20	4
File 21	5
File 22	2
File 23	3
File 24	2
File 25	1
File 26	2
File 27	3
File 28	2
File 29	1
File 30	2
Total Count	78

 

To calculate the DPMO
Total number of Defects (D) = 78                          
Number of units (N) = 30                           

Total number of defects Opportunities (O) =5

Defects Per Million Opportunities (DPMO) = 1000000 * (D/N*O) = 1000000 * [78/(30*5)] = 1000000 * 78/150 =520000

 

This is equal to 1.45 Sigma (with 1.5 shift) with yield at 48% and a 52 % defect which is much higher than the customer baselined -  10% defect allowance

 

 Now a process improvement was put in place. Coding Standards were introduced and a code review process was put in place. Now with this improvement in place, the opportunities for a defect to happen got reduced to two. 

 

S.No	Opportunity
1	Improper Relational Database handling
2	User Interface guidelines not properly followed

 

                                                                                                                                                                      

Number of units (N)=30  

Total number of defects Opportunities (O) =2

 

Files	Defects Count
File 1	0
File 2	0
File 3	0
File 4	0
File 5	0
File 6	0
File 7	0
File 8	0
File 9	1
File 10	0
File 11	0
File 12	0
File 13	0
File 14	0
File 15	0
File 16	0
File 17	0
File 18	0
File 19	0
File 20	0
File 21	0
File 22	0
File 23	0
File 24	0
File 25	0
File 26	0
File 27	0
File 28	0
File 29	0
File 30	1
Total Count	2

 

Total number of Defects (D) = 2. Note the 2 defects are due to the one file having database error and another file having User Interface guideline issue.                            

                                                  

Number of units (N) =30  

Total number of defects Opportunities (O) =1

 

Defects Per Million Opportunities (DPMO) = 1000000 * (D/N*O)    = 1000000 * [2/(30*1)] = 1000000 * 2/30) = 66666.66

 

This is equal to 3 Sigma (with 1.5 shift) with 93.3 % yield and 6.7 % defect, which is well below the 10% baseline allowance from the customer.

 

As we see, there is a drastic improvement in the yield and the process improvement made has virtually eliminated all but 2 defects.  So the state before and after the process improvement is vast The baseline can even be re-shifted to this new state(6.7%),  since the new improvement process is quite capable of producing lower amount of defects.

 

Conclusion

Thus we can observe how baselining helps us to know the current position of the process and how much we can improve on our process. We also saw how performance made, after an improvement is having vast difference with the performance made before improvement and how adjustment is made to have a proper comparison.

 

Situations, where performance after improvement is not comparable with the performance before improvement, may occure in those cases where the scenario of process changes over time. For example, if a new kind of fuel is introduced in market to reduce air pollution and the improvement has to be studied after one year of its introduction, it may be noticed that over the period of one year, number of vehicles using that fuel has also increased and hence total fuel consumption has increased. In such a scenario, if we want to compare the improvement in air quality due to introduction of new fuel, some adjustment in baseline performance of either of the two situations would have to be made i.e. either the baseline performance (say pollution level) of past has to be increased in proportion to the increase in fuel consumption or current baseline performance has to be reduced in proportion to the previous fuel consumption. In such situations, performance levels have to be adjusted for fair comparison.