Leaderboard

Case for Statistical Significance – an example Let’s consider the following data that is the age of 10 employees. 42, 35, 24, 31, 33, 41, 33, 31, 33, 32. Assume these 10 data points are a sample that represents a large population of say, over 5000 employees. Now, using this available information we are asked a question “whether the average age of the employees in this population can be considered equal to 30 years?” The quickest thing that anyone would do is to compute the average of the samples, which comes to 33.5. Since this is 3.5 more than 30, can we say that the population average age will be more than 30? These are situations where there is bound to be judgmental subjectivity and likelihood of reaching incorrect conclusions. This is a simple example of a situation where a test of hypothesis may be done and the concept of statistical significance helps to reach an objective conclusion. Statistical Significance – what does it imply? Statistical significance implies that the difference that is under evaluation, (whether it is a population average being compared to a specified value, or the averages of two populations are being compared, or the variances of two populations are being compared, etc.) can be considered as a difference that is significantly larger than what a chance cause variation would have caused. Since what we have is a sample data, it is to be noted that for different set of samples, the sample average is expected to vary with in certain limits for the same population (and same population average). The limits are governed by the variance of the population. The test of significance will evaluate, with the given set of data, whether the sample average is falling within the confidence limits or not. So long as the sample mean falls within the confidence limits, the conclusions will be that there not sufficient reason to believe that the population average represented by this sample is different from the specified value. Usage of Statistical Significance In today’s world the application of tests of significance has been simplified using statistical software such as Minitab. Once we give the inputs depending upon the case being studied, the application comes out with a P value, which is used to determine the significance of the results. Smaller the p-value, the evidence against the null hypothesis becomes stronger. Usually a p-value < 0.05 is used as the criteria for rejecting the null hypothesis; i.e. the difference is considered significant. As part to problem solving, tests of significance are integral part of Hypothesis testing, Analysis of Variance, Design of Experiments and other tools. It helps to take objective decisions with small samples. These methods are particularly useful during the Analyze phase where it helps to narrow down on short listed causes; and improve phase where the effectiveness of identified solutions could be validated.

In any business, performance is typically expected to vary over time and w.r.t. inputs. When comparing two performances, it would not be completely correct if a decision that the performances are different were to be taken based on comparison of just one or few data points from both the performances. Sampling errors should not influence the decision. Therefore, it is essential that the correctness of the decision taken should be sustainable over time. For the decision to be sustainable, data that reflect the sustainability of both the performances will be required. Once this data is available or is collected, the decision based on this data is also expected to sustain over time. The decision that is taken based on samples must hold good for the populations also. In other words, even after some unavoidable overlaps of both the performances, perhaps due to chance causes, the difference in the performances of the two populations must be visible, conspicuous and clearly discernible. In other words, the difference in the two performances need to be significantly different. But “significance” is quantitative and statistical. The significance of the difference is assessed from statistical data of the two performances. Statistically significant difference represents the clarity or discernibility of the difference between the two performances and the sustainability of this difference over time. Performances of two populations with a statistically significant difference will remain different over time unless there are some special causes in play on one or both of them. But how significant is significant? This depends on the objective of comparison and the stakes involved. The margin of error tolerable in taking a decision on the difference between the performances depends on these factors. For different combinations of conditions, this margin of error could be 1% or 5% or 10% or any other agreed number. This is the error involved in the decision to conclude that the two performances are significantly different based on the available statistics. Uses of the concept of Statistically Significant Difference in Problem Solving and Decision Making The uses of this key concept of “Statistically Significant Difference” to solve problems and take decisions are innumerable, a few of which are given below. 1. Comparison of performances between two or more a. Time periods b. Processes c. People d. Suppliers or Service Providers e. Applications 2. Assessing effectiveness of a. Training b. Improvements c. Corrective Actions d. Action taken on suspected root causes 3. Evaluating a. User ratings in market surveys against marketing campaigns b. Performances of new recruits against agreed targets In all the above cases, Hypothesis Testing can be effectively applied to assess the existence of a statistically significant difference.