What is the meaning of statistically significant difference? What are some of the most important ways to utilise this concept in problem solving and decision making? 

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time.  
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

In any business, performance is typically expected to vary over time and w.r.t. inputs. When comparing two performances, it would not be completely correct if a decision that the performances are different were to be taken based on comparison of just one or few data points from both the performances. Sampling errors should not influence the decision. Therefore, it is essential that the correctness of the decision taken should be sustainable over time.

 

For the decision to be sustainable, data that reflect the sustainability of both the performances will be required. Once this data is available or is collected, the decision based on this data is also expected to sustain over time. The decision that is taken based on samples must hold good for the populations also. In other words, even after some unavoidable overlaps of both the performances, perhaps due to chance causes, the difference in the performances of the two populations must be visible, conspicuous and clearly discernible. In other words, the difference in the two performances need to be significantly different.

 

But “significance” is quantitative and statistical. The significance of the difference is assessed from statistical data of the two performances. Statistically significant difference represents the clarity or discernibility of the difference between the two performances and the sustainability of this difference over time. Performances of two populations with a statistically significant difference will remain different over time unless there are some special causes in play on one or both of them.

 

But how significant is significant? This depends on the objective of comparison and the stakes involved. The margin of error tolerable in taking a decision on the difference between the performances depends on these factors. For different combinations of conditions, this margin of error could be 1% or 5% or 10% or any other agreed number. This is the error involved in the decision to conclude that the two performances are significantly different based on the available statistics.

 

Uses of the concept of Statistically Significant Difference in Problem Solving and Decision Making

The uses of this key concept of “Statistically Significant Difference” to solve problems and take decisions are innumerable, a few of which are given below.

 

1.      Comparison of performances between two or more

a.      Time periods

b.      Processes

c.      People

d.      Suppliers or Service Providers

e.      Applications

 

2.      Assessing effectiveness of

a.      Training

b.      Improvements

c.      Corrective Actions

d.      Action taken on suspected root causes

 

3.      Evaluating

a.      User ratings in market surveys against marketing campaigns

b.      Performances of new recruits against agreed targets

 

In all the above cases, Hypothesis Testing can be effectively applied to assess the existence of a statistically significant difference.

Statistically significant difference is the statistical confirmation that the difference seen between two sets of data points are significant. If we go by simple difference and conclude that two sets are different it may not help us understand if the difference is significant. Only if its statistically significant difference then it makes sense to draw conclusion about set of data points.

This is evaluated using hypothesis testing, where null hypothesis is taken as there is no statistically significant difference between two sets of data points, whereas alternate hypothesis is stated as there is statistically significant difference between the data points. Which hypothesis test to be chosen depends on the type of data sets and is a different topic of discussion. Based on hypothesis results obtained (p value) either of the hypothesis statement is proved.

One basic application of this concept is to evaluate, if the actions taken for a problem is effective. Only if it proves that the data sets before improvement and after improvement are having statistically significant difference then we will arrive at the fact that the action plans are effective, if not analysis of cause and action plan would need further investigation. This analysis also helps in understanding how significant is the difference by looking at standard deviations of data sets as well. Based on the statistical significance of the difference between two data sets, further decision making will be influenced.

Also this concept helps to make decision about the characteristics of the two population of data like, to select best vendor for the services based on their historical performance data. Also would help to make decision to select best performing product based on their historical performance.

Statistically significant difference means the difference may not be big enough to be importance to the business. The bigger the sample size used in an analysis, the smaller the deviation from the null hypothesis that may be detected. 

 

We use this concept in hypothesis testing. The equivalent to one minus the probability of a Type 2 error (1- beta) is called power. A higher power is associated with a higher probability of finding a statistically significant difference. Lack of power usually occurs with smaller sample sizes. The beta risk or consumer risk is the probability of failing to reject the null hypothesis when there is significant difference.Also the power of the sampling plan is defined as 1 - beta , hence the smaller the beta, larger the power.The product is passed on as meeting the acceptable quality level when in fact the product is bad. Typically the beta is 0.10% means there is 90% probability that we are rejecting the null when is is false which is correct decision.

Suppose we are conducting an experiment where we measure the weight of 100 random males and 100 random females. We find that for our sample the mean (average) weight of males is 70 kgs with a standard deviation (SD) of 20 kgs, For females we find that the mean weight is 60 kgs with a SD of 20 kgs. 
 

In our sample we find that males weigh more than females (70 kgs vs. 60 kgs), but can we extrapolate these results from our sample to the entire population (all males and all females in the world) and conclude that males weigh more than females on average? 
 

Statistical significance answers this question by telling us how likely (probability) it is that an alternate hypothesis (males DON'T weigh more than females) is true for the population. In our example its turns out that this probability (p-value) is 0.0005, which means it is extremely likely (99.9995% confidence) that our original hypothesis (males DO weigh more than females) is true. So we can say our results are 'statistically significant'.

 

Generally it is accepted that if p-value is less than 0.05 our result is statistically significant, and we can say with 95% confidence that our result will hold true for the population. Although this choice of 0.05 cutoff is completely subjective and arbitrary, we can define our own statistical significance based on how much confidence we want to have in your experiment. 

 

If on the other hand we got the same means and SDs after conducting this same experiment but this time on a sample of 10 males and 10 females, the p-value would be 0.28, and now we would only have 72% confidence that our hypothesis will hold true for the population, so our result is 'statistically insignificant'. 

Case for Statistical Significance – an example

Let’s consider the following data that is the age of 10 employees.

42, 35, 24, 31, 33, 41, 33, 31, 33, 32.

Assume these 10 data points are a sample that represents a large population of say, over 5000 employees. Now, using this available information we are asked a question “whether the average age of the employees in this population can be considered equal to 30 years?”

The quickest thing that anyone would do is to compute the average of the samples, which comes to 33.5. Since this is 3.5 more than 30, can we say that the population average age will be more than 30? These are situations where there is bound to be judgmental subjectivity and likelihood of reaching incorrect conclusions.

This is a simple example of a situation where a test of hypothesis may be done and the concept of statistical significance helps to reach an objective conclusion.

Statistical Significance – what does it imply?

Statistical significance implies that the difference that is under evaluation, (whether it is a population average being compared to a specified value, or the averages of two populations are being compared, or the variances of two populations are being compared, etc.) can be considered as a difference that is significantly larger than what a chance cause variation would have caused.

Since what we have is a sample data, it is to be noted that for different set of samples, the sample average is expected to vary with in certain limits for the same population (and same population average). The limits are governed by the variance of the population.

The test of significance will evaluate, with the given set of data, whether the sample average is falling within the confidence limits or not. So long as the sample mean falls within the confidence limits, the conclusions will be that there not sufficient reason to believe that the population average represented by this sample is different from the specified value.

Usage of Statistical Significance

In today’s world the application of tests of significance has been simplified using statistical software such as Minitab. Once we give the inputs depending upon the case being studied, the application comes out with a P value, which is used to determine the significance of the results. Smaller the p-value, the evidence against the null hypothesis becomes stronger. Usually a p-value < 0.05 is used as the criteria for rejecting the null hypothesis; i.e. the difference is considered significant.

As part to problem solving, tests of significance are integral part of Hypothesis testing, Analysis of Variance, Design of Experiments and other tools. It helps to take objective decisions with small samples. These methods are particularly useful during the Analyze phase where it helps to narrow down on short listed causes;   and improve phase where the effectiveness of identified solutions could be validated.

Statistical significance is most commonly used when dealing with quantitative methods. It is a result (usually a difference) that’s not attributed to chance. It means that if the Null Hypothesis is true (which means there really is no difference or that nothing happened or changed), there’s a low probability of getting a result that is large or larger. Two important factors to be considered here are

1.    Sampling Error. There is a possibility that the difference when measuring a sample of users is the result of random noise or chance fluctuations.

2.    Probability; never certainty. Statistics is about probability; not 100% certainty and of managing risk. What is the percent likely that a decision is wrong? 5%, 10%, 33% and what is the consequence of making a wrong choice?

If we do a large number of tests, falsely significant results are a problem. If there is a 95% chance of something being true that means there is a 5% chance of it being false. This means for every 100 tests that show results significant at the 95% level, the odds are that five of them do so falsely. The odds are that five tests would be falsely reported significant. So the more tests we do, the more there is a problem of false positives. We cannot tell which the false results are. They are just there. So how to solve this problem? One way is to limit the tests to a small group or to repeat the study to check if we get the same results. This may be practically not feasible in real life survey. Another method is to use the split halves technique where the sample can be randomly divided into 2 halves and a test can be conducted on each half.

Statistical significance does not mean practical significance. To declare practical significance, we need to determine whether the size of the difference is meaningful. Only by considering context can we determine whether a difference is practically significant; that is, whether it requires action. It also provides likely boundaries for any improvement to help in determining if a difference is really significant.

Thus Statistical significance plays a pivotal role in statistical hypothesis testing. It is used to determine whether the null hypothesis should be rejected or retained. For the null hypothesis to be rejected, an observed result has to be statistically significant, i.e.  p <α.  P (p-value) is the probability of observing an effect given that the null hypothesis is true and α is the significance level.

 

Statistically significant difference means that the two variables that we are comparing are having some difference caused because of some special cause and not because of chance.

 

1. We can use this analysis to verify whether the changes that are done in the process are actually giving the desired results or it is just random chance.

In a company, downtime caused due to Maintenance and electrical problems was 5%, after we have a new maintenance manager; he rolled out a new system of maintenance as a result of that downtime reduced to 3.5%. With this analysis we can statically prove that the reduction in downtime is because of the new process.

 

 

 

 

     2.      For measuring one quality parameter, we have two machines. We can use this analysis to check whether testing on type of machine gives different result.

From the analysis we can statistically prove that machines does affect the results that we are getting, because the p value is <0.5.

 

3. Similarly, if for an output Y, we have multiple X. we can also analyze which X are having statistical significant difference. and find the impact of that X with further analysis.

 

 

 

Statistically significant means that we are very sure that the statistic is reliable. It is a statistical term that gives the surety of a difference or relationship exists.

For example: Suppose we give 1000 employee an IQ test, and we are ask if there is a significant difference between male and female scores. The mean score for males is 98 and that of female is 100. We use independent groups t- test and find that the difference is significant or the 0.001 level. The difference between 98 and 100 on an IQ test is a very small difference.., which is not important

After finding a significant relationship, it is important to evaluate its strength which could be weak or strong; large or small. This depends upon the sample size. 

 

ONE TAILED OR TWO TAILED SIGNIFICANT TESTS: One tailed or two tailed significance depends on hypothesis. 

 

ONE TAILED SIGNIFICANT TEST: 

When the hypothesis states the direction of the difference it is said to be one tailed significant probability test.

Example: females will score significant higher than males in test.

Blue collar workers will not buy significantly more product than white collar.

In the above mentioned examples the null hypothesis predicts the direction of the difference.

 

TWO TAILED SIGNIFICANT TEST: 

A two tailed test would be used to test these null hypotheses.

For example: There is no significant difference in test scores between females and males. There is no significant difference between blue collar and white collar workers.

 

THE ONE TAILED PROBABILITY IS EXACTLY THE HALF THE VALUE OF THE TWO TAILED PROBABILITY.

 

 

 

 

 

 

 

 

 

Statistical Significance is mostly,used in health research to examines the difference between two sample groups, to determine if it is statistically significant. The difference between two groups is statistically significant if it cannot be explained by chance alone.

Statistical significance can be determined by arriving at the probability of error (p value) by the t ratio.

The generally accepted rule is that the difference between two groups (such as an experiment vs. control group) is judged to be statistically significant when p = 0.05 or less.

-         A 5% probability of occurrence of difference between 2 groups is conceived when the p value = 0.05 or more.

-         Only a 1% probability of occurrence of difference between 2 groups is conceived when the p value = 0.01.

The lesser the value of P below 0.05, the better the chances of the hypothesis not being null and hence the better option for the alternate hypothesis that can be pursued.

The whole exercise helps to determine whether an alternate Hypothesis is worth pursuing by research. The best way to move to , or consider a follow up study is to determine the statistical significance difference between 2 groups through a pilot study and then design a larger study. This gives better confidence on the conclusions.

 

Probability of rejecting null hypothesis when it is true is statistical significance i.e the chance of error allowed is significance level. It is denoted by greek letter “α”

 

 

·        P-value is used for decision making in a hypothesis testing.

·        If P value is less than or equal to a predetermined level of significance (α level), then null hypothesis is rejected and alternate is claimed.

·        If P value is greater than the α level, then null hypothesis is accepted and alternate cannot be claimed.

·        0.05 is the acceptance level of Type I error, thus any p-value less than 0.05 means we reject the null hypothesis.

·        If α is small, incorrectly rejecting the null hypothesis chances are less.

·        If α is large, incorrectly rejecting null hypothesis chances are high.

Thus choosing α level is very important for a data to check whether it is statistically significant or not to proceed further.

When we study the effect of some input parameters on output parameter and we try to find out that these input parameters affect output parameter in a significant way or not, we take help of statistics. Basically, the change in output parameter can be attributed to two things, either the change in output may be actually due to variation in input parameters or it may also be due to chance causes. To differentiate between these two possibilities we take help of statistics. Depending upon type of data various types of statistical tests are available to find the significance. For example, while using ANOVA we use F-test to compare two variances of output parameter. If the ratio of two variances is larger then a criterion then we can conclude with certain level of confidence (say 95 or 99%) that the difference in variance can be attributed to change in input parameters and is not due to chance causes (null hypothesis is void).

This concept of 'statistically significant' can be used in many real life problem solving and decision making techniques. It can be used to compare survey results, in Design of Experiments (DoE), in hypothesis testing etc.

Here it may be noted that this 'statistically significant' concept largely depent on sample size. If sample size is large then small difference can also be significant whereas in case if small sample the difference between two data sets has to be large to conclude 'statistical significance'. 

Statistically Significant Difference

By principle, a statistically significant difference does not attribute to a chance.  Sometimes its possible to detect significant difference between two populations when there is no practical importance.   So what would the driving factors to consider to convert statistically significant difference into a practically significant difference?  It depends on the context where this is achieved. However  Cost, Time, Resources would be some of the key elements in deciding that

Let us see how we can utilise this concept in Problem Solving and Decision Making with an example.

Eg: A Supermarket retailer decides to buy firecrackers for Diwali festival , so that he can sell to his customers. He prefers to buy from one of the two reputed brands which produces crackers- Brand A and Brand B. His marketing team told him that both are good brands and will have good quality crackers.

So from a hypothesis perspective, null hypothesis (hO) made by him was there would not be any or appreciable difference in terms of quality / sound for crackers of similar nature. Alternate hypothesis (ha): There is a difference in terms of quality / sound between the two brands and that Brand A is better than Brand B.   

     

Initially with a limited amount of samples that the retailer got, both the brands seemed to be ok in terms of quality/sound. But as more and more samples from different cracker of categories were put , the difference became significant . So it became apparent that null hypothesis had to be rejected and hence Alternate hypothesis was accepted.  Therefore, retailer came to the conclusion that Brand A is better than Brand B.

However the retailer do not want the Brand A crackers.  Because Brand A is too costlier than Brand B  for this difference in quality . So it is not worth for the retailer to pursue to buy crackers buy from Brand A.   So in this case, the statistical significance is not turning into a practical significance. The retailer took a decision with the help of the hypothesis testing with right sample size

 

Conclusion

As we have seen statistically Significant difference can aid in decision making and problem solving provided if we are able to focus on the factors that lead to practical significant difference

 

 

 

 

Question: What is the meaning of statistically significant difference? What some of the most important ways to utilize this concept in problem solving and decision making? 

Definition: A statistically significant is usually means statistically significant difference.

It is defined as the chance of existence of relationship between two or more variables is because of some other cause but not definitely by the random cause. The hypothesis tests are used to prove the relationship using various tests. It is interpreted by P Value. In general, when a p value is less than or = 5%, it is called statistically significant. It does not mean that the finding is important to consider or the decision making taken is reliable.

 

For Eg. 5000 coders are given enough training to check if there is any significant difference between male and female coder’s test scores. Lets say Mean for male is 97 and female is 99. We can use t test to compare the independent groups at .01 level of significance. The difference calculated is a very small difference. To say, it not even important as it is derived out of samples.

It is because when u have the larger group for study, the difference would be smaller, which means the result is real not by any chances.

 

Significance means Statistically it is important that the relationship exists between 2/more variables.

P- Value:

P-value is the level of marginal difference that it represents the chances / likelihood of the occurrence of a ny given event. When te p value is small it means there is a strong relationship / evidence in relation to alternative hypothesis.

One-Tailed and Two-Tailed Significance Tests

This is part of significant difference. It is important the hypothesis stated is of one tailed or two tailed. If it talks abnout the direction of relationship, it is one tailed probability. This is used to compare the group in one direction. Null hypothesis predicts the direction of the chance fixed.

For Eg. Males are generally stronger then females. Female coders score higher than male coders.

A two tailed would be stating its null hypothesis In the following way.

Eg. Males and females are equal. Null hypothesis states there is no significant difference.

Procedure Used to Test for Significance

1.     Decide on the alpha value.

2.     Conduct the study

3.     Calculate the sample statistic

4.     Compare the critical value obtained.

Thanks

Kavitha

 

 

Its means how much percentage of error we might take. If in any statistical tool its says .05 means the the reault is 95% accurate. Here we take p-value to consider. The smaller the sample size, the harder to get accurate result. Standard of significant is 5%, i.e 0.05. The more sample size the more better result we will get. 

Statistical Significance:

being of importance statistically.

 

meaning, the importance is emerged from  the statistics or of a data.

 

statistically significant difference,: 

The difference of the findings from different set of samples from the same population. ??

 

it is not necessary that the whole population is represented exactly by the sample. Even though the interpretation of a sample is considered as for the whole population, it could also sometimes leads to wrong or misleading results. 

 

If such a difference is statistically significant , or could show different statistically. 

 

It is always possible that there will always be difference in the results of the sample from the results of its population or from other similar population.

 

but why we say statistically significant difference is maybe because the difference is also measured statistically.

 

 

It is heartening to see that the complexity of the topic did not deter so many of us from taking a stab at the answer. That shows the true spirit of excellence! 

 

There were two parts to this question, and that was the tricky part. Not all attempts answered both parts - the meaning of statistically significant and practical application. Needless to say, articulation in your own words is a key, too.

 

The answer which was complete in all respects was by Mohan PB. Rajesh Gadgil and Venugopal R made a very good attempt too. Cheers to everyone who contributed.