Q 453. The P - value in a hypothesis test helps us to conclude. Assuming the confidence level as 95%, does a P value of 0.049 provide a different inference as compared to a P value of 0.02? Explain your response with an example.

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Imagine a pizza place claims their delivery times are 30 minutes or less on average but we believe it's much more than that. A hypothesis test is conducted because we believe that the null hypothesis, Ho, is that the mean delivery time is 30 minutes. The alternative hypothesis (Ha) is that the meantime is greater than 30 minutes.

 

We run the hypothesis test on some data and come up with a p-value of 0.02 in the first instance and in the next instance we got 0.05.

 

We typically reject the null hypothesis if this probability is below 0.05, but P-Values of 0.049 and 0.02 have varied values of Probability of rejecting a true null hypothesis stated by Colquhoun and Sellke et al.

 

P value	Probability of rejecting a true null hypothesis
0.05	At least 23% (and usually close to 50%)
0.01	A minimum of 7% (and typically close to 15%)

 

Above table shows that the decrease from the initial probability to the final probability of a true null depends on the P-value. 

It is clear from the above points how important it is to have lower P values and to have reproducibility rates that are higher with smaller P values.

 

 

P-Value. Under the assumption that the Null Hypothesis is true, the P-Value is the probability of getting test results at least as extreme as the results actually observed. Hence a P-Value of 0.049 indicates that the likelihood of obtaining a value as extreme as the test result is 4.9% whereas a P-Value of 0.02 indicates that the likelihood of obtaining a value as extreme as the test result is 2%. Hence, we can conclude, the smaller the P-Value, the smaller is the likelihood of the outcome to be at least as extreme the actual results obtained.

Statistical Significance: Statistical Significance (Alpha) is the probability of rejecting the Ho, given that it is true. When p <= Alpha, the results are statistically significant.

Type 1 and Type 2 Errors.

Type 1 Error is the rejection of a null hypothesis that is true. It is called a false positive. The probability of Type 1 error is Alpha.

Type 2 Error is not rejecting the null hypothesis is false. It is called a false negative. The probability of Type 2 error is Beta where the Power of the Test is 1 - Beta

In False positive/negative, the word false means that the conclusion drawn is false.

Type 1 and Type 2 are complementary. If you decrease one, the other will increase.

Example.

Case 1. Convicting an innocent defendant is a type 1 error, whereas acquitting a criminal is a type 2 error.

Case 2. Test results says you have corona virus but you don’t is type 1 error, whereas test results indicating that you don’t have the virus and you have the virus is type 2 error

In Case 1, we will want a low significance level, however in Case 2 we will want a high significance level.

Selection of Statistical Significance.

The Statistical Significance is chosen before the collection of data.

In Case 1, we will want a low significance level, however in Case 2 we will want a high significance level.

In Academic Settings once the Statistical Significance (Alpha) is determined, and the p-value calculated and compared to the Alpha, the Null Hypothesis is either rejected or we fail to reject the null hypothesis. The absolute difference between the p-value and α is inconsequential.

Having said that, (as against Scientific or Academic research), in a Managerial setting, say to accept a consignment based on the p value, the manager may do the following in case the p value is close to Alpha (0.048)

(a) accept the consignment with caution or 

(b) take another sample to confirm the findings or

(c) accept the consignment at a reduced price and sell it on a discount or

(d) have an alternative use for the consignment.

However, in case the reduced quality may have an adverse effect on the reputation of the firm, the Manager would then reject the consignment.

References

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

https://en.wikipedia.org/wiki/P-value

 

A hypothesis test is done to ascertain whether two variables (say Y and X) are related. i.e. whether the Y (also referred as 'output') is impacted by a change in X (also referred as input).

 

We do a trial by taking few varying samples and see if the metric of interest is showing a difference on the Y for different values of X.

 

For example, if we want to study whether the average productivity of a process is same or different for 'Day shift' and 'Night Shift', we would take samples of productivity numbers during Day and Night and compare the average productivities for Day with that of Night. In this example, the Y is the Productivity and X is the Shift (Day or Night).

 

If we observe a difference in the average productivity between the Day and Night shifts based on the sample, the question that arises is "Is this difference due to a sampling (chance cause) variation or really due to the change of shifts"?. The 'p' value which is an output that is obtained after performing the test of hypothesis, gives the probability that the difference could be due to 'chance causes'. Obviously if the p value is very high then, it makes sense to believe that the difference is more likely to be due to chance causes and not due to the change of the shifts. In the language of hypothesis testing, we say that we accept the Null Hypothesis, Ho.

 

On the other hand, if the p value is very low, it indicates that the probability that the difference is due to chance causes is very low and hence it is highly likely that the change of shifts has caused the difference in productivity levels. As per the hypothesis testing language, we say that we reject the Ho (or accept the alternate Hypothesis, Ha)

 

The practice is to fix a threshold for the p value, beyond which we consider that the difference on the Y is not due to X, but only sampling variation. This threshold is known as the 'alpha' value and the default alpha value is 0.05 (equivalent to 5% probability). This also means that the confidence level (1-alpha) is 95%.

 

Now, a p value of 0.049 indicates that there is a 4.9% chance that the difference is due to chance causes and hence 95.1% confidence that the difference is due to the change in the input (X) variable. Similarly, a p value of 0.02 indicates that there is a 2% chance that the difference is due to chance causes and hence 98% confidence that the difference is due to the input variable.

 

By fixing a confidence level of 95%, we are setting our threshold of 5% for the p value for recognizing the difference as significant, if the actual p value falls below this threshold.

 

In both the above cases, the basic inference based on a test of hypothesis would be the same i.e. the p value is lower than the alpha value (5%) and we would infer that the difference due to the X variable is significant and hence there is a relationship between the two variables.

 

If we need to prioritize the strength of significance levels, as is done, when hypothesis tests are used as part of an experimental analysis, the lower p value may be taken as more significant.

Kudos to Manish and Johanan for attempting this very tricky question. Since the confidence level of 95% is fixed, a

p-value of either 0.049 or 0.002 would yield the same inference. However, since smaller p-values provide increasing stronger evidence, as a researcher I would like to see it as small as possible 

 

There are no clear winners for this answer. 

 

The response from our in-house expert is a must read.