If the p value is higher than the acceptable type I error, it only means that you do not have enough evidence to be able to reject null hypothesis. It really does not mean that null is true.

Those who have not studied hypothesis tests, shall not be able to touch the topic (well, this is understandable). However, those who have gone through a Green Belt or a Black training may like to discuss and comment here. Please feel free to explain with example.

Agree with you VK.

A couple of things that needs to be kept in mind while doing hypothesis:

1. The concept of sample and population and the fact that whenever we take a sample there are chances that the test statistic will not be equal to the population parameter.
2. More often than not, we are more interested in the alternate hypothesis as that makes my life interesting

If after hypothesis, I get a high P, means the statistic from the sample in question is probably "insignificantly" different to the population parameter. however, there could be samples where the statistic could be "significantly" different from the population parameter.

 

Not sure if this example would fit in this case but recently came across a case where even after project completion the 2 proportion test was not showing significant improvement. The problem here was that the baseline data was for 6 months and the improvement data was for only 1 week. Once we compared the improvement data for 1 complete month, we could notice a significant improvement in the project. We concluded that we needed to have a more appropriate sample for the test to give "significant" results

Important observation there Mayank and your example fits perfectly here.

• Let us say a specific small difference (X) in proportions (or averages/medians) when comparing two populations (based on hypothesis test done with large sample data) is found to be significant.
  
• The same difference (X) may be absolutely insignificant when you take smaller sample sizes from the same two populations.
  
• This means if you work with insufficient sample sizes, small real differences in populations cannot be detected.
  
• Large sample sizes provide you with higher power (ability to detect small differences if they exist in populations)
  

Views, anyone?