You want output Y to improve and are looking for X that influences Y. Share examples when the outcome of a hypothesis test indicates statistical significance with respect to impact of X on Y but does not warrant a change in X

safety issue

When the cost of changing X outweighs the potential benefits gained in Y.

Given: X impacts Y. However, it doesn't indicate how X impacts Y.  A plausible scenario is that X can only impact Y adversely.

Any case where even though there is a statistical connection between X and Y, but your acceptable alpha is high enough that it will not change the total usable outcome of Y.  For instance, if one managed to proved the connection between X and Y, but it is not great enough to affect First Pass Yield on a production line.  

Difference is between statistical significance and practical significance.  If changing x, therefore y, does not have an impact that is important to the leaders of the business, then the efforts are better directed somewhere else.

 

An instance in which X significantly impacts Y but does not warrant changing X includes instances in which a manufacturer would have to compromise the integrity of their product for the sake of improving Y. For example, using copper wiring is significantly more costly to building a house than using aluminum wiring. However, the builder should not switch to aluminum because aluminum degrades significantly faster than copper and will demand more maintenance. Additionally, the interface of that aluminum with any non-aluminum wiring will create a voltage potential and possibly a fire hazard.

Let's say that Y = gas mileage. One factor X is drivers of cars. We find there's a statistical difference between drivers. But, we aren't going to change drivers. This factor X is a noise factor and can't be changed.

When the change in X would have a negative impact on safety, or when the cost of implementation of the change is prohibitive, when the change has a negative impact on another part of the system.

Share examples when the outcome of a hypothesis test indicates statistical significance with respect to impact of X on Y but does not warrant a change in X

 

• Cases where p is close to .05 the sample size may be small - need more data to make the decision
• A case where the X is at the limit of the specification or a threshold - cannot change X - machine temperature max is 100F  - may need to assess other variables like speed or air flow or
• A case where the cost to make the change to X is too costly versus the benefit of the change - business case not valid

Even though X may be shown to have a statistically significant effect on Y, it may not justify any changes in X if the marginal cost of the change in X exceeds the marginal benefit in Y.  Examples include going to higher quality materials that are expensive, but do not result in significant improvements in performance.

You may choose to not change x if the business case cannot be made.  It may not be practical to make changes to the X.  Statistical significance is not the same as practical significance.

When the change will have a negative impact (e.g. Safety, cost, an adjacent process)

Statistical significance between X and Y are important from data analysis point of view. However the correlation between X and Y is crucial significantly from Business consideration, we commonly refer that as Practical significance. 

One of the best example for this misnomer is Water purification system which are an important factor for many industries including Pharma. Chemical dosing is one of the X which impacts the Y i.e., hardness of water. However at certain level X can not be altered because of other considerations. Hence at a certain point the Changing X does not warrant a change in the Y.

 

Above example shows that, through Hypothesis testing, even though we can conclude that changing X impacts the Y statistically , however it does not matter in lieu of the practical significance.