Q 521. What is effect size in statistics? Should a researcher look the effect size or the p-value? Support your answers with examples.

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Effect size in statistics is refer to as difference between mean of experimental group and control group divided by standard deviation. it's a quantitative measure of magnitude of the experimental group. 

 

Both two variables (mean of experimental group and mean of Control group) have a direct relationship with each other and stronger relationship between two variables depends on effect size. The larger- the better.

 

Example- Effect of a therapy on treating a mental health problem. the effect size value will show whether the therapy had a small, medium or large effect on mental health problem.

 

The symbol of effect size is r2, the effect size of Pearson r correlation varies between -1 (a perfect negative correlation) to +1 (a perfect positive correlation)  

 

A researcher should look both r value and p-value because the p-value is not enough all the time as p-value sometimes interpreted as meaning there is a stronger relationship between two variables, however it is unlikely that the null hypothesis is true (less than 5%). The p value > 0.05 is the probability that null hypothesis is true.  so, a significant p- value tells us that an intervention works, whereas an effect size tells us how much its works.

What is Effect Size?

 

Effect Size is the minimum difference that the researcher wants to detect between study groups and is also known as minimum clinical relevant difference. One can estimate the effect size by pilot study, previously reported data or guess based on clinical experience.

To understand Effect Size better we can discuss example – Suppose Drug A effect on average blood pressure as reduction by 10mm of Hg and drug B effect reduction of 20mm Hg. Then, absolute Effect size will be 10mm of HG in this case. Effect size can be expressed as absolute or relative difference. In this example relative difference is 50% or 10/20. In continuous outcome effect size will be numerical and in binary outcome effect size will be yes/no, investigator can estimate the relevant difference between the event rates in both trials groups and select, for example a difference of 10% between  both the group as effect size. Effect size will also determine the sample size. As sample size is inversely proportional to the square of difference. I.e. If effect size is smaller then sample size would be large.

                                               

Why report Effect Size or P Value?

 

P value just inform the researcher weather effect exist of not, P value will not show size of effect. In report interpretation, both effect size and p value are essential to report. For this reason effect size should be reported in in report by researcher.

 

Example: Comparing two proportions

 

A study was conducted to see effectiveness of drug in preventing shivering. It was found effect off drug in reduction of shivering is 70% to 30%. It is considers as significant effect in shivering reduction. What would be sample size? if α =0.05, Power is 0.95

 

So in this example Effect Size is P1 – P2 = 70%-30% = 40%

And Sample Size using Minitab is

Test for Two Proportions

 

So sample Size is 32 nos. & effect Size is 40%.

If we change Effect Size to 10% than what will be sample Size?

So let us consider P1 = 70% and P2 =60%, so what will be sample size keeping other parameter same; Using Minitab:

 

 

 

So sample size increased from 32 to 490 nos, if effect size reduces 40% to 10%

if effect size is small and sample size is not enough than there is risk of Type 1 and Type 2 error. Means Type 1 error is rejecting null hypothesis when it is true and Type 2 error is rejecting alternate hypothesis when it is true. 

In above case, if we compare P1 and P2 than we may get P value < 0.05, compare to α = 0.05 which is significance level for decision making and reject Null hypothesis as Ho: P1=P2, & alternate Hypothesis will be true as Ha:P1>P2. So P value just tells whether there effect exists or not, but not size of effect. So it is necessary for researcher to report Effect size in their report.

 

Effect size indicates the practical significance of a research outcome. it tells you how meaningful the relationship between two variables or the difference between groups is. A large effect size means that a research finding has practical significance.

 

While statistical significance shows that an effect exists in a study, practical significance shows the effect is large enough to be meaningful in the real world. Statistical significance is denoted by p-value, whereas the practical significance is represented by effect size.

 

Statistical significance alone can be misleading as it is influenced by sample size i.e. increasing the sample size will always make it more likely to find a statistical significant effect, no matter how small the effect truly is in the real world. In contrast to this, effect size is independent of the sample size which makes it relevant to showcase in order to represent the practical significance of a finding.

 

Let us understand the difference in statistical & practical significance through an example:-

 

In a study, we are comparing two weight loss methods with 13000 subjects each in two groups. One group let’s say uses method I of weight loss & the other group uses method II of weight loss. Now basis the results, the mean weight loss in Kg for one group is 10.6 kg with standard deviation of 6.7 kg, which is marginally higher compared to the mean weight loss in Kg for the other group which is 10.5 kg with a standard deviation of 6.8 kg. Statistically these results are significant at p=0.01, however a difference of only 0.1 kg between the groups is negligible & doesn’t really tell you that which of the weight loss method is more effective. Here adding a measure of practical significance can showcase the differences in the two methods.

 

There are various measures of effect size. Let us see some of the common ones:-

 

Cohen’s d :

 

Cohen’s d is designed for comparing two groups, it basically takes the difference between two means & expresses them in standard deviation units. It shows how many standard deviation lie between two means. Cohen’s d is calculated with the below formula:-

 

d = (x̅1 - x̅2) / s where x1-bar is the mean of one group, x2-bar is the mean of the other group & s is the standard deviation. In general, greater the value of cohen’s d, the larger the effect size.

 

Considering the above weight loss example, let us calculate cohen’s d for both the groups:-

 

d = (10.6 - 10.5) / 6.8 = 0.015, now with this value of cohen’s d, there’s limited to no practical significance that one group findings are more effective than the other group’s findings.

 

Pearson’s r :

 

It is also known as the correlation coefficient & it measures the extent of a linear relationship between two variables. The main premise is to compute how much of the variability of one variable is determined by the variability of the other variable. A value of pearson’s r closer to -1 or +1 indicates a larger effect size.

 

Below is the representation of the magnitude of the effect size in terms of both Cohen’d d as well as Pearson’s r methods:-

 

Effect Size : Small, Cohen’s d : 0.2, Pearson’s r : +/- 0.1 to 0.3

Effect Size : Medium, Cohen’s d : 0.5, Pearson’s r : +/- 0.3 to 0.5

Effect Size : Large, Cohen’s d : >=0.8, Pearson’s r : >= 0.5 or <= -0.5 

 

It is always helpful to calculate effect size before commencing any study & post data collection completion. The reason behind this statement is that within an expected effect size, one can figure out the minimum sample size required in order to have enough statistical power to detect an effect of that magnitude. If we don’t ensure enough power in a study, we may not be able to detect a statistically significant result even though it has practical significance, thus it is helpful to perform a power analysis, so that one can use a set effect size & significance level to determine the required sample size.Once data is collected, one can calculate & report the actual effect size. 

Effect Size

Effect size may be a statistical conception that measures the strength of the relationship between two variables on a numeric scale. For case, if we've data on the peak of men and women and we notice that, on the average, men are high than women, the difference between the peak of Male and the height of Female is known as the effect size. The lesser the effect size, the lesser the peak difference between men and women will be. Statistic effect size helps us in determining if the difference is real or if it's thanks to a change of factors. In hypothesis testing, effect size, power, sample size, and important significance position are related to each other. In Meta- analysis, effect size is worried with different studies and also combines all the studies into single analysis. In statistics analysis, the effect size is usually measured in three ways (1) formalized mean difference, (2) odd rate, (3) correlation measure.

Effect size tells you ways significant the association between variables or the dissimilarity between clusters is.

 An outsized effect size means that exploration finding has practical significance, while a little effect size indicates limited practical operations.

Sorts of effect size

Pearson r correlation Pearson r correlation was developed by Karl Pearson, and it's most extensively utilized in statistics. This factor of effect size is signified by r. the worth of the effect size of Pearson r correlation varies between-1 to 1. Consistent with Cohen (1988, 1992), the effect size is low if the worth of r varies around0.1, medium if r varies around0.3, and enormous if r varies further than0.5. The Pearson correlation is reckoned using the subsequent formula.

 

Where
r = measure of correlation
N = number of dyads of scores
∑ xy = Add of the products of paired scores
∑ x = Add of x scores
∑ y = Add of y scores
∑ x2 = Add of squared x scores
∑ y2 = sum of squared y scores
Formalized means difference When a hunt study is grounded on the population mean and standard divagation, also the posterior system is used to know the effect size

The effect size of the population are frequently known by dividing the two population mean differences by their standard deviation.

Cohen’s d effect size Cohen’s d is understood as the difference of two population means and it's divided by the standard divagation from the data. Mathematically Cohen’s effect size is represented by
Where s can be calculated using this formula:

Glass’s Δ system of effect size this system is similar to the Cohen’s system, but during this system standard divagation is used for the alternate group. Mathematically this formula are frequently written as

Hedges ’ g system of effect size This system is that the modified system of Cohen’s d system. Hedges ’ g system of effect size are frequently written mathematically as follows

Where standard deviation can be calculated using this formula:

Cohen’s f2 system of effect size Cohen’s f2 system measures the effect size once we use styles like ANOVA, multiple correlation, etc. The Cohen’s f2 measure effect size for multiple retrogressions is defined because the following

Where R2 is that the squared multiple correlation.
Cramer’s φ or Cramer’s V system of effect size ki- forecourt is that the stylish statistic to measure the effect size for nominal data. In nominal data, when a variable has two orders, also Cramer’s phi is that the stylish statistic use. When these orders are relatively two, also Cramer’s V statistics will give the simplest result for nominal data.

What’s the distinction between statistical and practical significance?

While statistical significance shows that an influence occurs in a study, practical significance shows that the impact is large enough to be important within the real world. Statistical impact is denoted by p- values whereas practical impact is represented by effect sizes.

Why Is not the P Value Enough?

Statistical significance is the probability that the observed difference between two groups is due to chance. However, If the P value is larger than the nascence position chosen(eg. 05), any noted variation is expected to be explained by slice inconsistency. With a sufficiently large sample, a statistical test will nearly always demonstrate a significant difference, unless there's no effect whatsoever, that is, when the effect size is exactly zero; yet veritably small differences, indeed if significant, are frequently pointless. therefore, reporting only the significant P value for an analysis isn't acceptable for compendiums to completely understand the results.
For illustration, if a sample size is 10 000, a significant P value is likely to be set up indeed when the difference in issues between groups is negligible and may not justify an precious or time- consuming intervention over another. The position of significance by itself doesn't prognosticate effect size. Distinct significance tests, effect size is individual of sample size. Statistical impact, on the other hand, hangs upon both sample size and effect size. For this cause, P values are believed to be mystified because of their dependency on sample size. sometimes a statistically significant outcome means only that a vast sample size was used.

Illustration

a coin is flipped 100 times. Call XX the number of heads.
• Θ is the true and unknown probability of" heads".
• Null thesis the coin is unprejudiced( θ = 0.5 θ = 0.5).
• Our test is grounded on the distance from 50 to X d = | X −50| d = | X −50|. It's called the statistical summary. For illustration, if you get X = 33X = 33 heads, d = 17d = 17.

p- value is the probability, assuming the coin is unprejudiced, that you see what you see P( d ≥ 17| θ = 0.5) P( d ≥ 17| θ = 0.5). A small p- value means" if the coin was poisoned, what I see would be veritably doubtful".

You decide to set an arbitrary limit for the p- value, say α = 0.05 α = 0.05. also your test is completely defined
• if p- value> α> α, answer accept unprejudiced
• if p- value< α< α, answer reject unprejudiced
αα is also the probability of a false positive.

The true (and unknown) effect size then's ever| θ−0.5|| θ−0.5|. It's how important the null thesis is false. The better data (better power), the more you can descry the effect(reject null) for a small effect size.

Summary

Effect size helps compendiums understand the magnitude of differences set up, whereas statistical significance examines whether the findings are likely to be due to chance. Both are essential for compendiums to understand the full impact of your work.

 

 

What is effect size in statistics? Should a researcher look the effect size or the p-value? Support your answers with examples.

 

Effect size quantifies the relationship between the variable or the difference between the group means. It helps to put a numerical value to the relationship or the difference. It signifies the practical importance of the relationship or the difference i.e., the outcome.

 

It is very important for researchers to look for effect size in addition to p-value as the later only tells the statistical significance while the former signifies the practical importance. This helps to validate that the research is important for practical applications. In fact, reporting guidelines require effect size along with confidence level wherever possible in many cases. It is recommended to calculate effect size before starting the study as well as after collecting the data.

 

It the sample size is large and/or has low variability in data, hypothesis tests can produce significant p-values for trivial effects. But effect sizes indicate the magnitudes of those effects. By assessing the effect size, it can be determined if the effect is meaningful in the real world or trivial with no practical importance.

 

Effect size can be calculated in two ways – unstandardized or standardized. Unstandardized effect sizes use the units of the base data or variables. Standardized effect sizes do not use any units i.e., these are unitless. The effects magnitude becomes apparent when the unit of measurement is removed. It also helps to compare it to other findings without the need to be familiar with units to understand the results.

 

There are many measures of effect size but the 2 most popular are Cohen’s d and Pearson’s r.  Cohen’s d determines the extent of the difference between two groups and Pearson’s r determines the strength & direction of the relationship between two variables.

 

Cohen’s d – It is used to compare 2 groups. The difference in means of the 2 groups is divided by the standard deviation. The choice of standard deviation is up to the research design. It can be a pooled value based on data from 2 sets or can be from a control group or from pre-test data. It determines how many standard deviations lie between the two means.

 

Pearson’s r – It is also known as correlation coefficient, it measures the extent of a linear relationship between two variables. It has a complex formula and mostly calculated using statistical software. This signifies how much of the variability of one variable is determined by the variability of the other variable.

 

Cohen’s d can take on any number greater than 0, while Pearson’s r ranges between -1 and 1. The greater value for Cohen’s d, indicates larger effect size. For Pearson’s r, if the value is closer to 0, the smaller the effect size and a value closer to -1 or 1 indicates a higher effect size.

 

Cohen’s d is used frequently for studies which compares two groups. E.g., Comparing the impact of certain food supplements on weight loss or height increase, the impact of two different modes of teaching on 2 groups of students, impact of 2 different medicines on groups etc.

 

Pearson’s r is used in studies which assess the impact magnitude and direction of impact of one variable on another. E.g., impact of food supplement on a group for weight loss or height increase etc. and many more areas where cause and effect types of study is required.

 

Edited November 15, 20223 yr by Anuj Bhatnagar
Answer was shared in word document

Effect Size:

 

Effect size will give the practical significant and shows the strength between variables or between a group.

While hypothesis testing we check the statistical significance which shows the effect exists using the P-Value where as the practical significance shows how the effect is large enough and is represented by the effect size

Effect size helps to understand the magnitude of differences found, whereas statistical significance examines whether the findings are likely to be due to chance.

While a P value tells if an effect exists, the P value does not reveal the size of the effect. Hence, in reporting and interpreting studies, both the effect size and statistical significance (P value) are essential results to be reported.

 

When hypothesis test indicates statistically significant results, we can conclude that there is sufficient evidence that an effect exists in the population. This helps to rule out random sampling error as the culprit for an apparent effect in sample. But this doesn’t necessarily mean the effect size is meaningful in the real world.

To understand if a statistically significant result is practically important or not, the effect size which provides the magnitude of the effect comes in handy to answer crucial questions such as how much a particular treatment worked or how strong a relationship between a dependent and an independent variable is.

 

 

Consider the above example where the medication A is compared with medication B to understand which is more effective treating a disease. Assuming a p-value < 0.05 for both samples, provides an inference that there is a statistically significant difference in means between the treated and control groups in both cases. Now to understand which sample is better than the other, an effect size can be used. The effect size, calculated by subtracting the means indicate that the effect size for A is (20-5 = 15) is bigger than the effect size of B (16-14 = 2). To derive further insights and accuracy, it is also worthwhile to look into the confidence intervals of the effect size. In the above example, lets say, the CI for effect size A is [-4 to 25] and effect size B is [1 to 3]. Now, B looks more precise and statistically significant whereas A is imprecise and considering also includes 0 in its CI, can also result in a no treatment effect.  

Rahul Arora has provided the winning answer to this question. He has explained the concept, its comparison with p-value along with how it is measured in a concise and effective manner.