Q 441. While doing ANOVA or DOE, a researcher usually prefers a balanced design. However, in real world, we might get an unbalanced design. Illustrate the difference between the two using an example. What are the methods to deal with unbalanced designs?

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

When you want to compare two or more groups and those groups are of different sizes. Unbalanced design has unequal number of observations in some treatment groups. While balanced design has equal number of observations in all treatment groups. True of t-test, ANOVA and multivariate tests.

 

A simple example below, where 2 Teaching Techniques for 3 Languages compared between groups. Observations/Sample sizes remain same for balanced while for unbalanced we have variations.

 

 

Unbalanced is a problem because one of the assumptions for the t-test is “approximately equal sample size”, Divide the larger group by the smaller group, and make sure the ratio is less than 1.5.  Assuming that the groups are equal, we can do a t-test on an unequally sized groups. But it may be wrong.

 

Primary reason that t-test also assumes that the variances of the groups are equal. This is the assumption of “homogeneity of variance”. When sample sizes are equal than, unequal variances are not a problem. T-test is insensitive to Heteroscedasticity.

 

As the sample size become more unequal, the more that ratio differs then the t-test becomes increasingly sensitive to heteroscedasticity. If we compare two groups –40 (n=40)  and the other of 3 (n=3), that test will be highly sensitive to violation of variances.

 

Balanced design provides following advantage:

 

1)     Power of an ANOVA is highest when samples sizes are equal across all combinations and we have the best chances of detecting the differences among the mean across treatment combinations.

2)     Overall F-statistics of ANOVA is less sensitive to violation of equal variance.

 

There are various reasons why unbalanced design can occur. For ex. Individuals may decide to opt out,

 

Options with unbalanced design:

 

1) Fix your research group so that each group is equally sized. True for t-test, ANOVA and multivariate analysis. Get the group sizes equal.

 

2) Collect more data, especially in case of n=3, get another round of data collection

 

3) If the problem is of unequal sample size but you have plenty of cases – is to randomly sample the larger group to get an equally sized comparison group

 

4) Welsch’s t-test- Does not have the assumption of homogeneity of variances between groups, so unequal samples sizes will not affect like with student’s t-test.

 

A balanced design is said to be used in ANOVA and Design of Experiments, if it  has  same no of observations for all possibilities in level/ group combinations, while an unbalanced design, can have unequal number of observations. The Levels or groups are referred as different groups of observations for the same independent variable considered 
Eg:  A balanced design with  30 Sample Calls being monitored for each category of Issue type base calls monitored while an unbalanced design may have say 28 calls of diff issue category like Network issue call , Billing Issue call etc. 
Balanced designs are usually preferred while performing statistical tests, because 
1.    The test statistic is less dependent  or insensitive to  small deviations from the assumption of equal variances 
2.    The power of the test is maximized if the samples are of equal size i.e The test will have larger statistical power
3.    The factors are independent and so the variance can be decomposed into the individual contributions without confounding.
But practically much ever researchers attempt to set up a balanced design for an ANOVA/ DOW , there are several reasons why unbalanced design could pop in 
•    People opting out of the study in-between
•    Organization level changes in Team and Departments or processes 
•    The plat being shutdown and not being able to provide required samples to make the size equal. Etc. 
With all the above reasons Balances Design is prefered for ANOVA and DOE, but incase of circumstances where Un balanced design is to be used it requires to be  treated with a lot more care than balanced designs.

 

Single Factor Experiments - Unbalanced Data. This is when the number of observations taken within each treatment is different.

Advantages of using a Balanced Design / Disadvantages of Unbalanced Design

The test statistic is relatively insensitive to small departures from the assumption of equal variances for the treatments if the sample sizes are equal. (This is not the case with unequal size samples)

The Power of the Test is maximized if the samples are of equal size

In an Unbalanced Design ANOVA, a modification is made to the Sum of Squares formulas.

Factorial Design – Unbalanced Data

Reasons.

Designed as a balanced design initially, however, due to unforeseen problems in running the experiment, may result in loss of some observations

Designed as an unbalanced experiment intentionally.

This may be the case when certain treatment combinations may be more expensive or more difficult to run, hence fewer observations may be taken in these treatment combination cells.

This may be the case when some treatment combinations may be of greater interest to the experimenter as they may represent new or unexplored conditions, so the researcher may do more replication in these cells.

Unbalanced Design Examples

Proportional Data. Here the number of observations in any two rows or columns is proportional. In this case, normal ANOVA works with minor modifications for the sums of squares formula

Approximate Methods

When the unbalanced data is not far away from the balanced data, an approximation can be done to convert the unbalanced data to a balanced one. Some of the ways approximation are done is given below

Estimating the mission Observations. If only a few observations are different, a reasonable procedure for estimating the missing values can be done. For a model with interaction, the estimated value should reduce the Error Sum of Squares. This can be done by taking the average of the observations in Cell (2,2) having 3 observations (1 observation missing).

 

Setting Data Aside. In this case Cell (2,2) has one data point more than the other cells, we set aside one observation from Cell (2,2) in order to obtain a balanced design

Method for Unweighted Means. This method was introduced by Yates (1934) in which the cell averages are treated as data and subjected to standard balanced data analysis to obtain the Sum of Squares for rows, columns and interactions. This is an approximate procedure because the sums of squares of the rows, columns and interactions are not distributed as chi-square random variables.

Weighed Squares of Means Method. Also proposed by Yates (1934). In this method, the terms of the sums of squares are weighted in inverse proportions to their variance.

Exact Method

This is done when empty cells occur (n_ij = 0) or when n_ij are very different. Here we develop the sums of squares for testing the main effects and interactions by representing the ANOVA model as a regression model.

References

Design and Analysis of Experiments by Douglas C Montgomery, International Students Edition, Eight Edition

A balanced design for ANOVA /DOE would be when the sample sizes are equal across all groups.

And it is called as unbalanced when the sample sizes are not equal across the groups

 

In the illustrations below, ANOVA (“analysis of variance”) models are used to determine whether the average handling time (AHT) of different tenure buckets of staff are same.

 

For example, we perform a one-way ANOVA to determine if 3 different tenure groups differ in average handling time of a task.

 

The table shows a balanced data set for one-way ANOVA:

Groups:                “<6 months”   “6-12 months”        “>12 months”

Sample size:                 30                    30                            30

 

The table shows an unbalanced data set for one-way ANOVA, there are unequal sample sizes.

Groups:                “<6 months”   “6-12 months”        “>12 months”

Sample size:                 28                    30                            29 

 

Or if we perform a two-way ANOVA to determine if different combinations of tenure buckets and background have impact on average AHT of the staff:

 

The table shows a balanced data set for 2-way ANOVA:

Groups/Subgroups:        “<6 months”    “6-12 months”        “>12 months”

Insurance                                        30                  30                          30

Banking                                           30                  30                          30

Telecom                                          30                  30                          30

 

 

The table shows an unbalanced data set for 2-way ANOVA, there are unequal sample sizes.

Groups/Subgroups:        “<6 months”    “6-12 months”      “>12 months”

Insurance                                     30                   25                             29

Banking                                        28                   27                             25

Telecom                                       38                   29                             24

 

A Balanced Design preferred because the power of an ANOVA is highest when sample sizes are equal across all group/sub-groups combinations. When the power is highest, it is a best way to detect differences among the means across Tenure and staff background when the mean AHT are truly different. Also, the overall F-statistic of the ANOVA is less sensitive to deviations to the assumption of equal variance. Thus, a balanced design would result in a reliable test statistic.

However, it is difficult to achieve a balanced design, due to several factors like availability of staff for data capture, continuity of data capture or accuracy of data capture (more so in this illustration)

In a situation the design is unbalanced we can try the following:

1.       If the assumption of equal variance holds good and sample sizes across groups differ, we may proceed with an ANOVA/DOE anyway. Equal sample size or balanced design is not a mandatory requirement while it’s known that the statistical power to the test would be highest in a balanced design

2.       If the assumption of equal variance holds good and sample sizes across groups differ, we may estimate and represent the missing data points, we can use the mean or median of the observations. We have to be cautious while doing this and must ensure the sample sizes among the groups/subgroups are near to equal before we do this.

3.       If the sample sizes are not equal and the assumption of equal variances is also not holding good then we can perform a non-parametric test equivalent to ANOVA such as the (called Kruskal-Wallis ANOVA). This type of test would be better for unequal sample sizes and unequal variances across groups/subgroup combinations for handling time.

Johanan has provided the best answer to this question.

 

Answer from Sai Kotari is a must read.