Q 604. What is the difference between overall standard deviation and within standard deviation? Could there be a scenario where within standard deviation is greater than the overall standard deviation? Provide examples to support your answer.

 

Note for website visitors -

• This platform hosts two weekly questions, one on Tuesday and the other on Friday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be approved. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error prone as our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://crossplag.com/ai-content-detector/. Only answers with less than 15-20% AI-generated content will be approved.

Standard deviation represents the typical (average) distance from the central location we expect to observe. Its units are exactly the same as our original measurement.  Smaller the standard deviation, the better it is. Type of standard deviations are:

 

1.       Overall Standard deviation

Standard deviation is computed for all data points of long period like entire month. In this case, it represents actual variation of the process that the customer experiences over the time.

 

2.       Within Standard Deviation

It is an estimate of variation within the subgroup. This represents inherent variation of the process over a short period of time. This shows potential variation of the process if shifts and drifts between subgroups were eliminated.

 

“Below reference picture will articulate more visualization in terms of short-term samples and long-term samples."

 

 

Could there be a scenario where within standard deviation is greater than the overall standard deviation? 

Practically, Overall standard deviation should always be greater than or equal to the calculated within-standard deviation because standard deviation (overall) is affected by various causes for examples like “Equipment breakdown”, “Environment Effect”, “Unskilled operator deployment on machines’, “Difference between raw materials” and other’s special causes in long period of data. On the other side, standard deviation (within) is purely random.it is what we use to compare the inherent capability of different processes to meet a specified goal. 

Below graphical picture itself portrays the cause of higher overall standard deviation than within standard deviation with time extent.

 

 

Overall standard deviation is the standard deviation of all measurements and it is an estimate of overall process variation while within standard deviation is an estimate of variation within all sub groups. 

 

 

There can be cases where within standard deviation is higher than overall standard deviation. It indicates greater  variation within sub groups and that the process is not stable. It can also mean that our process can have other sources of variations in addition to variation with sub groups.

 

Sl. No	Overall standard deviation	Within Standard deviation
1	Considers all the readings from the measurements	It’s an estimate of the variation within the Subgroups
2	Also known as Inherent process variation	Also known as within-subgroup variation
3	Captures all sources of systematic variation	Captures nature and inherent variation of the process over a shorter period
4	Represents actual variation of the process	Represents the potential variation of the process
4	Represents using indices Pp and Ppk	Represents using Cp and Cpk
5	Standard Deviation Calculation-Population	Standard deviation calculation - Sample

 

 

Within Standard deviation will be higher, if the process capability is not stable i.e., lesser than one and the sample collected as subgroups are out of spec. But the overall variation in the process is somewhat stable and process needs to be centered.

 

The Standard Deviation (overall) would take all data points of the entire month and calculate one number as an output. It is calculated by considering all data points in the dataset, without regard to any specific subgroups or categories. The Standard Deviation (within) will calculate the standard deviation of each subgroup (the samples collected on the same day) and compare them with each other. Also, "Within" standard deviation is the standard deviation calculated without including 'special cause' variation in the calculation. This provides a significantly lower value in this case.

Now, let's consider whether there could be a scenario where within standard deviation is greater than the overall standard deviation. Yes, this is possible, and it typically occurs when there is significant variation between the subgroups or categories in the data.
Example: Let's say you are monitoring the daily production of a manufacturing plant over a month, and you collect data on the number of defective products produced each day. You have two production lines (Line A and Line in your plant.

•    Overall Standard Deviation: If you calculate the overall standard deviation for the entire month's data (combining data from both lines), it will give you a measure of the total variability in the production process.
•    Within Standard Deviation: However, if you calculate the within standard deviation separately for Line A and Line B, you might find that Line B has a much higher within standard deviation than Line A. This means that Line B has more variability or inconsistency in its daily production compared to Line A.

In this scenario, the within standard deviation for Line B is greater than the overall standard deviation for the entire plant because Line B's production process is less stable and exhibits more variation within its daily output.

To be honest, I am a little disappointed that there are no winners for this question. This is such a thought provoking question. While the published answers are correct, there is no winner to this question. 

 

Some scenarios where overall standard deviation can be less than within standard deviation

 

1. The rational sub-grouping has not been done correctly

2. Working with the old concept of long term vs short term and hence inducing a difference in the time period for data collection

3. Data does not follow normal distribution