Q 582. One prefers the data to follow normal distribution, and if it does not, then we transform it to normal. Usually Box Cox transformation will work. However, sometimes we prefer to use Johnson Transformation. What are the conditions where Johnson transformation will be preferred? What kind of statistical analysis can and cannot be performed on this transformed data?

 

Note for website visitors -

• This platform hosts two weekly questions, one on Tuesday and the other on Friday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday evening at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be approved. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error prone as our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://crossplag.com/ai-content-detector/. Only answers with less than 15-20% AI-generated content will be approved.

Wherever the Box Cox transformation may not be adequately sufficient, the below reasons lead to usage of Johnson Transformation.

Situation	Johnson Transformation
When the data exhibits both skewness (asymmetry) and kurtosis (peakedness)	Allows for more flexible adjustments to shape of the distribution and can handle a wider range of distributional shapes
Non normal Distributions	Can handle asymentric and multi modal distributions
Extreme values or outliers	Robust against outliers
When the variance of the residuals is unequal over a range of measured values	Can achieve more homogeneous variances

 

List of statistical analyses that may have limitations within Johnson Transformation:

1. Since the transformed values may not have direct interpretations in the original scale, obtaining exact p value may not be possible or can be tricky.

2. Constructing accurate confidence intervals for some parameters may be difficult due to the same reason as above.

3. Mann-Whitney U test or Kruskal-Wallis test do not assume normality. Since transformation may change the shape, it may violate the assumptions, thereby leading to incorrect validity of the tests.

4. While analyzing contingency tables, we assume categorical data under Chi-square test of independence. Johnson transformation may lead to compromising the assumptions of the tests.

5. Kaplan-Meier estimates or Cox proportional hazards models are commonly used for time-to-event data. Since these tests assume proportional hazards or data, it cannot be relied post transformation.

Whenever we need to transform non normal data to normal, we use Box Cox transformation or Johnson transformation.

 

Johnson transformation are generally used

·      When the non-normal data contains all data including negative data or zero

·      Box cox transformation results are not so productive

·      Data should be continuous

·      When dependent variables are right or left skewed

 

Johnson transformation generally can be used for predictive or prescriptive analysis since in practical scenarios, mostly sample data captured will non normal and is skewed (right or left). Statisticians transform these sample data to normal for further analysis to predict solutions about the entire population. Also, this transformation create new variables from existing ones to prescribe possible actions. Johnson transformation cannot be performed for descriptive analysis since this normality test is performed to normalize the existing data to prescribe or predict for possible solutions

Johnson Transformation

 

Johnson Transformation is a mathematical formula which is generally used to transform our existing data when required. Like, if we want to transform our existing data which is non normal then with the help of this tool or method, we can transform our data into normal distribution.

In this technique we can create new variables with the help of our existing variables.

This transformation technique starts with power transformation. Like, with the help of some mathematical formulas (logarithm, square root, or reciprocal). By applying this method data become more symmetric.

Then, after this step we perform second transformation which is called “normalizing transformation”.

There are other Johnson Transformation also available. Like

·       Johnson SB (bounded)

·       SL (unbounded),

·       SU (unbounded),

·       and SN (normal) distributions

 

In below conditions we can use Johnson transformation

·       When data is non normal

·       When data is skewed

·       When data has Heteroscedasticity situation

·       When data has outliers

 

It is generally suggesting that to use of this method to transform the data when there is a need of data to be transformed in to normal.

Statistical techniques which generally used:

·       Parametric tests

·       Confidence intervals

·       Hypothesis testing

·       Regression analysis

Statistical techniques which generally should not used:

·       Nonparametric tests

·       Multicollinearity assessment

 

Johnson Transformation is used in Hypothesis testing/ analysis in the following scenario:

 

When you need to run a statistical hypothesis test to determine if the means of the two processes are the same & if the data are not normally distributed , whereas you need a normally distributed data to perform the test.

 

For the calculation of CpK – transforming non- normal data into a normal distribution requires either Box-cox or the Johnson Transformation. This transformation is performed to apply certain statistical tools.so after transformation, for interpreting and making decisions ,interpreting  the original data( by back tracking )  and presenting to audience is better than interpreting the transformed data itself.

 

Conditions where Johnson’s Transformation will be preferred are:

1> when data contains any value including negative values and zero.( Boxcox cannot be used here).

2> when Box Cox algorithm does not define appropriateness towards transformation

3> when the transformed data need to preserve many of the features of original skewed distribution such as range & mode.

4>when non - parametric tests comparing median for a non -normal distribution may not be useful for statistical significance, it needs to use parametric tests with mean comparison for which data has to be normal. And to achieve this normality , there is a necessity to transform data using Johnson or Box cox .

 

In a Hypothesis testing , p value that we enter defines significance level of a normality test before & after transformation. usually Anderson – darling test is performed for normality check. A higher value of p makes the criteria for normality more rigorous. A lower value makes criteria for normality less.

 

Limitations of Johnson Transformation:

It would be confusing to interpret when we use the transformed data into a control chart. Since the original data is taken to the power of Lamda while doing the transformation , it cannot be on the same scale or units as the original data. So most of the cases Boxcox plot itself would be adequate to perform than a Johnson transformation. Because johnson more powerfully transforms the original data.

In some other cases , the usage of non – parametric statistical tools such as median can be more effective than  going for transformation. Example Turn around time of trucks bringing parts from suppliers to OEMs for vehicle assembly is non- normal.

In this case when you try to transform the data and plot on a control chart to monitor the variability , it has no meaning because already the data is transformed. Hence transforming data shall be used as a last option , if the statistical analysis you want to perform has normality as an underlying assumption .It depends on the statistical significance of the test to perform. One such case of calcium content across various oceans is non- normal distribution which needs Johnson transformation as some of the values can be zero as well.

 

Using Minitab for Johnson Transformation and interpreting the normality before vs after based on p value :

 

 

Johnson transformation is one of the type of mathematical transformation technique, which is used to transform non-normal data to follow a normal distribution through Johnson distribution system.

When the Box cox transformation is not adequate to use, and the data contains any values including negative as well, in such scenarios Johnson Transformation is preferred.

 

Johnson Transformation technique can be useful in the conditions like :

• When the original data set is following non normal distributions
• When the data set having negative values, also extreme outliers
• When the data has skewness or asymmetry distribution and kurtosis
• When it is required to preserve most of the features of original distribution
• When it is required to manage a wide range of shapes of distribution in data

Statistical analysis like normal probability test can be performed on Johnson transformed data along with Anderson Darling Test statistic, which can be compared with the original data and it’s probability to visualize the improvement or to understand whether the transformation is effective. Also Histogram plot can be done with the Johnson transformed data and it can be plotted to I-MR chart to check if the process is in control.

Definition	Johnson transformation is used to transform non-normal data to an approximate normal distribution. It helps on finding a suitable transformation which makes the data approximately normally distributed.
Conditions where Johnson transformation will be preferred	Non-symmetric distributions: The Johnson transformation is used when the data is not only skewed but is also exhibiting asymmetry beyond what the Box-Cox transformation can handle. It handles both positive and negative skewness, also complex distributions. Heteroscedasticity: If significant heteroscedasticity is present in the data, Johnson transformation could be preferred. The Johnson transformation can help stabilize the variance of the data. Specific distributional assumptions: In Johnson transformation we can choose from a range of parametric forms, like Johnson SB, SU, SL, and SN distributions, which corresponds to different shapes of transformed data. In presence of specific distributional assumptions or theoretical considerations which guides the analysis, Johnson transformation provides flexibility in fitting different distributions to the data.
What kind of statistical analysis can and cannot be performed on this transformed data	Johnson-transformed data can be used for parametric tests that assume normality. Examples:  t-tests, ANOVA, linear regression, and other parametric models. The Johnson transformation does not restrict the types of statistical analyses which can be performed on the transformed data. Some Examples as to Why Statistical Analysis cannot be performed with Johnson transformed data : Model assumptions: Certain statistical models have specific assumptions, Johnson transformed data might not meet those assumptions even after transformation. Example, if a statistical model assumes homoscedasticity (equal variances), but Johnson transformation does not sufficiently address heteroscedasticity in the data, the transformed data may violate the model assumption. Interpretation challenges: Interpretation of results obtained from Johnson transformed data may be complex due to the additional parameters in the transformation. Robustness of statistical tests: Some statistical tests may not be as robust or well-validated when applied to Johnson-transformed data. Johnson transformation is designed to approximate a normal distribution, usage of specific parametric form can impact the validity of certain statistical tests or underlying assumptions.

Johnson Transformation is preferred over Box Cox Transformation when the data has high or low skewness and kurtosis.

High skewness means distribution curve has longer tail on one end and shorter tail on the other. Kurtosis is used to described shape of probability distribution. Higher kurtosis refers to t-distribution with sharper peak and heavier tails compared to normal distribution. Lower kurtosis (platykurtic) refers to flatter peak and thinner tails.

Johnson Transformation can handle a wider range of distributions than Box Cox Transformation. Johnson transformation can be considered as an alternative when Box Cox transformation does not achieve the desired result. Box Cox transformation is limited to transforming data with positive values to fit a specified distribution shape such as normal, log-normal or gamma distribution.

 

Once the data has been transformed using Johnson Transformation, most statistical analyses can be performed on the transformed data as if it was normally distributed. However, the interpretation of the results need to be confirmed and validated in the context of original data.  

 

Conditions where Johnson transformation is preferred over Box Cox transformation: 

1. If the data exhibit non-normality, including high skewness and kurtosis and the Box Cox transformation does not sufficiently normalize the data.
2. It can handle situations where the data does not conform to common distributions assumed by Box Cox transformation (ones with positive skewness such as log-normal, gamma, exponential and Weibull distribution). Johnson transformation can accommodate broader range of distribution shapes with high or low kurtosis or skewness.
3. If the data exhibits multi-modality, has got multiple peaks and clusters
4. Johnson distribution is more suitable as it can handle both positive and negative values. Unlike Box Cox transformation where it assumes positive data values in order to apply logarithmic and power transformations.

Once the data has been transformed using Johnson transformation, various statistical analyses can be performed just like with normally distributed data.

1. Descriptive statistics such as mean, standard deviation, median and quartiles 
2. Parametric hypothesis tests such as t-tests or analysis of variance (ANOVA)
3. Process capability analysis to assess whether a process meets specification limits, such as Cp and Cpk, which assumes normality in process data
4. Regression models to examine the relationship between dependent variables and one or more independent variables.
5. Confidence intervals for population parameters (e.g. mean, proportion) based on transformed data

However, it is important to note that the interpretation of the results on transformed data may differ from the original scale. To ensure meaningful conclusions it is generally necessary to transform data back to original scale.

1. First, understand the purpose and rationale behind data transformation and consider how the transformation affects the distribution, scale and interpretation of data.
2. Conduct necessary statistical analysis, modeling or computation using the transformed data.
3. Examine the outcomes and draw conclusions based on transformed data.
4. Transform results back to original scale

1. Determine the type of Johnson transformation (SU, SB, SL, Sb) that was applied to the data
2. Obtain the transformation parameter lambda, gamma, xi, delta value
3. Apply inverse transformation formula involving parameters and transformed data points

5. Confirm the validity and relevance of the findings in the context of the original data set.

There is no correct answer provided for this question among the answers below. 

This question has two parts, and most respondents address the first reasonably. For the second part (what kind of statistical analysis can/ cannot be performed with transformed data), the correct response is as follows -

• Process Stability and Capability Analysis can be carried out using transformed data. The control limits are also based on transformed data when using control charts. When using capability analysis, the specification limits are transformed too. 
• Hypothesis testing is not done with transformed data but only with the original data. There are enough hypothesis tests (non-parametric tests) designed for non-normal data.
  • Imagine comparing averages of two data sets transformed through Johnson's transformation. This would be meaningless because the two data sets would have been transformed with different equations (Johnson's transformation generates an equation to transform the data).