Q 509.  Robert Abelson proposes the MAGIC criteria for making a persuasive statistical argument. Explain the criteria and establish the linkage of DMAIC project components to each of the letters?

 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening 5 PM as per Indian Standard Time. Questions launched on Tuesdays are open till Friday and questions launched on Friday are open till Tuesday. 
• When you respond to this question, your answer will not be visible till it is reviewed. Only non-plagiarised (plagiarism below 5-10%) responses will be approved. If you have doubts about plagiarism, please check your answer with a plagiarism checker tool like https://smallseotools.com/plagiarism-checker/ before submitting. 
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term

Robert Abelson in his book “Statistics as Principled Argument” has put forth the set of guidelines mentioned as the Magic Criteria. There are several properties of knowledge, and its analysis and presentation, that govern its persuasive force.

 

 

 

Magnitude:

 

“How big is that the effect? “

This is related to the DEFINE in the DMAIC methodology

 

The strength of a statistical argument is enhanced in unison with the quantitative magnitude of support for its qualitative claim. There are alternative ways to index magnitude, the foremost popular of which is the so-called” effect size”. within the basic case of the comparison between two means, effect size are often simply given as the difference between the means; often, however, this difference is split by the standard deviation of observations within groups.

 

Articulation:

“How specific is it?”

This is related to the MEASURE in the DMAIC methodology

By articulation, we ask the degree of comprehensible detail in which conclusions are phrased. Suppose, for instance , that the investigator is comparing the mean outcomes of 5 groups: A, B, C, D, E. The conclusion “there exist some systematic differences among these means" features a very minimum of articulation. a press release such as, ”means C, D, and E are each systematically above means A and B, although they're not reliably different from each other" contains more articulation. Still more would attach to a quantitative or near-quantitative specification of a pattern among the means, for instance ,” in moving from A to B to C to D to E, there's a steady increase in the respective means."

 

 

Generality:

 

“How generally does it apply?”

This is related to the ANALYSE in the DMAIC methodology

 

Generality refers to the breadth of applicability in the conclusions. The circumstances related to any given study are usually quite narrow, although investigators typically intend their arguments to apply more broadly. To support broad conclusions, it's necessary to include a wide range of contextual variations in a comprehensive research plan, or to cumulate outcome data from many interrelated but somewhat different studies, as are often done within the context of meta-analysis.

 

Interestingness:

 

Interesting effects are people who "have the potential, through empirical analysis, to vary what people believe about an important issue".

This is related to the IMPROVE in the DMAIC methodology

 

Philosophers, psychologists, et al. have pondered variously what it means for a story to be interesting or to have a point. Our view during this book is that for a statistical story to be theoretically interesting, it must have the potential, through empirical analysis, to vary what people believe about an important issue. This conceptual interpretation of statistical interestingness has several features requiring further explanation, The key ideas are change of belief—which typically entails surprising results—and the importance of the difficulty, which may be a function of the number of theoretical and applied propositions needing modification considering the new results.

 

Credibility:

 

“Credible claims are most compelling than the incredible ones.”

This is related to the CONTROL in the DMAIC methodology

 

Credibility refers to the believability of a search claim. It requires the methodological soundness, and theoretical coherence both. Claims supported sloppy experimental procedures or mistaken statistical analyses will fall victim to criticism by those with an interest in the results. Clues suggested by funny-looking data or wrongly framed procedures provide sceptics with information that something is amiss within the statistical analysis or research methodology.

The MAGIC criteria was put forth by Robert Abelson in his book “Statistics as Principled Argument” & is leveraged for making persuasive statistical argument.

 

The five letters in the MAGIC criteria are as explained below with analogy to DMAIC :-

 

The M in the acronym stands for Magnitude i.e. How big is the effect? - here we can tell how big an effect is through various measures of the effect size. It tells that big effects are impressive, small effects are not. Let’s take the scenario in the Improve phase of two DMAIC projects working on reducing the Vendor Payment Cycle Time of processes spread across two different locations of a company, now the first project has yielded a 40% cycle time reduction while the second one has yielded 10% reduction, thus clearly the magnitude of effect of the first project on the cycle time is much more substantial than the second project.

 

The A in the acronym stands for Articulation i.e. How precise stated it is? - it is measured in form of ticks & buts. A tick is a statement while but is an exception. The more precise the statement is i.e. more ticks the precise the statement is. From a DMAIC parlance this is analogous to the Define phase, where we are stating the problem statement. Here we are leveraging 4W 1H in order to have a precise problem statement framed in terms of What is the problem, Where it has occurred, When it has occurred, Who is impacted by this problem & How much is the magnitude of the problem. All these aspects helps us in creating a precise problem statement which becomes the basis for creating the goal statement so that everyone in the project team is calibrated.

 

The G in the acronym stands for Generality i.e. How widely does it apply? - it states that how broadly the empirical conclusion can be generalized, in other words it refers to how general an effect is?, this can be very general or can be very specific. Usually more general effects are of greater value than more specific ones. It we strike an analogy with DMAIC, this would refer to working out the scope of an improvement project. Too broad of a scope makes the project more impactful, although it adds to the complexity of the project as well & too narrow a scope makes the project less impactful.

 

The I in the acronym stands for Interesting i.e. How interesting it is? - it basically identifies the potential of an empirical finding to change people’s beliefs. This is analogous to the Analyze phase of a DMAIC project where we are leveraging hypothesis to identify which of the potential causes are significantly impacting the effect or the problem being focussed upon. The significant causes are then only considered for further root cause analysis & rest are ignored as they fail to prove that they have a significant impact on the problem.

 

The C in the acronym stands for Credibility i.e. How believable it is? - this means that the research method should be sound & disciplined, in other words, the more hard a result is to believe, the more stringent you have to be about the evidence supporting it. From a DMAIC perspective, this makes sense while planning for hypothesis tests, where it is of paramount importance that you precisely define your hypothesis & also calculate a statistically significant sample size considered both the type-1, type-2 errors & power of the test. This will ensure that the test undertaken will be effective in delivering accurate results.

In his book Statistics as Principled Argument, Robert Abelson outlines a series of standards known as the MAGIC criteria. In this work, he makes the argument that the purpose of statistical analysis should be to provide the MAGIC criteria as a means of making convincing assertions about the world.

 

Magnitude – What size is the impact? Effect size matters more than effect size, in my opinion.

Articulation – How detailed is it?Statements with greater precision have greater persuasive power.

 

Generality – How broadly is it applicable? Effects that are more widespread than others are more persuasive. Claims that appeal to a wider audience are more convincing.

 

Interestingness – Effects that "have the potential, through empirical investigation, to affect what people feel about an important subject" are particularly interesting, according to research. Effects that are more captivating than others are those that are more fascinating. Furthermore, unexpected results are more interesting than those that only corroborate what is already known.

 

Credibility – Claims that are credible rather than unbelievable are more persuasive. The credibility of the assertions made must be established by the researcher. Results that conflict with earlier findings are less reliable.

 

The linkage between MAGIC Criteria and DMAIC

 

Define- Declaring the business problem, aim, prospective resources, project scope, and high-level project timetable explicitly is the goal of this step. This can be related with “Magnitude” of MAGIC. This means how big is the effect? So it can be considered as declaring the business problem, aim, prospective resources

Measure- This step's objective is to evaluate how well the problem or goal has been specified. This is a data collecting stage whose goal is to create baselines for process performance. This can be related with “Articulation” of MAGIC criteria  How precisely stated is it?. The precision is one of the important criteria of measurement.

Analyze- This step's objectives are to find, validate, and choose the root cause for elimination. This can be related with “Interestingness” of MAGIC criteria . Effects that "have the potential, through empirical investigation, to affect what people feel about an important subject" are particularly interesting, according to research.

Improve- Finding, testing, and putting into practise a partial or comprehensive solution to the issue are the goals of this step. Depending on the circumstances. Determine innovative ways to reduce the main causes in order to solve and avoid process issues. This can be related with “Generality” of MAGIC criteria. Effects that are more common than others are more credible. Claims that appeal to a wider audience are more convincing.

Control- This step's goal is to integrate the changes and assure sustainability; this is also known as "making the change stick." This can be related with “Credibility” of MAGIC criteria Claims that are credible rather than unbelievable are more effective. The researcher must demonstrate the veracity of the claims made. So more credible means more acceptable so the solution will be more sustainable.

MAGIC Criteria

Book from Robert Abelson Book try to re-define the misunderstood field of statistics. P-values, z-scores, and t-tests aren’t mechanistic tools for quantifying the world.  

MAGIC Stand for magnitude, articulation, generality, interestingness, and credibility.

Magnitude – How big is the effect?.  We can tell size of an effect through various measure of effect. Bigger the effect better it is The difference between two outcomes in an experiment might seem large on its own (“effect size”) or because the intervention seemed small (“cause size”).

DMAIC Component – Hypothesis test the sample set which identify the effect of cause,  Cause and effect analysis

Articulation – How readily can the details be summarized in to specific principle. Which is details of precision or details.  In book he refer it as tick and buts. Statement- Tick , Exceptions- But. More the statement is better and less exception is better.

DMAIC Component : Definition of problem statement is very important to DMAIC project, Same level important is on define the Null and Alternate Hypothesis to success of the project.

Generality – How generally and widely we can apply the conclusion. Will it cover lot of cases or only few? This is specific to certain subset of population.

“Take multiple approaches to answering the same question and apply the same approach in different contexts.

DMAIC Component DOE, ANNOVA to try different possibility. Also scalability of solution. Even FMEA can be compared to generality to see the risk measure across population

Interestingness – How important is the issue addressed. How surprising is the conclusion and how much change in behavior it does? Interesting is very hard to measure precisely, but one way is to say how different the reported effect size is from what we thought it would be

DMAIC Component : Correlation test, Regression test, all this show the interestingness characteristics

“Credibility-  Given the method followed to gather and analyze the data. How trustable is the result based on the data. How much its contradicts with other understanding.

DMAIC Component : Gage R& R test will match the credibility test of the measure criteria

 

MAGIC criteria for making a persuasive statistical argument

The MAGIC criteria are the set of guidelines proposed by Robert Abelson in his book Statistics as Principled Argument. He presented MAGIC criteria for that the goals of Statistical analysis should be to make compelling claims about the world. It is an easy read with few formulas but with lots of wisdom.

MAGIC criteria

Magnitude- How big is the effect?

We can tell that how big an effect is through the various measures of effect size. We will get into some of these in the later diaries but some of the common ones are correlation coefficients. The difference between two means the big effects are impressive while the small effects are not. How big is big depends on the context and on what already known. If we find, for example that a new diet plan lets people lose (on average) 10 pounds in a month, that’s pretty big. 10 ounces in a month is pretty small but if it was a diet tested on rats, 10 ounces might be a lot.

Articulation- How precisely stated is it?

The Articulation is measured in what Abelson calls Ticks and Buts. A ‘tick’ is a statement and a ‘but’ is an exception in articulation. The more ticks the better, the fewer buts the better here in articulation. There are also the blobs which are masses of undifferentiated results. Blobs are as it might have guessed, bad.

Generality- How widely does it apply?

Generality refers to how general an effect is. Does it apply to all humans everywhere? That would be very general or does it apply only to people who have posted more diaries daily? That would be pretty specific. Usually, more general effects are of greater value than the more specific ones but should be sure that the study states how general it is.

Interestingness- How interesting is it?

Interestingness is very hard to measure precisely but one way is to say that how different the reported effect size is from what we thought it would be. For example, I once read a study that showed that black people on average earn less that white people. Upsetting but not interesting. Knew that already and the size of the difference was large (which is thought it would be) but not huge (which also knew because after all, even the average white person doesn’t earn all that much) but then it went on to say that while black men earned a lot less than white men (more than it was thought that the difference would be), black women and white women earned almost the same (that’s really interesting! it would have thought that black women earned much less than whites!)

Credibility- How believable is it?

The harder a result is to believe; the more stringent have to be about the evidence supporting it.

“The central theme is that good statistics involves principled argument that conveys an interesting and credible point.” said Sir Robert Abelson, while describing the intent of writing the book “Statistics as Principled Argument”.  Unfortunately, the statistical courses do not focus extensively or directionally influence user around the argumentative nature of claims they make using various statistical theories. This in turn leads to misleading, misinterpreted and misguided theories largely impacting the possible narrative.

Data in its various form is characterised with several properties and can be presented differently that widely changes the persuasive forces around acceptability, impact and usage. This is guided by Robert as MAGIC.

Intent of defining MAGIC was to allow the Statistical Analysis be used to make compelling or specific outcome and claims.

·         Magnitude – provide specific details around how BIG or SMALL the effect of analysis is. Larger the effect – more compelling it is

·         Articulation – Analysis outcome should be precise and restricted interpretation.

·         Generality – General application has faster acceptance and are more compelling.

·         Interestingness – Outcome should ideally result in change in the belief system in reference to the topic. More Interesting and Surprising effect – faster is the acceptance and larger compelling

·         Credibility – Refers to believability. There should be a method displayed and should result in a logical theory base.  Any conclusion which is contradictory to already existing belief will have slow acceptance and less compelling.

 

Since DMAIC Methodology is a data driven quality strategy used to improve processes, MAGIC Criteria has an influencing effect.

1)      Define – Project Charter VOC, VOP, VOE, VOB and Need for Project has compelling results on the Problem definition and its precise articulation, SMART GOAL and Expected Outcome be clearly defined along with project team, project plan. Historical Data around Metric significantly called out HOW BIG and SINCE WHEN the Problem is. Defining the scope is characterised by Generality principal to know coverage and restrictions around applicability. Communication plan and ARMI Chart articulates responsibility framework and ensures credibility, allowing Stakeholders to approve and make investment and at the same time team to put efforts in making it successful.

 

2)      Measure and Analyze – Process Map, Baselining, Fishbone or VMS approach, 5 WHY analysis tool and Process / Data Door approach makes the analysis and status quo meaningful, inclusive – since the entire team is included and believing in the change becomes easy. Articulation, Interestingness and Credibility are the narrative that best fits the Stage. Potential X’s along with occurrence data (Pareto Analysis) helps in providing meaningful insights - magnitude to the major contributors.

 

3)      Improve and Control – Optimizing Best Solution, FMEA, PHUG, Decision making matrix are some examples that explicitly calls out enhanced believability and interesting way to prove To be Status. Mistake Proofing, Standardization and Automation and phasing out each counter measure to documented SOP relates to compelled Articulation, tracking the improvement and showcasing the growth to Stakeholders is an excellent way to win trust and acceptance of concerned teams - Credibility.

 

While MAGIC Criteria has proved out to have its relevance and applicability across various streams including psychology, ecology, sociology, but due to its elementary composition, MAGIC and DMAIC goes a long way to Optime and Sustain Improvement. DMAIC follows the exact philosophy to highlight the Magnitude of the issue, point out variance and reasons - Articulation, method to identify solution and categorically improve and sustain performance.

 

“WHAT, BY WHEN, BY WHOM, HOW MUCH –is all inclusive and very well resonates with MAGIC”

 

Robert Abelson used the MAGIC criteria to theorize that the goals of the statistical anaylsis should be factual and present compelling accurate and precise claims with wider applicability and credibility.

 

The MAGIC acronyms stands for – Magnitude, Articulation, Generality, Interestingness and Credibility. Below is the detailed explanation of each along with the connectivity with DMAIC project component.

Magnitude:

It provides evidence on how big is the effect. In Supply chain and Logistics, the faster delivery of goods to the customer had negative impact of CO2 emissions. How much faster delivery? Or how much negative impact? does it vary from items types delivered? How does it compare with delivering through other mediums?

This is a classic input to the project charter to provide the impact and magnitude of the problem to be solved which is covered in Define stage but will also come across in Analyse phase once data is reviewed.

 

Articulation:

How specific is the problem in hand? Precise statements – 1 Day Delivery increases the CO2 emission by 20% reducing the packet induction by 10%.  More than 3 days delivery has no impact against the current benchmark set to 100% fill and induction.  If we consider the 3 days benchmark, will 2 day delivery be better than 1 day and by what extent. Should be also conclude that most deliveries should follow 3 days window as default.

This will be a part of MSA as well as setting limits on the data to be gathered and will be a part of Define and Measure phase.

 

Generality:

Can we generalise this conclusion? Does the Supply chain and emission be generalised for all categories and the regions.  What use cases are covered to gain maximum interest in benchmarking across a wide set of products?

This a is part of Analyse & Improve phase and can be also used to see where additional data points could be used.

 

Interestingness:

How important is the issue? How surprising is the conclusion? Interesting effects have potential through analysis. More compelling and surprising effects are important. The supply chain delivery impact might have the most impact in product mix which may remove the worry that might impact the delivery and no impact on CO2 emissions.

In DMAIC, This will be a part of the Analyse phase where surprising revelations help us to understand and discern standard directions to unique directions.

 

Credibility:

How much are the results credible from the results point of view?  How much constitute to the true results expected? And how much contradicted are our well-established understandings?

This is a part of Improve & control phase which provides an opportunity to improve our understanding and build iterative methodologies to dive Deep further to arrive at the best possible results expected.

 

How to use this Methodology:

Asking questions at every steps using the MAGIC acronym will help to remain true to the data and the provide a clear case of our understanding with might strengthen our case or will assist us to kill projects at the right time. This methodology is a good tool to look at the criteria as a group and build validate both data and outputs against the expected outcomes anticipated.

 

 

Many respondents have tried to establish a one to one linkage between DMAIC phases and MAGIC criteria. But it is many to many. Ashish Kumar Sharma's answer has established this many to many linkage and hence his answer has been selected as the winning answer!