Q 608. You are performing Attribute Agreement Analysis and the result indicates 'Caution'. What does caution mean? As a researcher what will be your next steps and why?

 

Note for website visitors -

• This platform hosts two weekly questions, one on Tuesday and the other on Friday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be approved. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error prone as our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://crossplag.com/ai-content-detector/. Only answers with less than 15-20% AI-generated content will be approved.

If an "Caution" results comes out as a inference in attribute agreement analysis, this is a signal to investigate the source of the incompatibility and make the necessary corrections. These improvements may include

1. Providing clearer instructions to raters

2. Additional training, or improving the rating process to achieve better agreement and reliability in rating characteristics.

 

 The ultimate goal is  a reliable measurement system to ensure consistent and accurate property evaluation, which is critical for quality control and decision-making processes.

 

• Kappa value = the degree of agreement made by multiple appraisers when assessing the same samples/parts

• How much better is your assessment compared to guessing – If you flipped a coin and you guessed heads or tails, you would be right about 50% of the time by chance

• Kappa >= 0.9  means that you can accept the result or there is good agreement

• Kappa  value between 0.7 and 0.9   means that you can accept the result but with a caution

• Kappa <=0.7 is not at all acceptable

Attribute Agreement Analysis (AAA) is a statistical method used to assess the level of agreement among different individuals. It is used in cases wherein measurement value has finite number of categories (discrete data). 
 

Parameter used to evaluate level of agreement
 

Fleiss Kappa 

    - Used for nominal data
    - Higher the value of Kappa, stronger the association
    - Absolute agreement between ratings
    - Ranges from 0 to 1 (although negative values is also possible)
 

Kendall's coefficient

    - Used for ordinal data
    - Higher the value, stronger the association
    - Association between ratings
    - Ranges from -1 to +1
 

Cohen's Kappa 

    - Similar to Fleiss Kappa except that there are only two individuals / evaluators
 

Meaning of caution
 

A caution results when Kappa value is greater than 0.7 but less than 0.9. It indicates a moderate level of inconsistency / disagreement among individuals in evaluation. It suggests there can be some issues with reliability of evaluation process. 
 

Steps to improve consistency
 

1. Source of disagreement
    Need to understand why there is a moderate level of inconsistency. If all evaluators have moderate scores, there might be a need to change the measurement system.
 

2. Evaluate training
    Review the training and instructions provided to evaluators. If only few evaluators have moderate scores, they can be trained. In case of poor between evaluators score, all evaluators can be trained. 
 

3. Addition of more evaluators
    Try adding more evaluators to improve the reliability. Rerun AAA to check if the change brings in improvement.

A lot of answers were dis-approved either due to high % of AI generated content or high plagiarism.

 

Among the published answers,  Keerthi Vasan has given the best answer.