Leaderboard

Attribute Agreement Analysis (AAA) is a statistical method used to assess the level of agreement among different individuals. It is used in cases wherein measurement value has finite number of categories (discrete data). Parameter used to evaluate level of agreement Fleiss Kappa - Used for nominal data - Higher the value of Kappa, stronger the association - Absolute agreement between ratings - Ranges from 0 to 1 (although negative values is also possible) Kendall's coefficient - Used for ordinal data - Higher the value, stronger the association - Association between ratings - Ranges from -1 to +1 Cohen's Kappa - Similar to Fleiss Kappa except that there are only two individuals / evaluators Meaning of caution A caution results when Kappa value is greater than 0.7 but less than 0.9. It indicates a moderate level of inconsistency / disagreement among individuals in evaluation. It suggests there can be some issues with reliability of evaluation process. Steps to improve consistency 1. Source of disagreement Need to understand why there is a moderate level of inconsistency. If all evaluators have moderate scores, there might be a need to change the measurement system. 2. Evaluate training Review the training and instructions provided to evaluators. If only few evaluators have moderate scores, they can be trained. In case of poor between evaluators score, all evaluators can be trained. 3. Addition of more evaluators Try adding more evaluators to improve the reliability. Rerun AAA to check if the change brings in improvement.