Q 486. In addition to Fleiss Kappa, Krippendorff's alpha is another metric used in Attribute Agreement Analysis. Explain the difference between the two. When will you prefer to use krippendorff's alpha?

Click here to see the best answer

Fleiss’ Kappa is another way to measure match of opinions between three or more raters. It is advisable to have Likert scale data or other closed-ended, ordinal scale or nominal scale (categorical) data. Similar to most correlation coefficients, Fleiss Kappa ranges from 0 to 1, where:

0 is no agreement/mismatch (or agreement that you would expect to find by chance),

1 is perfect agreement/perfect match

It is possible to have values < 1, meaning the values are less than expected by chance. For practical purposes, these values can be counted as 0, or no agreement. In general, a coefficient over .75 (75%) is considered as a “good” match, although what exactly is an “acceptable” level of agreement depends largely on particular field. In other words, check with experts or SME before concluding that a Fleiss’ kappa over .75 is acceptable.

In few cases, Fleiss’ Kappa can also return low values even when agreement is actually high. This is why it is less popular.

 

Difference between Fleiss Kappa & Krippendorff's alpha and reason to prefer to use krippendorff's alpha

 

Krippendorff’s alpha (also called Krippendorff’s Coefficient) is an alternative to Fleiss’ Kappa for determining inter-rater reliability.

Krippendorff’s alpha has below characteristics:

It ignores missing data entirely.

It has the ability to handle various size of samples/categories/numbers of raters.

It applies to any measurement level (i.e. (nominal, ordinal, interval, ratio).

It is commonly used in content analysis to quantify the extent of agreement between raters, & it differs from most other measures of inter-rater reliability because it calculates disagreement/mismatch (as opposed to agreement/match). This is one of the main reason why the statistic is more reliable, but some researchers report that in practice, the results from both alpha and kappa are similar (as explained by Dooley).

Computation of Krippendorff’s Alpha

The basic formula for calculating alpha is a ratio of observed disagreement & expected disagreement. The ratio is very simple, because the method is actually computationally complex, involving resampling methods like the bootstrap. This is a major disadvantage (explained by Osborne). We can get an idea of the computations involved from the following formula. These values range from 0 to 1, where 0 is perfect disagreement/mismatch and 1 is perfect agreement/match. 

 

 

The basic premise of conducting Attribute Agreement Analysis is to assess whether there is consistency amongst the appraisers in terms of assessing an attribute which is non-measurable in nature (i.e. Nominal, Ordinal, Binary etc) in terms of three aspects:-

 

• Agreement of appraisers within themselves i.e. Repeatability
• Agreement of appraisers between themselves i.e. Reproducibility
• Agreement of appraisers with the standard i.e. Accuracy

 

There are two popular measures of appraiser consistency / reliability i.e. Fleiss Kappa & Krippendorff’s Alpha values. However both are equally consistent measures when it comes to assessing the reliability of your measurement system, there are slight differences in these two measures. These differences are as mentioned below:-

 

• Fleiss Kappa is based on the concept of the ratio calculated between observed agreement(Pa) & agreement expected by chance(Pe) whereas Krippendorff's Alpha is based on the concept of ratio calculated between observed disagreement(Pa) & disagreement expected by chance(Pe). Mathematically both are calculated by the below formula:-

 

 

• Ranges for both the measures is from -1 to 1 with 1 indicating perfect agreement, 0 indicating no agreement & -1 denoting inverse agreement in case of Fleiss Kappa with an acceptable threshold value of 0.75 resembling significant agreement. However in the case of Krippendorff's Alpha an alpha value of 1 denoting perfect disagreement, 0 being no disagreement with an acceptable threshold value of 0.80 for significant disagreement.

 

• Fleiss Kappa is most suitable in case of nominal data while Krippendorff's Alpha has high flexibility as it can work with nominal, ordinal as well as metric data.

 

• In case of missing data Krippendorff's Alpha is the preferred option rather than Fleiss Kappa which cannot handle missing values & these missing values must be excluded from the data. Krippendorff's Alpha is said to be much more robust even if we have 50% of the values missing in our data & provides unbiased results.  

 

Based on the above facts it would be preferable to use Krippendorff's Alpha as the preferred statistic for measuring inter-appraiser reliability in situations where we have data other than nominal data, have multiple appraisers choosen randomly & the attribute agreement data is having missing values.

 

Fleiss Kappa & Krippendorff's Alpha are used to determine the level of agreement among multiple appraisers. However, Fleiss Kappa determines level of agreement & Krippendorff's Alpha determines the level of disagreement.

 

Fleiss Kappa can be used only for nominal data whereas Krippendorff's Alpha can be used for continuous data too. Secondly, Krippendorff's Alpha can be used in case of missing data.

Kappa is defined as the ratio of proportion of times that the appraisers agree to max proportion of times that the appraisers could agree.

 

Kappa ranges from -1 to 1

The larger the kappa, the more agreement in that category

For instance, Kappa value of 1 represents Absolute agreement

 

Below table represents commonly accepted values for reliability measures:

Cohen’s kappa Value Interpretation:

0.91 - 1.00 - Almost perfect

0.80 - 0.90 - Strong

0.60 - 0.79 - Moderate

0.40 - 0.59 - Weak

0.21 - 0.39 - Minimal

0.00 - 0.20 - None

 

Krippendorff’s alpha Value Interpretation:

0.80 - 1.00 - Reliable value

0.67 - 0.79 - Acceptable for tentative conclusions

0.00 - 0.66 - Not acceptable

 

Take Away:

With caution, Stat practitioners should primarily examine the marginal distribution and not uncritically interpret the kappa value whether it is high or low.  As prevalence, odds, raters independence, and the impact on diagnosis and other additional factors can have significant influence on the kappa statistics.

 

Kappa statistics represents the degree of absolute agreement amongst ratings and popular statistics includes that of,

Cohen’s kappa – Measures assessment agreement between two raters

Fleiss’s kappa – Generalization of Cohen’s kappa (>2 raters)

 

In most of the statistical tools, such as Minitab, by default Fleiss’s kappa is calculated for AAA (Attribute Agreement Analysis)

 

As we could note here, Fleiss’s kappa is based on the theory that the observed agreement is corrected for the agreement expected by chance.

However, on the contrary, Krippendorff’s alpha is based on the observed disagreement corrected for disagreement expected by chance.

 

Key Differences:

Fleiss’s kappa:

Cannot handle missing values

Expected agreement sample size is infinite

Best suited for Nominal data

 

Krippendorff’s alpha:

Can handle missing values

Actual sample size is considered

Can handle all data types

 

Both Fleiss’s kappa and Krippendorff’s alpha can be likewise recommended in the circumstance when the data is nominal and when there are no missing values.  

However, Krippendorff’s alpha statistics is preferred in below situations, viz.,

• Whenever the data is missing
• Higher than nominal order (ordinal, interval, ratio)
• When there is bias in the distribution of disagreements (even strong bias will not have any distorting effect)
• When different participants have different number of raters (usually when the number of raters is more than 2 and can be applied to any scale level)
• When there is incompatibility in obtaining observation ratios by pair counting in the small samples

 

Summary Table:

 

Final Take Away:

Before deep diving into the reliability data, it is recommended that based on the context, practitioners should select the index of Inter Coder Reliability based on data properties and assumptions, including the level of measurement of each variable to calculate the agreement and the number of coders.

 

Most of the times, it is difficult and complex to compute Krippendorff’s alpha statistics compared to Fleiss’s kappa, however Krippendorff’s alpha provides higher reliability, particularly when there are no perfect conditions for research.  

1. Conceptually the difference between the two metrics is as below:

Fleiss’ Kappa is based on the concept that the observed agreement is corrected for the agreement expected by chance. Whereas, Krippendorff’s alpha is based on the observed disagreement corrected for disagreement expected by chance.

 

2. Difference between the two metrics in observations is as below:

While for both, the point estimates of Fleiss’ K and Krippendorff’s alpha do not differ from each other in all scenarios. The difference lies in the cases of missing data (completely at random), where Krippendorff’s alpha provides stable estimates, while the Fleiss’ Kappa could potentially lead to biased estimates.

 

Hence Krippendorff’s alpha is preferred if the measurement scale is not nominal and/or missing values are present (completely at random). To make it more general, for those who are interested in a one-fits-all approach, Krippendorff’s alpha could be used as the measure of choice.

This was a slightly difficult question to answer and it is heartening to see correct answers. The best answer (for the content and the way it is structured) is from Rahul Arora.

 

Answer from Mohamed Asif is also a must read!