Jump to content

Kappa Value

 

Kappa Value is a statistic used to determine the goodness of the measurement system in Attribute Agreement Analysis. It is the proportion of times the appraisers agreed to the maximum proportion of the times they could agree (both corrected for chance agreement). It is used when the appraisers evaluate the same samples and give nominal or ordinal ratings. It ranges from -1 to 1. Higher the Kappa value, higher the agreement.
If Kappa = 1, perfect agreement exists
If Kappa = 0, the agreement is the same as would be expected by chance
If Kappa < 0, the agreement is weaker than expected by chance (Kappa rarely goes negative)

 

Kendall's Coefficient

 

Kendall's Coefficient of Concordance is a statistic used to determine the goodness of the measurement system in Attribute Agreement Analysis. It indicates the degree of association of ordinal assessments made by multiple appraisers when assessing the same samples. It is used instead of Kappa value when the rating scale is ordinal with more than 3 levels of ratings. Kendall's coefficient accounts for the order of the ratings (Kappa value does not account for it). It ranges from 0 to 1. Higher the coefficient, higher the agreement

 

An application-oriented question on the topic along with responses can be seen below. The best answer was provided by     
Amlan Dutt on 07th June  2019.

 

Applause for all the respondents- Amlan Dutt, Sreyash Sangam. 

Question

Q. 166  Kappa value is used to make inference about soundness of a Measurement System when Attribute Data is used. Kendall's coefficient is used to infer the soundness of Measurement System when ordinal Attribute Data is used. If one used Kappa value to make inferences for an Ordinal Attribute Data set, what errors are likely to happen?
 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

 

Share this post


Link to post
Share on other sites

3 answers to this question

Recommended Posts

  • 0

Consider two unbiased coins tossed 100 times simultaneouly and results put through Fleiss’ Kappa and Cohen’s Kappa (Kappa values range from –1 to +1) spitting out the results below; Since these are statistically independent events, any match will be purely due to chance alone.

p-values >0.6 tells (at 5% Type 1 Error Risk) fail to reject the null hypothesis that the agreement is due to chance alone. Notice Agreement is nearing 50%; thanks to Bernoulli random number option in Minitab.

So far so good.

image.png.c6e963a63d04255d68c67bf778c58edc.png

Let’s test it on liar’s dice!

Consider two dice (…from a Martian liar) which can show values from 0 to 7 tossed 100 times simultaneously.

It’s basically two random Poisson variable of 100 observations with mean=2 (mean=variance is a property but that’s another story)

As narrated before any match will be purely due to chance alone (…being random).

Presto! What went wrong?

image.png.b87269aa3b6cc418c165e9ef9ecf7166.png

The curious readers will notice for Responses "2" and "5" the p-values are below 0.06 (i.e. only 6% risk by chance alone). Which means we can reject the null hypothesis that the agreement is due to chance alone. i.e. numbers on 2 dice match NOT by chance for at least "2" and "5".

(The negative kappa values tell very poor inter-rater reliability)

While the fact is data has been randomly generated. Hence, any matched would be purely by chance alone.

These opposing statements indicate a null hypothesis accepted even though it’s wrong. i.e. alternate hypothesis is right. In other words, we have committed a Type 2 Error.

Why? Because of using Kappa instead of Kendall, Kendall give a Coefficient (values range from 0 to +1) but highly insignificant [0.4874]. i.e. Kendall tells it’s due to chance alone

Why? Because kappa treats all misclassifications equally, but Kendall's coefficients do not treat all misclassifications equally. i.e. Kendall's coefficient considers the consequences of misclassifying “4” as “0” more serious than as “3” (…being ordinal) while cute Kappa is not so serious.

Why? The devil is in the details; i.e. concordance and discordance. These 2 little mischiefs don’t appear in Kappa unlike Kendall’s formula.

Why? The first mention of a kappa-like statistic is attributed to Galton 1890s while Kendall’s work predominantly appear during 1930s. The last why was googled!

Share this post


Link to post
Share on other sites
  • 0

Basically data can be categorized into quantitative and qualitative. Qualitative data can be further classified into Nominal, Ordinal and and Binary. Each data has its own ways to get the inference of the data. 

 

Since the Kendall's coefficient is designed to identify any number of distinct outcome from the list of Ordinal attribute data, it become highly ineffective for the Kappa value to do so once it is used to make the inference from the set of Ordinal attribute data.

 

Ordinal data is one which has got some scale range in its preparation. For example: 1 being least, 2 for some other category,3,4,so on and 10 being the Most. The inference from  these data sets can be made properly by Kendall's coefficient.

 

Kappa value is useful in case of nominal attribute data sets wherein, where in the data has some names, symbols like Black, white; strong, weak etc. Here the data sets are organised into categorical form and different observers provides some kind of ranking to the data in terms of his/her understanding. Later on the hypothesis and actual observed data from various category is used to calculate the Kappa value.

But the Kendall's coefficient is useful in cases where the data are not categorical but having some kind of grades or sequences. Like, the taste choice of food in a restaurant can fall into this category like 1,2,3,4...n. This become more specific and quantified in terms of calculating the customer taste.  

Thus Kappa value is only useful when we have to get inference from the data which are having category not the grades, to ensure :

  • Good statistical reliability inference from the data
  • Effectiveness of the accuracy between hypothesis and actual value
  • Helps in effective utilization of statistical software tools like Mini tab, Sigma XL etc.

Share this post


Link to post
Share on other sites
  • 0

The chosen best answer is that of Amlan for providing a clear answer in an interesting way. Here is the crux of the answer and why Kappa may lead us to make erroneous inferences. Read through the complete answer!

 

On 6/7/2019 at 11:49 PM, Amlan Dutt said:

Because kappa treats all misclassifications equally, but Kendall's coefficients do not treat all misclassifications equally. i.e. Kendall's coefficient considers the consequences of misclassifying “4” as “0” more serious than as “3” (…being ordinal) while cute Kappa is not so serious.

 

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×
×
  • Create New...