Q. 166  Kappa value is used to make inference about soundness of a Measurement System when Attribute Data is used. Kendall's coefficient is used to infer the soundness of Measurement System when ordinal Attribute Data is used. If one used Kappa value to make inferences for an Ordinal Attribute Data set, what errors are likely to happen?
 

Note for website visitors - Two questions are asked every week on this platform. One on Tuesday and the other on Friday.

• All questions so far can be seen here - https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/
• Please visit the forum home page at https://www.benchmarksixsigma.com/forum/ to respond to the latest question open till the next Tuesday/ Friday evening as per Indian Standard Time.  
• The best answer is always shown at the top among responses and the author finds honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

 

Consider two unbiased coins tossed 100 times simultaneouly and results put through Fleiss’ Kappa and Cohen’s Kappa (Kappa values range from –1 to +1) spitting out the results below; Since these are statistically independent events, any match will be purely due to chance alone.

p-values >0.6 tells (at 5% Type 1 Error Risk) fail to reject the null hypothesis that the agreement is due to chance alone. Notice Agreement is nearing 50%; thanks to Bernoulli random number option in Minitab.

So far so good.

Let’s test it on liar’s dice!

Consider two dice (…from a Martian liar) which can show values from 0 to 7 tossed 100 times simultaneously.

It’s basically two random Poisson variable of 100 observations with mean=2 (mean=variance is a property but that’s another story)

As narrated before any match will be purely due to chance alone (…being random).

Presto! What went wrong?

The curious readers will notice for Responses "2" and "5" the p-values are below 0.06 (i.e. only 6% risk by chance alone). Which means we can reject the null hypothesis that the agreement is due to chance alone. i.e. numbers on 2 dice match NOT by chance for at least "2" and "5".

(The negative kappa values tell very poor inter-rater reliability)

While the fact is data has been randomly generated. Hence, any matched would be purely by chance alone.

These opposing statements indicate a null hypothesis accepted even though it’s wrong. i.e. alternate hypothesis is right. In other words, we have committed a Type 2 Error.

Why? Because of using Kappa instead of Kendall, Kendall give a Coefficient (values range from 0 to +1) but highly insignificant [0.4874]. i.e. Kendall tells it’s due to chance alone

Why? Because kappa treats all misclassifications equally, but Kendall's coefficients do not treat all misclassifications equally. i.e. Kendall's coefficient considers the consequences of misclassifying “4” as “0” more serious than as “3” (…being ordinal) while cute Kappa is not so serious.

Why? The devil is in the details; i.e. concordance and discordance. These 2 little mischiefs don’t appear in Kappa unlike Kendall’s formula.

Why? The first mention of a kappa-like statistic is attributed to Galton 1890s while Kendall’s work predominantly appear during 1930s. The last why was googled!

Basically data can be categorized into quantitative and qualitative. Qualitative data can be further classified into Nominal, Ordinal and and Binary. Each data has its own ways to get the inference of the data. 

 

Since the Kendall's coefficient is designed to identify any number of distinct outcome from the list of Ordinal attribute data, it become highly ineffective for the Kappa value to do so once it is used to make the inference from the set of Ordinal attribute data.

 

Ordinal data is one which has got some scale range in its preparation. For example: 1 being least, 2 for some other category,3,4,so on and 10 being the Most. The inference from  these data sets can be made properly by Kendall's coefficient.

 

Kappa value is useful in case of nominal attribute data sets wherein, where in the data has some names, symbols like Black, white; strong, weak etc. Here the data sets are organised into categorical form and different observers provides some kind of ranking to the data in terms of his/her understanding. Later on the hypothesis and actual observed data from various category is used to calculate the Kappa value.

But the Kendall's coefficient is useful in cases where the data are not categorical but having some kind of grades or sequences. Like, the taste choice of food in a restaurant can fall into this category like 1,2,3,4...n. This become more specific and quantified in terms of calculating the customer taste.  

Thus Kappa value is only useful when we have to get inference from the data which are having category not the grades, to ensure :

• Good statistical reliability inference from the data
• Effectiveness of the accuracy between hypothesis and actual value
• Helps in effective utilization of statistical software tools like Mini tab, Sigma XL etc.

The chosen best answer is that of Amlan for providing a clear answer in an interesting way. Here is the crux of the answer and why Kappa may lead us to make erroneous inferences. Read through the complete answer!

 

On 6/7/2019 at 11:49 PM, Amlan Dutt said:

Because kappa treats all misclassifications equally, but Kendall's coefficients do not treat all misclassifications equally. i.e. Kendall's coefficient considers the consequences of misclassifying “4” as “0” more serious than as “3” (…being ordinal) while cute Kappa is not so serious.