-------------------------------------------------------------------------------------------- corpora: BNC, verb: cry, number of patterns: 18, sample size: 50, annotators(4): AV EK JT SC -------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.s 3 4 4.s 5 5.a 6.s 7 7.a 14.a 14.s 17 18.s u x | # prob ------+---------------------------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - - - - - | 18 0.36 1.s | - 1 - - - - - - - - - - - - - - | 1 0.02 3 | - - 1 - - - - - - - - - - - - - | 1 0.02 4 | - - - 13 - - - - - - - - - - 2 - | 15 0.30 4.s | - - - - 2 - - - - - - - - - - - | 2 0.04 5 | - - - - - - - - - - - - - - - - | 0 0.00 5.a | - - - - - 1 - - - - - - - - - - | 1 0.02 6.s | - - - - 1 - - - - - - - - - - - | 1 0.02 7 | - - - - - - - - 2 - - - - - - - | 2 0.04 7.a | - - - - - - - - 1 - - - - - 1 - | 2 0.04 14.a | - - - - - - - - - - - - - - - - | 0 0.00 14.s | - - - - - - - - - - 1 - - - - - | 1 0.02 17 | - - - - - - - - - - - - 1 - - - | 1 0.02 18.s | - - - - - - - - - - - - 1 - - - | 1 0.02 u | - - - 1 - - - - - - - - - - - 1 | 2 0.04 x | - - - - - - - - - - - - - - - 2 | 2 0.04 ------+---------------------------------------------------------------------------------+------------ # | 18 1 1 14 3 1 0 0 3 0 1 0 2 0 3 3 | 50 prob | 0.36 0.02 0.02 0.28 0.06 0.02 0.00 0.00 0.06 0.00 0.02 0.00 0.04 0.00 0.06 0.06 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.742 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.s 3 4 4.s 5.a 5.c 6.s 7 7.a 14.s 17 18.s u x | # prob ------+----------------------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - - - - | 18 0.36 1.s | 1 - - - - - - - - - - - - - - | 1 0.02 3 | - - 1 - - - - - - - - - - - - | 1 0.02 4 | - - - 14 1 - - - - - - - - - - | 15 0.30 4.s | - - - - 2 - - - - - - - - - - | 2 0.04 5.a | - - - - - - 1 - - - - - - - - | 1 0.02 5.c | - - - - - - - - - - - - - - - | 0 0.00 6.s | - - - - 1 - - - - - - - - - - | 1 0.02 7 | - - - - - - - - 2 - - - - - - | 2 0.04 7.a | - - - - 1 - - - - - - - - 1 - | 2 0.04 14.s | - - - - - - - - - - - - - 1 - | 1 0.02 17 | - - - - - - - - - - - 1 - - - | 1 0.02 18.s | - - - - - - - - - - - 1 - - - | 1 0.02 u | - - - 1 - - - - - - - - - 1 - | 2 0.04 x | - - - - - - - - - - - - - - 2 | 2 0.04 ------+----------------------------------------------------------------------------+------------ # | 19 0 1 15 5 0 1 0 2 0 0 2 0 3 2 | 50 prob | 0.38 0.00 0.02 0.30 0.10 0.00 0.02 0.00 0.04 0.00 0.00 0.04 0.00 0.06 0.04 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.764 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.s 2 3 3.s 4 4.s 5 5.a 6.s 7 7.a 14.s 17 18.s u x | # prob ------+--------------------------------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - - - - - - | 18 0.36 1.s | - 1 - - - - - - - - - - - - - - - | 1 0.02 2 | - - - - - - - - - - - - - - - - - | 0 0.00 3 | - - - - 1 - - - - - - - - - - - - | 1 0.02 3.s | - - - - - - - - - - - - - - - - - | 0 0.00 4 | - - 1 - - 14 - - - - - - - - - - - | 15 0.30 4.s | - - - - - 1 1 - - - - - - - - - - | 2 0.04 5 | - - - - - - - - - - - - - - - - - | 0 0.00 5.a | - - - - - - - 1 - - - - - - - - - | 1 0.02 6.s | - - - - - - 1 - - - - - - - - - - | 1 0.02 7 | - - - - - - - - - - 2 - - - - - - | 2 0.04 7.a | - - - - - 1 - - - - - 1 - - - - - | 2 0.04 14.s | - - - - - - - - - - - - - - - 1 - | 1 0.02 17 | - - - - - - - - - - - - - 1 - - - | 1 0.02 18.s | - - - - - - - - - - - - - 1 - - - | 1 0.02 u | - - - - - 1 - - - - - - - - - 1 - | 2 0.04 x | - - - - - 1 - - - - - - - - - - 1 | 2 0.04 ------+--------------------------------------------------------------------------------------+------------ # | 18 1 1 0 1 18 2 1 0 0 2 1 0 2 0 2 1 | 50 prob | 0.36 0.02 0.02 0.00 0.02 0.36 0.04 0.02 0.00 0.00 0.04 0.02 0.00 0.04 0.00 0.04 0.02 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.735 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.s 3 4 4.s 5 5.c 7 14.a 17 u x | # prob ------+-------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - | 18 0.36 1.s | 1 - - - - - - - - - - - | 1 0.02 3 | - - 1 - - - - - - - - - | 1 0.02 4 | - - - 14 - - - - - - - - | 14 0.28 4.s | - - - - 3 - - - - - - - | 3 0.06 5 | - - - - - - 1 - - - - - | 1 0.02 5.c | - - - - - - - - - - - - | 0 0.00 7 | - - - - - - - 2 - - 1 - | 3 0.06 14.a | - - - - - - - - - - 1 - | 1 0.02 17 | - - - - - - - - - 2 - - | 2 0.04 u | - - - 1 2 - - - - - - - | 3 0.06 x | - - - - - - - - - - 1 2 | 3 0.06 ------+-------------------------------------------------------------+------------ # | 19 0 1 15 5 0 1 2 0 2 3 2 | 50 prob | 0.38 0.00 0.02 0.30 0.10 0.00 0.02 0.04 0.00 0.04 0.06 0.04 | 1 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.790 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.s 2 3 3.s 4 4.s 5 7 7.a 14.a 17 u x | # prob ------+-----------------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - - - | 18 0.36 1.s | - 1 - - - - - - - - - - - - | 1 0.02 2 | - - - - - - - - - - - - - - | 0 0.00 3 | - - - - 1 - - - - - - - - - | 1 0.02 3.s | - - - - - - - - - - - - - - | 0 0.00 4 | - - 1 - - 13 - - - - - - - - | 14 0.28 4.s | - - - - - 1 2 - - - - - - - | 3 0.06 5 | - - - - - - - 1 - - - - - - | 1 0.02 7 | - - - - - - - - 2 1 - - - - | 3 0.06 7.a | - - - - - - - - - - - - - - | 0 0.00 14.a | - - - - - - - - - - - - 1 - | 1 0.02 17 | - - - - - - - - - - - 2 - - | 2 0.04 u | - - - - - 3 - - - - - - - - | 3 0.06 x | - - - - - 1 - - - - - - 1 1 | 3 0.06 ------+-----------------------------------------------------------------------+------------ # | 18 1 1 0 1 18 2 1 2 1 0 2 2 1 | 50 prob | 0.36 0.02 0.02 0.00 0.02 0.36 0.04 0.02 0.04 0.02 0.00 0.04 0.04 0.02 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.736 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.s 2 3 3.s 4 4.s 5 5.c 7 7.a 17 u x | # prob ------+-----------------------------------------------------------------------+------------ 1 | 18 1 - - - - - - - - - - - - | 19 0.38 1.s | - - - - - - - - - - - - - - | 0 0.00 2 | - - - - - - - - - - - - - - | 0 0.00 3 | - - - - 1 - - - - - - - - - | 1 0.02 3.s | - - - - - - - - - - - - - - | 0 0.00 4 | - - 1 - - 14 - - - - - - - - | 15 0.30 4.s | - - - - - 3 2 - - - - - - - | 5 0.10 5 | - - - - - - - - - - - - - - | 0 0.00 5.c | - - - - - - - 1 - - - - - - | 1 0.02 7 | - - - - - - - - - 2 - - - - | 2 0.04 7.a | - - - - - - - - - - - - - - | 0 0.00 17 | - - - - - - - - - - - 2 - - | 2 0.04 u | - - - - - - - - - - 1 - 2 - | 3 0.06 x | - - - - - 1 - - - - - - - 1 | 2 0.04 ------+-----------------------------------------------------------------------+------------ # | 18 1 1 0 1 18 2 1 0 2 1 2 2 1 | 50 prob | 0.36 0.02 0.02 0.00 0.02 0.36 0.04 0.02 0.00 0.04 0.02 0.04 0.04 0.02 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.758 -------------------------------------------------------------------------------------------- Fleiss' kappa = 0.754 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK cry 50 18 4 0.742 40 0.800 AV_JT cry 50 18 4 0.764 41 0.820 AV_SC cry 50 18 4 0.735 40 0.800 EK_JT cry 50 18 4 0.790 42 0.840 EK_SC cry 50 18 4 0.736 40 0.800 JT_SC cry 50 18 4 0.758 41 0.820 ----------------------------------------------------------- cry AV EK JT SC AV - 0.742 0.764 0.735 EK - - 0.790 0.736 JT - - - 0.758 SC - - - -