-------------------------------------------------------------------------------------------- corpora: BNC, verb: ally, number of patterns: 6, sample size: 50, annotators(4): AV EK JT SC -------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.s 2 2.a 2.c 3 4 4.a 5 6 u x | # prob ------+-------------------------------------------------------------+------------ 1 | 1 - - - - - - - - - - - | 1 0.02 1.s | - - - - - - - - - - 1 - | 1 0.02 2 | - - 9 - - - - - - - - - | 9 0.18 2.a | - - - 2 - - - - - - - - | 2 0.04 2.c | - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - 12 - - - - - - | 12 0.24 4 | 1 - - - - - 13 - - - - - | 14 0.28 4.a | - - - - - 4 1 - - - - - | 5 0.10 5 | - - - - - - - - 1 - - - | 1 0.02 6 | - - - - - 1 - - - - - - | 1 0.02 u | - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - - 3 | 3 0.06 ------+-------------------------------------------------------------+------------ # | 2 0 10 2 0 17 14 0 1 0 1 3 | 50 prob | 0.04 0.00 0.20 0.04 0.00 0.34 0.28 0.00 0.02 0.00 0.02 0.06 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.774 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.s 2 2.a 2.c 3 4 4.a 5 6 x | # prob ------+-------------------------------------------------------------+------------ 1 | 1 - - - - - - - - - - - | 1 0.02 1.a | - - - - - - - - - - - - | 0 0.00 1.s | - 1 - - - - - - - - - - | 1 0.02 2 | 3 - - 6 - - - - - - - - | 9 0.18 2.a | - - - - - - 1 - 1 - - - | 2 0.04 2.c | - - - - 1 - - - - - - - | 1 0.02 3 | - - - - - - 12 - - - - - | 12 0.24 4 | 4 - - - - - - 10 - - - - | 14 0.28 4.a | - - - - - - 4 - - - - 1 | 5 0.10 5 | - - - - - - - - - 1 - - | 1 0.02 6 | - - - - - - - - - - 1 - | 1 0.02 x | - - - - - - - - - - - 3 | 3 0.06 ------+-------------------------------------------------------------+------------ # | 8 1 0 6 1 0 17 10 1 1 1 4 | 50 prob | 0.16 0.02 0.00 0.12 0.02 0.00 0.34 0.20 0.02 0.02 0.02 0.08 | 1 sample size: 50 number of agreements: 34 ( 0.68) number of disagreements: 16 ( 0.32) Cohen's kappa = 0.614 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.s 2 2.a 2.c 3 4 4.a 5 5.f 6 u x | # prob ------+------------------------------------------------------------------+------------ 1 | 1 - - - - - - - - - - - - | 1 0.02 1.s | - - - 1 - - - - - - - - - | 1 0.02 2 | - - 9 - - - - - - - - - - | 9 0.18 2.a | - - - - - 1 - - - - - 1 - | 2 0.04 2.c | - - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - 12 - - - - - - - | 12 0.24 4 | - - - - - 1 12 - - 1 - - - | 14 0.28 4.a | - - - - - 4 1 - - - - - - | 5 0.10 5 | - - - - - - - - 1 - - - - | 1 0.02 5.f | - - - - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - - 1 - - | 1 0.02 u | - - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - - - 3 | 3 0.06 ------+------------------------------------------------------------------+------------ # | 1 0 9 2 0 18 13 0 1 1 1 1 3 | 50 prob | 0.02 0.00 0.18 0.04 0.00 0.36 0.26 0.00 0.02 0.02 0.02 0.02 0.06 | 1.00 sample size: 50 number of agreements: 39 ( 0.78) number of disagreements: 11 ( 0.22) Cohen's kappa = 0.726 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 2 2.a 3 4 4.a 5 6 u x | # prob ------+--------------------------------------------------------+------------ 1 | 1 - - - - 1 - - - - - | 2 0.04 1.a | - - - - - - - - - - - | 0 0.00 2 | 3 - 6 1 - - - - - - - | 10 0.20 2.a | - - - - 1 - 1 - - - - | 2 0.04 3 | - - - - 16 - - - 1 - - | 17 0.34 4 | 4 - - - - 9 - - - - 1 | 14 0.28 4.a | - - - - - - - - - - - | 0 0.00 5 | - - - - - - - 1 - - - | 1 0.02 6 | - - - - - - - - - - - | 0 0.00 u | - 1 - - - - - - - - - | 1 0.02 x | - - - - - - - - - - 3 | 3 0.06 ------+--------------------------------------------------------+------------ # | 8 1 6 1 17 10 1 1 1 0 4 | 50 prob | 0.16 0.02 0.12 0.02 0.34 0.20 0.02 0.02 0.02 0.00 0.08 | 1.00 sample size: 50 number of agreements: 36 ( 0.72) number of disagreements: 14 ( 0.28) Cohen's kappa = 0.646 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 2 2.a 3 4 5 5.f 6 u x | # prob ------+---------------------------------------------------+------------ 1 | 1 - - - - - 1 - - - | 2 0.04 2 | - 9 1 - - - - - - - | 10 0.20 2.a | - - - 1 - - - - 1 - | 2 0.04 3 | - - - 16 - - - 1 - - | 17 0.34 4 | - - - 1 13 - - - - - | 14 0.28 5 | - - - - - 1 - - - - | 1 0.02 5.f | - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - - | 0 0.00 u | - - 1 - - - - - - - | 1 0.02 x | - - - - - - - - - 3 | 3 0.06 ------+---------------------------------------------------+------------ # | 1 9 2 18 13 1 1 1 1 3 | 50 prob | 0.02 0.18 0.04 0.36 0.26 0.02 0.02 0.02 0.02 0.06 | 1 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.816 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 2 2.a 3 4 4.a 5 5.f 6 u x | # prob ------+-------------------------------------------------------------+------------ 1 | 1 - 3 - - 4 - - - - - - | 8 0.16 1.a | - - - 1 - - - - - - - - | 1 0.02 2 | - - 6 - - - - - - - - - | 6 0.12 2.a | - - - 1 - - - - - - - - | 1 0.02 3 | - - - - 17 - - - - - - - | 17 0.34 4 | - - - - 1 8 - - 1 - - - | 10 0.20 4.a | - - - - - - - - - - 1 - | 1 0.02 5 | - - - - - - - 1 - - - - | 1 0.02 5.f | - - - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - 1 - - | 1 0.02 u | - - - - - - - - - - - - | 0 0.00 x | - - - - - 1 - - - - - 3 | 4 0.08 ------+-------------------------------------------------------------+------------ # | 1 0 9 2 18 13 0 1 1 1 1 3 | 50 prob | 0.02 0.00 0.18 0.04 0.36 0.26 0.00 0.02 0.02 0.02 0.02 0.06 | 1 sample size: 50 number of agreements: 38 ( 0.76) number of disagreements: 12 ( 0.24) Cohen's kappa = 0.698 -------------------------------------------------------------------------------------------- Fleiss' kappa = 0.710 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK ally 50 6 4 0.774 41 0.820 AV_JT ally 50 6 4 0.614 34 0.680 AV_SC ally 50 6 4 0.726 39 0.780 EK_JT ally 50 6 4 0.646 36 0.720 EK_SC ally 50 6 4 0.816 43 0.860 JT_SC ally 50 6 4 0.698 38 0.760 ----------------------------------------------------------- ally AV EK JT SC AV - 0.774 0.614 0.726 EK - - 0.646 0.816 JT - - - 0.698 SC - - - -