--------------------------------------------------------------------------------------------- corpora: BNC, verb: wake, number of patterns: 10, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 2 2.f 4 4.a 4.f 5 6 6.a 8 9 10 12 u | # prob ------+------------------------------------------------------------------+------------ 2 | 16 - 1 - - - - - - 1 - - 1 | 19 0.38 2.f | - - - - - - - - - - - - - | 0 0.00 4 | 1 - 11 - 1 - - - - - - - - | 13 0.26 4.a | - - 3 - - - - - - - - - - | 3 0.06 4.f | - - - - - - - - - - - - - | 0 0.00 5 | 1 1 - - - 3 - - - - - - - | 5 0.10 6 | - - - - - - 1 - - - - - - | 1 0.02 6.a | - - 1 - - - - - - - - - - | 1 0.02 8 | - - - - - - - - 1 - - - - | 1 0.02 9 | 1 - - - - - - - - 1 - - - | 2 0.04 10 | - - - - - - - - - - 1 - - | 1 0.02 12 | 1 - - - - 1 - - - - - 2 - | 4 0.08 u | - - - - - - - - - - - - - | 0 0.00 ------+------------------------------------------------------------------+------------ # | 20 1 16 0 1 4 1 0 1 2 1 2 1 | 50 prob | 0.40 0.02 0.32 0.00 0.02 0.08 0.02 0.00 0.02 0.04 0.02 0.04 0.02 | 1.00 sample size: 50 number of agreements: 36 ( 0.72) number of disagreements: 14 ( 0.28) Cohen's kappa = 0.627 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 2 4 4.a 5 6 6.a 8 9 9.f 10 11.f 12 x | # prob ------+------------------------------------------------------------------+------------ 2 | 15 1 - - - - - 1 - 1 1 - - | 19 0.38 4 | 1 12 - - - - - - - - - - - | 13 0.26 4.a | - 1 2 - - - - - - - - - - | 3 0.06 5 | 1 - - 3 - - - - 1 - - - - | 5 0.10 6 | - - - 1 - - - - - - - - - | 1 0.02 6.a | - 1 - - - - - - - - - - - | 1 0.02 8 | - - - - - - 1 - - - - - - | 1 0.02 9 | 1 - - - - - - 1 - - - - - | 2 0.04 9.f | - - - - - - - - - - - - - | 0 0.00 10 | - - - - - - - - - - - - 1 | 1 0.02 11.f | - - - - - - - - - - - - - | 0 0.00 12 | 1 - - 1 - - - - - - - 2 - | 4 0.08 x | - - - - - - - - - - - - - | 0 0.00 ------+------------------------------------------------------------------+------------ # | 19 15 2 5 0 0 1 2 1 1 1 2 1 | 50 prob | 0.38 0.30 0.04 0.10 0.00 0.00 0.02 0.04 0.02 0.02 0.02 0.04 0.02 | 1.00 sample size: 50 number of agreements: 36 ( 0.72) number of disagreements: 14 ( 0.28) Cohen's kappa = 0.631 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 2 4 4.a 5 5.a 6 6.a 8 9 10 12 | # prob ------+--------------------------------------------------------+------------ 2 | 18 1 - - - - - - - - - | 19 0.38 4 | 1 12 - - - - - - - - - | 13 0.26 4.a | - 2 1 - - - - - - - - | 3 0.06 5 | 1 - - 4 - - - - - - - | 5 0.10 5.a | - - - - - - - - - - - | 0 0.00 6 | - - - - 1 - - - - - - | 1 0.02 6.a | - 1 - - - - - - - - - | 1 0.02 8 | - - - - - - - 1 - - - | 1 0.02 9 | 2 - - - - - - - - - - | 2 0.04 10 | - - - - - - - - - 1 - | 1 0.02 12 | 1 - - 1 - - - - - - 2 | 4 0.08 ------+--------------------------------------------------------+------------ # | 23 16 1 5 1 0 0 1 0 1 2 | 50 prob | 0.46 0.32 0.02 0.10 0.02 0.00 0.00 0.02 0.00 0.02 0.04 | 1.00 sample size: 50 number of agreements: 39 ( 0.78) number of disagreements: 11 ( 0.22) Cohen's kappa = 0.697 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 2 2.f 4 4.a 4.f 5 6 8 9 9.f 10 11.f 12 u x | # prob ------+----------------------------------------------------------------------------+------------ 2 | 19 - - - - - - - - - 1 - - - - | 20 0.40 2.f | - - - - - - - - - 1 - - - - - | 1 0.02 4 | - - 14 2 - - - - - - - - - - - | 16 0.32 4.a | - - - - - - - - - - - - - - - | 0 0.00 4.f | - - 1 - - - - - - - - - - - - | 1 0.02 5 | - - - - - 4 - - - - - - - - - | 4 0.08 6 | - - - - - 1 - - - - - - - - - | 1 0.02 8 | - - - - - - - 1 - - - - - - - | 1 0.02 9 | - - - - - - - - 2 - - - - - - | 2 0.04 9.f | - - - - - - - - - - - - - - - | 0 0.00 10 | - - - - - - - - - - - - - - 1 | 1 0.02 11.f | - - - - - - - - - - - - - - - | 0 0.00 12 | - - - - - - - - - - - - 2 - - | 2 0.04 u | - - - - - - - - - - - 1 - - - | 1 0.02 x | - - - - - - - - - - - - - - - | 0 0.00 ------+----------------------------------------------------------------------------+------------ # | 19 0 15 2 0 5 0 1 2 1 1 1 2 0 1 | 50 prob | 0.38 0.00 0.30 0.04 0.00 0.10 0.00 0.02 0.04 0.02 0.02 0.02 0.04 0.00 0.02 | 1.00 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.784 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 2 2.f 4 4.a 4.f 5 5.a 6 8 9 10 12 u | # prob ------+------------------------------------------------------------------+------------ 2 | 20 - - - - - - - - - - - - | 20 0.40 2.f | - - - - - 1 - - - - - - - | 1 0.02 4 | - - 15 1 - - - - - - - - - | 16 0.32 4.a | - - - - - - - - - - - - - | 0 0.00 4.f | - - 1 - - - - - - - - - - | 1 0.02 5 | - - - - - 4 - - - - - - - | 4 0.08 5.a | - - - - - - - - - - - - - | 0 0.00 6 | - - - - - - 1 - - - - - - | 1 0.02 8 | - - - - - - - - 1 - - - - | 1 0.02 9 | 2 - - - - - - - - - - - - | 2 0.04 10 | - - - - - - - - - - 1 - - | 1 0.02 12 | - - - - - - - - - - - 2 - | 2 0.04 u | 1 - - - - - - - - - - - - | 1 0.02 ------+------------------------------------------------------------------+------------ # | 23 0 16 1 0 5 1 0 1 0 1 2 0 | 50 prob | 0.46 0.00 0.32 0.02 0.00 0.10 0.02 0.00 0.02 0.00 0.02 0.04 0.00 | 1.00 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.801 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 2 4 4.a 5 5.a 8 9 9.f 10 11.f 12 x | # prob ------+-------------------------------------------------------------+------------ 2 | 19 - - - - - - - - - - - | 19 0.38 4 | - 15 - - - - - - - - - - | 15 0.30 4.a | - 1 1 - - - - - - - - - | 2 0.04 5 | - - - 4 1 - - - - - - - | 5 0.10 5.a | - - - - - - - - - - - - | 0 0.00 8 | - - - - - 1 - - - - - - | 1 0.02 9 | 2 - - - - - - - - - - - | 2 0.04 9.f | - - - 1 - - - - - - - - | 1 0.02 10 | 1 - - - - - - - - - - - | 1 0.02 11.f | 1 - - - - - - - - - - - | 1 0.02 12 | - - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - 1 - - - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 23 16 1 5 1 1 0 0 1 0 2 0 | 50 prob | 0.46 0.32 0.02 0.10 0.02 0.02 0.00 0.00 0.02 0.00 0.04 0.00 | 1.00 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.777 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.717 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK wake 50 10 4 0.627 36 0.720 AV_JT wake 50 10 4 0.631 36 0.720 AV_SC wake 50 10 4 0.697 39 0.780 EK_JT wake 50 10 4 0.784 42 0.840 EK_SC wake 50 10 4 0.801 43 0.860 JT_SC wake 50 10 4 0.777 42 0.840 ----------------------------------------------------------- wake AV EK JT SC AV - 0.627 0.631 0.697 EK - - 0.784 0.801 JT - - - 0.777 SC - - - -