--------------------------------------------------------------------------------------------- corpora: BNC, verb: tell, number of patterns: 19, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 1.c 1.s 2 2.a 3 6 7 7.s 8 13 14 15 u | # prob ------+----------------------------------------------------------------------------+------------ 1 | 22 1 - 6 - - - - - - - - - - 1 | 30 0.60 1.a | - - - - - - - - - - - - - - - | 0 0.00 1.c | - 1 - - - - - - - - - - - - - | 1 0.02 1.s | - - - - - - - - - - - - - - - | 0 0.00 2 | - - - - 1 - - - - - - - - - - | 1 0.02 2.a | - - - - 1 - - - - - - - - - - | 1 0.02 3 | - - - - - - 4 - - - - - - - - | 4 0.08 6 | - - - - - - - 2 - - - - - - - | 2 0.04 7 | - - - - - - - - 4 - 1 - - - - | 5 0.10 7.s | - - - - - - - - 1 - - - - - - | 1 0.02 8 | - - - - - - - - - - - - - - - | 0 0.00 13 | - - - - - - - - - - - 4 - - - | 4 0.08 14 | - - - - - - - - - - - - - - - | 0 0.00 15 | - - - - - - - - - - - - 1 - - | 1 0.02 u | - - - - - - - - - - - - - - - | 0 0.00 ------+----------------------------------------------------------------------------+------------ # | 22 2 0 6 2 0 4 2 5 0 1 4 1 0 1 | 50 prob | 0.44 0.04 0.00 0.12 0.04 0.00 0.08 0.04 0.10 0.00 0.02 0.08 0.02 0.00 0.02 | 1.00 sample size: 50 number of agreements: 37 ( 0.74) number of disagreements: 13 ( 0.26) Cohen's kappa = 0.634 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.c 1.s 2 2.a 3 6 7 7.s 13 14 15 x | # prob ------+------------------------------------------------------------------+------------ 1 | 28 - 2 - - - - - - - - - - | 30 0.60 1.c | 1 - - - - - - - - - - - - | 1 0.02 1.s | - - - - - - - - - - - - - | 0 0.00 2 | - - - 1 - - - - - - - - - | 1 0.02 2.a | - - - 1 - - - - - - - - - | 1 0.02 3 | 1 - - - - 3 - - - - - - - | 4 0.08 6 | - - - - - - 1 - - - - - 1 | 2 0.04 7 | - - - - - - - 4 1 - - - - | 5 0.10 7.s | - - - - - - - 1 - - - - - | 1 0.02 13 | - - - - - - - - - 4 - - - | 4 0.08 14 | - - - - - - - - - - - - - | 0 0.00 15 | - - - - - - - - - - 1 - - | 1 0.02 x | - - - - - - - - - - - - - | 0 0.00 ------+------------------------------------------------------------------+------------ # | 30 0 2 2 0 3 1 5 1 4 1 0 1 | 50 prob | 0.60 0.00 0.04 0.04 0.00 0.06 0.02 0.10 0.02 0.08 0.02 0.00 0.02 | 1.00 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.708 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.c 2 2.a 3 6 7 7.a 7.s 13 14 15 | # prob ------+-------------------------------------------------------------+------------ 1 | 30 - - - - - - - - - - - | 30 0.60 1.c | 1 - - - - - - - - - - - | 1 0.02 2 | - - 1 - - - - - - - - - | 1 0.02 2.a | - - 1 - - - - - - - - - | 1 0.02 3 | - - - - 4 - - - - - - - | 4 0.08 6 | - - - - - 2 - - - - - - | 2 0.04 7 | - - - - - - 3 2 - - - - | 5 0.10 7.a | - - - - - - - - - - - - | 0 0.00 7.s | - - - - - - 1 - - - - - | 1 0.02 13 | - - - - - - - - - 4 - - | 4 0.08 14 | - - - - - - - - - - - - | 0 0.00 15 | - - - - - - - - - - 1 - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 31 0 2 0 4 2 4 2 0 4 1 0 | 50 prob | 0.62 0.00 0.04 0.00 0.08 0.04 0.08 0.04 0.00 0.08 0.02 0.00 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.802 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.s 2 3 6 7 7.s 8 13 14 u x | # prob ------+------------------------------------------------------------------+------------ 1 | 22 - - - - - - - - - - - - | 22 0.44 1.a | 2 - - - - - - - - - - - - | 2 0.04 1.s | 4 - 2 - - - - - - - - - - | 6 0.12 2 | - - - 2 - - - - - - - - - | 2 0.04 3 | 1 - - - 3 - - - - - - - - | 4 0.08 6 | - - - - - 1 - - - - - - 1 | 2 0.04 7 | - - - - - - 4 1 - - - - - | 5 0.10 7.s | - - - - - - - - - - - - - | 0 0.00 8 | - - - - - - 1 - - - - - - | 1 0.02 13 | - - - - - - - - - 4 - - - | 4 0.08 14 | - - - - - - - - - - 1 - - | 1 0.02 u | 1 - - - - - - - - - - - - | 1 0.02 x | - - - - - - - - - - - - - | 0 0.00 ------+------------------------------------------------------------------+------------ # | 30 0 2 2 3 1 5 1 0 4 1 0 1 | 50 prob | 0.60 0.00 0.04 0.04 0.06 0.02 0.10 0.02 0.00 0.08 0.02 0.00 0.02 | 1.00 sample size: 50 number of agreements: 39 ( 0.78) number of disagreements: 11 ( 0.22) Cohen's kappa = 0.689 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 1.s 2 3 6 7 7.a 8 13 14 u | # prob ------+-------------------------------------------------------------+------------ 1 | 22 - - - - - - - - - - - | 22 0.44 1.a | 2 - - - - - - - - - - - | 2 0.04 1.s | 6 - - - - - - - - - - - | 6 0.12 2 | - - - 2 - - - - - - - - | 2 0.04 3 | - - - - 4 - - - - - - - | 4 0.08 6 | - - - - - 2 - - - - - - | 2 0.04 7 | - - - - - - 4 1 - - - - | 5 0.10 7.a | - - - - - - - - - - - - | 0 0.00 8 | - - - - - - - 1 - - - - | 1 0.02 13 | - - - - - - - - - 4 - - | 4 0.08 14 | - - - - - - - - - - 1 - | 1 0.02 u | 1 - - - - - - - - - - - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 31 0 0 2 4 2 4 2 0 4 1 0 | 50 prob | 0.62 0.00 0.00 0.04 0.08 0.04 0.08 0.04 0.00 0.08 0.02 0.00 | 1.00 sample size: 50 number of agreements: 39 ( 0.78) number of disagreements: 11 ( 0.22) Cohen's kappa = 0.687 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.s 2 3 6 7 7.a 7.s 13 14 x | # prob ------+--------------------------------------------------------+------------ 1 | 29 - - 1 - - - - - - - | 30 0.60 1.s | 2 - - - - - - - - - - | 2 0.04 2 | - - 2 - - - - - - - - | 2 0.04 3 | - - - 3 - - - - - - - | 3 0.06 6 | - - - - 1 - - - - - - | 1 0.02 7 | - - - - - 4 1 - - - - | 5 0.10 7.a | - - - - - - - - - - - | 0 0.00 7.s | - - - - - - 1 - - - - | 1 0.02 13 | - - - - - - - - 4 - - | 4 0.08 14 | - - - - - - - - - 1 - | 1 0.02 x | - - - - 1 - - - - - - | 1 0.02 ------+--------------------------------------------------------+------------ # | 31 0 2 4 2 4 2 0 4 1 0 | 50 prob | 0.62 0.00 0.04 0.08 0.04 0.08 0.04 0.00 0.08 0.02 0.00 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.802 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.715 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK tell 50 19 4 0.634 37 0.740 AV_JT tell 50 19 4 0.708 41 0.820 AV_SC tell 50 19 4 0.802 44 0.880 EK_JT tell 50 19 4 0.689 39 0.780 EK_SC tell 50 19 4 0.687 39 0.780 JT_SC tell 50 19 4 0.802 44 0.880 ----------------------------------------------------------- tell AV EK JT SC AV - 0.634 0.708 0.802 EK - - 0.689 0.687 JT - - - 0.802 SC - - - -