--------------------------------------------------------------------------------------------- corpora: BNC, verb: deny, number of patterns: 10, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 2 2.c 3 3.a 5 6 7 8 8.f 9 9.a 9.f 10 11 11.s 12.s | # prob ------+---------------------------------------------------------------------------------+------------ 2 | 9 - - - - - - - - 1 - - - - - - | 10 0.20 2.c | - - - - - - - - - 1 - - - - - - | 1 0.02 3 | - - 6 - - - - - - - - - - - - - | 6 0.12 3.a | - - 1 - - - - - - - - - - - - - | 1 0.02 5 | - - - - 2 - - - - - - - - - - - | 2 0.04 6 | - - - - - 1 - - - 1 - - - - - - | 2 0.04 7 | - - - - - - 1 - - - - - 1 - - - | 2 0.04 8 | - - - - - - - - 1 - - - - - - - | 1 0.02 8.f | - - - - - - - - - - - - - - - - | 0 0.00 9 | - - - - 1 - - - - 12 - - - - - 1 | 14 0.28 9.a | - - - - - - - - - 1 - - - - - - | 1 0.02 9.f | - - - - - - - - - 1 - - - - - - | 1 0.02 10 | - - - - - - - - - - - - 5 - - - | 5 0.10 11 | - - - - - - - - - - - - 1 2 - - | 3 0.06 11.s | - - - - - - - - - - - - 1 - - - | 1 0.02 12.s | - - - - - - - - - - - - - - - - | 0 0.00 ------+---------------------------------------------------------------------------------+------------ # | 9 0 7 0 3 1 1 0 1 17 0 0 8 2 0 1 | 50 prob | 0.18 0.00 0.14 0.00 0.06 0.02 0.02 0.00 0.02 0.34 0.00 0.00 0.16 0.04 0.00 0.02 | 1.00 sample size: 50 number of agreements: 38 ( 0.76) number of disagreements: 12 ( 0.24) Cohen's kappa = 0.711 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 2 2.c 3 3.a 5 6 7 8 9 9.a 9.f 10 11 11.s 12 u | # prob ------+---------------------------------------------------------------------------------+------------ 2 | 6 - - - - - - - 3 - - - - - 1 - | 10 0.20 2.c | - - - - - - - - 1 - - - - - - - | 1 0.02 3 | - - 6 - - - - - - - - - - - - - | 6 0.12 3.a | 1 - - - - - - - - - - - - - - - | 1 0.02 5 | 1 - - - - - - - - - - 1 - - - - | 2 0.04 6 | - - - - - 1 - - - - - 1 - - - - | 2 0.04 7 | - - - - - - 1 - - - - 1 - - - - | 2 0.04 8 | - - - - - - - - - - - 1 - - - - | 1 0.02 9 | - - 2 - - - - - 11 - - 1 - - - - | 14 0.28 9.a | - - - - - - - - - - - - - - - 1 | 1 0.02 9.f | - - - - - - - - - - - - - - - 1 | 1 0.02 10 | - - - - - - - - - - - 5 - - - - | 5 0.10 11 | - - - - - - - - - - - 2 1 - - - | 3 0.06 11.s | - - - - - - - - - - - 1 - - - - | 1 0.02 12 | - - - - - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - - - - - - - - - | 0 0.00 ------+---------------------------------------------------------------------------------+------------ # | 8 0 8 0 0 1 1 0 15 0 0 13 1 0 1 2 | 50 prob | 0.16 0.00 0.16 0.00 0.00 0.02 0.02 0.00 0.30 0.00 0.00 0.26 0.02 0.00 0.02 0.04 | 1.00 sample size: 50 number of agreements: 31 ( 0.62) number of disagreements: 19 ( 0.38) Cohen's kappa = 0.545 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 2 2.c 3 3.a 5 6 7 8 9 9.a 9.f 10 10.s 11 11.s | # prob ------+----------------------------------------------------------------------------+------------ 2 | 8 - 1 - - - - - 1 - - - - - - | 10 0.20 2.c | - - - - - - - - 1 - - - - - - | 1 0.02 3 | - - 6 - - - - - - - - - - - - | 6 0.12 3.a | - - 1 - - - - - - - - - - - - | 1 0.02 5 | - - - - 1 - - - 1 - - - - - - | 2 0.04 6 | - - - - - 1 - - 1 - - - - - - | 2 0.04 7 | - - - - - - 1 - - - - 1 - - - | 2 0.04 8 | - - - - - - - 1 - - - - - - - | 1 0.02 9 | - - - - 1 - - - 13 - - - - - - | 14 0.28 9.a | - - - - - - - - 1 - - - - - - | 1 0.02 9.f | - - - - - - - - 1 - - - - - - | 1 0.02 10 | - - - - - - - - - - - 4 1 - - | 5 0.10 10.s | - - - - - - - - - - - - - - - | 0 0.00 11 | - - - - - - - - - - - 1 - 2 - | 3 0.06 11.s | - - - - - - - - - - - 1 - - - | 1 0.02 ------+----------------------------------------------------------------------------+------------ # | 8 0 8 0 2 1 1 1 19 0 0 7 1 2 0 | 50 prob | 0.16 0.00 0.16 0.00 0.04 0.02 0.02 0.02 0.38 0.00 0.00 0.14 0.02 0.04 0.00 | 1.00 sample size: 50 number of agreements: 37 ( 0.74) number of disagreements: 13 ( 0.26) Cohen's kappa = 0.684 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 2 3 5 6 7 8.f 9 10 11 12 12.s u | # prob ------+-------------------------------------------------------------+------------ 2 | 5 - - - - - 3 - - 1 - - | 9 0.18 3 | 1 6 - - - - - - - - - - | 7 0.14 5 | 1 - - - - - 1 1 - - - - | 3 0.06 6 | - - - 1 - - - - - - - - | 1 0.02 7 | - - - - 1 - - - - - - - | 1 0.02 8.f | - - - - - - - 1 - - - - | 1 0.02 9 | 1 2 - - - - 11 1 - - - 2 | 17 0.34 10 | - - - - - - - 8 - - - - | 8 0.16 11 | - - - - - - - 1 1 - - - | 2 0.04 12 | - - - - - - - - - - - - | 0 0.00 12.s | - - - - - - - 1 - - - - | 1 0.02 u | - - - - - - - - - - - - | 0 0.00 ------+-------------------------------------------------------------+------------ # | 8 8 0 1 1 0 15 13 1 1 0 2 | 50 prob | 0.16 0.16 0.00 0.02 0.02 0.00 0.30 0.26 0.02 0.02 0.00 0.04 | 1.00 sample size: 50 number of agreements: 33 ( 0.66) number of disagreements: 17 ( 0.34) Cohen's kappa = 0.577 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 2 3 5 6 7 8 8.f 9 10 10.s 11 12.s | # prob ------+-------------------------------------------------------------+------------ 2 | 8 - - - - - - 1 - - - - | 9 0.18 3 | - 7 - - - - - - - - - - | 7 0.14 5 | - - 2 - - - - 1 - - - - | 3 0.06 6 | - - - 1 - - - - - - - - | 1 0.02 7 | - - - - 1 - - - - - - - | 1 0.02 8 | - - - - - - - - - - - - | 0 0.00 8.f | - - - - - 1 - - - - - - | 1 0.02 9 | - 1 - - - - - 16 - - - - | 17 0.34 10 | - - - - - - - - 7 1 - - | 8 0.16 10.s | - - - - - - - - - - - - | 0 0.00 11 | - - - - - - - - - - 2 - | 2 0.04 12.s | - - - - - - - 1 - - - - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 8 8 2 1 1 1 0 19 7 1 2 0 | 50 prob | 0.16 0.16 0.04 0.02 0.02 0.02 0.00 0.38 0.14 0.02 0.04 0.00 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.849 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 2 3 5 6 7 8 9 10 10.s 11 12 u | # prob ------+-------------------------------------------------------------+------------ 2 | 5 2 1 - - - - - - - - - | 8 0.16 3 | - 6 - - - - 2 - - - - - | 8 0.16 5 | - - - - - - - - - - - - | 0 0.00 6 | - - - 1 - - - - - - - - | 1 0.02 7 | - - - - 1 - - - - - - - | 1 0.02 8 | - - - - - - - - - - - - | 0 0.00 9 | 3 - 1 - - - 11 - - - - - | 15 0.30 10 | - - - - - 1 3 7 1 1 - - | 13 0.26 10.s | - - - - - - - - - - - - | 0 0.00 11 | - - - - - - - - - 1 - - | 1 0.02 12 | - - - - - - 1 - - - - - | 1 0.02 u | - - - - - - 2 - - - - - | 2 0.04 ------+-------------------------------------------------------------+------------ # | 8 8 2 1 1 1 19 7 1 2 0 0 | 50 prob | 0.16 0.16 0.04 0.02 0.02 0.02 0.38 0.14 0.02 0.04 0.00 0.00 | 1.00 sample size: 50 number of agreements: 32 ( 0.64) number of disagreements: 18 ( 0.36) Cohen's kappa = 0.548 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.651 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK deny 50 10 4 0.711 38 0.760 AV_JT deny 50 10 4 0.545 31 0.620 AV_SC deny 50 10 4 0.684 37 0.740 EK_JT deny 50 10 4 0.577 33 0.660 EK_SC deny 50 10 4 0.849 44 0.880 JT_SC deny 50 10 4 0.548 32 0.640 ----------------------------------------------------------- deny AV EK JT SC AV - 0.711 0.545 0.684 EK - - 0.577 0.849 JT - - - 0.548 SC - - - -