-------------------------------------------------------------------------------------------- corpora: BNC, verb: say, number of patterns: 14, sample size: 50, annotators(4): AV EK JT SC -------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 4 6 7 11 12 14 x | # prob ------+----------------------------------------------+------------ 1 | 38 - - 1 - - - - - | 39 0.78 1.a | - 1 - - - - - - - | 1 0.02 4 | - - 1 - - - - - - | 1 0.02 6 | - - - - - - - - - | 0 0.00 7 | - - - - 1 - - - - | 1 0.02 11 | - - - - - 5 - - - | 5 0.10 12 | - 1 - - - - - - - | 1 0.02 14 | - - - - - - - 1 - | 1 0.02 x | - - - - - - - - 1 | 1 0.02 ------+----------------------------------------------+------------ # | 38 2 1 1 1 5 0 1 1 | 50 prob | 0.76 0.04 0.02 0.02 0.02 0.10 0.00 0.02 0.02 | 1.00 sample size: 50 number of agreements: 48 ( 0.96) number of disagreements: 2 ( 0.04) Cohen's kappa = 0.899 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.c 4 7 11 11.s 12 14 14.a u x | # prob ------+-------------------------------------------------------------+------------ 1 | 38 - 1 - - - - - - - - - | 39 0.78 1.a | - - - - - - - - - 1 - - | 1 0.02 1.c | - - - - - - - - - - - - | 0 0.00 4 | - - - 1 - - - - - - - - | 1 0.02 7 | - - - - 1 - - - - - - - | 1 0.02 11 | - - - - - 3 2 - - - - - | 5 0.10 11.s | - - - - - - - - - - - - | 0 0.00 12 | 1 - - - - - - - - - - - | 1 0.02 14 | - - - - - - - - 1 - - - | 1 0.02 14.a | - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - 1 - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 39 0 1 1 1 3 2 0 1 1 1 0 | 50 prob | 0.78 0.00 0.02 0.02 0.02 0.06 0.04 0.00 0.02 0.02 0.02 0.00 | 1 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.688 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 4 6 7 11 12 14 x | # prob ------+----------------------------------------------+------------ 1 | 38 - - 1 - - - - - | 39 0.78 1.a | - 1 - - - - - - - | 1 0.02 4 | - - 1 - - - - - - | 1 0.02 6 | - - - - - - - - - | 0 0.00 7 | - - - - 1 - - - - | 1 0.02 11 | - - - - - 5 - - - | 5 0.10 12 | - 1 - - - - - - - | 1 0.02 14 | - - - - - - - 1 - | 1 0.02 x | - - - - - - - - 1 | 1 0.02 ------+----------------------------------------------+------------ # | 38 2 1 1 1 5 0 1 1 | 50 prob | 0.76 0.04 0.02 0.02 0.02 0.10 0.00 0.02 0.02 | 1.00 sample size: 50 number of agreements: 48 ( 0.96) number of disagreements: 2 ( 0.04) Cohen's kappa = 0.899 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.c 4 6 7 11 11.s 14 14.a u x | # prob ------+-------------------------------------------------------------+------------ 1 | 37 - 1 - - - - - - - - - | 38 0.76 1.a | 1 - - - - - - - - 1 - - | 2 0.04 1.c | - - - - - - - - - - - - | 0 0.00 4 | - - - 1 - - - - - - - - | 1 0.02 6 | 1 - - - - - - - - - - - | 1 0.02 7 | - - - - - 1 - - - - - - | 1 0.02 11 | - - - - - - 3 2 - - - - | 5 0.10 11.s | - - - - - - - - - - - - | 0 0.00 14 | - - - - - - - - 1 - - - | 1 0.02 14.a | - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - 1 - | 1 0.02 ------+-------------------------------------------------------------+------------ # | 39 0 1 1 0 1 3 2 1 1 1 0 | 50 prob | 0.78 0.00 0.02 0.02 0.00 0.02 0.06 0.04 0.02 0.02 0.02 0.00 | 1 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.650 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 4 6 7 11 14 x | # prob ------+-----------------------------------------+------------ 1 | 38 - - - - - - - | 38 0.76 1.a | - 2 - - - - - - | 2 0.04 4 | - - 1 - - - - - | 1 0.02 6 | - - - 1 - - - - | 1 0.02 7 | - - - - 1 - - - | 1 0.02 11 | - - - - - 5 - - | 5 0.10 14 | - - - - - - 1 - | 1 0.02 x | - - - - - - - 1 | 1 0.02 ------+-----------------------------------------+------------ # | 38 2 1 1 1 5 1 1 | 50 prob | 0.76 0.04 0.02 0.02 0.02 0.10 0.02 0.02 | 1.00 sample size: 50 number of agreements: 50 ( 1.00) number of disagreements: 0 ( 0.00) Cohen's kappa = 1.000 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.c 4 6 7 11 11.s 14 14.a u x | # prob ------+-------------------------------------------------------------+------------ 1 | 37 1 - - 1 - - - - - - - | 39 0.78 1.a | - - - - - - - - - - - - | 0 0.00 1.c | 1 - - - - - - - - - - - | 1 0.02 4 | - - - 1 - - - - - - - - | 1 0.02 6 | - - - - - - - - - - - - | 0 0.00 7 | - - - - - 1 - - - - - - | 1 0.02 11 | - - - - - - 3 - - - - - | 3 0.06 11.s | - - - - - - 2 - - - - - | 2 0.04 14 | - - - - - - - - 1 - - - | 1 0.02 14.a | - 1 - - - - - - - - - - | 1 0.02 u | - - - - - - - - - - - 1 | 1 0.02 x | - - - - - - - - - - - - | 0 0.00 ------+-------------------------------------------------------------+------------ # | 38 2 0 1 1 1 5 0 1 0 0 1 | 50 prob | 0.76 0.04 0.00 0.02 0.02 0.02 0.10 0.00 0.02 0.00 0.00 0.02 | 1 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.650 -------------------------------------------------------------------------------------------- Fleiss' kappa = 0.798 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK say 50 14 4 0.899 48 0.960 AV_JT say 50 14 4 0.688 44 0.880 AV_SC say 50 14 4 0.899 48 0.960 EK_JT say 50 14 4 0.650 43 0.860 EK_SC say 50 14 4 1.000 50 1.000 JT_SC say 50 14 4 0.650 43 0.860 ----------------------------------------------------------- say AV EK JT SC AV - 0.899 0.688 0.899 EK - - 0.650 1.000 JT - - - 0.650 SC - - - -