---------------------------------------------------------------------------------------------- corpora: BNC, verb: enlist, number of patterns: 5, sample size: 47, annotators(4): AV EK JT SC ---------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.s 2 2.a 3 3.a 4 5 | # prob ------+-----------------------------------------+------------ 1 | 11 - - - 1 - - - | 12 0.26 1.s | - - - - - - - - | 0 0.00 2 | - - 16 - - - - - | 16 0.34 2.a | - - 2 1 - - - - | 3 0.06 3 | - 1 - - 6 1 - - | 8 0.17 3.a | - - - - 1 2 - - | 3 0.06 4 | - - - - - - 2 - | 2 0.04 5 | - - - - - - - 3 | 3 0.06 ------+-----------------------------------------+------------ # | 11 1 18 1 8 3 2 3 | 47 prob | 0.23 0.02 0.38 0.02 0.17 0.06 0.04 0.06 | 1 sample size: 47 number of agreements: 41 ( 0.87) number of disagreements: 6 ( 0.13) Cohen's kappa = 0.834 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.s 2 2.a 3 3.a 4 4.a 5 5.a x | # prob ------+-------------------------------------------------------------+------------ 1 | 7 3 1 - - - - - - - 1 - | 12 0.26 1.a | - - - - - - - - - - - - | 0 0.00 1.s | - - - - - - - - - - - - | 0 0.00 2 | - - - 15 1 - - - - - - - | 16 0.34 2.a | - - - 1 - - - - 1 - 1 - | 3 0.06 3 | - - - - - 5 2 - - - 1 - | 8 0.17 3.a | - - - - - - 2 - - 1 - - | 3 0.06 4 | - - - - - - - 1 1 - - - | 2 0.04 4.a | - - - - - - - - - - - - | 0 0.00 5 | - - - - - - - - - 1 - 2 | 3 0.06 5.a | - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - - - | 0 0.00 ------+-------------------------------------------------------------+------------ # | 7 3 1 16 1 5 4 1 2 2 3 2 | 47 prob | 0.15 0.06 0.02 0.34 0.02 0.11 0.09 0.02 0.04 0.04 0.06 0.04 | 1.00 sample size: 47 number of agreements: 31 ( 0.66) number of disagreements: 16 ( 0.34) Cohen's kappa = 0.584 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 2 2.a 3 3.a 3.c 4 5 | # prob ------+-----------------------------------------+------------ 1 | 11 - - 1 - - - - | 12 0.26 2 | - 16 - - - - - - | 16 0.34 2.a | - 2 - - - - 1 - | 3 0.06 3 | - - - 7 - 1 - - | 8 0.17 3.a | - - - 1 - 2 - - | 3 0.06 3.c | - - - - - - - - | 0 0.00 4 | - - - - - - 2 - | 2 0.04 5 | - - - 1 - - - 2 | 3 0.06 ------+-----------------------------------------+------------ # | 11 18 0 10 0 3 3 2 | 47 prob | 0.23 0.38 0.00 0.21 0.00 0.06 0.06 0.04 | 1.00 sample size: 47 number of agreements: 38 ( 0.81) number of disagreements: 9 ( 0.19) Cohen's kappa = 0.751 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.s 2 2.a 3 3.a 4 4.a 5 5.a x | # prob ------+-------------------------------------------------------------+------------ 1 | 7 3 1 - - - - - - - - - | 11 0.23 1.a | - - - - - - - - - - - - | 0 0.00 1.s | - - - - - - 1 - - - - - | 1 0.02 2 | - - - 16 1 - - - - - 1 - | 18 0.38 2.a | - - - - - - - - 1 - - - | 1 0.02 3 | - - - - - 5 - - - 1 2 - | 8 0.17 3.a | - - - - - - 3 - - - - - | 3 0.06 4 | - - - - - - - 1 1 - - - | 2 0.04 4.a | - - - - - - - - - - - - | 0 0.00 5 | - - - - - - - - - 1 - 2 | 3 0.06 5.a | - - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - - - - | 0 0.00 ------+-------------------------------------------------------------+------------ # | 7 3 1 16 1 5 4 1 2 2 3 2 | 47 prob | 0.15 0.06 0.02 0.34 0.02 0.11 0.09 0.02 0.04 0.04 0.06 0.04 | 1 sample size: 47 number of agreements: 33 ( 0.70) number of disagreements: 14 ( 0.30) Cohen's kappa = 0.631 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.s 2 2.a 3 3.a 3.c 4 5 | # prob ------+----------------------------------------------+------------ 1 | 11 - - - - - - - - | 11 0.23 1.s | - - - - - - 1 - - | 1 0.02 2 | - - 18 - - - - - - | 18 0.38 2.a | - - - - - - - 1 - | 1 0.02 3 | - - - - 8 - - - - | 8 0.17 3.a | - - - - 1 - 2 - - | 3 0.06 3.c | - - - - - - - - - | 0 0.00 4 | - - - - - - - 2 - | 2 0.04 5 | - - - - 1 - - - 2 | 3 0.06 ------+----------------------------------------------+------------ # | 11 0 18 0 10 0 3 3 2 | 47 prob | 0.23 0.00 0.38 0.00 0.21 0.00 0.06 0.06 0.04 | 1 sample size: 47 number of agreements: 41 ( 0.87) number of disagreements: 6 ( 0.13) Cohen's kappa = 0.831 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.s 2 2.a 3 3.a 3.c 4 4.a 5 5.a x | # prob ------+------------------------------------------------------------------+------------ 1 | 7 - - - - - - - - - - - - | 7 0.15 1.a | 3 - - - - - - - - - - - - | 3 0.06 1.s | 1 - - - - - - - - - - - - | 1 0.02 2 | - - - 16 - - - - - - - - - | 16 0.34 2.a | - - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - 5 - - - - - - - | 5 0.11 3.a | - - - - - 1 - 3 - - - - - | 4 0.09 3.c | - - - - - - - - - - - - - | 0 0.00 4 | - - - - - - - - 1 - - - - | 1 0.02 4.a | - - - - - - - - 2 - - - - | 2 0.04 5 | - - - - - 1 - - - - 1 - - | 2 0.04 5.a | - - - 1 - 2 - - - - - - - | 3 0.06 x | - - - - - 1 - - - - 1 - - | 2 0.04 ------+------------------------------------------------------------------+------------ # | 11 0 0 18 0 10 0 3 3 0 2 0 0 | 47 prob | 0.23 0.00 0.00 0.38 0.00 0.21 0.00 0.06 0.06 0.00 0.04 0.00 0.00 | 1.00 sample size: 47 number of agreements: 30 ( 0.64) number of disagreements: 17 ( 0.36) Cohen's kappa = 0.553 ---------------------------------------------------------------------------------------------- Fleiss' kappa = 0.693 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK enlist 47 5 4 0.834 41 0.872 AV_JT enlist 47 5 4 0.584 31 0.660 AV_SC enlist 47 5 4 0.751 38 0.809 EK_JT enlist 47 5 4 0.631 33 0.702 EK_SC enlist 47 5 4 0.831 41 0.872 JT_SC enlist 47 5 4 0.553 30 0.638 ----------------------------------------------------------- enlist AV EK JT SC AV - 0.834 0.584 0.751 EK - - 0.631 0.831 JT - - - 0.553 SC - - - -