---------------------------------------------------------------------------------------------- corpora: BNC, verb: access, number of patterns: 8, sample size: 50, annotators(4): AV EK JT SC ---------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 1.s 2 3 3.f 4 u x | # prob ------+----------------------------------------------+------------ 1 | 24 - - 1 - - - - - | 25 0.50 1.a | - - - - - - - - - | 0 0.00 1.s | - - 1 - - - - - - | 1 0.02 2 | - - - 2 - - - - - | 2 0.04 3 | - - - - 1 - - - 1 | 2 0.04 3.f | - 1 - - - - - - - | 1 0.02 4 | - - - - - - 2 - - | 2 0.04 u | - 1 - - - - - - - | 1 0.02 x | 3 1 - - - 1 - - 11 | 16 0.32 ------+----------------------------------------------+------------ # | 27 3 1 3 1 1 2 0 12 | 50 prob | 0.54 0.06 0.02 0.06 0.02 0.02 0.04 0.00 0.24 | 1.00 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.722 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.s 2 2.a 3 3.f 4 5.a 6 u x | # prob ------+-------------------------------------------------------------+------------ 1 | 21 3 - - - - - 1 - - - - | 25 0.50 1.a | - - - - - - - - - - - - | 0 0.00 1.s | 1 - - - - - - - - - - - | 1 0.02 2 | 1 - - - 1 - - - - - - - | 2 0.04 2.a | - - - - - - - - - - - - | 0 0.00 3 | - - - - - 1 - - - 1 - - | 2 0.04 3.f | - - - - - - 1 - - - - - | 1 0.02 4 | - - - - - 1 - 1 - - - - | 2 0.04 5.a | - - - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - 1 - - - | 1 0.02 x | 1 1 - 1 1 1 - - - - - 11 | 16 0.32 ------+-------------------------------------------------------------+------------ # | 24 4 0 1 2 3 1 2 1 1 0 11 | 50 prob | 0.48 0.08 0.00 0.02 0.04 0.06 0.02 0.04 0.02 0.02 0.00 0.22 | 1 sample size: 50 number of agreements: 35 ( 0.70) number of disagreements: 15 ( 0.30) Cohen's kappa = 0.562 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.s 2 3 3.f 4 4.a u x | # prob ------+----------------------------------------------+------------ 1 | 25 - - - - - - - - | 25 0.50 1.s | 1 - - - - - - - - | 1 0.02 2 | 1 - 1 - - - - - - | 2 0.04 3 | - - - 1 - - - - 1 | 2 0.04 3.f | - - - - 1 - - - - | 1 0.02 4 | - - - - - 1 1 - - | 2 0.04 4.a | - - - - - - - - - | 0 0.00 u | - - - - - - - 1 - | 1 0.02 x | 2 - 1 - - - - 1 12 | 16 0.32 ------+----------------------------------------------+------------ # | 29 0 2 1 1 1 1 2 13 | 50 prob | 0.58 0.00 0.04 0.02 0.02 0.02 0.02 0.04 0.26 | 1 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.743 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.s 2 2.a 3 3.f 4 5.a 6 x | # prob ------+--------------------------------------------------------+------------ 1 | 20 4 - - - - - 1 - - 2 | 27 0.54 1.a | - - - - 1 - 1 - 1 - - | 3 0.06 1.s | 1 - - - - - - - - - - | 1 0.02 2 | 2 - - - 1 - - - - - - | 3 0.06 2.a | - - - - - - - - - - - | 0 0.00 3 | - - - - - - - - - 1 - | 1 0.02 3.f | - - - - - - - - - - 1 | 1 0.02 4 | - - - - - 1 - 1 - - - | 2 0.04 5.a | - - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - - - | 0 0.00 x | 1 - - 1 - 2 - - - - 8 | 12 0.24 ------+--------------------------------------------------------+------------ # | 24 4 0 1 2 3 1 2 1 1 11 | 50 prob | 0.48 0.08 0.00 0.02 0.04 0.06 0.02 0.04 0.02 0.02 0.22 | 1 sample size: 50 number of agreements: 29 ( 0.58) number of disagreements: 21 ( 0.42) Cohen's kappa = 0.381 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 1.s 2 3 3.f 4 4.a u x | # prob ------+---------------------------------------------------+------------ 1 | 26 - - - - - - - - 1 | 27 0.54 1.a | - - - 1 - 1 - - 1 - | 3 0.06 1.s | 1 - - - - - - - - - | 1 0.02 2 | 2 - - 1 - - - - - - | 3 0.06 3 | - - - - 1 - - - - - | 1 0.02 3.f | - - - - - - - - 1 - | 1 0.02 4 | - - - - - - 1 1 - - | 2 0.04 4.a | - - - - - - - - - - | 0 0.00 u | - - - - - - - - - - | 0 0.00 x | - - - - - - - - - 12 | 12 0.24 ------+---------------------------------------------------+------------ # | 29 0 0 2 1 1 1 1 2 13 | 50 prob | 0.58 0.00 0.00 0.04 0.02 0.02 0.02 0.02 0.04 0.26 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.710 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 2 2.a 3 3.f 4 4.a 5.a 6 u x | # prob ------+-------------------------------------------------------------+------------ 1 | 23 - - - - - - - - - - 1 | 24 0.48 1.a | 4 - - - - - - - - - - - | 4 0.08 2 | - - - - - - - - - - - 1 | 1 0.02 2.a | - - 2 - - - - - - - - - | 2 0.04 3 | - - - - - - - 1 - - - 2 | 3 0.06 3.f | - - - - - 1 - - - - - - | 1 0.02 4 | 1 - - - - - 1 - - - - - | 2 0.04 4.a | - - - - - - - - - - - - | 0 0.00 5.a | - - - - - - - - - - 1 - | 1 0.02 6 | - - - - 1 - - - - - - - | 1 0.02 u | - - - - - - - - - - - - | 0 0.00 x | 1 - - - - - - - - - 1 9 | 11 0.22 ------+-------------------------------------------------------------+------------ # | 29 0 2 0 1 1 1 1 0 0 2 13 | 50 prob | 0.58 0.00 0.04 0.00 0.02 0.02 0.02 0.02 0.00 0.00 0.04 0.26 | 1.00 sample size: 50 number of agreements: 34 ( 0.68) number of disagreements: 16 ( 0.32) Cohen's kappa = 0.516 ---------------------------------------------------------------------------------------------- Fleiss' kappa = 0.600 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK access 50 8 4 0.722 41 0.820 AV_JT access 50 8 4 0.562 35 0.700 AV_SC access 50 8 4 0.743 42 0.840 EK_JT access 50 8 4 0.381 29 0.580 EK_SC access 50 8 4 0.710 41 0.820 JT_SC access 50 8 4 0.516 34 0.680 ----------------------------------------------------------- access AV EK JT SC AV - 0.722 0.562 0.743 EK - - 0.381 0.710 JT - - - 0.516 SC - - - -