---------------------------------------------------------------------------------------------- corpora: BNC, verb: arrive, number of patterns: 6, sample size: 50, annotators(4): AV EK JT SC ---------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 1.f 2 3 4 6 | # prob ------+------------------------------------+------------ 1 | 29 - 1 - - - - | 30 0.60 1.a | - 1 - - - - - | 1 0.02 1.f | - - - - - - - | 0 0.00 2 | - - - 8 - - - | 8 0.16 3 | - - - - 8 - - | 8 0.16 4 | - - - - - 1 - | 1 0.02 6 | - - - - - - 2 | 2 0.04 ------+------------------------------------+------------ # | 29 1 1 8 8 1 2 | 50 prob | 0.58 0.02 0.02 0.16 0.16 0.02 0.04 | 1.00 sample size: 50 number of agreements: 49 ( 0.98) number of disagreements: 1 ( 0.02) Cohen's kappa = 0.967 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.f 2 2.a 3 3.a 4 5.a 6 | # prob ------+---------------------------------------------------+------------ 1 | 28 1 1 - - - - - - - | 30 0.60 1.a | - - - - - - - - 1 - | 1 0.02 1.f | - - - - - - - - - - | 0 0.00 2 | - 1 - 6 1 - - - - - | 8 0.16 2.a | - - - - - - - - - - | 0 0.00 3 | 1 - - - - 3 4 - - - | 8 0.16 3.a | - - - - - - - - - - | 0 0.00 4 | - - - - - - - 1 - - | 1 0.02 5.a | - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - 2 | 2 0.04 ------+---------------------------------------------------+------------ # | 29 2 1 6 1 3 4 1 1 2 | 50 prob | 0.58 0.04 0.02 0.12 0.02 0.06 0.08 0.02 0.02 0.04 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.678 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 1.f 2 3 3.a 4 5.a 6 u | # prob ------+---------------------------------------------------+------------ 1 | 29 - 1 - - - - - - - | 30 0.60 1.a | - - - - - - - 1 - - | 1 0.02 1.f | - - - - - - - - - - | 0 0.00 2 | - - 1 6 - - - - - 1 | 8 0.16 3 | - - - - 7 1 - - - - | 8 0.16 3.a | - - - - - - - - - - | 0 0.00 4 | - - - - - - 1 - - - | 1 0.02 5.a | - - - - - - - - - - | 0 0.00 6 | - - - - - - - - 2 - | 2 0.04 u | - - - - - - - - - - | 0 0.00 ------+---------------------------------------------------+------------ # | 29 0 2 6 7 1 1 1 2 1 | 50 prob | 0.58 0.00 0.04 0.12 0.14 0.02 0.02 0.02 0.04 0.02 | 1.00 sample size: 50 number of agreements: 45 ( 0.90) number of disagreements: 5 ( 0.10) Cohen's kappa = 0.836 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.f 2 2.a 3 3.a 4 5.a 6 | # prob ------+---------------------------------------------------+------------ 1 | 28 1 - - - - - - - - | 29 0.58 1.a | - - - - - - - - 1 - | 1 0.02 1.f | - - 1 - - - - - - - | 1 0.02 2 | - 1 - 6 1 - - - - - | 8 0.16 2.a | - - - - - - - - - - | 0 0.00 3 | 1 - - - - 3 4 - - - | 8 0.16 3.a | - - - - - - - - - - | 0 0.00 4 | - - - - - - - 1 - - | 1 0.02 5.a | - - - - - - - - - - | 0 0.00 6 | - - - - - - - - - 2 | 2 0.04 ------+---------------------------------------------------+------------ # | 29 2 1 6 1 3 4 1 1 2 | 50 prob | 0.58 0.04 0.02 0.12 0.02 0.06 0.08 0.02 0.02 0.04 | 1.00 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.715 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 1.f 2 3 3.a 4 5.a 6 u | # prob ------+---------------------------------------------------+------------ 1 | 29 - - - - - - - - - | 29 0.58 1.a | - - - - - - - 1 - - | 1 0.02 1.f | - - 1 - - - - - - - | 1 0.02 2 | - - 1 6 - - - - - 1 | 8 0.16 3 | - - - - 7 1 - - - - | 8 0.16 3.a | - - - - - - - - - - | 0 0.00 4 | - - - - - - 1 - - - | 1 0.02 5.a | - - - - - - - - - - | 0 0.00 6 | - - - - - - - - 2 - | 2 0.04 u | - - - - - - - - - - | 0 0.00 ------+---------------------------------------------------+------------ # | 29 0 2 6 7 1 1 1 2 1 | 50 prob | 0.58 0.00 0.04 0.12 0.14 0.02 0.02 0.02 0.04 0.02 | 1.00 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.871 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.f 2 2.a 3 3.a 4 5.a 6 u | # prob ------+--------------------------------------------------------+------------ 1 | 28 - - - - 1 - - - - - | 29 0.58 1.a | 1 - 1 - - - - - - - - | 2 0.04 1.f | - - 1 - - - - - - - - | 1 0.02 2 | - - - 6 - - - - - - - | 6 0.12 2.a | - - - - - - - - - - 1 | 1 0.02 3 | - - - - - 3 - - - - - | 3 0.06 3.a | - - - - - 3 1 - - - - | 4 0.08 4 | - - - - - - - 1 - - - | 1 0.02 5.a | - - - - - - - - 1 - - | 1 0.02 6 | - - - - - - - - - 2 - | 2 0.04 u | - - - - - - - - - - - | 0 0.00 ------+--------------------------------------------------------+------------ # | 29 0 2 6 0 7 1 1 1 2 1 | 50 prob | 0.58 0.00 0.04 0.12 0.00 0.14 0.02 0.02 0.02 0.04 0.02 | 1.00 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.780 ---------------------------------------------------------------------------------------------- Fleiss' kappa = 0.806 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK arrive 50 6 4 0.967 49 0.980 AV_JT arrive 50 6 4 0.678 40 0.800 AV_SC arrive 50 6 4 0.836 45 0.900 EK_JT arrive 50 6 4 0.715 41 0.820 EK_SC arrive 50 6 4 0.871 46 0.920 JT_SC arrive 50 6 4 0.780 43 0.860 ----------------------------------------------------------- arrive AV EK JT SC AV - 0.967 0.678 0.836 EK - - 0.715 0.871 JT - - - 0.780 SC - - - -