-------------------------------------------------------------------------------------------- corpora: BNC, verb: hail, number of patterns: 9, sample size: 50, annotators(4): AV EK JT SC -------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.s 4 5 8 u x | # prob ------+------------------------------------+------------ 1 | 30 - 4 - - - - | 34 0.68 1.s | 3 - - - - - - | 3 0.06 4 | - - 1 - - - - | 1 0.02 5 | - - - 3 - - - | 3 0.06 8 | - - - - 3 - - | 3 0.06 u | 2 - - - - 1 1 | 4 0.08 x | - - - - - - 2 | 2 0.04 ------+------------------------------------+------------ # | 35 0 5 3 3 1 3 | 50 prob | 0.70 0.00 0.10 0.06 0.06 0.02 0.06 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.608 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.s 4 5 8 u x | # prob ------+-----------------------------------------+------------ 1 | 33 1 - - - - - - | 34 0.68 1.a | - - - - - - - - | 0 0.00 1.s | - - 3 - - - - - | 3 0.06 4 | - - - 1 - - - - | 1 0.02 5 | - - - - 3 - - - | 3 0.06 8 | - - - - - 3 - - | 3 0.06 u | - - 2 - - - 1 1 | 4 0.08 x | - - - - - - - 2 | 2 0.04 ------+-----------------------------------------+------------ # | 33 1 5 1 3 3 1 3 | 50 prob | 0.66 0.02 0.10 0.02 0.06 0.06 0.02 0.06 | 1 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.850 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.s 4 5 8 u x | # prob ------+------------------------------------+------------ 1 | 34 - - - - - - | 34 0.68 1.s | 3 - - - - - - | 3 0.06 4 | - - 1 - - - - | 1 0.02 5 | - - - 3 - - - | 3 0.06 8 | - - - - 3 - - | 3 0.06 u | 2 - - - - 1 1 | 4 0.08 x | - - - - - - 2 | 2 0.04 ------+------------------------------------+------------ # | 39 0 1 3 3 1 3 | 50 prob | 0.78 0.00 0.02 0.06 0.06 0.02 0.06 | 1 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.738 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.s 4 5 8 u x | # prob ------+-----------------------------------------+------------ 1 | 29 1 5 - - - - - | 35 0.70 1.a | - - - - - - - - | 0 0.00 1.s | - - - - - - - - | 0 0.00 4 | 4 - - 1 - - - - | 5 0.10 5 | - - - - 3 - - - | 3 0.06 8 | - - - - - 3 - - | 3 0.06 u | - - - - - - 1 - | 1 0.02 x | - - - - - - - 3 | 3 0.06 ------+-----------------------------------------+------------ # | 33 1 5 1 3 3 1 3 | 50 prob | 0.66 0.02 0.10 0.02 0.06 0.06 0.02 0.06 | 1 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.619 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 4 5 8 u x | # prob ------+-------------------------------+------------ 1 | 35 - - - - - | 35 0.70 4 | 4 1 - - - - | 5 0.10 5 | - - 3 - - - | 3 0.06 8 | - - - 3 - - | 3 0.06 u | - - - - 1 - | 1 0.02 x | - - - - - 3 | 3 0.06 ------+-------------------------------+------------ # | 39 1 3 3 1 3 | 50 prob | 0.78 0.02 0.06 0.06 0.02 0.06 | 1 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.819 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.s 4 5 8 u x | # prob ------+-----------------------------------------+------------ 1 | 33 - - - - - - - | 33 0.66 1.a | 1 - - - - - - - | 1 0.02 1.s | 5 - - - - - - - | 5 0.10 4 | - - - 1 - - - - | 1 0.02 5 | - - - - 3 - - - | 3 0.06 8 | - - - - - 3 - - | 3 0.06 u | - - - - - - 1 - | 1 0.02 x | - - - - - - - 3 | 3 0.06 ------+-----------------------------------------+------------ # | 39 0 0 1 3 3 1 3 | 50 prob | 0.78 0.00 0.00 0.02 0.06 0.06 0.02 0.06 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.747 -------------------------------------------------------------------------------------------- Fleiss' kappa = 0.727 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK hail 50 9 4 0.608 40 0.800 AV_JT hail 50 9 4 0.850 46 0.920 AV_SC hail 50 9 4 0.738 44 0.880 EK_JT hail 50 9 4 0.619 40 0.800 EK_SC hail 50 9 4 0.819 46 0.920 JT_SC hail 50 9 4 0.747 44 0.880 ----------------------------------------------------------- hail AV EK JT SC AV - 0.608 0.850 0.738 EK - - 0.619 0.819 JT - - - 0.747 SC - - - -