----------------------------------------------------------------------------------------------- corpora: BNC, verb: enlarge, number of patterns: 4, sample size: 50, annotators(4): AV EK JT SC ----------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 2 3 4 | # prob ------+--------------------------+------------ 1 | 27 - - - - | 27 0.54 1.a | 6 - - - - | 6 0.12 2 | - - 4 - - | 4 0.08 3 | - - - 4 - | 4 0.08 4 | 1 - - - 8 | 9 0.18 ------+--------------------------+------------ # | 34 0 4 4 8 | 50 prob | 0.68 0.00 0.08 0.08 0.16 | 1.00 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.763 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.c 2 3 4 x | # prob ------+------------------------------------+------------ 1 | 14 - 1 - - 12 - | 27 0.54 1.a | 3 2 - - - 1 - | 6 0.12 1.c | - - - - - - - | 0 0.00 2 | - - - 3 - - 1 | 4 0.08 3 | - - - - 4 - - | 4 0.08 4 | - - - 1 - 5 3 | 9 0.18 x | - - - - - - - | 0 0.00 ------+------------------------------------+------------ # | 17 2 1 4 4 18 4 | 50 prob | 0.34 0.04 0.02 0.08 0.08 0.36 0.08 | 1.00 sample size: 50 number of agreements: 28 ( 0.56) number of disagreements: 22 ( 0.44) Cohen's kappa = 0.401 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 2 3 4 | # prob ------+--------------------------+------------ 1 | 27 - - - - | 27 0.54 1.a | 6 - - - - | 6 0.12 2 | - - 4 - - | 4 0.08 3 | - - - 4 - | 4 0.08 4 | 3 - - - 6 | 9 0.18 ------+--------------------------+------------ # | 36 0 4 4 6 | 50 prob | 0.72 0.00 0.08 0.08 0.12 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.688 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.c 2 3 4 x | # prob ------+------------------------------------+------------ 1 | 17 2 1 - - 14 - | 34 0.68 1.a | - - - - - - - | 0 0.00 1.c | - - - - - - - | 0 0.00 2 | - - - 3 - - 1 | 4 0.08 3 | - - - - 4 - - | 4 0.08 4 | - - - 1 - 4 3 | 8 0.16 x | - - - - - - - | 0 0.00 ------+------------------------------------+------------ # | 17 2 1 4 4 18 4 | 50 prob | 0.34 0.04 0.02 0.08 0.08 0.36 0.08 | 1.00 sample size: 50 number of agreements: 28 ( 0.56) number of disagreements: 22 ( 0.44) Cohen's kappa = 0.370 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 2 3 4 | # prob ------+---------------------+------------ 1 | 33 - - 1 | 34 0.68 2 | - 4 - - | 4 0.08 3 | - - 4 - | 4 0.08 4 | 3 - - 5 | 8 0.16 ------+---------------------+------------ # | 36 4 4 6 | 50 prob | 0.72 0.08 0.08 0.12 | 1 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.833 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.c 2 3 4 x | # prob ------+------------------------------------+------------ 1 | 17 - - - - - - | 17 0.34 1.a | 2 - - - - - - | 2 0.04 1.c | 1 - - - - - - | 1 0.02 2 | 1 - - 3 - - - | 4 0.08 3 | - - - - 4 - - | 4 0.08 4 | 14 - - - - 4 - | 18 0.36 x | 1 - - 1 - 2 - | 4 0.08 ------+------------------------------------+------------ # | 36 0 0 4 4 6 0 | 50 prob | 0.72 0.00 0.00 0.08 0.08 0.12 0.00 | 1 sample size: 50 number of agreements: 28 ( 0.56) number of disagreements: 22 ( 0.44) Cohen's kappa = 0.371 ----------------------------------------------------------------------------------------------- Fleiss' kappa = 0.536 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK enlarge 50 4 4 0.763 43 0.860 AV_JT enlarge 50 4 4 0.401 28 0.560 AV_SC enlarge 50 4 4 0.688 41 0.820 EK_JT enlarge 50 4 4 0.370 28 0.560 EK_SC enlarge 50 4 4 0.833 46 0.920 JT_SC enlarge 50 4 4 0.371 28 0.560 ----------------------------------------------------------- enlarge AV EK JT SC AV - 0.763 0.401 0.688 EK - - 0.370 0.833 JT - - - 0.371 SC - - - -