--------------------------------------------------------------------------------------------- corpora: BNC, verb: cool, number of patterns: 14, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 2 2.a 4 4.a 6 7 10 13 17 x | # prob ------+--------------------------------------------------------+------------ 1 | 15 - - - - - - - - - - | 15 0.30 2 | - 6 - - - - - - - - - | 6 0.12 2.a | - 1 - - - - - - - - - | 1 0.02 4 | - - - - - - - - - - - | 0 0.00 4.a | - - - 2 1 - - - - - - | 3 0.06 6 | - - - - - 1 1 - - - - | 2 0.04 7 | - - - - - - 1 - - - - | 1 0.02 10 | - - - - - - - 2 - - - | 2 0.04 13 | 1 - - - - - - - 13 - - | 14 0.28 17 | - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - 4 | 4 0.08 ------+--------------------------------------------------------+------------ # | 16 7 0 2 1 1 2 2 13 2 4 | 50 prob | 0.32 0.14 0.00 0.04 0.02 0.02 0.04 0.04 0.26 0.04 0.08 | 1.00 sample size: 50 number of agreements: 45 ( 0.90) number of disagreements: 5 ( 0.10) Cohen's kappa = 0.875 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 2 2.a 2.f 4.a 6 7 7.a 10 11 13 17 x | # prob ------+------------------------------------------------------------------+------------ 1 | 13 - - - - - - - - - 2 - - | 15 0.30 2 | - 3 - - - - - - - 3 - - - | 6 0.12 2.a | - 1 - - - - - - - - - - - | 1 0.02 2.f | - - - - - - - - - - - - - | 0 0.00 4.a | - - - 1 1 - - 1 - - - - - | 3 0.06 6 | - - - - - 2 - - - - - - - | 2 0.04 7 | - - - - - - 1 - - - - - - | 1 0.02 7.a | - - - - - - - - - - - - - | 0 0.00 10 | - - - - - - - - 2 - - - - | 2 0.04 11 | - - - - - - - - - - - - - | 0 0.00 13 | - - - - - - - - - - 14 - - | 14 0.28 17 | - - - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - - - 4 | 4 0.08 ------+------------------------------------------------------------------+------------ # | 13 4 0 1 1 2 1 1 2 3 16 2 4 | 50 prob | 0.26 0.08 0.00 0.02 0.02 0.04 0.02 0.02 0.04 0.06 0.32 0.04 0.08 | 1.00 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.802 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 2 2.a 4.a 6 7 10 11 13 17 x | # prob ------+--------------------------------------------------------+------------ 1 | 15 - - - - - - - - - - | 15 0.30 2 | - 4 - - - - - 2 - - - | 6 0.12 2.a | - 1 - - - - - - - - - | 1 0.02 4.a | - - - 3 - - - - - - - | 3 0.06 6 | - - - - 2 - - - - - - | 2 0.04 7 | - - - - - 1 - - - - - | 1 0.02 10 | - - - - - - 2 - - - - | 2 0.04 11 | - - - - - - - - - - - | 0 0.00 13 | 1 - - - - - - - 13 - - | 14 0.28 17 | - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - 4 | 4 0.08 ------+--------------------------------------------------------+------------ # | 16 5 0 3 2 1 2 2 13 2 4 | 50 prob | 0.32 0.10 0.00 0.06 0.04 0.02 0.04 0.04 0.26 0.04 0.08 | 1.00 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.900 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 2 2.f 4 4.a 6 7 7.a 10 11 13 17 x | # prob ------+------------------------------------------------------------------+------------ 1 | 13 - - - - - - - - - 3 - - | 16 0.32 2 | - 4 - - - - - - - 3 - - - | 7 0.14 2.f | - - - - - - - - - - - - - | 0 0.00 4 | - - - - 1 - - 1 - - - - - | 2 0.04 4.a | - - 1 - - - - - - - - - - | 1 0.02 6 | - - - - - 1 - - - - - - - | 1 0.02 7 | - - - - - 1 1 - - - - - - | 2 0.04 7.a | - - - - - - - - - - - - - | 0 0.00 10 | - - - - - - - - 2 - - - - | 2 0.04 11 | - - - - - - - - - - - - - | 0 0.00 13 | - - - - - - - - - - 13 - - | 13 0.26 17 | - - - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - - - 4 | 4 0.08 ------+------------------------------------------------------------------+------------ # | 13 4 1 0 1 2 1 1 2 3 16 2 4 | 50 prob | 0.26 0.08 0.02 0.00 0.02 0.04 0.02 0.02 0.04 0.06 0.32 0.04 0.08 | 1.00 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.753 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 2 4 4.a 6 7 10 11 13 17 x | # prob ------+--------------------------------------------------------+------------ 1 | 16 - - - - - - - - - - | 16 0.32 2 | - 5 - - - - - 2 - - - | 7 0.14 4 | - - - 2 - - - - - - - | 2 0.04 4.a | - - - 1 - - - - - - - | 1 0.02 6 | - - - - 1 - - - - - - | 1 0.02 7 | - - - - 1 1 - - - - - | 2 0.04 10 | - - - - - - 2 - - - - | 2 0.04 11 | - - - - - - - - - - - | 0 0.00 13 | - - - - - - - - 13 - - | 13 0.26 17 | - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - 4 | 4 0.08 ------+--------------------------------------------------------+------------ # | 16 5 0 3 2 1 2 2 13 2 4 | 50 prob | 0.32 0.10 0.00 0.06 0.04 0.02 0.04 0.04 0.26 0.04 0.08 | 1.00 sample size: 50 number of agreements: 45 ( 0.90) number of disagreements: 5 ( 0.10) Cohen's kappa = 0.876 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 2 2.f 4.a 6 7 7.a 10 11 13 17 x | # prob ------+-------------------------------------------------------------+------------ 1 | 13 - - - - - - - - - - - | 13 0.26 2 | - 4 - - - - - - - - - - | 4 0.08 2.f | - - - 1 - - - - - - - - | 1 0.02 4.a | - - - 1 - - - - - - - - | 1 0.02 6 | - - - - 2 - - - - - - - | 2 0.04 7 | - - - - - 1 - - - - - - | 1 0.02 7.a | - - - 1 - - - - - - - - | 1 0.02 10 | - - - - - - - 2 - - - - | 2 0.04 11 | - 1 - - - - - - 2 - - - | 3 0.06 13 | 3 - - - - - - - - 13 - - | 16 0.32 17 | - - - - - - - - - - 2 - | 2 0.04 x | - - - - - - - - - - - 4 | 4 0.08 ------+-------------------------------------------------------------+------------ # | 16 5 0 3 2 1 0 2 2 13 2 4 | 50 prob | 0.32 0.10 0.00 0.06 0.04 0.02 0.00 0.04 0.04 0.26 0.04 0.08 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.852 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.843 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK cool 50 14 4 0.875 45 0.900 AV_JT cool 50 14 4 0.802 42 0.840 AV_SC cool 50 14 4 0.900 46 0.920 EK_JT cool 50 14 4 0.753 40 0.800 EK_SC cool 50 14 4 0.876 45 0.900 JT_SC cool 50 14 4 0.852 44 0.880 ----------------------------------------------------------- cool AV EK JT SC AV - 0.875 0.802 0.900 EK - - 0.753 0.876 JT - - - 0.852 SC - - - -