--------------------------------------------------------------------------------------------- corpora: BNC, verb: claim, number of patterns: 9, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.c 2 2.a 3 3.a 4 7 u | # prob ------+----------------------------------------------+------------ 1 | 25 - - - 1 - - - 2 | 28 0.56 1.c | 1 - - - - - - - - | 1 0.02 2 | - - 8 - - - - - - | 8 0.16 2.a | - - 1 - - - - - - | 1 0.02 3 | - - - - 5 - - - - | 5 0.10 3.a | - - - - - 1 - - - | 1 0.02 4 | - - - - - - 2 - - | 2 0.04 7 | - - - - - - - 2 - | 2 0.04 u | 1 - - - - 1 - - - | 2 0.04 ------+----------------------------------------------+------------ # | 27 0 9 0 6 2 2 2 2 | 50 prob | 0.54 0.00 0.18 0.00 0.12 0.04 0.04 0.04 0.04 | 1 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.785 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.c 2 2.a 2.c 3 3.a 4 7 8 8.a u | # prob ------+------------------------------------------------------------------+------------ 1 | 26 - - - - - 1 - - - 1 - - | 28 0.56 1.a | - - - - - - - - - - - - - | 0 0.00 1.c | 1 - - - - - - - - - - - - | 1 0.02 2 | - - - 7 - 1 - - - - - - - | 8 0.16 2.a | - - - 1 - - - - - - - - - | 1 0.02 2.c | - - - - - - - - - - - - - | 0 0.00 3 | - 1 - - - - 3 - - - 1 - - | 5 0.10 3.a | - - - - - - 1 - - - - - - | 1 0.02 4 | - - - - - - - - 2 - - - - | 2 0.04 7 | - - - - - - - - - 2 - - - | 2 0.04 8 | - - - - - - - - - - - - - | 0 0.00 8.a | - - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - - - - 1 1 | 2 0.04 ------+------------------------------------------------------------------+------------ # | 27 1 0 8 0 1 5 0 2 2 2 1 1 | 50 prob | 0.54 0.02 0.00 0.16 0.00 0.02 0.10 0.00 0.04 0.04 0.04 0.02 0.02 | 1.00 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.726 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 1.c 2 2.a 3 3.a 4 7 u | # prob ------+---------------------------------------------------+------------ 1 | 26 1 - - - 1 - - - - | 28 0.56 1.a | - - - - - - - - - - | 0 0.00 1.c | 1 - - - - - - - - - | 1 0.02 2 | - - - 8 - - - - - - | 8 0.16 2.a | - - - 1 - - - - - - | 1 0.02 3 | - - - - - 5 - - - - | 5 0.10 3.a | - - - - - 1 - - - - | 1 0.02 4 | - - - - - - - 2 - - | 2 0.04 7 | - - - - - - - - 2 - | 2 0.04 u | 1 - - - - 1 - - - - | 2 0.04 ------+---------------------------------------------------+------------ # | 28 1 0 9 0 8 0 2 2 0 | 50 prob | 0.56 0.02 0.00 0.18 0.00 0.16 0.00 0.04 0.04 0.00 | 1 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.781 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 2 2.c 3 3.a 4 7 8 8.a u | # prob ------+--------------------------------------------------------+------------ 1 | 25 - - - - - - - 1 - 1 | 27 0.54 1.a | - - - - - - - - - - - | 0 0.00 2 | - - 8 1 - - - - - - - | 9 0.18 2.c | - - - - - - - - - - - | 0 0.00 3 | - 1 - - 4 - - - 1 - - | 6 0.12 3.a | - - - - 1 - - - - 1 - | 2 0.04 4 | - - - - - - 2 - - - - | 2 0.04 7 | - - - - - - - 2 - - - | 2 0.04 8 | - - - - - - - - - - - | 0 0.00 8.a | - - - - - - - - - - - | 0 0.00 u | 2 - - - - - - - - - - | 2 0.04 ------+--------------------------------------------------------+------------ # | 27 1 8 1 5 0 2 2 2 1 1 | 50 prob | 0.54 0.02 0.16 0.02 0.10 0.00 0.04 0.04 0.04 0.02 0.02 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.729 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 2 3 3.a 4 7 u | # prob ------+-----------------------------------------+------------ 1 | 26 - - 1 - - - - | 27 0.54 1.a | - - - - - - - - | 0 0.00 2 | - - 9 - - - - - | 9 0.18 3 | 1 - - 5 - - - - | 6 0.12 3.a | - - - 2 - - - - | 2 0.04 4 | - - - - - 2 - - | 2 0.04 7 | - - - - - - 2 - | 2 0.04 u | 1 1 - - - - - - | 2 0.04 ------+-----------------------------------------+------------ # | 28 1 9 8 0 2 2 0 | 50 prob | 0.56 0.02 0.18 0.16 0.00 0.04 0.04 0.00 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.813 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 2 2.c 3 4 7 8 8.a u | # prob ------+---------------------------------------------------+------------ 1 | 26 1 - - - - - - - - | 27 0.54 1.a | - - - - 1 - - - - - | 1 0.02 2 | - - 8 - - - - - - - | 8 0.16 2.c | - - 1 - - - - - - - | 1 0.02 3 | 1 - - - 4 - - - - - | 5 0.10 4 | - - - - - 2 - - - - | 2 0.04 7 | - - - - - - 2 - - - | 2 0.04 8 | - - - - 2 - - - - - | 2 0.04 8.a | - - - - 1 - - - - - | 1 0.02 u | 1 - - - - - - - - - | 1 0.02 ------+---------------------------------------------------+------------ # | 28 1 9 0 8 2 2 0 0 0 | 50 prob | 0.56 0.02 0.18 0.00 0.16 0.04 0.04 0.00 0.00 0.00 | 1 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.754 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.764 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK claim 50 9 4 0.785 43 0.860 AV_JT claim 50 9 4 0.726 41 0.820 AV_SC claim 50 9 4 0.781 43 0.860 EK_JT claim 50 9 4 0.729 41 0.820 EK_SC claim 50 9 4 0.813 44 0.880 JT_SC claim 50 9 4 0.754 42 0.840 ----------------------------------------------------------- claim AV EK JT SC AV - 0.785 0.726 0.781 EK - - 0.729 0.813 JT - - - 0.754 SC - - - -