----------------------------------------------------------------------------------------------- corpora: BNC, verb: furnish, number of patterns: 7, sample size: 50, annotators(4): AV EK JT SC ----------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 2 3 4 5 5.a 6 u | # prob ------+----------------------------------------------+------------ 1 | 5 - - - - - - - - | 5 0.10 1.a | - 1 - - - - - - - | 1 0.02 2 | - - 7 - - - - - - | 7 0.14 3 | - - - 2 - - - - - | 2 0.04 4 | - - - - 18 - - - 2 | 20 0.40 5 | 10 - - - - 3 1 - - | 14 0.28 5.a | - - - - - - - - - | 0 0.00 6 | - - - - - - - 1 - | 1 0.02 u | - - - - - - - - - | 0 0.00 ------+----------------------------------------------+------------ # | 15 1 7 2 18 3 1 1 2 | 50 prob | 0.30 0.02 0.14 0.04 0.36 0.06 0.02 0.02 0.04 | 1.00 sample size: 50 number of agreements: 37 ( 0.74) number of disagreements: 13 ( 0.26) Cohen's kappa = 0.670 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 2 3 4 5 6 | # prob ------+------------------------------------+------------ 1 | 3 - - - 2 - - | 5 0.10 1.a | - 1 - - - - - | 1 0.02 2 | - - 7 - - - - | 7 0.14 3 | - - - 2 - - - | 2 0.04 4 | - - - - 20 - - | 20 0.40 5 | - - - - - 14 - | 14 0.28 6 | - - - - - - 1 | 1 0.02 ------+------------------------------------+------------ # | 3 1 7 2 22 14 1 | 50 prob | 0.06 0.02 0.14 0.04 0.44 0.28 0.02 | 1.00 sample size: 50 number of agreements: 48 ( 0.96) number of disagreements: 2 ( 0.04) Cohen's kappa = 0.944 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 2 3 4 5 6 u | # prob ------+-----------------------------------------+------------ 1 | 4 - - - - - - 1 | 5 0.10 1.a | 1 - - - - - - - | 1 0.02 2 | - - 7 - - - - - | 7 0.14 3 | - - - 2 - - - - | 2 0.04 4 | - - - - 20 - - - | 20 0.40 5 | 1 - - - - 13 - - | 14 0.28 6 | - - - - - - 1 - | 1 0.02 u | - - - - - - - - | 0 0.00 ------+-----------------------------------------+------------ # | 6 0 7 2 20 13 1 1 | 50 prob | 0.12 0.00 0.14 0.04 0.40 0.26 0.02 0.02 | 1.00 sample size: 50 number of agreements: 47 ( 0.94) number of disagreements: 3 ( 0.06) Cohen's kappa = 0.918 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 2 3 4 5 5.a 6 u | # prob ------+----------------------------------------------+------------ 1 | 3 - - - 2 10 - - - | 15 0.30 1.a | - 1 - - - - - - - | 1 0.02 2 | - - 7 - - - - - - | 7 0.14 3 | - - - 2 - - - - - | 2 0.04 4 | - - - - 18 - - - - | 18 0.36 5 | - - - - - 3 - - - | 3 0.06 5.a | - - - - - 1 - - - | 1 0.02 6 | - - - - - - - 1 - | 1 0.02 u | - - - - 2 - - - - | 2 0.04 ------+----------------------------------------------+------------ # | 3 1 7 2 22 14 0 1 0 | 50 prob | 0.06 0.02 0.14 0.04 0.44 0.28 0.00 0.02 0.00 | 1.00 sample size: 50 number of agreements: 35 ( 0.70) number of disagreements: 15 ( 0.30) Cohen's kappa = 0.618 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 2 3 4 5 5.a 6 u | # prob ------+----------------------------------------------+------------ 1 | 5 - - - - 9 - - 1 | 15 0.30 1.a | 1 - - - - - - - - | 1 0.02 2 | - - 7 - - - - - - | 7 0.14 3 | - - - 2 - - - - - | 2 0.04 4 | - - - - 18 - - - - | 18 0.36 5 | - - - - - 3 - - - | 3 0.06 5.a | - - - - - 1 - - - | 1 0.02 6 | - - - - - - - 1 - | 1 0.02 u | - - - - 2 - - - - | 2 0.04 ------+----------------------------------------------+------------ # | 6 0 7 2 20 13 0 1 1 | 50 prob | 0.12 0.00 0.14 0.04 0.40 0.26 0.00 0.02 0.02 | 1.00 sample size: 50 number of agreements: 36 ( 0.72) number of disagreements: 14 ( 0.28) Cohen's kappa = 0.642 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 2 3 4 5 6 u | # prob ------+-----------------------------------------+------------ 1 | 3 - - - - - - - | 3 0.06 1.a | 1 - - - - - - - | 1 0.02 2 | - - 7 - - - - - | 7 0.14 3 | - - - 2 - - - - | 2 0.04 4 | 1 - - - 20 - - 1 | 22 0.44 5 | 1 - - - - 13 - - | 14 0.28 6 | - - - - - - 1 - | 1 0.02 u | - - - - - - - - | 0 0.00 ------+-----------------------------------------+------------ # | 6 0 7 2 20 13 1 1 | 50 prob | 0.12 0.00 0.14 0.04 0.40 0.26 0.02 0.02 | 1.00 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.889 ----------------------------------------------------------------------------------------------- Fleiss' kappa = 0.773 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK furnish 50 7 4 0.670 37 0.740 AV_JT furnish 50 7 4 0.944 48 0.960 AV_SC furnish 50 7 4 0.918 47 0.940 EK_JT furnish 50 7 4 0.618 35 0.700 EK_SC furnish 50 7 4 0.642 36 0.720 JT_SC furnish 50 7 4 0.889 46 0.920 ----------------------------------------------------------- furnish AV EK JT SC AV - 0.670 0.944 0.918 EK - - 0.618 0.642 JT - - - 0.889 SC - - - -