--------------------------------------------------------------------------------------------- corpora: BNC, verb: smell, number of patterns: 9, sample size: 50, annotators(4): AV EK JT SC --------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 2 3 4 4.s 5 5.s 8 9 x | # prob ------+--------------------------------------------------------+------------ 1 | 15 - - - - - - - - - - | 15 0.30 1.a | 2 1 - - - - - - - - - | 3 0.06 2 | - - 2 - - - - - - - - | 2 0.04 3 | - - - 1 - - - - - - - | 1 0.02 4 | - - - - 5 - - - - - - | 5 0.10 4.s | - - - - - - - - - - - | 0 0.00 5 | - - - - 1 1 9 - - - - | 11 0.22 5.s | - - - - - - 1 - - - - | 1 0.02 8 | - - - - - - - - 1 - - | 1 0.02 9 | 1 - - - - - - - - 1 - | 2 0.04 x | - - - - - - 1 - - - 8 | 9 0.18 ------+--------------------------------------------------------+------------ # | 18 1 2 1 6 1 11 0 1 1 8 | 50 prob | 0.36 0.02 0.04 0.02 0.12 0.02 0.22 0.00 0.02 0.02 0.16 | 1.00 sample size: 50 number of agreements: 43 ( 0.86) number of disagreements: 7 ( 0.14) Cohen's kappa = 0.825 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 2 2.a 3 4 4.a 4.s 5 5.a 5.s 5.f 8 9 x | # prob ------+----------------------------------------------------------------------------+------------ 1 | 12 1 - - - - - - - - - - - 2 - | 15 0.30 1.a | 3 - - - - - - - - - - - - - - | 3 0.06 2 | - - 1 1 - - - - - - - - - - - | 2 0.04 2.a | - - - - - - - - - - - - - - - | 0 0.00 3 | - - - - 1 - - - - - - - - - - | 1 0.02 4 | - - - - - 1 2 1 1 - - - - - - | 5 0.10 4.a | - - - - - - - - - - - - - - - | 0 0.00 4.s | - - - - - - - - - - - - - - - | 0 0.00 5 | - - - - - 1 - 1 7 2 - - - - - | 11 0.22 5.a | - - - - - - - - - - - - - - - | 0 0.00 5.s | - - - - - - - - 1 - - - - - - | 1 0.02 5.f | - - - - - - - - - - - - - - - | 0 0.00 8 | - - - - - - - - - - - 1 - - - | 1 0.02 9 | 1 - - - - - - - - - - - - 1 - | 2 0.04 x | - - - - - - - - 1 - - - - - 8 | 9 0.18 ------+----------------------------------------------------------------------------+------------ # | 16 1 1 1 1 2 2 2 10 2 0 1 0 3 8 | 50 prob | 0.32 0.02 0.02 0.02 0.02 0.04 0.04 0.04 0.20 0.04 0.00 0.02 0.00 0.06 0.16 | 1.00 sample size: 50 number of agreements: 31 ( 0.62) number of disagreements: 19 ( 0.38) Cohen's kappa = 0.538 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 1.f 2 3 4 4.s 5 5.s 8 9 x | # prob ------+-------------------------------------------------------------+------------ 1 | 15 - - - - - - - - - - - | 15 0.30 1.a | 2 1 - - - - - - - - - - | 3 0.06 1.f | - - - - - - - - - - - - | 0 0.00 2 | - - 1 1 - - - - - - - - | 2 0.04 3 | - - - - 1 - - - - - - - | 1 0.02 4 | - - - - - 5 - - - - - - | 5 0.10 4.s | - - - - - - - - - - - - | 0 0.00 5 | - - - - - - 2 9 - - - - | 11 0.22 5.s | - - - - - - - 1 - - - - | 1 0.02 8 | - - - - - - - - - 1 - - | 1 0.02 9 | 1 - - - - - - - - - 1 - | 2 0.04 x | - - - - - - - 1 - - - 8 | 9 0.18 ------+-------------------------------------------------------------+------------ # | 18 1 1 1 1 5 2 11 0 1 1 8 | 50 prob | 0.36 0.02 0.02 0.02 0.02 0.10 0.04 0.22 0.00 0.02 0.02 0.16 | 1.00 sample size: 50 number of agreements: 42 ( 0.84) number of disagreements: 8 ( 0.16) Cohen's kappa = 0.800 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 2 2.a 3 4 4.a 4.s 5 5.a 5.f 8 9 x | # prob ------+-----------------------------------------------------------------------+------------ 1 | 15 1 - - - - - - - - - - 2 - | 18 0.36 1.a | 1 - - - - - - - - - - - - - | 1 0.02 2 | - - 1 1 - - - - - - - - - - | 2 0.04 2.a | - - - - - - - - - - - - - - | 0 0.00 3 | - - - - 1 - - - - - - - - - | 1 0.02 4 | - - - - - 2 2 1 1 - - - - - | 6 0.12 4.a | - - - - - - - - - - - - - - | 0 0.00 4.s | - - - - - - - 1 - - - - - - | 1 0.02 5 | - - - - - - - - 9 2 - - - - | 11 0.22 5.a | - - - - - - - - - - - - - - | 0 0.00 5.f | - - - - - - - - - - - - - - | 0 0.00 8 | - - - - - - - - - - 1 - - - | 1 0.02 9 | - - - - - - - - - - - - 1 - | 1 0.02 x | - - - - - - - - - - - - - 8 | 8 0.16 ------+-----------------------------------------------------------------------+------------ # | 16 1 1 1 1 2 2 2 10 2 1 0 3 8 | 50 prob | 0.32 0.02 0.02 0.02 0.02 0.04 0.04 0.04 0.20 0.04 0.02 0.00 0.06 0.16 | 1.00 sample size: 50 number of agreements: 38 ( 0.76) number of disagreements: 12 ( 0.24) Cohen's kappa = 0.703 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 1.f 2 3 4 4.s 5 8 9 x | # prob ------+--------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - | 18 0.36 1.a | - 1 - - - - - - - - - | 1 0.02 1.f | - - - - - - - - - - - | 0 0.00 2 | - - 1 1 - - - - - - - | 2 0.04 3 | - - - - 1 - - - - - - | 1 0.02 4 | - - - - - 5 1 - - - - | 6 0.12 4.s | - - - - - - 1 - - - - | 1 0.02 5 | - - - - - - - 11 - - - | 11 0.22 8 | - - - - - - - - 1 - - | 1 0.02 9 | - - - - - - - - - 1 - | 1 0.02 x | - - - - - - - - - - 8 | 8 0.16 ------+--------------------------------------------------------+------------ # | 18 1 1 1 1 5 2 11 1 1 8 | 50 prob | 0.36 0.02 0.02 0.02 0.02 0.10 0.04 0.22 0.02 0.02 0.16 | 1.00 sample size: 50 number of agreements: 48 ( 0.96) number of disagreements: 2 ( 0.04) Cohen's kappa = 0.949 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.f 2 2.a 3 4 4.a 4.s 5 5.a 5.f 8 9 x | # prob ------+----------------------------------------------------------------------------+------------ 1 | 15 1 - - - - - - - - - - - - - | 16 0.32 1.a | 1 - - - - - - - - - - - - - - | 1 0.02 1.f | - - - - - - - - - - - - - - - | 0 0.00 2 | - - - 1 - - - - - - - - - - - | 1 0.02 2.a | - - 1 - - - - - - - - - - - - | 1 0.02 3 | - - - - - 1 - - - - - - - - - | 1 0.02 4 | - - - - - - 1 - 1 - - - - - - | 2 0.04 4.a | - - - - - - 2 - - - - - - - - | 2 0.04 4.s | - - - - - - 1 - 1 - - - - - - | 2 0.04 5 | - - - - - - 1 - - 9 - - - - - | 10 0.20 5.a | - - - - - - - - - 2 - - - - - | 2 0.04 5.f | - - - - - - - - - - - - 1 - - | 1 0.02 8 | - - - - - - - - - - - - - - - | 0 0.00 9 | 2 - - - - - - - - - - - - 1 - | 3 0.06 x | - - - - - - - - - - - - - - 8 | 8 0.16 ------+----------------------------------------------------------------------------+------------ # | 18 1 1 1 0 1 5 0 2 11 0 0 1 1 8 | 50 prob | 0.36 0.02 0.02 0.02 0.00 0.02 0.10 0.00 0.04 0.22 0.00 0.00 0.02 0.02 0.16 | 1.00 sample size: 50 number of agreements: 37 ( 0.74) number of disagreements: 13 ( 0.26) Cohen's kappa = 0.678 --------------------------------------------------------------------------------------------- Fleiss' kappa = 0.746 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK smell 50 9 4 0.825 43 0.860 AV_JT smell 50 9 4 0.538 31 0.620 AV_SC smell 50 9 4 0.800 42 0.840 EK_JT smell 50 9 4 0.703 38 0.760 EK_SC smell 50 9 4 0.949 48 0.960 JT_SC smell 50 9 4 0.678 37 0.740 ----------------------------------------------------------- smell AV EK JT SC AV - 0.825 0.538 0.800 EK - - 0.703 0.949 JT - - - 0.678 SC - - - -