------------------------------------------------------------------------------------------------ corpora: BNC, verb: breathe, number of patterns: 17, sample size: 50, annotators(4): AV EK JT SC ------------------------------------------------------------------------------------------------ +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 2 2.f 3 3.a 4 9 9.f 11 11.s 12 u x | # prob ------+------------------------------------------------------------------+------------ 1 | 17 - - - - - - - - - - - - | 17 0.34 2 | - 6 1 - - - - - - - - - - | 7 0.14 2.f | - - - - - - - - - - - - - | 0 0.00 3 | - - - - - - - - - - - - - | 0 0.00 3.a | - - - - - - - - - - - - - | 0 0.00 4 | - - - - - 2 - - - - - - - | 2 0.04 9 | 4 - - - - - 13 1 - - - - - | 18 0.36 9.f | - - - - - - - - - - - - - | 0 0.00 11 | - - - 1 - - - - - - - 1 - | 2 0.04 11.s | - - - - 1 - - - - - - - - | 1 0.02 12 | - - - - - - - - - - - 2 - | 2 0.04 u | - - - - - - - - - - - - - | 0 0.00 x | - - - - - - - 1 - - - - - | 1 0.02 ------+------------------------------------------------------------------+------------ # | 21 6 1 1 1 2 13 2 0 0 0 3 0 | 50 prob | 0.42 0.12 0.02 0.02 0.02 0.04 0.26 0.04 0.00 0.00 0.00 0.06 0.00 | 1 sample size: 50 number of agreements: 38 ( 0.76) number of disagreements: 12 ( 0.24) Cohen's kappa = 0.678 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 2 2.f 4 9 9.f 11 11.a 11.s 12 x | # prob ------+--------------------------------------------------------+------------ 1 | 17 - - - - - - - - - - | 17 0.34 2 | - 6 1 - - - - - - - - | 7 0.14 2.f | - - - - - - - - - - - | 0 0.00 4 | - - - 2 - - - - - - - | 2 0.04 9 | 1 - - - 16 1 - - - - - | 18 0.36 9.f | - - - - - - - - - - - | 0 0.00 11 | - - - - - - 2 - - - - | 2 0.04 11.a | - - - - - - - - - - - | 0 0.00 11.s | - - - - - - - 1 - - - | 1 0.02 12 | - - - - - - - - - 2 - | 2 0.04 x | - - - - 1 - - - - - - | 1 0.02 ------+--------------------------------------------------------+------------ # | 18 6 1 2 17 1 2 1 0 2 0 | 50 prob | 0.36 0.12 0.02 0.04 0.34 0.02 0.04 0.02 0.00 0.04 0.00 | 1 sample size: 50 number of agreements: 45 ( 0.90) number of disagreements: 5 ( 0.10) Cohen's kappa = 0.864 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 2 2.a 3 4 9 11 11.s 12 u x | # prob ------+--------------------------------------------------------+------------ 1 | 17 - - - - - - - - - - | 17 0.34 2 | - 6 1 - - - - - - - - | 7 0.14 2.a | - - - - - - - - - - - | 0 0.00 3 | - - - - - - - - - - - | 0 0.00 4 | - - - - 2 - - - - - - | 2 0.04 9 | 1 - - - - 16 - - - 1 - | 18 0.36 11 | - - - - - - 2 - - - - | 2 0.04 11.s | - - - 1 - - - - - - - | 1 0.02 12 | - - - - - - - - 2 - - | 2 0.04 u | - - - - - - - - - - - | 0 0.00 x | - - - - - - - - - 1 - | 1 0.02 ------+--------------------------------------------------------+------------ # | 18 6 1 1 2 16 2 0 2 2 0 | 50 prob | 0.36 0.12 0.02 0.02 0.04 0.32 0.04 0.00 0.04 0.04 0.00 | 1 sample size: 50 number of agreements: 45 ( 0.90) number of disagreements: 5 ( 0.10) Cohen's kappa = 0.865 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 2 2.f 3 3.a 4 9 9.f 11 11.a 12 u | # prob ------+-------------------------------------------------------------+------------ 1 | 18 - - - - - 3 - - - - - | 21 0.42 2 | - 6 - - - - - - - - - - | 6 0.12 2.f | - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - - - - 1 - - - | 1 0.02 3.a | - - - - - - - - - 1 - - | 1 0.02 4 | - - - - - 2 - - - - - - | 2 0.04 9 | - - - - - - 13 - - - - - | 13 0.26 9.f | - - - - - - 1 1 - - - - | 2 0.04 11 | - - - - - - - - - - - - | 0 0.00 11.a | - - - - - - - - - - - - | 0 0.00 12 | - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - 1 - 2 - | 3 0.06 ------+-------------------------------------------------------------+------------ # | 18 6 1 0 0 2 17 1 2 1 2 0 | 50 prob | 0.36 0.12 0.02 0.00 0.00 0.04 0.34 0.02 0.04 0.02 0.04 0.00 | 1.00 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.758 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 2 2.a 2.f 3 3.a 4 9 9.f 11 12 u | # prob ------+-------------------------------------------------------------+------------ 1 | 18 - - - - - - 3 - - - - | 21 0.42 2 | - 6 - - - - - - - - - - | 6 0.12 2.a | - - - - - - - - - - - - | 0 0.00 2.f | - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - - - - - 1 - - | 1 0.02 3.a | - - - - 1 - - - - - - - | 1 0.02 4 | - - - - - - 2 - - - - - | 2 0.04 9 | - - - - - - - 13 - - - - | 13 0.26 9.f | - - - - - - - - - - - 2 | 2 0.04 11 | - - - - - - - - - - - - | 0 0.00 12 | - - - - - - - - - - - - | 0 0.00 u | - - - - - - - - - 1 2 - | 3 0.06 ------+-------------------------------------------------------------+------------ # | 18 6 1 0 1 0 2 16 0 2 2 2 | 50 prob | 0.36 0.12 0.02 0.00 0.02 0.00 0.04 0.32 0.00 0.04 0.04 0.04 | 1.00 sample size: 50 number of agreements: 39 ( 0.78) number of disagreements: 11 ( 0.22) Cohen's kappa = 0.705 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 2 2.a 2.f 3 4 9 9.f 11 11.a 12 u | # prob ------+-------------------------------------------------------------+------------ 1 | 18 - - - - - - - - - - - | 18 0.36 2 | - 6 - - - - - - - - - - | 6 0.12 2.a | - - - - - - - - - - - - | 0 0.00 2.f | - - 1 - - - - - - - - - | 1 0.02 3 | - - - - - - - - - - - - | 0 0.00 4 | - - - - - 2 - - - - - - | 2 0.04 9 | - - - - - - 16 - - - - 1 | 17 0.34 9.f | - - - - - - - - - - - 1 | 1 0.02 11 | - - - - - - - - 2 - - - | 2 0.04 11.a | - - - - 1 - - - - - - - | 1 0.02 12 | - - - - - - - - - - 2 - | 2 0.04 u | - - - - - - - - - - - - | 0 0.00 ------+-------------------------------------------------------------+------------ # | 18 6 1 0 1 2 16 0 2 0 2 2 | 50 prob | 0.36 0.12 0.02 0.00 0.02 0.04 0.32 0.00 0.04 0.00 0.04 0.04 | 1.00 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.892 ------------------------------------------------------------------------------------------------ Fleiss' kappa = 0.793 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK breathe 50 17 4 0.678 38 0.760 AV_JT breathe 50 17 4 0.864 45 0.900 AV_SC breathe 50 17 4 0.865 45 0.900 EK_JT breathe 50 17 4 0.758 41 0.820 EK_SC breathe 50 17 4 0.705 39 0.780 JT_SC breathe 50 17 4 0.892 46 0.920 ----------------------------------------------------------- breathe AV EK JT SC AV - 0.678 0.864 0.865 EK - - 0.758 0.705 JT - - - 0.892 SC - - - -