---------------------------------------------------------------------------------------------- corpora: BNC, verb: submit, number of patterns: 5, sample size: 50, annotators(4): AV EK JT SC ---------------------------------------------------------------------------------------------- +-----------------------+ | Annotators: AV vs. EK | +-----------------------+ | 1 1.a 1.c 2 4 5 5.a | # prob ------+------------------------------------+------------ 1 | 30 - 1 - - - - | 31 0.62 1.a | - - - - - - - | 0 0.00 1.c | - 1 - - - - - | 1 0.02 2 | - - - 12 - - - | 12 0.24 4 | - - - - 1 1 - | 2 0.04 5 | - - - - - 3 - | 3 0.06 5.a | - - - - - 1 - | 1 0.02 ------+------------------------------------+------------ # | 30 1 1 12 1 5 0 | 50 prob | 0.60 0.02 0.02 0.24 0.02 0.10 0.00 | 1.00 sample size: 50 number of agreements: 46 ( 0.92) number of disagreements: 4 ( 0.08) Cohen's kappa = 0.858 +-----------------------+ | Annotators: AV vs. JT | +-----------------------+ | 1 1.a 1.c 2 2.s 4 4.a 5 5.a | # prob ------+----------------------------------------------+------------ 1 | 29 1 - 1 - - - - - | 31 0.62 1.a | - - - - - - - - - | 0 0.00 1.c | - 1 - - - - - - - | 1 0.02 2 | - 1 - 10 1 - - - - | 12 0.24 2.s | - - - - - - - - - | 0 0.00 4 | - - - - - 2 - - - | 2 0.04 4.a | - - - - - - - - - | 0 0.00 5 | - - - - - 2 1 - - | 3 0.06 5.a | - - - - - - - 1 - | 1 0.02 ------+----------------------------------------------+------------ # | 29 3 0 11 1 4 1 1 0 | 50 prob | 0.58 0.06 0.00 0.22 0.02 0.08 0.02 0.02 0.00 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.691 +-----------------------+ | Annotators: AV vs. SC | +-----------------------+ | 1 1.a 1.c 2 2.s 4 5 5.a | # prob ------+-----------------------------------------+------------ 1 | 29 1 1 - - - - - | 31 0.62 1.a | - - - - - - - - | 0 0.00 1.c | 1 - - - - - - - | 1 0.02 2 | - - - 11 1 - - - | 12 0.24 2.s | - - - - - - - - | 0 0.00 4 | - - - - - 1 1 - | 2 0.04 5 | - - - - - - 3 - | 3 0.06 5.a | - - - - - - 1 - | 1 0.02 ------+-----------------------------------------+------------ # | 30 1 1 11 1 1 5 0 | 50 prob | 0.60 0.02 0.02 0.22 0.02 0.02 0.10 0.00 | 1.00 sample size: 50 number of agreements: 44 ( 0.88) number of disagreements: 6 ( 0.12) Cohen's kappa = 0.789 +-----------------------+ | Annotators: EK vs. JT | +-----------------------+ | 1 1.a 1.c 2 2.s 4 4.a 5 | # prob ------+-----------------------------------------+------------ 1 | 28 1 - 1 - - - - | 30 0.60 1.a | - 1 - - - - - - | 1 0.02 1.c | 1 - - - - - - - | 1 0.02 2 | - 1 - 10 1 - - - | 12 0.24 2.s | - - - - - - - - | 0 0.00 4 | - - - - - 1 - - | 1 0.02 4.a | - - - - - - - - | 0 0.00 5 | - - - - - 3 1 1 | 5 0.10 ------+-----------------------------------------+------------ # | 29 3 0 11 1 4 1 1 | 50 prob | 0.58 0.06 0.00 0.22 0.02 0.08 0.02 0.02 | 1 sample size: 50 number of agreements: 41 ( 0.82) number of disagreements: 9 ( 0.18) Cohen's kappa = 0.697 +-----------------------+ | Annotators: EK vs. SC | +-----------------------+ | 1 1.a 1.c 2 2.s 4 5 | # prob ------+------------------------------------+------------ 1 | 29 1 - - - - - | 30 0.60 1.a | 1 - - - - - - | 1 0.02 1.c | - - 1 - - - - | 1 0.02 2 | - - - 11 1 - - | 12 0.24 2.s | - - - - - - - | 0 0.00 4 | - - - - - 1 - | 1 0.02 5 | - - - - - - 5 | 5 0.10 ------+------------------------------------+------------ # | 30 1 1 11 1 1 5 | 50 prob | 0.60 0.02 0.02 0.22 0.02 0.02 0.10 | 1.00 sample size: 50 number of agreements: 47 ( 0.94) number of disagreements: 3 ( 0.06) Cohen's kappa = 0.896 +-----------------------+ | Annotators: JT vs. SC | +-----------------------+ | 1 1.a 1.c 2 2.s 4 4.a 5 | # prob ------+-----------------------------------------+------------ 1 | 27 1 1 - - - - - | 29 0.58 1.a | 2 - - 1 - - - - | 3 0.06 1.c | - - - - - - - - | 0 0.00 2 | 1 - - 10 - - - - | 11 0.22 2.s | - - - - 1 - - - | 1 0.02 4 | - - - - - 1 - 3 | 4 0.08 4.a | - - - - - - - 1 | 1 0.02 5 | - - - - - - - 1 | 1 0.02 ------+-----------------------------------------+------------ # | 30 1 1 11 1 1 0 5 | 50 prob | 0.60 0.02 0.02 0.22 0.02 0.02 0.00 0.10 | 1 sample size: 50 number of agreements: 40 ( 0.80) number of disagreements: 10 ( 0.20) Cohen's kappa = 0.666 ---------------------------------------------------------------------------------------------- Fleiss' kappa = 0.764 Overview of inter-annotator agreement: ----------------------------------------------------------- double verb size #N #A Cohens agrs % AV_EK submit 50 5 4 0.858 46 0.920 AV_JT submit 50 5 4 0.691 41 0.820 AV_SC submit 50 5 4 0.789 44 0.880 EK_JT submit 50 5 4 0.697 41 0.820 EK_SC submit 50 5 4 0.896 47 0.940 JT_SC submit 50 5 4 0.666 40 0.800 ----------------------------------------------------------- submit AV EK JT SC AV - 0.858 0.691 0.789 EK - - 0.697 0.896 JT - - - 0.666 SC - - - -