Results
CoNLL 2009 Shared Task Evaluation Results
Output of all the systems - Download
In the following
zip archive (120 MB) you will
find outputs of all the participating systems for all the tasks they
participated in. The systems use the 'anonymised' names they used during
the task. The translation to 'real' names, as they are used throughout this
document, is also included in the archive.
Description
Green numbers indicate the best score for the system.
@ denotes the best score for the column, i.e. for the language.
List of Tables
In-domain Evaluation Data
Closed challenge
Open challenge
Out-of-domain Evaluation Data
Closed challenge
Open challenge
Precision and Recall
Closed challenge
Open challenge
System names, authors, and description papers
Closed Challenge, Joint Task, Macro F1 Score
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish |
|---|
| 1 | Che | 82.64 | 81.84
| @ 76.38
| @ 83.27
| 87.00
| @ 82.44
| @ 85.65
| 81.90
|
| 2 | Chen | 82.52 | @ 83.01
| 76.23
| 80.87
| @ 87.69
| 81.22
| 85.28
| @ 83.31
|
| 3 | Merlo | 82.14 | 82.66
| 76.15
| 83.21
| 86.03
| 79.59
| 84.91
| 82.43
|
| 4 | Bohnet | 80.85 | 80.44
| 75.91
| 79.57
| 85.14
| 81.60
| 82.51
| 80.75
|
| 5 | Asahara | 78.43 | 75.91
| 73.43
| 81.43
| 86.40
| 69.84
| 84.86
| 77.12
|
| 6 | Brown | 77.27 | 77.40
| 72.12
| 75.66
| 83.98
| 77.86
| 76.65
| 77.21
|
| 7 | Zhang | 76.49 | 75.00
| 73.42
| 76.93
| 82.88
| 73.76
| 78.17
| 75.25
|
| 8 | Dai | 73.98 | 72.09
| 72.72
| 67.14
| 81.89
| 75.00
| 80.89
| 68.14
|
| 9 | Lu Li | 73.97 | 71.32
| 65.53
| 75.85
| 81.92
| 70.93
| 80.49
| 71.72
|
| 10 | Lluís | 71.49 | 56.64
| 66.18
| 75.95
| 81.69
| 72.31
| 81.76
| 65.91
|
| 11 | Vallejo | 70.81 | 73.75
| 67.16
| 60.50
| 78.19
| 67.51
| 77.75
| 70.78
|
| 12 | Ren | 67.81 | 59.42
| 75.90
| 60.18
| 77.83
| 65.77
| 77.63
| 57.96
|
| 13 | Zeman | 51.07 | 49.61
| 43.50
| 57.95
| 50.27
| 49.57
| 57.69
| 48.90
|
Closed Challenge, SRL-only Task, Semantic Labeled F1
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish |
|---|
| 1 | Zhao | 80.47 | @ 80.32
| 77.72
| 85.19
| 85.44
| 75.99
| @ 78.15
| @ 80.46
|
| 2 | Nugues | 80.31 | 80.01
| @ 78.60
| @ 85.41
| @ 85.63
| @ 79.71
| 76.30
| 76.52
|
| 3 | Meza-Ruiz | 77.46 | 78.00
| 77.73
| 75.75
| 83.34
| 73.52
| 76.00
| 77.91
|
| 4 | Baoli Li | 69.26 | 74.06
| 70.37
| 57.46
| 69.63
| 67.76
| 72.03
| 73.54
|
| 5 | Moreau | 66.49 | 65.60
| 67.37
| 71.74
| 72.14
| 66.50
| 57.75
| 64.33
|
| 6 | Täckström | 61.27 | 57.11
| 63.41
| 71.05
| 67.64
| 53.42
| 54.74
| 61.51
|
| 7 | Lin | 57.18 | 61.70
| 70.33
| 60.43
| 65.66
| 59.51
| 23.78
| 58.87
|
Closed Challenge, Joint Task, Labeled Syntactic Accuracy
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish
|
|---|
| — | P-columns | 84.54
| 86.13
| 78.46
| 78.46
| 85.50
| 85.93
| 91.12
| 86.20
|
| 1 | Merlo | 85.77 | @ 87.86
| 76.11
| @ 80.38
| 88.79
| 87.29
| 92.34
| @ 87.64
|
| 2 | Bohnet | 85.68 | 86.35
| 76.51
| 80.11
| @ 89.88
| @ 87.48
| 92.21
| 87.19
|
| 3 | Che | 85.23 | 86.56
| 75.49
| 80.01
| 88.48
| 86.19
| @ 92.57
| 87.33
|
| 4 | Chen | 85.04 | 85.88
| 75.67
| 79.70
| 89.19
| 86.24
| 92.32
| 86.29
|
| 5 | Zhang | 82.64 | 82.67
| 73.63
| 75.58
| 87.90
| 84.57
| 91.47
| 82.69
|
| 6 | Brown | 81.67 | 82.61
| 71.81
| 73.29
| 87.47
| 82.31
| 91.71
| 82.46
|
| 7 | Asahara | 81.16 | 79.48
| 72.66
| 78.17
| 88.54
| 75.85
| 91.69
| 81.74
|
| 8 | Lluís | 79.11 | 64.21
| 70.53
| 75.00
| 87.48
| 81.94
| 91.55
| 83.09
|
| 9 | Lu Li | 78.17 | 75.68
| 63.95
| 72.60
| 86.61
| 79.85
| 91.26
| 77.21
|
| 10 | Ren | 78.01 | 77.84
| @ 79.17
| 57.30
| 87.57
| 76.63
| 91.47
| 76.11
|
| 11 | Dai | 77.11 | 77.85
| 73.86
| 58.69
| 84.81
| 82.51
| 90.43
| 71.64
|
| 12 | Vallejo | 72.54 | 77.33
| 67.58
| 49.41
| 80.35
| 73.88
| 86.17
| 73.07
|
| 13 | Zeman | 64.92 | 67.68
| 49.48
| 57.06
| 61.82
| 69.79
| 82.66
| 65.98
|
Closed Challenge, Both Tasks, Semantic Labeled F1
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish |
|---|
| 1 | 1 (SRLonly) | Zhao | 80.47 | @ 80.32
| 77.72
| 85.19
| 85.44
| 75.99
| 78.15
| @ 80.46
|
| 2 | 2 (SRLonly) | Nugues | 80.31 | 80.01
| @ 78.60
| 85.41
| 85.63
| @ 79.71
| 76.30
| 76.52
|
| 3 | 1 (Joint) | Chen | 79.96 | 80.10
| 76.77
| 82.04
| @ 86.15
| 76.19
| 78.17
| 80.29
|
| 4 | 2 (Joint) | Che | 79.94 | 77.10
| 77.15
| @ 86.51
| 85.51
| 78.61
| @ 78.26
| 76.47
|
| 5 | 3 (Joint) | Merlo | 78.42 | 77.44
| 76.05
| 86.02
| 83.24
| 71.78
| 77.23
| 77.19
|
| 6 | 3 (SRLonly) | Meza-Ruiz | 77.46 | 78.00
| 77.73
| 75.75
| 83.34
| 73.52
| 76.00
| 77.91
|
| 7 | 4 (Joint) | Bohnet | 76.00 | 74.53
| 75.29
| 79.02
| 80.39
| 75.72
| 72.76
| 74.31
|
| 8 | 5 (Joint) | Asahara | 75.65 | 72.35
| 74.17
| 84.69
| 84.26
| 63.66
| 77.93
| 72.50
|
| 9 | 6 (Joint) | Brown | 72.85 | 72.18
| 72.43
| 78.02
| 80.43
| 73.40
| 61.57
| 71.95
|
| 10 | 7 (Joint) | Dai | 70.78 | 66.34
| 71.57
| 75.50
| 78.93
| 67.43
| 71.02
| 64.64
|
| 11 | 8 (Joint) | Zhang | 70.31 | 67.34
| 73.20
| 78.28
| 77.85
| 62.95
| 64.71
| 67.81
|
| 12 | 9 (Joint) | Lu Li | 69.72 | 66.95
| 67.06
| 79.08
| 77.17
| 61.98
| 69.58
| 66.23
|
| 13 | 4 (SRLonly) | Baoli Li | 69.26 | 74.06
| 70.37
| 57.46
| 69.63
| 67.76
| 72.03
| 73.54
|
| 14 | 10 (Joint) | Vallejo | 68.95 | 70.14
| 66.71
| 71.49
| 75.97
| 61.01
| 68.82
| 68.48
|
| 15 | 5 (SRLonly) | Moreau | 66.49 | 65.60
| 67.37
| 71.74
| 72.14
| 66.50
| 57.75
| 64.33
|
| 16 | 11 (Joint) | Lluís | 63.06 | 46.79
| 59.72
| 76.90
| 75.86
| 62.66
| 71.60
| 47.88
|
| 17 | 6 (SRLonly) | Täckström | 61.27 | 57.11
| 63.41
| 71.05
| 67.64
| 53.42
| 54.74
| 61.51
|
| 18 | 7 (SRLonly) | Lin | 57.18 | 61.70
| 70.33
| 60.43
| 65.66
| 59.51
| 23.78
| 58.87
|
| 19 | 12 (Joint) | Ren | 56.69 | 41.00
| 72.58
| 62.82
| 67.56
| 54.31
| 58.73
| 39.80
|
| 20 | 13 (Joint) | Zeman | 32.14 | 24.19
| 34.71
| 58.13
| 36.05
| 16.44
| 30.13
| 25.36
|
Open Challenge, Joint Task, Macro F1 Score
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish |
|---|
| 1 | Che | 82.70 | @ 81.97
| @ 76.42
| @ 83.31
| @ 87.05
| @ 82.44
| @ 85.70
| @ 82.00
|
| 2 | Zhang | 76.72 | 75.00
| 73.42
| 76.93
| 83.02
| 74.44
| 78.78
| 75.47
|
Open Challenge, SRL-only Task, Semantic Labeled F1
No such system.
Open Challenge, Joint Task, Labeled Syntactic Accuracy
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish
|
|---|
| — | P-columns | 84.54
| 86.13
| 78.46
| 78.46
| 85.50
| 85.93
| 91.12
| 86.20
|
| 1 | Che | 85.23 | @ 86.56
| @ 75.49
| @ 80.01
| @ 88.48
| @ 86.19
| @ 92.57
| @ 87.33
|
| 2 | Zhang | 82.64 | 82.67
| 73.63
| 75.58
| 87.90
| 84.57
| 91.47
| 82.69
|
Open Challenge, Both Tasks, Semantic Labeled F1
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish |
|---|
| 1 | 1 (Joint) | Che | 80.06 | @ 77.36
| @ 77.23
| @ 86.57
| @ 85.61
| @ 78.61
| @ 78.35
| @ 76.66
|
| 2 | 2 (Joint) | Zhang | 70.78 | 67.34
| 73.20
| 78.28
| 78.13
| 64.31
| 65.95
| 68.24
|
Closed Challenge, Out-of-domain Data, Joint Task, Macro F1 Score
| Rank | System | Average | Czech-ood | English-ood | German-ood |
|---|
| 1 | Che | 77.19 | 80.66
| 77.71
| @ 73.19
|
| 2 | Chen | 76.51 | 79.12
| @ 78.51
| 71.89
|
| 3 | Merlo | 75.93 | @ 80.70
| 75.76
| 71.32
|
| 4 | Bohnet | 74.37 | 77.20
| 75.55
| 70.35
|
| 5 | Lluís | 71.92 | 74.56
| 73.91
| 67.30
|
| 6 | Brown | 71.84 | 71.81
| 75.16
| 68.55
|
| 7 | Lu Li | 71.79 | 74.81
| 73.69
| 66.86
|
| 8 | Asahara | 71.46 | 78.63
| 76.70
| 59.04
|
| 9 | Zhang | 71.26 | 74.53
| 74.29
| 64.96
|
| 10 | Dai | 67.47 | 65.63
| 74.34
| 62.45
|
| 11 | Zeman | 43.67 | 54.49
| 48.56
| 27.97
|
Closed Challenge, Out-of-domain Data, SRL-only Task, Semantic Labeled F1
| Rank | System | Average | Czech-ood | English-ood | German-ood |
|---|
| 1 | Nugues | 74.38 | 83.93
| @ 73.31
| @ 65.89
|
| 2 | Zhao | 74.34 | @ 85.44
| @ 73.31
| 64.26
|
| 3 | Meza-Ruiz | 70.52 | 77.34
| 71.86
| 62.37
|
| 4 | Täckström | 60.61 | 71.59
| 59.82
| 50.43
|
| 5 | Baoli Li | 60.24 | 59.62
| 62.02
| 59.08
|
| 6 | Moreau | 59.66 | 72.51
| 57.34
| 49.12
|
Closed Challenge, Out-of-domain Data, Joint Task, Labeled Syntactic Accuracy
| Rank | System | Average | Czech-ood | English-ood | German-ood
|
|---|
| — | P-columns | 73.96
| 74.37
| 77.06
| 70.44
|
| 1 | Bohnet | 78.79 | 76.40
| @ 82.64
| @ 77.34
|
| 2 | Merlo | 78.01 | @ 76.41
| 80.84
| 76.77
|
| 3 | Chen | 77.96 | 75.58
| 82.38
| 75.93
|
| 4 | Che | 77.90 | 76.03
| 81.57
| 76.11
|
| 5 | Zhang | 75.95 | 71.29
| 81.50
| 75.06
|
| 6 | Lluís | 75.09 | 72.11
| 80.92
| 72.25
|
| 7 | Brown | 74.82 | 70.60
| 80.67
| 73.19
|
| 8 | Asahara | 74.29 | 73.27
| 82.39
| 67.21
|
| 9 | Lu Li | 73.92 | 69.81
| 80.09
| 71.86
|
| 10 | Dai | 66.74 | 55.50
| 78.95
| 65.78
|
| 11 | Zeman | 50.20 | 51.45
| 62.83
| 36.31
|
Closed Challenge, Out-of-domain Data, Both Tasks, Semantic Labeled F1
| Rank | Rank in task | System | Average | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (Joint) | Che | 76.38 | 85.26
| 73.82
| @ 70.07
|
| 2 | 2 (Joint) | Chen | 75.01 | 82.66
| @ 74.58
| 67.78
|
| 3 | 1 (SRLonly) | Nugues | 74.38 | 83.93
| 73.31
| 65.89
|
| 4 | 2 (SRLonly) | Zhao | 74.34 | @ 85.44
| 73.31
| 64.26
|
| 5 | 3 (Joint) | Merlo | 73.63 | 84.99
| 70.65
| 65.25
|
| 6 | 3 (SRLonly) | Meza-Ruiz | 70.52 | 77.34
| 71.86
| 62.37
|
| 7 | 4 (Joint) | Bohnet | 69.94 | 78.01
| 68.44
| 63.36
|
| 8 | 5 (Joint) | Lu Li | 69.61 | 79.80
| 67.21
| 61.83
|
| 9 | 6 (Joint) | Brown | 68.85 | 73.03
| 69.61
| 63.92
|
| 10 | 7 (Joint) | Lluís | 68.74 | 77.01
| 66.88
| 62.34
|
| 11 | 8 (Joint) | Asahara | 68.48 | 84.00
| 70.99
| 50.44
|
| 12 | 9 (Joint) | Dai | 67.99 | 75.71
| 69.67
| 58.60
|
| 13 | 10 (Joint) | Zhang | 66.57 | 77.78
| 67.07
| 54.87
|
| 14 | 4 (SRLonly) | Täckström | 60.61 | 71.59
| 59.82
| 50.43
|
| 15 | 5 (SRLonly) | Baoli Li | 60.24 | 59.62
| 62.02
| 59.08
|
| 16 | 6 (SRLonly) | Moreau | 59.66 | 72.51
| 57.34
| 49.12
|
| 17 | 11 (Joint) | Zeman | 32.86 | 56.83
| 31.77
| 9.98
|
Open Challenge, Out-of-domain Data, Joint Task, Macro F1 Score
| Rank | System | Average | Czech-ood | English-ood | German-ood |
|---|
| 1 | Che | 77.15 | @ 80.63
| @ 77.63
| @ 73.20
|
| 2 | Zhang | 72.03 | 74.53
| 74.81
| 66.74
|
Open Challenge, Out-of-domain Data, SRL-only Task, Semantic Labeled F1
No such system.
Open Challenge, Out-of-domain Data, Joint Task, Labeled Syntactic Accuracy
| Rank | System | Average | Czech-ood | English-ood | German-ood
|
|---|
| — | P-columns | 73.96
| 74.37
| 77.06
| 70.44
|
| 1 | Che | 77.90 | @ 76.03
| @ 81.57
| @ 76.11
|
| 2 | Zhang | 75.95 | 71.29
| 81.50
| 75.06
|
Open Challenge, Out-of-domain Data, Both Tasks, Semantic Labeled F1
| Rank | Rank in task | System | Average | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (Joint) | Che | 76.32 | @ 85.21
| @ 73.66
| @ 70.09
|
| 2 | 2 (Joint) | Zhang | 68.10 | 77.78
| 68.11
| 58.42
|
Closed Challenge, Semantic Labeled Precision for All Systems
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (Joint) | Che | 83.29 | 78.59
| 81.66
| @ 89.12
| 86.66
| @ 82.09
| 87.26
| 77.65
| 87.32
| 75.78
| @ 75.60
|
| 2 | 1 (SRLonly) | Nugues | 83.27 | 81.01
| 82.42
| 88.11
| 87.46
| 81.16
| 83.87
| 78.87
| 86.11
| 76.04
| 70.97
|
| 3 | 2 (SRLonly) | Zhao | 83.10 | @ 82.91
| 80.42
| 88.17
| 88.22
| 77.23
| 81.66
| @ 83.08
| @ 88.55
| 77.18
| 68.51
|
| 4 | 2 (Joint) | Chen | 82.15 | 82.87
| 78.57
| 83.07
| @ 88.67
| 77.83
| 81.42
| 82.64
| 82.99
| @ 77.73
| 70.79
|
| 5 | 3 (SRLonly) | Meza-Ruiz | 81.82 | 80.39
| @ 82.66
| 79.43
| 87.49
| 77.04
| 84.88
| 80.85
| 80.61
| @ 77.73
| 69.13
|
| 6 | 3 (Joint) | Merlo | 81.60 | 79.08
| 80.93
| 87.45
| 84.92
| 75.60
| 83.75
| 79.44
| 85.90
| 72.89
| 75.19
|
| 7 | 4 (Joint) | Asahara | 77.63 | 72.85
| 76.18
| 85.63
| 85.42
| 68.41
| 82.04
| 72.88
| 84.71
| 72.58
| 57.42
|
| 8 | 4 (SRLonly) | Baoli Li | 77.37 | 77.10
| 81.54
| 69.85
| 79.11
| 72.36
| 84.23
| 77.41
| 71.81
| 70.86
| 65.80
|
| 9 | 5 (Joint) | Bohnet | 76.95 | 74.98
| 77.17
| 79.93
| 81.21
| 75.35
| 75.51
| 74.51
| 77.96
| 69.28
| 62.85
|
| 10 | 6 (Joint) | Brown | 73.87 | 72.40
| 73.35
| 79.47
| 83.28
| 73.35
| 63.45
| 71.79
| 73.45
| 71.86
| 64.23
|
| 11 | 7 (Joint) | Dai | 73.14 | 66.25
| 72.86
| 79.34
| 81.65
| 70.13
| 78.04
| 63.71
| 78.93
| 72.77
| 67.23
|
| 12 | 8 (Joint) | Vallejo | 72.60 | 72.49
| 68.59
| 75.68
| 79.04
| 65.15
| 77.66
| 69.62
| — | — | — |
| 13 | 9 (Joint) | Lu Li | 71.75 | 67.54
| 69.97
| 80.58
| 79.89
| 63.40
| 74.13
| 66.73
| 81.55
| 70.63
| 63.62
|
| 14 | 10 (Joint) | Lluís | 71.73 | 65.32
| 79.80
| 77.54
| 78.30
| 64.41
| 79.25
| 57.50
| 76.54
| 68.07
| 63.93
|
| 15 | 11 (Joint) | Zhang | 71.41 | 67.40
| 74.74
| 79.30
| 79.05
| 62.57
| 68.97
| 67.87
| 77.76
| 68.04
| 54.50
|
| 16 | 5 (SRLonly) | Moreau | 70.98 | 68.35
| 72.75
| 75.23
| 76.01
| 68.96
| 68.92
| 66.66
| 75.21
| 63.42
| 54.43
|
| 17 | 12 (Joint) | Ren | 65.31 | 41.34
| 75.45
| 68.83
| 76.57
| 62.96
| @ 91.93
| 40.11
| — | — | — |
| 18 | 6 (SRLonly) | Lin | 63.18 | 62.08
| 76.30
| 66.70
| 74.67
| 65.84
| 37.56
| 59.08
| — | — | — |
| 19 | 7 (SRLonly) | Täckström | 55.81 | 54.15
| 52.82
| 72.79
| 59.62
| 48.06
| 44.68
| 58.58
| 72.82
| 54.02
| 47.35
|
| 20 | 13 (Joint) | Zeman | 23.88 | 15.49
| 25.57
| 50.28
| 27.10
| 9.46
| 22.68
| 16.57
| 48.98
| 23.85
| 5.57
|
Closed Challenge, Semantic Labeled Recall for All Systems
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (SRLonly) | Zhao | 78.01 | 77.89
| 75.20
| 82.41
| 82.83
| 74.80
| 74.94
| 78.00
| 82.54
| 69.82
| 60.51
|
| 2 | 1 (Joint) | Chen | 77.89 | 77.51
| 75.05
| 81.03
| 83.77
| 74.61
| @ 75.17
| @ 78.07
| 82.33
| 71.69
| 65.02
|
| 3 | 2 (SRLonly) | Nugues | 77.65 | @ 79.04
| 75.12
| 82.88
| 83.87
| @ 78.31
| 69.99
| 74.31
| 81.86
| 70.76
| 61.49
|
| 4 | 2 (Joint) | Che | 76.99 | 75.68
| 73.11
| 84.04
| @ 84.39
| 75.42
| 70.95
| 75.32
| 83.30
| @ 71.95
| @ 65.29
|
| 5 | 3 (Joint) | Merlo | 75.56 | 75.87
| 71.73
| @ 84.64
| 81.63
| 68.33
| 71.65
| 75.05
| @ 84.09
| 68.55
| 57.63
|
| 6 | 4 (Joint) | Bohnet | 75.10 | 74.09
| 73.50
| 78.13
| 79.59
| 76.09
| 70.21
| 74.12
| 78.06
| 67.63
| 63.88
|
| 7 | 5 (Joint) | Asahara | 73.84 | 71.86
| 72.27
| 83.77
| 83.14
| 59.52
| 74.20
| 72.13
| 83.29
| 69.48
| 44.98
|
| 8 | 3 (SRLonly) | Meza-Ruiz | 73.62 | 75.75
| 73.36
| 72.40
| 79.56
| 70.30
| 68.81
| 75.19
| 74.31
| 66.80
| 56.82
|
| 9 | 6 (Joint) | Brown | 71.89 | 71.97
| 71.52
| 76.62
| 77.78
| 73.44
| 59.80
| 72.10
| 72.62
| 67.48
| 63.61
|
| 10 | 7 (Joint) | Zhang | 69.29 | 67.28
| 71.72
| 77.28
| 76.68
| 63.34
| 60.96
| 67.75
| 77.80
| 66.12
| 55.24
|
| 11 | 4 (SRLonly) | Täckström | 68.97 | 60.41
| @ 79.31
| 69.38
| 78.16
| 60.14
| 70.65
| 64.75
| 70.40
| 67.02
| 53.94
|
| 12 | 8 (Joint) | Dai | 68.69 | 66.42
| 70.32
| 72.02
| 76.39
| 64.94
| 65.16
| 65.61
| 72.74
| 66.83
| 51.93
|
| 13 | 9 (Joint) | Lu Li | 67.85 | 66.37
| 64.38
| 77.63
| 74.63
| 60.63
| 65.56
| 65.74
| 78.12
| 64.11
| 60.13
|
| 14 | 10 (Joint) | Vallejo | 65.76 | 67.94
| 64.93
| 67.75
| 73.13
| 57.36
| 61.80
| 67.38
| — | — | — |
| 15 | 5 (SRLonly) | Baoli Li | 62.97 | 71.25
| 61.89
| 48.80
| 62.18
| 63.71
| 62.92
| 70.04
| 50.97
| 55.15
| 53.61
|
| 16 | 6 (SRLonly) | Moreau | 62.72 | 63.06
| 62.73
| 68.56
| 68.65
| 64.20
| 49.70
| 62.16
| 70.00
| 52.33
| 44.76
|
| 17 | 11 (Joint) | Lluís | 57.33 | 36.45
| 47.71
| 76.27
| 73.57
| 61.00
| 65.29
| 41.01
| 77.50
| 65.74
| 60.84
|
| 18 | 12 (Joint) | Zeman | 56.20 | 55.23
| 53.99
| 68.88
| 53.85
| 62.60
| 44.88
| 53.98
| 67.69
| 47.57
| 47.91
|
| 19 | 7 (SRLonly) | Lin | 52.96 | 61.32
| 65.23
| 55.24
| 58.58
| 54.28
| 17.40
| 58.66
| — | — | — |
| 20 | 13 (Joint) | Ren | 51.32 | 40.66
| 69.92
| 57.78
| 60.45
| 47.75
| 43.15
| 39.50
| — | — | — |
Closed Challenge, Macro Precision for Joint Task Systems
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | Che | 84.26 | 82.58
| @ 78.58
| @ 84.56
| 87.57
| @ 84.14
| 89.92
| 82.49
| @ 81.68
| 78.68
| 75.86
|
| 2 | Merlo | 83.68 | 83.47
| 78.52
| 83.91
| 86.86
| 81.44
| 88.05
| 83.54
| 81.16
| 76.86
| @ 75.98
|
| 3 | Chen | 83.60 | @ 84.38
| 77.12
| 81.39
| @ 88.93
| 82.03
| 86.87
| @ 84.47
| 79.28
| @ 80.05
| 73.36
|
| 4 | Bohnet | 81.31 | 80.67
| 76.84
| 80.02
| 85.54
| 81.41
| 83.86
| 80.85
| 77.18
| 75.96
| 70.10
|
| 5 | Asahara | 79.40 | 76.16
| 74.42
| 81.90
| 86.98
| 72.13
| 86.87
| 77.31
| 78.99
| 77.48
| 62.32
|
| 6 | Brown | 77.77 | 77.50
| 72.58
| 76.38
| 85.38
| 77.83
| 77.58
| 77.13
| 72.02
| 76.27
| 68.71
|
| 7 | Zhang | 77.03 | 75.03
| 74.19
| 77.44
| 83.47
| 73.57
| 80.22
| 75.28
| 74.52
| 74.77
| 64.78
|
| 8 | Lluís | 75.42 | 64.77
| 75.16
| 76.27
| 82.89
| 73.17
| 85.40
| 70.29
| 74.32
| 74.49
| 68.09
|
| 9 | Dai | 75.13 | 72.05
| 73.36
| 69.02
| 83.23
| 76.32
| 84.23
| 67.67
| 67.22
| 75.86
| 66.51
|
| 10 | Lu Li | 74.96 | 71.61
| 66.96
| 76.59
| 83.25
| 71.63
| 82.69
| 71.97
| 75.68
| 75.36
| 67.74
|
| 11 | Vallejo | 72.57 | 74.91
| 68.09
| 62.55
| 79.69
| 69.52
| 81.91
| 71.34
| — | — | — |
| 12 | Ren | 71.66 | 59.59
| 77.31
| 63.07
| 82.07
| 69.79
| @ 91.70
| 58.11
| — | — | — |
| 13 | Zeman | 44.40 | 41.59
| 37.53
| 53.67
| 44.46
| 39.62
| 52.67
| 41.27
| 50.22
| 43.34
| 20.94
|
Closed Challenge, Macro Recall for Joint Task Systems
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | Chen | 81.46 | 81.69
| @ 75.36
| 80.36
| @ 86.48
| 80.43
| @ 83.75
| @ 82.18
| 78.95
| @ 77.03
| 70.48
|
| 2 | Che | 81.11 | 81.12
| 74.30
| 82.02
| 86.43
| 80.80
| 81.76
| 81.32
| 79.67
| 76.76
| @ 70.70
|
| 3 | Merlo | 80.66 | @ 81.86
| 73.92
| @ 82.51
| 85.21
| 77.81
| 81.99
| 81.35
| @ 80.25
| 74.70
| 67.20
|
| 4 | Bohnet | 80.39 | 80.22
| 75.00
| 79.12
| 84.73
| @ 81.79
| 81.21
| 80.65
| 77.23
| 75.14
| 70.61
|
| 5 | Asahara | 77.50 | 75.67
| 72.47
| 80.97
| 85.84
| 67.69
| 82.95
| 76.94
| 78.28
| 75.93
| 56.09
|
| 6 | Brown | 76.78 | 77.29
| 71.67
| 74.96
| 82.63
| 77.88
| 75.75
| 77.28
| 71.61
| 74.08
| 68.40
|
| 7 | Zhang | 75.97 | 74.97
| 72.68
| 76.43
| 82.29
| 73.96
| 76.21
| 75.22
| 74.55
| 73.81
| 65.15
|
| 8 | Lu Li | 73.01 | 71.02
| 64.17
| 75.12
| 80.62
| 70.24
| 78.41
| 71.48
| 73.97
| 72.10
| 66.00
|
| 9 | Dai | 72.90 | 72.13
| 72.09
| 65.36
| 80.60
| 73.73
| 77.79
| 68.62
| 64.12
| 72.89
| 58.86
|
| 10 | Vallejo | 69.15 | 72.63
| 66.26
| 58.58
| 76.74
| 65.62
| 73.98
| 70.22
| — | — | — |
| 11 | Lluís | 68.22 | 50.33
| 59.12
| 75.64
| 80.52
| 71.47
| 78.42
| 62.05
| 74.80
| 73.33
| 66.54
|
| 12 | Ren | 64.66 | 59.25
| 74.54
| 57.54
| 74.01
| 62.19
| 67.31
| 57.81
| — | — | — |
| 13 | Zeman | 60.56 | 61.46
| 51.74
| 62.97
| 57.83
| 66.20
| 63.77
| 59.98
| 59.57
| 55.20
| 42.11
|
Open Challenge, Semantic Labeled Precision for All Systems
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (Joint) | Che | 83.44 | @ 78.86
| @ 81.71
| @ 89.19
| @ 86.78
| @ 82.09
| @ 87.59
| @ 77.88
| @ 87.28
| @ 75.59
| @ 75.73
|
| 2 | 2 (Joint) | Zhang | 71.88 | 67.40
| 74.74
| 79.30
| 79.17
| 64.12
| 70.13
| 68.29
| 77.76
| 69.00
| 58.61
|
Open Challenge, Semantic Labeled Recall for All Systems
| Rank | Rank in task | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | 1 (Joint) | Che | 77.07 | @ 75.90
| @ 73.21
| @ 84.10
| @ 84.47
| @ 75.42
| @ 70.88
| @ 75.48
| @ 83.25
| @ 71.83
| @ 65.24
|
| 2 | 2 (Joint) | Zhang | 69.76 | 67.28
| 71.72
| 77.28
| 77.12
| 64.51
| 62.24
| 68.20
| 77.80
| 67.24
| 58.23
|
Open Challenge, Macro Precision for Joint Task Systems
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | Che | 84.34 | @ 82.71
| @ 78.60
| @ 84.60
| @ 87.63
| @ 84.14
| @ 90.08
| @ 82.60
| @ 81.65
| @ 78.58
| @ 75.92
|
| 2 | Zhang | 77.26 | 75.03
| 74.19
| 77.44
| 83.54
| 74.34
| 80.80
| 75.49
| 74.52
| 75.25
| 66.83
|
Open Challenge, Macro Recall for Joint Task Systems
| Rank | System | Average | Catalan | Chinese | Czech | English | German | Japanese | Spanish | Czech-ood | English-ood | German-ood |
|---|
| 1 | Che | 81.15 | @ 81.23
| @ 74.35
| @ 82.05
| @ 86.47
| @ 80.80
| @ 81.73
| @ 81.40
| @ 79.64
| @ 76.70
| @ 70.68
|
| 2 | Zhang | 76.20 | 74.97
| 72.68
| 76.43
| 82.51
| 74.54
| 76.85
| 75.44
| 74.55
| 74.37
| 66.64
|
System names, authors, and description papers
| System name |
Description paper |
| Zhao |
Multilingual Dependency Learning: A Huge Feature Engineering
Method to Semantic Dependency Parsing
Hai Zhao (赵海), Wenliang Chen (陈文亮),
Chunyu Kit, Guodong Zhou |
| Nugues |
Multilingual Semantic Role Labeling
Anders Björkelund, Love Hafdell, Pierre
Nugues |
| Chen |
Multilingual Dependency Learning: Exploiting Rich Features for
Tagging Syntactic and Semantic Dependencies
Hai Zhao (赵海), Wenliang Chen (陈文亮),
Jun’ichi Kazama , Kiyotaka Uchimoto, and Kentaro Torisawa |
| Che |
Multilingual Dependency-based Syntactic and Semantic
Parsing
Wanxiang Che, Zhenghua Li, Yongqiang Li,
Yuhang Guo, Bing Qin, Ting Liu |
| Merlo |
A Latent Variable Model of Synchronous Syntactic-Semantic
Parsing for Multiple Languages
Andrea Gesmundo, James Henderson, Paola
Merlo, Ivan Titov |
| Meza-Ruiz |
Multilingual Semantic Role Labelling with Markov
Logic
Ivan Meza-Ruiz, Sebastian
Riedel |
| Bohnet |
Efficient Parsing of Syntactic and Semantic Dependency
Structures
Bernd Bohnet |
| Asahara |
Multilingual Syntactic-Semantic Dependency Parsing with
Three-Stage Approximate Max-Margin Linear Models
Yotaro Watanabe, Masayuki Asahara and
Yuji Matsumoto |
| Dai |
An Iterative Approach for Joint Dependency Parsing and Semantic
Role Labeling
Qifeng Dai, Enhong Chen, Liu
Shi |
| Zhang |
Hybrid Multilingual Parsing with HPSG for
SRL
Yi Zhang, Rui Wang, Stephan
Oepen |
| Lu Li |
A Joint Syntactic and Semantic Dependency Parsing System based
on Maximum Entropy Models
Buzhou Tang, Lu Li, Xinxin Li, Xuan Wang,
Xiaolong Wang |
| Baoli Li |
Exploring Multilingual Semantic Role
Labeling
Baoli Li, Martin Emms, Saturnino Luz,
Carl Vogel |
| Vallejo |
Joint memory-based learning of syntactic and semantic
dependencies in multiple languages
Roser Morante, Vincent Van Asch, Antal
van den Bosch |
| Moreau |
The Crotal SRL System : a Generic Tool Based on Tree-structured
CRF
Erwan Moreau, Isabelle Tellier |
| Lluís |
A Second-Order Joint Eisner Model for Syntactic and Semantic
Dependency Parsing
Xavier Lluís, Stefan Bott, Lluís
Màrquez |
| Täckström |
Multilingual semantic parsing with a pipeline of linear
classifiers
Oscar Täckström |
| Ren |
Parsing Syntactic and Semantic Dependencies for Multiple
Languages with A Pipeline Approach
Han Ren, Donghong Ji Jing Wan, Mingyao
Zhang |
| Zeman |
A Simple Generative Pipeline Approach to Dependency Parsing and
Semantic Role Labeling
Daniel Zeman |
| Brown |
no paper |
| Lin |
no paper |
