Table 5.
Inter-rater agreement for real-time assessment: Cohen’s kappa with 95% confidence intervals and percentage agreement for criteria (n = 20 except * n = 18)
Test
Criteria
Left-hand side
Kappa (95% CI)
Right-hand side
Kappa (95% CI)
Criteria agreement (%)
1
A
.86
(.59-1.00)
.88
(.64-1.00)
95.0
B
.55
(.19-.90)
.38
(-.02-.78)
75.0
C
.74
(.40-1.00)
.74
(.40-1.00)
90.0
D
.58
(.19-.98)
.70
(.39-1.00)
85.0
E
.66
(.31-1.00)
.43
(.01-.85)
80.0
2
A
.22
(-.33-.76)
.00
N/A
82.5
B
.57
(.14-1.00)
.32
(-.27-.90)
85.0
C
.12
(-.28-0.52)
.27
(-.17-.71)
75.0
D
.88
(.64-1.00)
.76
(.45-1.00)
92.5
E
.76
(.45-1.00)
.38
(-.07-.82)
82.5
3
A
.77 (.35-1.00)
95.0
B
.80 (.54-1.00)
90.0
C
.00
N/A
.34 *
(-.17-0.85)
84.2
4
A
1.00 (1.00-1.00)
100.0
B
N/A
N/A
N/A
N/A
100.0
C
1.00
(1.00-1.00)
.64
(.01-1.00)
97.5
D
.69
(.38-1.00)
.68
(.35-1.00)
85.0
5
A
N/A
N/A
.00
N/A
97.5
B
.63
(.24-1.00)
.74
(.40-1.00)
87.5
C
.38
(-.07-.82)
.39
(-.03-.81)
75.0
D
.24
(-.17-.65)
.13
(-.33-.58)
67.5
7
A
.00
N/A
.00
N/A
95.0
B
.77
(.35-1.00)
.44
(-.20-1.00)
92.5
C
.77
(.35-1.00)
.86
(.59-1.00)
95.0
D
.66 (.308-1.00)
85.0
E
.89
(.67-1.00)
.58
(.22-.95)
87.5
8
A
N/A N/A
100.0
B
.80
(.54-1.00)
.89
(.67-1.00)
92.5
C
N/A
N/A
N/A
N/A
100.0
D
.69
(.29-1.00)
.78
(.50-1.00)
90.0
9
A
.29
(-.14-.73)
.63
(.24-1.00)
77.5
B
.68
(.35-1.00)
.60
(.26-.94)
82.5
C
.55
(.19-.90)
.89
(.67-1.00)
87.5
D
.38
(-.12-.87)
.15
(-.28-.57)
72.5
E
.35
(-.07-.76)
.30
(-.06-.66)
67.5
10
A
.44
(-.20-1.00)
.000
N/A
92.5
B
.39
(-.09-.86)
.615
(.15-1.00)
85.0
C
.50
(.12-.88)
.600
(.25-.95)
77.5
D
.64
(.01-1.00)
1.000
(1.00-1.00)
97.5
E
.60
(.24-.95)
.634
.28-.99)
82.5
NB. Test 6 from the Performance Matrix was not included in the study, for reasons explained in the methodology section of the text