Table 5. Inter-rater agreement for real-time assessment: Cohen’s kappa with 95% confidence intervals and percentage agreement for criteria (n = 20 except * n = 18)
| Test |
Criteria |
Left-hand side |
Kappa (95% CI) |
Right-hand side |
Kappa (95% CI) |
Criteria agreement (%) |
| 1 |
A |
.86 |
(.59-1.00) |
.88 |
(.64-1.00) |
95.0 |
| B |
.55 |
(.19-.90) |
.38 |
(-.02-.78) |
75.0 |
| C |
.74 |
(.40-1.00) |
.74 |
(.40-1.00) |
90.0 |
| D |
.58 |
(.19-.98) |
.70 |
(.39-1.00) |
85.0 |
| E |
.66 |
(.31-1.00) |
.43 |
(.01-.85) |
80.0 |
| 2 |
A |
.22 |
(-.33-.76) |
.00 |
N/A |
82.5 |
| B |
.57 |
(.14-1.00) |
.32 |
(-.27-.90) |
85.0 |
| C |
.12 |
(-.28-0.52) |
.27 |
(-.17-.71) |
75.0 |
| D |
.88 |
(.64-1.00) |
.76 |
(.45-1.00) |
92.5 |
| E |
.76 |
(.45-1.00) |
.38 |
(-.07-.82) |
82.5 |
| 3 |
A |
|
.77 (.35-1.00) |
|
95.0 |
| B |
|
.80 (.54-1.00) |
|
90.0 |
| C |
.00 |
N/A |
.34 * |
(-.17-0.85) |
84.2 |
| 4 |
A |
|
1.00 (1.00-1.00) |
|
100.0 |
| B |
N/A |
N/A |
N/A |
N/A |
100.0 |
| C |
1.00 |
(1.00-1.00) |
.64 |
(.01-1.00) |
97.5 |
| D |
.69 |
(.38-1.00) |
.68 |
(.35-1.00) |
85.0 |
| 5 |
A |
N/A |
N/A |
.00 |
N/A |
97.5 |
| B |
.63 |
(.24-1.00) |
.74 |
(.40-1.00) |
87.5 |
| C |
.38 |
(-.07-.82) |
.39 |
(-.03-.81) |
75.0 |
| D |
.24 |
(-.17-.65) |
.13 |
(-.33-.58) |
67.5 |
| 7 |
A |
.00 |
N/A |
.00 |
N/A |
95.0 |
| B |
.77 |
(.35-1.00) |
.44 |
(-.20-1.00) |
92.5 |
| C |
.77 |
(.35-1.00) |
.86 |
(.59-1.00) |
95.0 |
| D |
|
.66 (.308-1.00) |
|
85.0 |
| E |
.89 |
(.67-1.00) |
.58 |
(.22-.95) |
87.5 |
| 8 |
A |
|
N/A N/A |
|
100.0 |
| B |
.80 |
(.54-1.00) |
.89 |
(.67-1.00) |
92.5 |
| C |
N/A |
N/A |
N/A |
N/A |
100.0 |
| D |
.69 |
(.29-1.00) |
.78 |
(.50-1.00) |
90.0 |
| 9 |
A |
.29 |
(-.14-.73) |
.63 |
(.24-1.00) |
77.5 |
| B |
.68 |
(.35-1.00) |
.60 |
(.26-.94) |
82.5 |
| C |
.55 |
(.19-.90) |
.89 |
(.67-1.00) |
87.5 |
| D |
.38 |
(-.12-.87) |
.15 |
(-.28-.57) |
72.5 |
| E |
.35 |
(-.07-.76) |
.30 |
(-.06-.66) |
67.5 |
| 10 |
A |
.44 |
(-.20-1.00) |
.000 |
N/A |
92.5 |
| B |
.39 |
(-.09-.86) |
.615 |
(.15-1.00) |
85.0 |
| C |
.50 |
(.12-.88) |
.600 |
(.25-.95) |
77.5 |
| D |
.64 |
(.01-1.00) |
1.000 |
(1.00-1.00) |
97.5 |
| E |
.60 |
(.24-.95) |
.634 |
.28-.99) |
82.5 |
NB. Test 6 from the Performance Matrix was not included in the study, for reasons explained in the methodology section of the text
