The ability to react following the processing and integration of relevant visual cues within a changing environment is a key determinant of sporting success (Adam et al., 1992). Individuals who possess the ability to process a greater amount of visual information in a shorter period of time may have a competitive advantage over their slower counterparts (Spiteri et al., 2013), allowing for the facilitation of both decision making ability (Mori et al., 2002), and motor response time (Ando et al., 2001). Consequently, the ability to reliably assess reaction time (RT) may support the evaluation of athletic ability.

To date, several laboratory assessments have been utilized to evaluate and quantify changes in RT performance (Ando et al., 2001; Hoffman et al., 2010; 2012; Hultsch et al., 2002; Li et al., 2000; Mori et al., 2002; Schatz et al., 2006; Stuss et al., 1989; Williams et al., 2005). Acute changes in RT performance are commonly quantified with finger tapping tests, simple reaction time (SRT) tests and/or choice reaction time tests (CRT) via a computer integrated; touch sensitive, visual light system. Such tests have also been administered in conjunction with a battery of other psychological tests to aid in the quantification of changes in cognitive performance (Schatz et al., 2006; Williams et al., 2005). However, information regarding the reliability of many of these reaction tests is lacking. The internal test-retest reliability of reaction devices are rarely reported in the literature, while emerging reliability studies on RT test devices show unacceptable (≤0.53) intraclass correlation coefficients (Eckner et al., 2011; Mercer et al., 2009).

Recently, the Dynavision™ D2 Visuomotor Training Device (D2) has emerged as a tool in the assessment of RT (Hoffman et al., 2012; Wells et al., 2013). The D2 is a light-training reaction device, developed to train sensory motor integration through the visual system (Wells et al., 2013). In addition to high performance training and evaluation, the D2 is also marketed as a standard device for neuro-rehabilitation, stroke recovery, and concussion evaluation. It is also used to assess visual and motor reaction to both central and peripheral stimuli, with a capacity to integrate increasing levels of cognitive challenges.

Previous studies utilizing the D2 have shown high intraclass correlation coefficients (ICC’s; 0.79 - 0.97) in a number of tests of varying complexity, indicating strong test-retest reliability (Klavora et al., 1995; Wells et al., 2013). In addition, strong correlations with conventional psychomotor tests demanding similar visuomotor skills and psychomotor ability have also been demonstrated (Vesia et al., 2008). However; to our knowledge, these studies have not been repeated. In addition, all prior studies have utilized either ICC version 3,1, or declined to delineate the ICC version utilized (Klavora et al., 1994; 1995; Wells et al., 2013). As such, reliability data for the D2 generated from these studies is not generalizable beyond the confines of that study since the effect of trials is either fixed or unknown (Weir, 2005). Therefore, the purpose of this study was to establish the reliability of three D2 RT tasks utilizing ICC version 2,1 (ICC_2,1), which considers the effect of trials to be a random factor since the trials are a sample of possible levels.

Forty-two recreationally active individuals (22 men, 20 women) volunteered to participate in this study (n = 42; age: 23.41 ± 4.84 years; height: 1.72 ± 0.11 m; mass: 76.62 ± 18.26 Kg; body fat %: 19.46 ± 8.90). The research protocol was approved by the University Institutional Review Board. Following an explanation of all procedures, risks, and benefits associated with the experimental protocol, each participant gave his or her informed consent to participate in this study. Participants were healthy college students with no prior experience with the test apparatus. All participants reported having no vision problems, other than that correctable with prescription lenses. Prescription lenses (glasses and/or contact lenses) were permitted. Use of prescriptive lenses was standardized across test sessions. Participants were also instructed not to consume caffeine at least 5 hours prior to testing. This was verbally confirmed prior to each test session, and again at the conclusion of the experimental protocol.

Participants reported to the Human Performance Laboratory (HPL) on six separate occasions, with at least 48 hours between each session. During each of the six sessions, participants completed three consecutive visuomotor tasks of increasing complexity. Reaction time was assessed using the Dynavision™ D2 Visuomotor Training Device (D2; Dynavision International LLC, West Chester, OH). The D2 device was previously described (Wells et al., 2013). Briefly, the D2 is a light-training reaction device, developed to train sensory motor integration through the visual system (see Figure 1). It consists of a board (4 x 4 ft.) that can be raised or lowered relative to the height of the participant. The board contains 64 target buttons arranged into five concentric circles that can be illuminated to serve as a stimulus for the participant, and contains an LCD display above the inner most ring of target buttons. The LCD display is utilized to place a cognitive stressor on the participant during testing and provide a 5 s visual countdown to the start of a test.

Upon arriving at the HPL for the first visit, participants were familiarized with the D2, and were provided detailed verbal instructions on both the testing protocol and how to complete each of the three tasks from a standardized script. For each task, participants were instructed to take an athletic stance, consisting of flexed knees, low center of gravity and upright posture. The D2 board was then raised or lowered to the height of the participant, such that the LCD screen was approximately at eye level, and the outer-most target buttons were within hands reach. Following verbal instruction, participants completed a shortened practice trial of each test. Upon completion of the practice protocol, participants began the first round of testing. Participants then returned to the HPL on five additional occasions to complete the identical testing protocol. Practice trials were not permitted during sessions two through six; however, instructions were repeated during each session. Lighting in the testing room standardized for all participants across all six tests.

The first assessment (Choice Reaction Test; CRT) measured the participant’s visual and motor RT to a visual stimulus with the dominant hand. The test was initiated when a participant placed and held his or her hand on an illuminated “home” button. At this point, a single button would light up (visual stimulus) in one of four locations adjacent to the home button on the same horizontal plane. Once the participant recognized the stimulus, they were required to leave the “home” button, strike the stimulus and return back to the “home” button. Visual RT was measured as the amount of time it took to identify the stimulus and initiate a reaction by leaving the “home” button. Motor response time was measured as the amount of time it took to physically strike the illuminated button following the initial visual reaction and is measured as the amount of time between the hand leaving the “home” button and striking the stimulus. Time was measured to the nearest one hundredth of a second. Participants were instructed to respond to the stimulus as quickly as possible. For the practice trial, participants completed a shortened version of the test consisting of three stimuli. For the subsequent six test trials, participants were required to respond to 10 stimuli, which is consistent with standard protocol for this test to generate an average reaction time. For each test trial, if a stimulus was missed, the test was repeated until an error free trial was achieved to avoid inflated RT. An error free trial was defined as successful completion of the task without misplacement of the hand or failure to strike the stimulus on the initial attempt.

The second assessment (Reactive; Mode A) measured the participant’s ability to react to a stimulus as it changed positions on the board. Following a 5 s visual countdown on the board’s LCD screen, an initial stimulus would present on the D2 in a random location. The stimulus remained illuminated until it was struck by the participant. The stimulus would then appear at another random location. The participant was instructed to successfully identify and strike as many stimuli as possible within 60 s with both hands. Participants were advised to utilize their peripheral vision, keep their hands raised as opposed to down by their sides and avoid crossing the hands over the body. In addition, participants were informed that the stimulus could be struck with any part of the hand. The number of hits and the average time per hit were recorded for each participant.

The third assessment (Reactive with cognitive stress; Mode B) was similar to the previous measure in Mode A in that participants were required to react to a visual stimulus with both hands, as it changed positions on the board. The difference between the two assessments was that participants were asked to verbally recite a five-digit number that was presented on the LCD screen of the apparatus. The five-digit number was presented a total of 11 times throughout the 60 s test and remained for 0.75 s each time. Additionally, the visual stimulus remained illuminated for only one second before changing location requiring the participant to be increasingly reactive in identifying the stimulus. The participant was instructed to successfully identify and strike each stimulus before it changed position, score as many strikes as possible within 60 s and successfully recite all eleven 5-digit numbers. The number of successful hits was recorded for each participant. As in Mode A, participants were advised to utilize their peripheral vision, keep their hands raised, avoid crossing the hands over the body and use any part of the hand they desired.

A repeated measures analysis of variance (ANOVA) was used to detect differences in RT performance across the six trials. In the event of a significant F ratio, a Bonferroni post hoc analysis was used for pairwise comparisons. Homogeneity of variance was assessed using Mauchly’s test of sphericity. In the event homogeneity of variance was violated, a Greenhouse-Geisser adjustment was used. As recommended by Weir (Weir, 2005), intraclass correlation coefficients (ICC_2,1), standard error of measurement (SEM), standard error of measurement as a percent of the grand mean (%SEM) minimum difference (MD) and minimum difference as a percent of the grand mean (%MD) were calculated to determine the reliability of the D2 device. Briefly, SEM is an indication of the precision of a score; while MD represents the threshold score for a deviation from baseline to be considered ‘real’. MD, %MD and %SEM are a function of reliability and provide practically applicable data. For all statistical tests, a criterion alpha level of p ≤ 0.05 was used to determine statistical significance. Data are reported as mean ± SD.

Performance data for the six RT trials are presented in Table 1. The repeated measures ANOVA showed a significant time effect for Visual RT (p = 0.001). Post hoc analysis of the six sessions showed that session 1 was not significantly different from session 2 (p = 1.00), session 3 (p = 0.439) or session 4 (p = 1.00). However session 1 was significantly different from session 5 (p = 0.013) and session 6 (p = 0.024). There were no other significant differences between time points for visual RT. Motor RT also showed a significant effect for time (p = < 0.001). Post hoc analysis of the six sessions showed that session 1 was not significantly different from session 2 (p = 1.00), session 3 (p = 0.305) or session 5 (p = 0.142). However, session 1 was significantly different from session 4 (p = 0.012) and session 6 (p = 0.004). There were no other significant differences between time points for motor RT.

The repeated measures ANOVA showed a significant time effect for the number of hits in Mode A. Post hoc analysis of the six sessions showed that session 1 was significantly different from all other sessions (p = < .001). Significant differences were observed between sessions 1- 2 (p = < 0.001), sessions 2 - 3 (p = 0.001) and sessions 5 - 6 (p = 0.002). However, sessions 3 - 4, and sessions 4 - 5 were not significantly different from each other (p = 0.108 and p = 1.00 respectively).

A significant time effect was also seen for the average RT per hit in Mode A. Post hoc analysis of the six sessions showed that session 1 was significantly different from all other sessions (p = ≤ 0.001). Significant differences were observed between sessions 1 - 2 (p = 0.001), sessions 2 - 3 (p = < 0.001), and sessions 5 - 6 (p = 0.005). However, sessions 3 - 4, and sessions 4 - 5 were not significantly different from each other (p = 0.190 and p = 1.00 respectively).

The repeated measures ANOVA showed a significant time effect for the number of hits in Mode B. Post hoc analysis of the six sessions showed that session 1 was significantly different from all other sessions (p = < 0.001). Significant differences were observed between sessions 1 - 2 (p = < 0.001) and sessions 2 - 3 (p = < 0.001). However sessions 3 - 4, sessions 4 - 5, and sessions 5 - 6 were not significantly different from each other (p values = 0.120, 1.00 and 0.104 respectively).

A significant time effect was also seen for the average RT per hit in Mode B. Post hoc analysis of the six sessions showed that session 1 was significantly different from all other sessions (p = < .001). Significant differences were observed between sessions 1 - 2 (p = < 0.001) and sessions 3 - 4 (p = < 0.001). However sessions 2 - 3, sessions 4 - 5, and sessions 5 - 6 were not significantly different from each other (p values = 0.611, 1.00 and 0.95 respectively).

Reliability data are presented in Table 2. Intraclass correlation coefficients (ICC_2,1) were calculated between the two sessions (separated by 48-hours) with the least significant difference in score to avoid the influence of learning and training curves. In the CRT task, the repeated measures ANOVA showed no significant differences between consecutive sessions. Visual RT showed strong reliability (ICC_2,1 = 0.84), while motor RT showed moderate reliability (ICC_2,1 = 0.63). Reliability for Visual and Motor RT were taken between sessions 2 and 3 (p = 1.00), allowing for 1 familiarization trial. For Mode A and Mode B, the repeated measures ANOVA showed significant differences in score during the first 3 sessions. Both Mode A (Hits: ICC_2,1 = 0.75, Avg.: ICC_2,1 = 0.68) and Mode B (Hits: ICC_2,1 = 0.73, Avg.: ICC_2,1 = 0.72) showed moderate to strong reliability. Reliability for Mode A and Mode B was taken between sessions 4 and 5 allowing for 3 familiarization trials (Mode A: p = 1.00 and p = 1.00; Mode B: p = 1.00 and p = 1.00).

Results of this study indicate that the Dynavision™ D2 is a reliable instrument to assess RT. Moderate to strong reliability (ICC_2,1: 0.73 - 0.84), within acceptable ranges (Portney and Watkins, 2000), were demonstrated for visual RT (ICC_2,1: 0.84, SEM: 0.033), and reactive ability (Mode A hits - ICC_2,1: 0.75, SEM: 5.44; Mode B hits - ICC_2,1: 0.73, SEM: 8.57), providing support for the use of the D2 in reaction assessment of recreationally active young adults. Motor RT (ICC_2,1: 0.63, SEM: 0.035s) showed fair reliability, while average RT per hit for Modes A and B showed moderate reliability (ICC_2,1: 0.68, SEM: 0.43 s and ICC_2,1: 0.72, SEM: 0.03 s respectively).

Previous studies investigating the reliability of the D2 have utilized ICC version 3,1, or the version used was not specified. ICC version 3,1 assumes that trials are a fixed effect, and thus only reliable within the confines of a particular study. In contrast, ICC version 2,1 contends that trials are a random effect, and as such, the test-retest reliability of a device determined by a sample of individuals is representative of that population. As such, reliability data for the D2 generated from previous studies are not generalizable beyond the procedures, testers and sample population measured. This study is the first to provide reliability data for the D2 that may be generalized to other laboratories testing recreationally active young adults. Nonetheless, our results show similar reliability to previous ICC_3,1 data produced by our lab (Wells et al., 2013; Visual RT: ICC_3,1: 0.84, SEM: 0.021 s; Mode A: ICC_3,1: 0.80, SEM: 5.59 hits; Mode B: ICC_3,1: 0.82, SEM: 6.83 hits) providing further evidence for test-retest reliability. Observed differences between the two studies may be a result of the ICC version utilized, the use of different familiarization protocols or the number of trials employed.

Initial improvements in RT performance on the D2 device have been attributed to systematic error of a learning effect (Klavora et al., 1994; 1995). Therefore, familiarization trials may be necessary to eliminate this effect. In the CRT task, there were no significant differences between consecutive trials for visual or motor RT, indicating that a learning curve was not present, possibly due to a lack of task complexity (Wells et al., 2013). However, our data indicate a significant learning effect was present for Mode A and Mode B tasks. Significant differences were observed in trials 1 - 3 for both Mode A and Mode B. As such, our results suggest that only one familiarization trial is necessary for CRT assessment to learn the test protocol, while three familiarization trials are needed for Mode A and Mode B before a subsequent reliable baseline score can be established. This is consistent with Klavora et al., 1994 who acknowledge the presence of a learning curve through three trials in a similar Mode B test.

Klavora et al., 1995 reported an ICC of 0.88 for Mode A between trial 1 and 5, and 0.92 for Mode B for trials 2 through 5. They observed no significant differences in performance beyond trial 2 for both tasks. In contrast, our ICC’s for Mode A and B were calculated between the first two trials without systematic error (Trial 4 & 5). Possible differences in observed learning curves and ICC’s may be the result of differing familiarization protocols. Following a shortened practice test at trial 1 for each task, we utilized a protocol consisting of one full-length test for subsequent trials (2-6) with no preceding practice test. In contrast, Klavora et al., 1995 utilized a modified familiarization protocol consisting of an individualized series of 30 s practice trials until the number of hits was less than, equal to or no more than one greater than the number of hits in the preceding trial. In addition, each subsequent trial was preceded by a 30 second practice trial. Should this protocol be generalizable to a similar population, the individualized nature of the familiarization protocol abates the convenience and practicality of mass testing. In addition, while this methodology may have yielded acceptable ICC’s, it does not fully delineate the characterization of the learning curve. As such, it is impossible to distinguish where a true baseline value lies. Prior investigations regarding the reliability and learning curve of the CRT task are lacking.

Notwithstanding the learning curve, significant time effects were observed for all RT tasks. In the CRT task, significant increases in performance were observed following session 5 for visual RT, while a significant increase in performance was observed following sessions 4 and 6 for motor RT. In addition, a significant secondary increase in performance was observed between trials 5 and 6 for Mode A. This is in contrast to Klavora et al., 1995, who did not observe a training effect with Mode A testing. These results suggest that continuous training with the D2 results in a training effect that leads to increased task performance. Further increases in Mode B performance were not observed beyond trial 4. However, it may be possible that the complexity of the task may delay any associated training curve. Consequently, the possibility of a training curve should be taken into account when continual testing is desired, especially with clinical and intervention studies. Future studies should attempt to characterize the training curve associated with the D2.

Our results suggest that the Dynavision™ D2 is a reliable means through which to assess RT. This is the first study we are aware of to examine D2 reliability in terms of results that are generalizable to recreationally active young adults. These data may be used by other laboratories. It is recommended that one familiarization trial is necessary for CRT assessment to learn the test protocol, while three familiarization trials are needed for Mode A and Mode B before a subsequent reliable baseline score can be established. Investigations utilizing the D2 should account for the possibility of a training curve when performing continuous testing. Future studies should investigate continuous trials beyond the six presented herein to further characterize the training effect.