Soccer imposes significant metabolic load on players (Stolen et al., 2005). Due to continuous change in playing direction and running speeds from low intensity to all-out efforts (Wing et al., 2020), both the aerobic and anaerobic system play an important factor in soccer specific fitness (Stolen et al., 2005). With maximum oxygen uptake (VO_2max) as the most prominent parameter reflecting the capacity of the aerobic system (Poole and Richardson, 1997), assessing this value in soccer players is crucial to quantify cardiovascular fitness. There is a close association of a sufficient VO_2max with running distances at high-intensity speed in official matches (Bradley et al., 2013) and is therefore often a primary focus in the training and assessment of soccer players (Slimani et al., 2019; Clemente et al., 2023; Wisloeff et al., 1998). Additionally, the assessment of VO_2max of soccer players is important to retrospectively elucidate effectiveness, and to prospectively prescribe training programs to enhance performance optimally (Wisloeff et al., 1998; Clemente et al., 2023; Helgerud et al., 2001).

The gold-standard to assess VO_2max is connected to ramp testing on a treadmill employing respiratory gas analysis. However, this procedure is time-consuming, costly and requires specialized equipment and personnel. Furthermore, ramp-testing necessitates the player's maximal effort, which can disrupt training and recovery routines. It has been noted that maximal or near-maximal fitness tests, like ramp-tests, are often considered unsuitable for regular use in practical settings by many practitioners, due to their interference with ongoing training schedules or competitions (Schimpchen et al., 2023).

To address these drawbacks, the scientific community developed and explored simpler methods. For instance, a submaximal cycle test designed to predict V̇O_2max was introduced by Åstrand and Ryhming in 1954 (Åstrand and Ryhming, 1954). In soccer, running based field tests such as the YoYo Intermittent Recovery Test 2 (YYIR2) have become popular for indirectly assessing VO_2max. The test is widely employed due to its simplicity, -cost-effectiveness and allows to test multiple players at the same time (Bangsbo et al., 2008). Additionally, the YYIR2 was designed to reflect the demands of team sports such as soccer with special regards to their intermittent nature (Thomas et al., 2006). The YYIR2 provides VO_2max with good-to-excellent test-retest reliability in team sport athletes as recently outlined in a review (Grgic et al., 2019). However, the YYIR2, also requires an all-out effort which impairs the (frequent) assessment of e.g. VO_2max within preparation phase and soccer season. Additionally, factors affecting the results of the YYIR2 include nutritional status (Grgic et al., 2019), time of day at which testing is conducted (Chtourou et al., 2012), provision of verbal encouragement (Currell and Jeukendrup, 2008), and it is currently unclear whether ground surface characteristics influence test results (Grgic et al., 2019). Consequently, the YYIR2 has some disadvantages in practice and overcoming (some of) these might allow more frequent and easier assessment of VO_2max.

With technological progress, particularly in chip miniaturization (Waldrop, 2016), contemporary wearable technologies like smartwatches are now available. These devices provide users with V̇O_2max if the smartwatch is worn during outdoor runs at submaximal intensities. (Garmin Ltd, 2019). Although assessing VO_2max with a smartwatch during submaximal outdoor runs appears promising, studies have revealed significant variability in the criterion-related concurrent validity of parameters provided by consumer-grade wearables (Düking et al., 2016; Peake et al., 2018). Many, although not all wearables do not provide valid parameters in many scenarios and populations (Düking et al., 2016; Peake et al., 2018) yet are marketed with bold marketing claims due to a largely unregulated market (Sperlich and Holmberg, 2017).

Therefore, evaluating the criterion-related concurrent validity of parameters from wearables is crucial for their application in sports practice. For VO_2max assessment, companies producing wearables such as smartwatches advancements smartwatch technologies claim to estimate VO_2max through non-exhaustive, easy-to-administer tests in their marketing. Recent analyses indicate that runners can accurately assess their VO_2max with a smartwatch during submaximal outdoor runs, with an error margin of 5.7% compared to respiratory gas analysis (Düking et al., 2022).

Currently there is no available evidence on the validity of the smartwatch “Garmin Forerunner 245” to assess VO2max using a non-fatiguing test performed as a warm-up prior to the usual soccer training.

Addressing this research gap could aid practitioners in selecting the most effective methodology for VO_2max assessment in highly trained soccer players, whether it be through respiratory gas analysis, the YYIR2 test, or smartwatch-assessed VO_2max. Therefore, the aim was to assess the criterion related concurrent validity of VO_2max estimation provided by an end consumer grade smartwatch andthe YYIR2 in comparison to respiratory gas analysis. Based on prior studies of VO_2max assessment using smartwatches (which showed an error of 5.7% in runners) (Düking et al., 2022) and the YYIR2 (Grgic et al., 2019) in varied populations, our hypotheses are: i) the error in V̇O_2max assessment via smartwatch will be approximately 5-6%, and ii) the error margin in VO_2max estimation through smartwatch will be comparable to that of the YYIR2 test.

All players visited the laboratory to determine anthropometric data, maximum values of heart rate (HR_max) and VO_2max, which was assessed during a ramp test treadmill protocol and by employing the criterion breath-by-breath gas analysis. On another day, subjects performed one outdoor YYIR2 as this test is often used by practitioners when no laboratory with respiratory gas analysis is available or if such assessments of V̇O_2max are deemed unfeasible e.g. due to time constraints. On two other days, the players performed outdoor runs during the warm-up period of the regular soccer training while wearing the smartwatch. Figure 1 illustrates the experimental approach.

24 healthy and injury free national level male youth soccer players (mean age 17.3 ± 1.3 years, body height 178.1 ± 6.2 cm, body mass 71.5 ± 8.6 kg) and/or if necessary, their legal guardians were informed about all experimental procedures and consented to participate in the study. The players were recruited from a youth soccer academy. Players were eligible for inclusion if they had been part of the academy for over three years. However, players with wrist tattoos were excluded due to potential interference with the smartwatch's optical sensor (Bent et al., 2020). According to a recent classification framework, this group of players were categorized as Tier 3 athletes (highly trained/national level athletes) (McKay et al., 2022). As there are no ICCs available prior to our study, we performed a sample size calculation following data collection in our study using Arifin’s web-based sample size calculator (Arifin, 2018) with parameters set as follows: ICC, ρ0 = 0.70 (Nunnally, 1978); ρ1 = 0.54 (own calculation, see Table 1), α = 0.05, 1 - β = 0.80, k = 2, dropout = 0% (as no participant dropped out of our study). A final sample size of 115 participants was calculated and our result should be seen as a convenient sample as highly trained, national level youth soccer players are few in numbers and difficult to motivate to participate in research studies. All experimental procedures took place within the players’ usual training and testing routines. The study was approved by the Faculty’s Exercise Science and Training Ethical Committee of the University of Würzburg and performed following the declaration of Helsinki.

All players performed a ramp protocol followed by a verification phase on a motorized treadmill (Mercury, h/p/cosmos sports and Medical GmbH, Nussdorf-Traunstein, Germany) to assess VO_2max. The initial treadmill speed was set to 7 km·h^-1 increasing every minute by 1 km·h^-1 until volitional exhaustion. Overall, exhaustion was verified when three of the four following criteria were met: 1) plateau in VO₂, that is, an increase < 1.0 mL·min_-¹·kg^-1 despite an increase in velocity; 2) respiratory exchange ratio >1.1; 3) ratings of perceived exertion > 18; and 4) peak blood lactate (peak lactate) > 6 mmol·L^-1 after exercise. Addressing concerns that these criteria alone might not adequately determine VO_2max, a second trial to verify V̇O_2max was initiated three minutes after the completion of the ramp test, as suggested in existing literature (Poole and Jones, 2017). This trial was performed at a velocity of 1 km·h^-1 higher than that achieved during the final velocity of the ramp test and all participants were encouraged verbally to run for as long as possible.

Portable breath-by-breath analysis (Metamax 3B, CORTEX Biophysik GmbH, Leipzig, Germany) employing standard algorithms assessed oxygen consumption serving as the criterion measure. The breath-by-breath analyzer was calibrated prior to each individual test using a calibration gas (15.8% O₂, 5% CO₂ in N; Praxair, Düsseldorf, Germany) targeting the range of anticipated fractional gas concentrations and a precision 3L syringe. The oxygen sensor of this portable breath-by-breath gas analyzer provides reliable data with technical measurement error below 2% (Macfarlane and Wong, 2012). Additionally, a recent article by van Hooren (van Hooren et al., 2024) examined error rates of various gas analyzers. Across different intensities, the Cortex Metamax 3B used in this study exhibited an error rate of 1.64 ± 1.87%, which is among the lowest error rates observed for the investigated devices.

The YYIR2 involved progressively increasing running speed over 2 × 20 m, interspersed with a 10-s period of jogging around a marker placed 5 m behind the finish line after each 40 m. The time frame was controlled by audio signals. The test ended when the participant stopped voluntarily or was unable to complete the shuttle run in time due to exhaustion on two consecutive occasions. The primary outcome of this test was the total distance covered (Bangsbo et al., 2008). A recent review summarized the literature on the reliability of the YYIR2 and found that ICCs ranged from 0.86 to 0.96 and coefficients of variation ranged from 4.2 to 12.7% (Grgic et al., 2019).

We employed a multi-sensor wrist worn smartwatch (Forerunner 245, Garmin, Olathe, USA, Software:13.00). This smartwatch was chosen because it claims to provide the user with a V̇O_2max value and the manufacturer is among the top selling brands in the worldwide market for smartwatches (Statista Market Insights, 2023). The smartwatch features an optical heart rate (HR) sensor and a GPS receiver unit. To test the smartwatch in a scenario which resembles practice, we used and programmed the smartwatch as indicated by the manufacturer. Additionally, as the information which is provided by the manufacturer is also available to the soccer players, this is likely how the smartwatch would be used in real life scenarios. Each player’s individual ramp-test-derived HR_max was inserted into the smartwatch’s software. The specific algorithms utilized to assess VO_2max are not publicly disclosed by the manufacturer. However, according to the manufacturer, reliable HR and GPS-derived velocity data segments from individual runs are utilized to estimate VO_2max (Garmin Ltd, 2019)_. The manufacturer’s instructions to assess V̇O_2max with the smartwatch indicates that a person should run outdoors for at least 10 min with a HR “several minutes” above 70% of the HR_max (Garmin Ltd, 2019). Therefore, each participant wore the smartwatch while performing a warm-up run prior to their usual soccer training on the soccer pitch. To meet the manufacturers recommendations, the warm-up run was performed at an individually constant pace (which was checked by the coach by assessing GPS-derived velocity on the smartwatch) and the experienced coach ensured that the individuals HR was above 70% HR_max for several minutes by checking the HR of players throughout the run. The manufacturer indicates that the VO_2max assessment might improve following “a couple” of runs (Garmin Ltd, 2019). Therefore, out of the tested athletes and prior to any data analysis, 16 randomly selected athletes wore the smartwatch on a second occasion approx. 3-5 days after their first run.

To investigate the validity of the VO_2max provided by the smartwatch and the YYIR2, we compared values of the smartwatch and YYIR2 against the respiratory gas analysis using different statistical metrics: i) mean absolute percentage error, ii) Bland-Altmann analysis and ii) Intraclass Correlation Coefficients (ICC).

Data was checked for normal distribution using the Kolmogorov-Smirnoff-Test which revealed normal distribution for all variables (p ≥ 0.05).

To provide an indicator of the overall measurement error, mean absolute percent errors (MAPE) were calculated as previously performed (Lee et al., 2014). We followed previously performed statistical analysis to investigate validity to increase comparability of results between studies.

MAPE was calculated as average of absolute difference between the smartwatch and the respiratory gas analysis divided by the respiratory gas analysis values, multiplied by 100. The MAPE was calculated as it is a more conservative estimate of error that takes into account both over- and underestimation (Lee et al., 2014).

As previously performed (Mayorga-Vega et al., 2024), we calculated an ICC using a two-way random effects model with absolute agreement and single measurements [also known as ICC (2.1)] (Koo and Li, 2016) and was interpreted as follows:<0.5 poor, 0.5-0.75 moderate, 0.75-0.09 good, and 0.90-1.00 excellent (Koo and Li, 2016).

Bland–Altman plots were used to calculate the average difference and corresponding 95% limits of agreement.

No missing data or dropouts were reported. VO_2max as measured by respiratory gas analysis was 56.6 ± 4.9 ml·min^–1·kg^–1 (Figure 2). The players achieved a VO_2max of 54.2 ± 1.7 ml·min^–1·kg^–1 as assessed by the YYIR2. Table 2 summarizes the duration and covered distance of the runs performed while the players wore the smartwatch.

The Bland-Altman analysis comparing the different V̇O_2max assessments are displayed in in Figure 3. The average difference and 95% limits of agreement revealed by the Bland-Altman analysis when comparing VO2max values from smartwatch run 1 with the respiratory gas analysis, smartwatch run 2 with the respiratory gas analysis and the YYIR2 with the respiratory gas analysis are -3.16 ml·min^–1·kg^–1(-15.7 ml·min^–1·kg^–1; 9.3 ml·min^–1·kg^–1); 0.12 ml·min^–1·kg^–1 (-8.94 ml·min^–1·kg^–1; 9.17 ml·min^–1·kg^–1) and -2.40 ml·min^–1·kg^–1 (12.06 ml·min^–1·kg^–1; 7.25 ml·min^–1·kg^–1), respectively.

Table 1 presents Intraclass Correlation Coefficient and the Mean Absolute Percentage Error.

The aim of the present study was to assess the validity of VO_2max estimation provided by a consumer-grade smartwatch and the YYIR2 in comparison to breath-by-breath-analysis. The main findings of the present study showed that when performing two runs with the smartwatch to assess VO2max, the mean absolute percentage error decrease to 1.06% and agreement with respiratory gas analysis increases to 0.12 ml·min^–1·kg^–1 across the tested V̇O2max range, but comparably large confidence intervals indicate that individual values might show larger errors. Additionally, the mean absolute percentage error was 4.2% and showed to have poor agreement with respiratory has analysis.

The different statistical approaches employed in our study (i.e. mean absolute percentage error, intra-class correlation coefficient and Bland-Altmann analysis) are congruent, showing that it is favorable to perform two runs with the herein used Smartwatch to increase validity of provided VO₂max values.

Our findings expand upon existing research in different populations, which demonstrated a MAPE of 5.6% for V̇O_2max assessment using a smartwatch in runners (Düking et al., 2022). Interestingly, a previous study (Düking et al., 2022) revealed no decrease in MAPE with an increase in the number of runs as seen in our results. As the algorithms employed by the smartwatch are not disclosed by the manufacturer in detail, we can only speculate why in the present study the MAPE decreases with one more run. Potentially the algorithm needs a sufficient amount of individual data to calculate VO_2max properly which was not available in the present data, but in the data of previous publications. Another differentiating reason is that in our study, we included participants HR_max, which was not done in the previously published study with runners (Düking et al., 2022) which could affect VO_2max assessment. Consequently, it seems reasonable to manually insert HR_max when aiming to use the smartwatch to assess VO_2max, and based on our data, it appears advisable to conduct multiple runs with the smartwatch to reduce error rates. Remarkably, even with a relatively low MAPE, the ICCs for V̇O_2max assessment following one or two runs using the smartwatch are classified as "poor" and "moderate." This may be attributed to limited variability in our data, as suggested by the standard deviation in Table 1, and a small sample size, which are known to influence the ICC (Koo and Li, 2016). Consequently, future studies should investigate the ICC in larger groups of individuals and with more heterogenous data.

For the YYIR2, our study showed an ICC of 0.17 [95%CI: -5.7; 0.6] and a MAPE of 4.2%; ~2.3 ±·min^–1·kg^–1. A recent review summarized the literature on the reliability of the YYIR2 and found that ICCs ranged from 0.86 to 0.96 and coefficients of variation ranged from 4.2 to 12.7% (Grgic et al., 2019).

Our differing results for the YYIR2 compared to existing literature are speculative, but similar to our rationale regarding the ICCs for smartwatch-assessed V̇O_2max, we assume that the lower ICC for YYIR2-assessed V̇O_2max in our study may be due to our small sample size and homogeneous data (Koo and Li, 2016).

When comparing V̇O_2max assessment errors between YYIR2 and the smartwatch, YYIR2 shows a lower MAPE with only one smartwatch run. However, with two smartwatch runs, the MAPE for smartwatch-assessed V̇O_2max is smaller than that for YYIR2-assessed V̇O_2max.

The results of the smartwatch derived VO_2max are limited to runs performed during a warm-up of highly trained soccer players and generalization to other populations or settings should be performed with caution. Given the improvements in validity-related parameters after two runs performed with the smartwatch shown in our study, it could be that validity further increases with additional runs. In our experiments, we limited the number of runs to two because a previous analysis did not demonstrate any improvement in the validity of VO_2max with additional runs (Düking et al., 2022). Therefore, we did not assume that increasing the number of runs would improve the validity of VO_2max. Future studies should verify if validity of smartwatch assessed VO_2max increases with additional runs. As our aim was to assess concurrent criterion related validity of smartwatch derived V̇O_2max values, we did not assess other relevant criteria, such as reliability (Currell and Jeukendrup, 2008) or sensitivity (Düking et al., 2018) which future studies need to assess. Our results are limited to the herein investigated smartwatch, software version and the investigated VO_2max parameter range and additionally should be seen with caution due to a small sample size. VO_2max assessments of other models or software versions and other VO_2max parameter ranges may differ from these results. As alterations in algorithms do not have to be disclosed in detail since the device is a non-medical grade product, future smartwatch versions might yield different results for VO_2max assessment. Future research needs to continuously evaluate newer models of smartwatches and would be eased if manufacturers would have to disclose algorithm details and potential alterations.

We showed that when performing two runs with the smartwatch to assess VO2max, the mean absolute percentage error decrease to 1.06%, and agreement with respiratory gas analysis increases to 0.12 ml·min^–1·kg^–1 across the tested V̇O2max range, but comparably large confidence intervals indicate that individual values might show larger errors. Additionally, the mean absolute percentage error of the YYIR2 was 4.2% and showed to have poor agreement with respiratory has analysis. If VO_2max is assessed with the smartwatch, we advise to perform (at least) two runs to decrease the MAPE of the results. If practitioners are aware of the MAPE after two non-fatiguing runs during the warm-up prior to soccer practice and take this error into account, practitioners might use the herein investigated smartwatch (and software version) to assess VO_2max in soccer players within a VO_2max range of 49-69 ml·min^–1·kg^–1_. To eliminate potential inaccuracies in individual measurements, professionals should meticulously review the V̇O_2max results displayed by the smartwatch. If the data appears implausible, verification through gold-standard procedures, such as an all-out ramp test using a respiratory gas analysis, is recommended.