Poor physical fitness and low serum vitamin D are known to be modifiable risk factors for cognitive declines with normal aging. We investigated the association of physical fitness and serum vitamin D levels with global cognitive function in older adults. In this cross-sectional study, a total of 412 older Korean adults (108 men aged 74.4 ± 6.0 years and 304 women aged 73.1 ± 5.4 years) completed the Korean version of Mini-Mental State Examination (MMSE) to assess global cognitive performance and the senior fitness test to assess strength, flexibility, agility, and endurance domains of physical fitness. Body mass index, percent body fat, serum vitamin D, geriatric depression scale (GDS), level of education, smoking, and history of cardiovascular or cerebrovascular disease were also assessed as covariates. Age, sex, GDS, and body fatness were negatively associated with MMSE-based cognitive performance. Serum vitamin D and physical fitness were positively associated with MMSE-based cognitive performance. Multivariate linear regression showed that agility (partial R² = -0.184, p = 0.029) and endurance (partial R² = 0.191, p = 0.022) domains of physical fitness along with serum vitamin D (partial R² = 0.210, p = 0.012) were significant predictors for global cognitive performance after controlling for covariates (i.e., age, sex, education, GDS, body fatness, and comorbidity index). The current findings of the study suggest that promotion of physical fitness and vitamin D supplementation should be key components of interventions to prevent cognitive decline with normal aging.