High levels of cortisol, a sign of potential hypothalamic-pituitary-adrenal (HPA) axis dysregulation, have been associated with poor cognitive outcomes in older adults. Most cortisol research has focused on hippocampal-related abilities such as episodic memory; however, the presence of glucocorticoid receptors in the human prefrontal cortex suggests that cortisol regulation is likely to be associated with prefrontally-mediated executive function abilities. We hypothesized that elevated cortisol levels would be associated with poorer frontal-executive function in addition to episodic memory. We assessed cortisol from 15 saliva samples paralleling individual diurnal rhythms across three non-consecutive days in a group of 778 middle-aged twin men ages 51 to 60. Cognitive domains created from 24 standard measures included: general cognitive ability, verbal and visual-spatial ability, verbal and visual-spatial memory, short-term/immediate memory, working memory, executive function, verbal fluency, abstract reasoning, and psychomotor processing speed. Adjusting for general cognitive ability at age 20, age, race, and multiple health and lifestyle indicators, higher levels of average area-under-the-curve cortisol output across three days were significantly associated with poorer performance in three domains: executive (primarily set-shifting) measures, processing speed, and visual-spatial memory. In a 35-year longitudinal component of the study, we also found that general cognitive ability at age 20 was a significant predictor of midlife cortisol levels. These results possibly support the notion that glucocorticoid exposure is associated with cognitive functions that are mediated by frontal-striatal systems, and is not specific to hippocampal-dependent memory. The results also suggest that the direction of effect is complex.