People with Absolute Pitch can categorize musical pitches without a reference, whereas people with tone-color synesthesia can see colors when hearing music. Both of these special populations perceive music in an above-normal manner. In this study we asked whether AP possessors and tone-color synesthetes might recruit specialized neural mechanisms during music listening. Furthermore, we tested the degree to which neural substrates recruited for music listening may be shared between these special populations. AP possessors, tone-color synesthetes, and matched controls rated the perceived arousal levels of musical excerpts in a sparse-sampled fMRI study. Both APs and synesthetes showed enhanced superior temporal gyrus (STG, secondary auditory cortex) activation relative to controls during music listening, with left-lateralized enhancement in the APs and right-lateralized enhancement in the synesthetes. When listening to highly arousing excerpts, AP possessors showed additional activation in the left STG whereas synesthetes showed enhanced activity in the bilateral lingual gyrus and inferior temporal gyrus (late visual areas). Results support both shared and distinct neural enhancements in AP and synesthesia: common enhancements in early cortical mechanisms of perceptual analysis, followed by relative specialization in later association and categorization processes that support the unique behaviors of these special populations during music listening.