Autism spectrum disorders (ASD) are associated with differences in sensory sensitivity and affective response to sensory stimuli, the neural basis of which is still largely unknown. We used psychophysics and functional magnetic resonance imaging (fMRI) to investigate responses to somatosensory stimulation with three textured surfaces that spanned a range of roughness and pleasantness in a sample of adults with ASD and a control group. While psychophysical ratings of roughness and pleasantness were largely similar across the two groups, the ASD group gave pleasant and unpleasant textures more extreme average ratings than did controls. In addition, their ratings for a neutral texture were more variable than controls, indicating they are less consistent in evaluating a stimulus that is affectively ambiguous. Changes in brain blood oxygenation level-dependent (BOLD) signal in response to stimulation with these textures differed substantially between the groups, with the ASD group exhibiting diminished responses compared to the control group, particularly for pleasant and neutral textures. For the most unpleasant texture, the ASD group exhibited greater BOLD response than controls in affective somatosensory processing areas such as the posterior cingulate cortex and the insula. The amplitude of response in the insula in response to the unpleasant texture was positively correlated with social impairment as measured by the Autism Diagnostic Interview-Revised (ADI-R). These results suggest that people with ASD tend to show diminished response to pleasant and neutral stimuli, and exaggerated limbic responses to unpleasant stimuli, which may contribute to diminished social reward associated with touch, perpetuating social withdrawal, and aberrant social development.