Older male premutation carriers of the FMR1 gene are associated with the risk of developing a late-onset neurodegenerative disorder, fragile X-associated tremor/ataxia syndrome (FXTAS). Although previous postmortem and in vivo MRI studies have indicated white matter pathology, the regional selectivity of abnormalities, as well as their relationship with molecular variables of the FMR1 gene, has not been investigated. In this study, we used diffusion tensor imaging (DTI) to study male premutation carriers with and without FXTAS and healthy gender-matched controls. We performed a tract of interest analysis for fractional anisotropy (FA), axial and radial diffusivities of major white matter tracts in the cerebellar-brainstem and limbic systems. Compared with healthy controls, patients with FXTAS showed significant reductions of FA in multiple white matter tracts, including the middle cerebellar peduncle (MCP), superior cerebellar peduncle, cerebral peduncle, and the fornix and stria terminalis. Significant reduction of FA in these tracts were confirmed by a voxel-wise analysis using Tract-Based Spatial Statistics. Analysis of axial and radial diffusivities showed significant elevation of these measures in MCP even among premutation carriers without FXTAS. Furthermore, regression analyses demonstrated clear inverted U-shaped relationship between CGG repeat size and axial and radial diffusivities in MCP. These results provide new evidence from DTI for white matter abnormalities in the cerebellar-brainstem and limbic systems among individuals with the fragile X premutation, and suggest the involvement of molecular mechanisms related to the FMR1 gene in their white matter pathology.