Fear of cancer recurrence (FCR) is a common and persistent concern among breast cancer survivors. Little is known about factors associated with FCR in women with ductal carcinoma in situ (DCIS) or early invasive breast cancer (EIBC). Women with first primary DCIS, or stages I–IIA breast cancer were prospectively enrolled in a quality-of-life study and completed interviews at 4–6 weeks, 6 months, and 2 years after definitive surgical treatment. In three stepwise multivariable linear regression models, including both time-independent and time-varying variables measured at each respective interview, we identified independent correlates of mean FCR scores (range 1–6) using four items from the Concern About Recurrence Scale (CARS) at 2-year follow-up. Of 506 disease-free patients at 2-year follow-up (mean [SD] age, 58 [10] years; 81% White; 34% DCIS), the average FCR score of 2.0 was low. However, 145 (29%) reported moderate-to-high levels of FCR (scores 3.0–6.0). All three models showed that younger age, stage IIA breast cancer (vs. DCIS), lower social support, and elevated anxiety were consistently associated with higher FCR at 2-year follow-up (each P < 0.05; final models R² = 0.25–0.32). DCIS patients reported lower FCR than stage IIA patients (each P ≤ 0.01) but had similar FCR as stage I patients. Although mean FCR was low, 29% of DCIS and EIBC survivors reported moderate-to-high levels of FCR at 2-year follow-up. Management of anxiety, provision of social support, and patient education may help reduce FCR among DCIS and EIBC survivors, especially among younger survivors.