Obese individuals exhibit an increase in pancreatic β-cell mass; conversely, scarce nutrition during pregnancy has been linked to β-cell insufficiency in the offspring (reviewed in [1, 2]). These phenomena are thought to be mediated mainly through effects on β-cell proliferation, since a nutrient sensitive β-cell progenitor population in the pancreas has not been identified. Here, we employed the FUCCI (Fluorescent Ubiquitination-based Cell Cycle Indicator) system to investigate β-cell replication in real-time, and found that high nutrient concentrations induce rapid β-cell proliferation. Importantly, we found that high nutrient concentrations also stimulate β-cell differentiation from progenitors in the intrapancreatic duct (IPD). Using a new zebrafish line where β-cells are constitutively ablated, we further show that β-cell loss and high nutrient intake synergistically activate these progenitors. At the cellular level, this activation process causes ductal cell reorganization as it stimulates their proliferation and differentiation. Notably, we link the nutrient-dependent activation of these progenitors to a down-regulation of Notch signaling specifically within the IPD. Furthermore, we show that the nutrient sensor mechanistic Target Of Rapamycin (mTOR) is required for endocrine differentiation from the IPD under physiological conditions as well as in the diabetic state. This study thus reveals critical insights into how cells modulate their plasticity in response to metabolic cues and identifies nutrient sensitive progenitors in the mature pancreas.