Learned associations between object properties, such as weight and size, allow for quick and accurate manipulations of objects that we encounter repeatedly. This integration of learned sensory information reduces the overall computational load of our visuomotor system when interacting with familiar objects. In the laboratory, even novel associations can be quickly established after only brief training. Haffenden and Goodale in J Cogn Neurosci 12:950–964 (2000) found that learned associations between color and size affected grip scaling for manual estimations of size and visually guided grasping. But, how specific are these learned associations? In the current study, lighter-shaded “untrained” target objects were added to Haffenden and Goodale’s color-size association paradigm to determine if the learned associations made by the perception and action systems are equally tolerant to within-category color changes. During perceptual estimations, training was generalized within color categories––manual estimations of size were influenced by both the trained and lighter-shaded untrained colors. In contrast, grasping was not influenced by the untrained colored blocks. These results demonstrate how the perception and action systems differ in their incorporation of learned perceptual information. In contrast to the object specific associations needed for grasping, our perceptual system is more categorical and uses generalized perceptual grouping strategies when relying on learned color information.