Polyphenols are natural antioxidant compounds ubiquitously found in plants and, thus, ever present in human nutrition (tea, wine, chocolate, fruits and vegetables are typical examples of polyphenol-rich foods). Widespread evidence indicate that polyphenols exert strong antioxidant, anti-inflammatory, anti-microbial and anti-cancer activities, and thus, they are generally regarded to as all-purpose beneficial nutraceuticals or supplements whose use can only have a positive influence on the body. A closer look to the large body of results of years of investigations, however, present a more complex scenario where polyphenols exert different and, sometimes, paradoxical effects depending on dose, target system and cell type and the biological status of the target cell. Particularly, the immunomodulatory potential of polyphenols presents two opposite faces to researchers trying to evaluate their usability in future cancer therapies: on one hand, these compounds could be beneficial suppressors of peri-tumoral inflammation that fuels cancer growth. On the other hand, they might suppress immunotherapeutic approaches and give rise to immunosuppressive cell clones that, in turn, would aid tumor growth and dissemination. In this review, we summarize knowledge of the immunomodulatory effects of polyphenols with a particular focus on cancer microenvironment and immunotherapy, highlighting conceptual pitfalls and delicate cell-specific effects in order to aid the design of future therapies involving polyphenols as chemoadjuvants.