Photosynthesis operates at the bottom of the food chain to convert the energy of light into carbohydrates at a remarkable global rate of about 130 TW. Nonetheless, the overall photosynthetic process has a conversion efficiency of a few percent at best, significantly less than bottom-up photovoltaic cells. The primary photosynthetic steps, consisting of light harvesting and charge separation, are often presented as having near-unity quantum efficiency but this holds only true under ideal conditions. In this review, we discuss the importance of energy loss mechanisms to establish robustness in photosynthetic light harvesting. Thermal energy dissipation of light-harvesting complexes (LHCs) in different environments is investigated and the relationships and contrasts between concentration quenching of high pigment concentrations, photoprotection (non-photochemical quenching), quenching due to protein aggregation, and fluorescence blinking are discussed. The role of charge-transfer states in light harvesting and energy dissipation is highlighted and the importance of controlled protein structural disorder to switch the light-harvesting antennae between effective light harvesters and efficient energy quenchers is underscored. The main LHC of plants, LHCII, is used as a prime example.