Across cultures, people associate colours with emotions. Here, we test the hypothesis that one driver of this cross-modal correspondence is the physical environment we live in. We focus on a prime example – the association of yellow with joy, – which conceivably arises because yellow is reminiscent of life-sustaining sunshine and pleasant weather. If so, this association should be especially strong in countries where sunny weather is a rare occurrence. We analysed yellow-joy associations of 6625 participants from 55 countries to investigate how yellow-joy associations varied geographically, climatologically, and seasonally. We assessed the distance to the equator, sunshine, precipitation, and daytime hours. Consistent with our hypotheses, participants who live further away from the equator and in rainier countries are more likely to associate yellow with joy. We did not find associations with seasonal variations. Our findings support a role for the physical environment in shaping the affective meaning of colour.