Agricultural productivity is sensitive to temperature and precipitation extremes, which are increasing with climate change. It is well-established that planting and growing season weather affects crop yields, but conditions in other seasons may also be important. We generate a county-level dataset that links yields for six major crops in the US with agriculture-relevant weather variables for four distinct seasons (planting, growing, harvest, and non-growing) over the years 1983 to 2021. The data include binned temperature variables, precipitation, and the Palmer Drought Severity Index (PDSI). We demonstrate that models using weather conditions from all four crop calendar seasons segments outperform those using only growing season or yearlong data at predicting yields, highlighting the importance of considering the impact of non-growing season weather on agricultural productivity.