Terrestrial ecosystem gross primary productivity (GPP) is the largest land-atmosphere carbon flux and the primary mechanism of photosynthetic fixation of atmospheric CO2 into plant biomass. Anomalous rainfall events have been shown to have a great impact on the global carbon cycle. However, less is known about the impact of these events on GPP, especially in Africa, where in situ observations are sparse. Here, we use a suite of satellite and other geospatial data to examine the responses of major ecosystems in Africa to anomalous rainfall events from 2003 to 2017. Our results reveal that higher-than-average groundwater storage in tropical ecosystems offsets the rainfall deficit during the dry years. While the inter-annual variations in GPP in semi-arid ecosystems are controlled by near surface soil water, deeper soil moisture and groundwater control the inter-annual variability of the GPP in dense tropical forests. Our study highlights the critical role of groundwater in buffering rainfall shortages and continued availability of near-surface water to plants through dry spells.