We present a new method for atmospheric correction of remote Visible Shortwave Infrared (VSWIR) imaging spectroscopy. Our approach fits a combined model of atmospheric scattering, absorption, and surface reflectance across the solar reflected interval from 380 to 2500 nm. This can estimate spectrally-broad atmospheric perturbations such as aerosol effects that are difficult to retrieve with narrow spectral windows. A probabilistic formulation from Optimal Estimation inversion theory accounts for uncertainties in model parameters and measurement noise. This paper presents a field experiment using NASA's Next Generation Visible/Near Infrared Imaging Spectrometer (AVIRIS-NG) with analysis of retrieval accuracy and information content. The inversion outperforms traditional approaches, achieving mean reflectance accuracy of 1.0% on diverse validation surfaces. Predicted posterior distributions fully explain the observed discrepancies, demonstrating the first closed uncertainty budget for VSWIR imaging spectrometer atmospheric correction. This shows the potential of combined surface/atmosphere fitting to advance the accuracy and statistical rigor of remote reflectance measurements.