The Geostationary Environment Monitoring Spectrometer (GEMS) onboard the Geostationary Korean Multi-Purpose Satellite 2B was successfully launched in February 2020. GEMS is a hyperspectral spectrometer measuring solar irradiance and Earth radiance in the wavelength range of 300 to 500 nm. This paper introduces the spectral calibration algorithm for GEMS, which uses a nonlinear least-squares approach. Sensitivity tests for a series of unknown algorithm parameters such as spectral range for fitting, spectral response function (SRF), and reference spectrum were conducted using the synthetic GEMS spectrum prepared with the ground-measured GEMS SRF. The test results show that the required accuracy of 0.002 nm is achievable provided the SRF and the high-resolution reference spectrum are properly prepared. Such a satisfactory performance is possible mainly due to the inclusion of additional fitting parameters of spectral scales (shift, squeeze, and high order shifts) and SRF (width, shape and asymmetry). For the application to the actual GEMS data, in-orbit SRF is to be monitored using an analytic SRF function and the measured GEMS solar irradiance, while a reference spectrum is going to be selected during the instrument in-orbit test. The calibrated GEMS data is expected to be released by the end of 2020.