Object-Based Verification of a Prototype Warn-on-Forecast System

The core information for this publication's citation.: 
Skinner, P., D. Wheatley, K. Knopfmeier, A. Reinhart, J. Choate, T. A. Jones, G. Creager, D. Dowell, C. Alexander, T. Ladwig, L. Wicker, P. Heinselman, P. Minnis, and R. Palikonda (2018), Object-Based Verification of a Prototype Warn-on-Forecast System, Skinner Et Al., doi:10.1175/WAF-D-18-0020.1.

An object-based verification methodology for the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been developed and applied to 32 cases between December 2015 and June 2017. NEWS-e forecast objects of composite reflectivity and 30-min updraft helicity swaths are matched to corresponding reflectivity and rotation track objects in Multi-Radar Multi-Sensor system data on space and time scales typical of a National Weather Service warning. Object matching allows contingency-table-based verification statistics to be used to establish baseline performance metrics for NEWS-e thunderstorm and mesocyclone forecasts. NEWS-e critical success index (CSI) scores of reflectivity (updraft helicity) forecasts decrease from approximately 0.7 (0.4) to 0.4 (0.2) over 3 h of forecast time. CSI scores decrease through the forecast period, indicating that errors do not saturate during the 3-h forecast. Lower verification scores for rotation track forecasts are primarily a result of a high-frequency bias. Comparison of different system configurations used in 2016 and 2017 shows an increase in skill for 2017 reflectivity forecasts, attributable mainly to improvements in the forecast initial conditions. A small decrease in skill in 2017 rotation track forecasts is likely a result of sample differences between 2016 and 2017. Although large case-to-case variation is present, evidence is found that NEWS-e forecast skill improves with increasing object size and intensity.

PDF of Publication: 
Download from publisher's website.
Research Program: 
Modeling Analysis and Prediction Program (MAP)