Forecaster Use and Evaluation of Real-Time 3DVAR Analyses during Severe Thunderstorm and Tornado Warning Operations in the Hazardous Weather Testbed (2024)

a. Data collection from forecasters

The primary feedback from the forecasters was qualitative in nature and acquired through a variety of processes, including participant (forecaster) observation by the project leads (e.g., Morss and Ralph 2007), forecaster surveys (e.g., Lusk et al. 1999; Heinselman et al. 2009), discussion and interviews between the lead scientists and forecasters during and postevent (e.g., Heinselman et al. 2012), and direct forecaster comments and screenshots collected via blogs posted by the individual forecasters during an event [capturing the event evolution and warning-decision process similar to Andra et al. (2002)]. Each of these methods is described in greater detail below within the context of daily operations in the HWT.

Forecasters typically worked in pairs with each pair acting as a different NWS WFO from around the CONUS. On extremely busy weather days, two sets of forecasters may work in different sectors of a WFO. Forecasters were asked not only to issue relevant warnings using WarnGen (a warning generation tool) within the AWIPS platform, but also to blog about their warning decision or items they found interesting during their storm interrogation period. Typically, one forecaster would concentrate on the warning products while the other would generate the blog posts (note: the use of the blog was increased in 2012 following an initial review of data from the 2011 experiment). The forecasters had the ability to take screenshots of their desktop (and were encouraged to do so anytime they saw a feature of interest) and use that image or series of images for inclusion in a blog post. These blog posts not only provided an opportunity to understand what products or combination of products forecasters were using in real time, but also a description of why forecasters were using those specific products and a deeper understanding how they were using the products within their warning-decision process.

Throughout an event, project scientists and developers were also able to have brief conversations with the forecasters, either to clarify what the forecasters were examining or to understand why a forecaster found a product useful (or not). Additionally, forecasters completed online surveys at the end of every day; the surveys contained a series of product rankings (using a Likert scale) and open-ended questions about product use and understanding. The majority of the questions from the survey were open ended, meaning that the analysis of these data was completed via coding and categorizing the forecaster responses (e.g., Boyatzis 1998; Creswell 2002). Responses were incrementally categorized through two techniques. First, an automated text analysis in which repeated answers (singular words or phrases) were grouped together followed by a manual categorization of repeated themes. Second, codes (automated and manually constructed) were organized using a tree structure to easily view grouped themes. Examples of repeated themes included warning decision, time or time scale, situational awareness, visualization, trends, latency, and confidence. A debriefing session covering the previous day of operations began every shift; this focused on a review of the 3DVAR products and how they were utilized within the context of the particular event. Finally, every week of the experiment concluded with an open discussion period between all the scientists/developers and forecasters to retrieve any final thoughts and opinions from using the data during a number of different settings and events.

This study uses a trade-off between realism and control that numerous operational impact studies have dealt with in the past (e.g., Stewart et al. 1989), namely in this case, “control” was given up to produce a realistic operational setting for forecaster evaluation. This issue was alleviated slightly by the number and variety of types of cases evaluated by a wide range of NWS forecasters (skill and experience included) such that feedback should be applicable to the larger field of NWS severe weather operations, not just one or two types of case studies. However, the current experiment implementation does produce the inherent limitation that quantifiable statistics cannot be related as they would from a study using a controlled, repeatable, real-time simulation. Finally, the forecasts and warnings issued by the HWT participants were not compared to forecasts and warnings made from the actual WFO nor were verification skill scores computed due to the variety of uncontrolled factors involved in HWT operations including, but not limited to, individual warning-forecaster experience, knowledge of the local area, and differences in regional storm spotter information and emergency management information.

b. Weekly schedule and forecaster expertise

During the EWP, between four and six NWS forecasters visited the test bed each week. In 2011, the forecasters were split into shifts, early (1000–1800 local time) and late (1300–2100 local time) Tuesday–Thursday, with the late shift typically focusing on 3DVAR and other convection and warning-based products. Monday was used primarily as a training day, as forecasters were introduced to each of the new products. This period contained discussion with the project scientists followed by a period of hands-on training within the AWIPS platform using the Warning Event Simulator (WES) to examine the products within the context of the 19 May 2010 tornadic supercell event in central Oklahoma. In 2012, the schedule was updated such that the forecasters all worked the same 8-h shift beginning as early as noon or as late as 1400 local time dependent upon the expected timing of the severe weather on a given day. Additionally, in 2012, the majority of the training, including the hands-on WES event (switched to the 24 May 2011 central Oklahoma tornadic supercell event), was completed prior to arrival in Norman for the experiment, freeing an additional day for real-time product evaluation.

There was a wide range in forecaster warning expertise among the HWT participants, as participants included NWS journeymen and lead forecasters as well as science operations officers and meteorologists in charge. The overall experience level was weighted by forecasters with a greater number of years of NWS warning experience. In 2011, 27.3% of survey respondents were from forecasters with more than 20 yr of experience, 13.6% with 16–20 yr, 20.5% with 11–15 yr, 31.8% with 6–10 yr, and 6.8% with 5 yr or less of warning experience. The distribution of experience was modified slightly in 2012: 28.2% of surveys were completed by forecasters with more than 20 yr of experience, 19.4% with 16–20 yr, 6.8% with 11–15 yr, 21.4% with 6–10 yr, and 24.3% with 5 yr or less of warning experience.

c. Cases evaluated

A variety of weather and storm modes were examined during the 2011 and 2012 EWPs. On almost all operational days in the HWT, we were able to operate in a region where severe or marginally severe activity occurred. In addition to the training cases reviewed by all forecasters, forecasters evaluated 14 different real-time events occurring on nine separate days during the 2011 EWP and 26 different events occurring over 18 days during the 2012 EWP. Potential bias was introduced via forecaster expectation of the data because the training case for all forecasters included an ideal case for supercell storms surrounded by multiple radars, while the real-time events included tornadic supercells, squall lines, and multicell and isolated storms with varying radar coverage patterns; the dates and event types are summarized in Table 1.

a. Forecaster surveys

At the end of every shift in the HWT, forecasters were asked to complete an online survey regarding the various experimental products they evaluated that day. The 3DVAR section contained six questions, each to be evaluated within the context of the particular event worked during that shift. Five of the six questions were open ended:

1. Did the 3DVAR products provide a realistic visualization of the storm structure and morphology? Why or why not?

2. Did access to the 3DVAR products add value to your warning–decision making process? Please explain.

3. What do you feel were the strengths and weaknesses of the 3DVAR analysis?

4. Please describe any additional related features, visualizations, or products that you would like to see developed.

5. In future years, we will be implementing a forecast step that will provide similar types of products that you have seen this week from the 3DVAR analysis (e.g., updraft intensity, vorticity, wind field, etc.), but from ensembles of 5–20 min in the future. Do you envision your warning–decision making process changing with access to these products? If so, how?

This final question was only answered once by every forecaster on the final survey of the week. All other questions were available after every event. In addition to the open-ended questions, forecasters were also asked to rank each product from “not at all useful” to “extremely useful,” choosing from nine different options, as well as “N/A” if a forecaster did not evaluate that product for a particular shift. In total, the survey was completed 41 times in 2011 and 88 times in 2012.

Results from the Likert scale question are shown in Fig. 2. The average rankings from the question can help quantify when the product was used and how frequently the forecaster found it useful. Due to the lack of use combined with low average rankings, the downdraft, downdraft track, and updraft volume products were removed between the 2011 and 2012 experiments. Overall, the updraft products were the highest rated over both years of the experiment with a majority of respondents ranking it between “very useful” and “extremely useful.” During 2011, the vorticity products (composite, low level, midlevel, and tracks) were used more often and ranked markedly higher than in the 2012 responses. This is likely due in part to the development of an updraft helicity product (which combines updraft and vorticity into a single grid) between the 2011 and 2012 experiments, but possibly also due to the slightly higher percentage (44% versus 37%) of supercell-focused events. In 2011, the high N/A response to the usefulness of the 2D wind vectors was primarily because the product was not available in AWIPS, though the forecasters could view the product on the situational awareness display in the HWT via the WDSSII platform. In 2012, with the introduction of AWIPS2 into the HWT, the vectors could be viewed directly on the forecaster display and the overall utilization of the product increased, though the responses showed a more varied degree of actual usefulness. Though the majority of forecasters used the simulated reflectivity product, there was a large amount of spread regarding how useful forecasters actually found that product, with the average ranking it only “moderately useful.”

From the coded responses to the open-ended questions, we summarize that the forecasters found the 3DVAR products provided a realistic representation of the storms viewed, but with some possible qualifications. One representative forecaster response exemplified some of these qualifications: “Yes [the 3DVAR products were realistic], but only when convection was relatively close to the radar, and two or more radars were able to look at the storm. Storms too far away from the radar (i.e., lowest slice greater than 8 kft) were not represented well by 3DVAR.” Almost all of the forecaster responses noted that the 3DVAR products added value to the warning-decision process through added confidence and continuity on the warnings. This added confidence was cited in survey responses by forecasters during both years of the experiment. Specifically, forecasters found that the 3DVAR products “acted like an algorithm that helped highlight attention to areas that deserved closer inspection using the base data” and “the combined nature of the products (using multiple radars) also served to highlight storms of interest that might not be well sampled by one radar, but were by another.”

However, the main weakness of the product, as noted by multiple respondents, was the latency of the product. In 2011, multiple separate survey responses noted that it was difficult to use the products within a warning context due specifically to latency issues. Additionally, some forecasters noted that “severe thresholds” varied within the products from case to case. One respondent worried that the access to the 3DVAR synthesis products and trends may discourage forecasters from performing a complete analysis of the storms using base radar data such as reflectivity and radial velocity. A final weakness of 3DVAR products within the survey results regards the spatial resolution: some forecasters believed the spatial resolution of the products to be too low (especially when compared to current WSR-88D data) and that increased resolution is desired in future implementations (though not at the cost of additional latency).

The majority of forecasters either noted that they were happy with the current display options or skipped the question regarding additional visualizations and features. However, the answers that were provided did contain the same themes: a simple visualization of a “goodness” of the analysis (e.g., how many radars are incorporated into the current analysis) and an easy way to visualize the three-dimensional nature of the products (e.g., an AWIPS-style “all tilts” visualization). The former would provide the forecaster with an understanding of how much he or she could trust the current analysis and the latter would provide the forecaster with the ability to scroll through all levels instead of loading each separately in different panels.

Responses to the final question (answered 14 times in 2011 and 28 times in 2012) noted that added confidence and situational awareness would be the likely use of 15–20-min forecast ensemble information of the same format. While none of the forecasters could see themselves removing radar base data from the actual warning-decision process, numerous respondents mentioned the likelihood of providing additional lead time, assistance in determining the type of warning (e.g., severe thunderstorm versus tornado), and guidance on whether to extend or terminate a current warning. These statements were often made in conjunction with the qualification of “if the system is proven reliable.” Multiple respondents also noted that the products and display will need to be simple and easy to understand; if not, it could muddle the analysis and actually delay warning times.

b. Examples from real-time analyses

As previously noted, the real-time blogging by forecasters allowed the project scientists to review cases postevent with forecaster screenshots and notes providing unbiased insights into which products were used and how they were utilized during storm interrogation. Forecasters were encouraged to utilize the blog during both years; however, after reviewing 2011 data, project scientists in the EWP placed greater emphasis on the blogging aspect in 2012. Typically, forecaster comments concentrated on current analyses from storm interrogation and often explained which factors and products contributed to a warning decision. Below are two examples of forecaster comments within the context of the actual operational event.

c. Synthesis of forecaster discussion

Discussion with forecasters postevent and during the weekly debriefing session concentrated on how they integrated the 3DVAR products into their warning procedures and how, if at all, the availability of these products modified their storm interrogation and warning-decision process, as well as any difficulties they encountered. Similar to details provided via the surveys and blog posts, the discussion illustrated that forecasters found the 3DVAR products provided a clearer situational awareness picture, were easy to integrate, and provided additional confidence during the warning-decision process.

Most forecasters greatly utilized both the updraft and vorticity products during warning operations. The updraft helicity and storm-top divergence products introduced in 2012 were also well utilized. Forecasters found these products best highlighted the most intense areas of the storm and provided information on cycling mesocyclones. Forecasters particularly utilized these products (both alone and in combination) when trying to diagnose a large number of storms and when “sitting on the fence” about issuing a warning. Additionally, forecasters mentioned that both the updraft and vertical vorticity products were more efficient to view than existing algorithms to diagnose storm intensity and rotation.

Within AWIPS/AWIPS2, forecasters had the option of combining the 3DVAR products with a multitude of sensors and products, both experimental and those currently available in NWS forecast offices. Many forecasters found that combining the updraft or updraft track products with lightning (total and cloud to ground) and/or MESH plots (instant and track) provided a more complete view of the storm intensity and trends. The correlations they saw between these different products typically provided increased confidence in the storm diagnosis or warning decision. However, some forecasters were concerned about the time lag and resulting displacement from current WSR-88D base data, while a few found the latency to be bad enough that it was impossible to use the 3DVAR data in a real-time sense and useless for issuing warnings.

Another display combination that was often utilized included radial velocity from multiple elevations from the nearest WSR-88D radar and vertical vorticity from the 3DVAR analysis (e.g., Fig. 6). Forecasters relied on this type of display to provide guidance on how the 3DVAR analyses of vertical vorticity compared to single-radar velocity trends, as well as to provide a visualization of the latency of the product relative to the current WSR-88D scans.

A reoccurring theme in weekly debriefs was the variance in specific values and the quality of the 3DVAR products when storms were close to a radar or when multiple radars were incorporated into the analysis. Forecasters noted that the supercell events occurring on 24 May and 9 June 2011 seem to be handled particularly well within the analyses, perhaps due to their close proximity to the WSR-88D. Specifically, the rapid increase in strength of both the vorticity and vertical velocity for the 9 June 2011 supercell was the deciding factor for one forecaster to issue a tornado warning on the storm approaching Wichita, Kansas. Because the 3DVAR analyses were developed with the intent of diagnosing supercells (Gao et al. 2013), it is not surprising that it handled these events much better than other cases. However, the forecasters were not limited to supercell events and these other storm modes depicted some of the limitations of the data at this point. Storms that were not quite as intense or at farther distances from the radar often contained poor artifacts of the analysis. Specifically, on 17 May 2011 in the mid-Atlantic region, interacting storms and shallow convection often contained large up- and downdrafts in regions of clear air between storms or at unrealistic heights above the storm (e.g., large downdraft above storm top). It is likely the 3DVAR cost function constrained by mass continuity produced some of these artifacts, particularly in regions of poor resolution away from a radar. Additionally, sidelobe contamination and other radar data quality issues would occasionally make it into the analysis. However, none of the forecasters noted these as significant enough problems to remove all value from using the 3DVAR analyses.

Multiple forecasters commented that they would like the ability to display and toggle through the 3DVAR products in a manner similar to an all-tilts feature available for radar data in AWIPS. The all-tilts option allows the forecaster to simply hit the up arrow to move spatially from one radar tilt (e.g., 0.5°) to the next highest (0.9°). Similarly, forecasters would like to move from one level of vertical velocity grids to the next highest with just the up and down arrows.

A recommendation for the creation of an updraft helicity, similar to that currently available from ensemble and short-range forecast models such as the High-Resolution Rapid Refresh (HRRR), and storm-top divergence products was the result of discussion periods with the forecasters in 2011. As seen from the surveys, these products were particularly well utilized during 2012 operations. Due to limits on the screen space a single forecaster has access to during warning operations, combination products, such as updraft helicity, can quickly become heavily relied upon by forecasters.

Finally, the weekly summaries captured the ease with which the forecasters were able to integrate the 3DVAR products into the warning-decision process. It was noted, “the multiple-radar integration really helped fill in gaps of having to analyze multiple radars separately, especially where storms from one radar were in the purple haze of the range-folded obscuration.” Additionally, the vorticity products (both low and midlevel) reinforced where the strongest rotation was tracking and were of particular value in determining whether storm tracks were deviating, which guided polygon placement.