A New Method for Evaluating Regional Air Quality Forecasts

1. A New Method for Evaluating Regional Air Quality Forecasts Helen Dacre Suzanne Gray, Stephen Belcher Heini Wernli

2. Talk Outline • Studying pollution transport using the UM • Frontal flows • Convection and coastal processes • Qualitative model evaluation • Quantitative model evaluation • European Tracer EXperiment • SAL evaluation method • Future work • MACC AQ model inter-comparison

3. Pollution Transport Mixing Convection Advection Boundary layer ~1km

4. Frontal Flows Surface pressure analysis 00UTC on 15/11/94 Release site

5. Frontal Flows 10.0 10.0 ngm-3 ngm-3 1.00 1.00 0.10 0.10 T+12 T+24 0.01 0.01 T+12 T+24 Height (km) Height (km) Dacre (2009) Cold front

6. Convection and Coastal Processes UK Met Office surface pressure analysis 00UTC on 9/5/05

7. Convection and Coastal Processes Tracer in free troposphere integrated over height wind direction 17 UTC 17 UTC kg/m2 sea land coast

8. AMPEP Observations (Aircraft Measurements of chemical Processing and Export fluxes of Pollutants) AMPEP flight path AMPEP flight height wind direction I D

9. Boundary layer profile in central England Model tracer profile AMPEP CO profile 17:24 UTC boundary layer top background concentration Dacre et al. (2007)

10. Mass conservation

11. Mass Conservation 200 Mass/total input mass (%) 100 Constant emission Time evolution of the total amount of tracer in the UM. Release over 1 grid box (red), UM conserved tracer (blue)

12. European tracer experiment (ETEX)

13. ETEX • Aim: To evaluate the ability of a variety of long-range dispersion models to predict pollution concentrations across Europe • Inert and non-depositing tracer released from NW France • Tracer perfluoromethylcyclopentane (PMCP) • 168 surface station measurements

14. ETEX Observational Network Release site

15. ETEX 1 Overview AVHRR infrared 08UTC 23/10/94 Synoptic Analysis at 00UTC 24/10/94

16. Observed Tracer Observed non-zero tracer concentrations 24, 36 and 48 hours after the start of the tracer release.

17. Traditional Evaluation 50km UM concentration ng/m3 12km UM concentration ng/m3 NMSE Observed concentration ng/m3 Observed concentration ng/m3 Scatter plot of observed vs modelled tracer concentration (a) 50km, (b) 12km. Zero concentrations are set to 0.01 ng m-3, dotted line shows factor of 2. Time since start of release (hours)

18. SAL Evaluation Method (Wernli et al 2008) • Takes into account the spatial correlation existing within the tracer field. • Considers aspects of the Structure, Amplitude and Location of the tracer field independently. • Interpolate observations and modelled concentrations onto same grid. • Define threshold to identify features

19. Observed and Modelled Tracer T+12 T+36 T+60 T+12 T+36 T+60 Observed (top) and 12km modelled (bottom) tracer concentrations interpolated onto a 1o1o grid, 12, 36 and 60 hours after the start of the tracer release.

20. SAL : Amplitude Component • The amplitude component quantifies errors in the over- or under-prediction of the tracer field. • Where N is the number of gridcells in the domain, Qi,j is the concentration in each gridcell. • -2 ≤ A ≤ +2 and 0 denotes a perfect amplitude forecast.

21. SAL Diagnostics Amplitude Time since start of release (hours) Observed UM NAME

22. SAL: Location Component • The location component quantifies errors in the position of the tracer field. • L1 measures the normalised distance between the centres of mass of the predicted and observed field. • L2 measures the averaged distance between the centre of mass of the total field and individual objects. • 0 ≤ L ≤ 2 but 0 does not necessarily denote a perfect location forecast.

23. SAL Diagnostics ~422 km d=2816 km ~282 km Location ~141 km Time since start of release (hours) Observed UM NAME

24. SAL: Structure Component • The structure component quantifies errors in the predicted shape and size of the tracer field. • -2 ≤ S ≤ +2 and 0 denotes a perfect structure forecast. • + S → model predicts widespread tracer in a situation of small tracer features. • - S → model predicts a small peaked feature in a situation of a large flat feature.

25. Idealised Example of Structure S>0 S=0 S<0

26. SAL Diagnostics Structure Time since start of release (hours) Observed UM NAME

27. Model Physics Amplitude Time since start of release (hours) NAME UM Height (m) Dacre (2010)

28. Quantitative Evaluation Summary • SAL quantitative results are close to a subjective visual inspection of the predictions. • SAL can be used to; • Identify differences in tracer transport between models. • Compare forecasts from different resolution simulations. • Evaluate model performance over longer time periods and hence identify systematic errors. • Evaluate ensemble predictions with varying meteorology or emissions

29. MACC Ensemble 11 air quality model forecasts of hourly and daily mean SO2, PM10, O3, NO2 and CO. 9 offline CTM’s, 2 ‘online’ Met-CTM’s and ensemble mean.

30. AirBase Observation Network European AirBase network. >7000 surface stations recording hourly and daily mean SO2, PM10, O3, NO2, CO

31. Future Work • Do AQ models with common met input data show similar SAL performance? • Online vs offline models • Meteorological resolution • How is the forecast skill of air quality models related to the forecast skill of the driving met model? • Low-level wind skill and SAL performance for long-lived pollutants • SAL precipitation performance and SAL performance for soluble species • Dependence on synoptic situation

32. Extra figures

33. AMPEP Observations (Aircraft Measurements of chemical Processing and Export fluxes of Pollutants) • Direct measurements of the mass budgets of pollutants in the boundary layer over the UK Upwind air measurements (background concentrations) Z Downwind air measurements (mass flux of pollution coming off UK)

35. UM Tracer UM tracer concentration interpolated onto 1o ×1o lat/lon grid 12, 24, 36, 48 and 60 hours after the start of the tracer release.

36. ‘Double Penalty’ Problem Observed 12km UM 50km UM NMSE Time since start of release (hours)

37. New Evaluation Methods • Neighbourhood based • Consider neighbouring observation in space and time thus relaxing the requirements for perfect matching (Roberts, 2007). • Need dense network of observations such as radar. • Feature based • Identify features in predicted and observed fields and assess different attributes associated with each pair of forecast-observed features. • Need observations on a regular grid but do not need such a dense network of observations.

38. Frontal Flows Schematic Release site

40. Future Work – Multi-model evaluation SAL: aLMo 24hr precipitation SAL: ECMWF 24hr precipitation 2 2 0 0 Amplitude Amplitude -2 -2 -2 2 -2 2 Structure Structure SAL diagrams for 2001-2004 summer precipitation forecasts in Elbe region. Every dot shows S, A and L values for a particular day (Wernli et al. 2008).

41. Model Physics Amplitude Time series of amplitude component for UM (solid), UM with no convection (dotted), NAME (dashed) and NAME with no convection (dash-dot) simulations.