Third SRNWP Workshop on Mesoscale Verification

17-18 May 2006, Sofia (Bulgaria)

Actions and Recommendations (preliminary version)

1. Verification scores should be accompanied by their statistical significance

It is a fact that the statistical results (moments or scores) of model performance presented in conferences or published in scientific journals are rarely accompanied by their statistical significance or, which is the same, by their significance level.

For example, when results are presented as 1-D curves, points stand not very often on vertical line segments whose length would inform us about the statistical significance of the values

This contrasts with other sciences where it is self-evident to indicate the statistical significance of results presented.

Without information on the statistical significance the part of the chance is unknown.

The drawback is that the determination of statistically significant results will generally imply the necessity of a larger number of realisations (in NWP, of model integrations) that we are used to perform. And if we want to render small differences in the results statistically significant, the necessary number of realisations (model integrations!) can become enormous.

The same is true is for the coefficient of correlation: in order to have a narrow confidence interval around the value of your correlation coefficient, you must perform a large number of experiments (in NWP, they would probably be model integrations).

This recommendation should encourage the scientists in NWP - particularly the scientists active in verification - to accompany their results with a measure of their statistical significance whenever possible.

2. Guide-lines for the verification of the PEPS forecasts

The SRNWP-DWD multi-model EPS - baptised PEPS (Poor man EPS) - is working with great stability and reliability in its pre-operational phase.

PEPS web site: http://www.dwd.de/en/FundE/Projekte/PEPS/index.htm

Each participating NMS (all the European NMS active in NWP!) has received a password allowing it to look at the operational forecasts.

Over Germany, the system has already reached an "operational status" as the forecasts used are calibrated (with the Bayesian Model Averaging method) and the results are verified.

It would be very useful and even necessary for the assessment of the quality of the method and for the determination of the future development works if each NMS would verify the PEPS probabilistic forecasts over its territory. But this work will only be meaningful if the PEPS forecasts are verified everywhere in the same way.

Action: The DWD will issue guide-lines specifying how the verification of the PEPS probabilistic forecasts should be conducted (parameters, ranges, scores, verification periods, etc) by the National Meteorological Service over their respective territory.

3. Radar precipitation information: composites

Radar information will become very important in the km-scale modeling for precipitation validation and verification as well as, in a later future, for data assimilation.

For many works, conglomerated radar data in so called "composites" will be the adequate tool. But skepticism has been expressed by the audience about the quality of today's radar composites.

Firstly, nobody was aware of a comprehensive work devoted to the assessment of the quality of the radar composites. It has been claimed that even when the radar data have been previously calibrated with rain gauges the quality of the composites remains uncertain.

Although today's radar composites present shortcomings (a colleague said that on the European composite we can at first glance recognize the radar of his country!) it would nevertheless be very useful to have free access to them, particularly when we know that their quality will surely improve in the future.

But the dissemination of the radar composites of the Programme OPERA has been blocked by the EUMETNET Council at its 23rd Meeting (14 December 2004).

Concerning the concern of the NWP verification specialists about the quality of the composites, the SRNWP Coordinator will discuss this point in a meeting between the EUMETNET Programmes OPERA and SRNWP that he will organize in the fourth quarter of 2006 or in the first quarter of 2007.

Action: The SRNWP Coordinator must

- continue and if possible intensify his action for the removal of the ban decided by the EUMETNET Council on the dissemination of the radar composites produced by the Meteorological Office on behalf of the OPERA Programme

- organize in 2006 4th quarter or in 2007 1st quarter a meeting between the EUMETNET Programmes OPERA and SRNWP. The aim of this meeting is to permit to the NWP specialists in data assimilation and verification to meet the radar specialists.

4. Expression of the verification of deterministic forecasts with probabilistic scores

Kees Kok from KNMI presented a talk titled "Probabilistic Approach in the Verification of Deterministic High Resolution Model Output".

Kees' method produces verification results of deterministic forecasts in terms of the Brier scores and ROC curves. The method seems to be most suitable for convective precipitations, as the double penalty problem does not exist and the skill of small scale information (scattered showers) can be assessed.

The recommendation passed is that this verification method should be included in the standard verification packages used for deterministic forecasting.

The problem is that the slides presented at the meeting give a good overview but do not yield enough information to develop the method.

Jean Quiby

SRNWP Programme Coordinator