Verification Session: Final Discussion


Monday 8th of October 2007, 17h30-17h50

Chair: Gerard Cats


The Local Organizer has had the very good idea to put in the programme of the EWGLAM Meeting a session devoted to Verification.

The aim was that each Consortium should present its strategy for verification.

The Consortia, with the exception of Aladin, have done this.


The main point of this session has nevertheless been the presentation by Terry Davies - on behalf of Clive Wilson - of a revised Proposal for a EUMETNET Programme on Verification.

To read the first version of this new Proposal, click here.

We recall that the first proposal has eventually not been presented to Council, as - among other reasons - it has been judged by its main authors (JoŽl Stein and Clive Wilson) too ambitious. Moreover, the necessity of a common verification package (as specified in the Proposal) could not be assessed.


Differences between the new and the old (non-submitted) Proposal

1.†††††††† The main difference is that in the new Proposal the Programme Verification is no longer foreseen as a large and single programme, but will develop as a succession of smaller programmes, each programme complementing the previous one.

2.†††††††† The old Proposal had two main aims: Development of a common verification package and realization of an operational model intercomparison.

For the first stage in the new Proposal, the development of a common verification package will not be considered. The aim of the first stage will only be to operate an intercomparison of the main European forecasting systems: Aladin, Cosmo, Hirlam and Unified Model.


Characteristics of the first Programme stage

It must be cheap, therefore practical and pragmatic. There is presently no chance to bring through the EUMETNET Council a second (after Interoperability) expensive NWP Programme.

The models foreseen for the intercomparison are:

- The Met Office NAE - 12km (integrated at the Met Office)

- The Hirlam reference - 16km (integrated at the FMI)†††††††††

- The Aladin France - 9km (integrated at Meteo-France)

- The COSMO-EU - 7km (integrated at the DWD)

The verification will be done on the common area of these 4 models.

It has been proposed to use as verification package an extension of the package used at the Met Office for the European Precipitation comparison.


Ideas for later Programme stages

- Addition of higher resolution forecasts up to the km-scale to the intercomparison

- Investigation of new verification methods (e.g. fuzzy logic, scale intensities, ...)

- Creation of a hub for non-GTS data, particularly for the precipitations (at ECMWF or at a NMS).



The discussion that took place at the end of the session concentrated almost exclusively on the new Verification Proposal.


Some points had to be explained as the size of the common area, the format to be used for the output fields, etc. It is clear that several points remain to be specified.


The main points of discussion have been:


Intrinsic difficulties of the verification.

- It has been made clear that what the verification results show are the performance of a numerical forecast system and not the quality of a particular model. All the components of a NWP system matter, as, for example, the cut-off time.

- Even on the same integration domain, it is a problem to compare models of different resolutions. The best strategy seems to remain to downgrade by averaging the model resolutions towards the coarsest resolution of the models involved and to verify over this resolution (and to hope that the models computed with higher resolutions will show better results!)


Relation with the ECMWF

It has been asked whether the ECMWF forecasts should also be included in the intercomparison.

Nobody opposed a participation of the ECMWF. However, our intercomparison should be limited to the very short and short-range, where we can take advantage of the use of fresh analyses. We should concentrate our verification on the boundary layer and on the precipitations. As our LAMs have a higher resolution than the ECMWF global model, the better representation of orography should give us better precipitations.



Next steps

The Met Office is asked to complete the revised Proposal.

Then the Proposal will be reviewed by the SRNWP Expert Team for "Diagnostics, Validation and Verification". The aim remains that the Met Office presents the Proposal at the spring meeting of the EUMETNET Council.



For the minutes:

Jean Quiby

SRNWP Programme Manager