Report on the
First SRNWP Workshop on Mesoscale Verification
23-24 April 2001, KNMI, De Bilt, The Netherlands
This workshop, organized in the KNMI premises, has been attended by
23 participants coming from the following Consortia: HIRLAM: 14 (of them
5 KNMI); COSMO: 4; UKMO: 2; ALADIN: 0; LACE: 2. As no "List of participants"
was available at the meeting, some of these figures are maybe not correct.
Eight contributions (one had to be cancelled due to illness) have been presented. This small number gave us plenty of time for discussions.
The meeting has been opened by Leo Hafkenscheid, Head of the Observations
and Models Division at KNMI, who confirmed that the purpose of this meeting
must be to foster the exchange of information and experience on mesoscale
verification because the new non hydrostatic mesoscale models cannot be
verified with the same techniques as models with grid distances of, say,
20 km or more. Leo explicitly encouraged the audience to find new methods
and new ways of verifying the meso-scale models.
Then the SRNWP Coordinator recalled briefly why he proposed at the 1999 SRNWP Annual Meeting in Bratislava the creation of a Lead Centre for Verification and that HIRLAM/KNMI volunteered to take over this responsibility.
Summary of the presentations
The interest was primarily focused on the verification of the precipitations.
A point of discussion was whether the precipitations should be verified with the radar or with the rain-gauges. The unanimous conclusion was that it is pointless to play one against the other: we need both and this still for a long time to go. As one colleague put it: we will always use the rain-gauges, at least for calibration of the radar data!
It seems that the radar information should not be used operationally for verification without correction (or calibration) by rain-gauges. FMI and UKMO were clear on that point.
When rain-gauges are used, it was clear for everybody that, on the mesoscale,
the pairwise (one grid-point / one station) verification is inapplicable
not only for the convective precipitations but also for the stratiform
ones (frontal precipitations are normally not homogeneous; they have rain
Spatial averaging is a necessity. But the difficulty is to know how large must be the verification area on which model and observed precipitations must be averaged.
From the presentation of the ARPA-SMR (Italy), we saw that, for a given catchment area in the Piedmont Region over which model precipitations have been verified by averaging, the larger the number of stations used, the better was the fit with the model. This is an important result.
Another important topic of discussion in connection with the verification
of the model precipitations was:
how do we get rid of the double penalty?
The double penalty occurs when forecast precipitations occur, but to early or to late, i.d. not in the right verification time window, or not at the correct place. In both cases the model is more penalized than by producing no rain.
It seems clear that this problem cannot be resolved by the use of scores based on contingency tables.
The only hope for solving this problem properly has been given by the MeteoSwiss talk on pattern correlation technique: radar patterns are compared with precipitation charts. A phase shift could be quantitatively assessed by the distance of the centres of gravity of the two precipitation patterns. But this is not enough: the pattern have to look similarly. This could be measured by the correlation coefficient.
Another main point has been the verification of extreme events. This
is a difficult problem because extreme events are, by definition, rare
and thus not suited for statistical treatment. Nevertheless the presentation
of the results of the KNMI warnings was interesting: it showed a very good
(too good!) false alarm rate, but a too low (i.e. not so good) probability
of detection. This is not due to an insufficient forecast skill but to
a pure human problem: KNMI is reluctant to issue alarms because it does
not want to give unjustified fears to the Dutch population.
Discussions in the working groups
Two groups have been formed:
Group 1 had to deal with the technical and practical sides of the verification
Group 2 was a "think tank" which had to consider basic questions
Verification of the cloudiness
The verification of the cloudiness was considered important because of the role that cloudiness plays in the subjective perception of the weather and because of its impact on the sensible and latent heat surface fluxes.
But cloudiness is very difficult to verify accurately.
The group recommends verification of the cloudiness with the satellite imagery and not with the SYNOP information although today only UKMO and MeteoSwiss (among the NWS represented at the meeting) have an operational verification of the cloudiness with METEOSAT.
It would be very interesting to investigate a verification using the radiances at the TOA (top of the atmosphere).
Verification of the surface processes
Our soil models are today rather sophisticated. In the most elaborated schemes, a grid cell contains tiles of different soil properties. But for the verification of the 2m temperature and humidity, we simply take the grid point value which is a weighted average of the contributions of all the tiles.
What we should do should be to verify only the values coming from the tile containing the observing station.
As a consequence, it would be meaningful if, next to latitude, longitude and height, we would also have physiographical information, as soil-type, for the surface stations.
Verification of the precipitations
For the verification of the precipitations, next to the points already mentioned above, Group 1 has looked for other methods of verification:
- instead of verifying 2D precipitation fields, one could compare 3D radar reflectivity fields (we would need an observation operator for radar reflectivities).
- satellites: the use of microwave channels allows the detection of precipitations. This is already effective in the TRMM satellite (TRMM = Tropical Rainfall Measuring Mission). Very probably not applicable over land but could be very useful over the seas at mid-latitudes. A drawback of the method is that there is no provision to guaranty that the inferred precipitation has reached the ground.
These two possibilities have been qualified as worth an investigation.
The possibility of using surface run-off water has also been discussed. The conclusion was that this cannot be a good method because the rivers are today in the largest part of Europe regulated. Moreover, the time lag between the occurrence of the precipitation and the rise of the river water level can be very variable, depending, among others, on the soil conditions.
This group has asked itself what mesoscale verification is about and
has tried to find procedures that would explicitly verify the meso-scale
structures. As the traditional methods (e.g. mean, rms, scores from a contigency
table) do not discriminate between scales, it would be interesting to know,
by a not so good forecast, on which scale the model went wrong: did it
already on the large-scale or went it wrong specifically on smaller scales?
Nobody will be surprised to learn that the method the group recommends
is Fourier decomposition.
Final discussion and recommendations
The final discussion was mainly centered around two points:
- exchange of observation data
- comparison of model results
Exchange of observation data
This discussion is summarized under the form of the recommendation which has been proposed unanimously by the Workshop:
- a better spatial and temporal resolution for the observations allows a more accurate verification of the meso-scale models (and would also allow a better definition of the initial conditions on the meso-scale),
- WMO Resolution 40 recommends free exchange of only the 3-hourly SYNOPs, a restriction which hinders an accurate verification of the short living meso-scale phenomenons and of the daily cycle of temperature and humidity, as with one observation every three hours, neither the maxima nor the minima of temperature can show up,
- already today in Europe a large number of SYNOP stations report hourly and this number is still increasing,
the participants of the Workshop recommend that
- the SRNWP Project Manager, after consultation with the EUMETNET Coordinating Officer, submits to the EUMETNET Council a proposition that would for the EUMETNET Members not only relax the WMO Resolution 40 but also strongly encourage the largest possible exchange of the SYNOP observations.
Comparison of model results
This topic gave rise to an interesting and engaged discussion that can be summarised as follows:
If a comparison of forecasts of our models should take place, the following conditions must be met:
- The purpose of the comparison must not be to distribute medals. The purpose of a comparison can only be to learn more about the NWP and about our respective models.
- The comparison must be accurate and homogeneous. That means that it can de facto be done by one institution only in order to guarantee that exactly the same procedure is applied to every model. If the NWS would simply deliver their results, homogeneity is not fulfilled because, as one example, the criteria for the rejection of observations are not unified.
- the comparison should be made on common areas. It is not possible to compare results for Northern Europe with those valid for a Mediterranean area.
- the choice of the methods of verification will be delicate. For precipitations, meso-scale modellers will favour averaging on areas smaller than the ones preferred by modellers using coarser resolutions because the former have to justify the higher resolution.
The Lead Centre for Meso-scale Verification should study possibilities of organising a meaningful model result comparison and presents ideas at the next workshop.
The EWGLAM NWP verification scheme is maybe "de jure" still active but
is "de facto" dead. From the NWS represented at the Workshop, it looked
like as only SMHI is still sending verification results to DWD (most of
the delegates were not aware of that action).
The SRNWP Programme Manager will officially inform the DWD that the Workshop recommends to the SRNWP members to stop sending model results and that DWD is relieved from the SRNWP Network of any commitment concerning a comparison of NWP model results.
The Workshop recommends that the Swiss Federal Office for Meteorology and Climatology (in short: MeteoSwiss) continues its development work in the verification of the precipitations by pattern comparison technique.
The presented contributions will be published in a "HIRLAM Technical Report".
The publication of the presentations in this high standard series will undoubtedly increase the visibility of this and of the future workshops.
It has been decided to hold the second workshop in one year time. It will be organised by the Lead Centre and will very probably take place in De Bilt in Spring 2002. But this is not a prejudice: it must not be inferred that this meeting will be annual in the future.
For the report:
SRNWP Project Manager