EUMETNET
SRNWP Programme
Report on the
First SRNWP Workshop on Mesoscale Verification
23-24 April 2001, KNMI, De Bilt, The Netherlands
This workshop, organized in the KNMI premises, has been attended by
23 participants coming from the following Consortia: HIRLAM: 14 (of them
5 KNMI); COSMO: 4; UKMO: 2; ALADIN: 0; LACE: 2. As no "List of participants"
was available at the meeting, some of these figures are maybe not correct.
Eight contributions (one had to be cancelled due to illness) have been
presented. This small number gave us plenty of time for discussions.
The meeting has been opened by Leo Hafkenscheid, Head of the Observations
and Models Division at KNMI, who confirmed that the purpose of this meeting
must be to foster the exchange of information and experience on mesoscale
verification because the new non hydrostatic mesoscale models cannot be
verified with the same techniques as models with grid distances of, say,
20 km or more. Leo explicitly encouraged the audience to find new methods
and new ways of verifying the meso-scale models.
Then the SRNWP Coordinator recalled briefly why he proposed at the
1999 SRNWP Annual Meeting in Bratislava the creation of a Lead Centre for
Verification and that HIRLAM/KNMI volunteered to take over this responsibility.
Summary of the presentations
The interest was primarily focused on the verification of the precipitations.
A point of discussion was whether the precipitations should be verified
with the radar or with the rain-gauges. The unanimous conclusion was that
it is pointless to play one against the other: we need both and this still
for a long time to go. As one colleague put it: we will always use the
rain-gauges, at least for calibration of the radar data!
It seems that the radar information should not be used operationally
for verification without correction (or calibration) by rain-gauges. FMI
and UKMO were clear on that point.
When rain-gauges are used, it was clear for everybody that, on the mesoscale,
the pairwise (one grid-point / one station) verification is inapplicable
not only for the convective precipitations but also for the stratiform
ones (frontal precipitations are normally not homogeneous; they have rain
bands).
Spatial averaging is a necessity. But the difficulty is to know how
large must be the verification area on which model and observed precipitations
must be averaged.
From the presentation of the ARPA-SMR (Italy), we saw that, for a given
catchment area in the Piedmont Region over which model precipitations have
been verified by averaging, the larger the number of stations used, the
better was the fit with the model. This is an important result.
Another important topic of discussion in connection with the verification
of the model precipitations was:
how do we get rid of the double penalty?
The double penalty occurs when forecast precipitations occur, but to
early or to late, i.d. not in the right verification time window, or not
at the correct place. In both cases the model is more penalized than by
producing no rain.
It seems clear that this problem cannot be resolved by the use of scores
based on contingency tables.
The only hope for solving this problem properly has been given by the
MeteoSwiss talk on pattern correlation technique: radar patterns are compared
with precipitation charts. A phase shift could be quantitatively assessed
by the distance of the centres of gravity of the two precipitation patterns.
But this is not enough: the pattern have to look similarly. This could
be measured by the correlation coefficient.
Another main point has been the verification of extreme events. This
is a difficult problem because extreme events are, by definition, rare
and thus not suited for statistical treatment. Nevertheless the presentation
of the results of the KNMI warnings was interesting: it showed a very good
(too good!) false alarm rate, but a too low (i.e. not so good) probability
of detection. This is not due to an insufficient forecast skill but to
a pure human problem: KNMI is reluctant to issue alarms because it does
not want to give unjustified fears to the Dutch population.
Discussions in the working groups
Two groups have been formed:
Group 1 had to deal with the technical and practical sides of the verification
Group 2 was a "think tank" which had to consider basic questions
Group 1
Verification of the cloudiness
The verification of the cloudiness was considered important because
of the role that cloudiness plays in the subjective perception of the weather
and because of its impact on the sensible and latent heat surface fluxes.
But cloudiness is very difficult to verify accurately.
The group recommends verification of the cloudiness with the satellite
imagery and not with the SYNOP information although today only UKMO and
MeteoSwiss (among the NWS represented at the meeting) have an operational
verification of the cloudiness with METEOSAT.
It would be very interesting to investigate a verification using the
radiances at the TOA (top of the atmosphere).
Verification of the surface processes
Our soil models are today rather sophisticated. In the most elaborated
schemes, a grid cell contains tiles of different soil properties. But for
the verification of the 2m temperature and humidity, we simply take the
grid point value which is a weighted average of the contributions of all
the tiles.
What we should do should be to verify only the values coming from the
tile containing the observing station.
As a consequence, it would be meaningful if, next to latitude, longitude
and height, we would also have physiographical information, as soil-type,
for the surface stations.
Verification of the precipitations
For the verification of the precipitations, next to the points
already mentioned above, Group 1 has looked for other methods of verification:
- instead of verifying 2D precipitation fields, one could compare 3D
radar reflectivity fields (we would need an observation operator for radar
reflectivities).
- satellites: the use of microwave channels allows the detection of
precipitations. This is already effective in the TRMM satellite (TRMM =
Tropical Rainfall Measuring Mission). Very probably not applicable over
land but could be very useful over the seas at mid-latitudes. A drawback
of the method is that there is no provision to guaranty that the inferred
precipitation has reached the ground.
These two possibilities have been qualified as worth an investigation.
The possibility of using surface run-off water has also been discussed.
The conclusion was that this cannot be a good method because the rivers
are today in the largest part of Europe regulated. Moreover, the time lag
between the occurrence of the precipitation and the rise of the river water
level can be very variable, depending, among others, on the soil conditions.
Group 2
This group has asked itself what mesoscale verification is about and
has tried to find procedures that would explicitly verify the meso-scale
structures. As the traditional methods (e.g. mean, rms, scores from a contigency
table) do not discriminate between scales, it would be interesting to know,
by a not so good forecast, on which scale the model went wrong: did it
already on the large-scale or went it wrong specifically on smaller scales?
Nobody will be surprised to learn that the method the group recommends
is Fourier decomposition.
Final discussion and recommendations
The final discussion was mainly centered around two points:
- exchange of observation data
- comparison of model results
Exchange of observation data
This discussion is summarized under the form of the recommendation
which has been proposed unanimously by the Workshop:
Considering that
- a better spatial and temporal resolution for the observations allows
a more accurate verification of the meso-scale models (and would also allow
a better definition of the initial conditions on the meso-scale),
- WMO Resolution 40 recommends free exchange of only the 3-hourly SYNOPs,
a restriction which hinders an accurate verification of the short living
meso-scale phenomenons and of the daily cycle of temperature and humidity,
as with one observation every three hours, neither the maxima nor the minima
of temperature can show up,
- already today in Europe a large number of SYNOP stations report hourly
and this number is still increasing,
the participants of the Workshop recommend that
- the SRNWP Project Manager, after consultation with the EUMETNET Coordinating
Officer, submits to the EUMETNET Council a proposition that would for the
EUMETNET Members not only relax the WMO Resolution 40 but also strongly
encourage the largest possible exchange of the SYNOP observations.
Comparison of model results
This topic gave rise to an interesting and engaged discussion that
can be summarised as follows:
If a comparison of forecasts of our models should take place, the following
conditions must be met:
- The purpose of the comparison must not be to distribute medals. The
purpose of a comparison can only be to learn more about the NWP and about
our respective models.
- The comparison must be accurate and homogeneous. That means that
it can de facto be done by one institution only in order to guarantee that
exactly the same procedure is applied to every model. If the NWS would
simply deliver their results, homogeneity is not fulfilled because, as
one example, the criteria for the rejection of observations are not unified.
- the comparison should be made on common areas. It is not possible
to compare results for Northern Europe with those valid for a Mediterranean
area.
- the choice of the methods of verification will be delicate. For precipitations,
meso-scale modellers will favour averaging on areas smaller than the ones
preferred by modellers using coarser resolutions because the former have
to justify the higher resolution.
Recommendation
The Lead Centre for Meso-scale Verification should study possibilities
of organising a meaningful model result comparison and presents ideas at
the next workshop.
The EWGLAM NWP verification scheme is maybe "de jure" still active but
is "de facto" dead. From the NWS represented at the Workshop, it looked
like as only SMHI is still sending verification results to DWD (most of
the delegates were not aware of that action).
Recommendation
The SRNWP Programme Manager will officially inform the DWD that the
Workshop recommends to the SRNWP members to stop sending model results
and that DWD is relieved from the SRNWP Network of any commitment concerning
a comparison of NWP model results.
Fourth recommendation
The Workshop recommends that the Swiss Federal Office for Meteorology
and Climatology (in short: MeteoSwiss) continues its development work in
the verification of the precipitations by pattern comparison technique.
Proceedings
The presented contributions will be published in a "HIRLAM Technical
Report".
The publication of the presentations in this high standard series will
undoubtedly increase the visibility of this and of the future workshops.
Next meeting
It has been decided to hold the second workshop in one year time. It
will be organised by the Lead Centre and will very probably take place
in De Bilt in Spring 2002. But this is not a prejudice: it must not be
inferred that this meeting will be annual in the future.
For the report:
J. Quiby
SRNWP Project Manager