Proposal for a EUMETNET / SRNWP Programme for
Interoperability between the European Modelling Systems
by
Ulrich Schättler (DWD – COSMO); Chairman of the Redaction Committee
François Bouttier (Météo-France – ALADIN)
Xiaohua Yang (DMI – HIRLAM)
Radmila Brozkova (CHMI – LACE)
Alan Radford (Met Office)
Initiated by “A Vision for Numerical Weather Prediction in Europe”, a document of Dr. David Rogers, former Chief Executive of the UK Meteorological Office, a discussion about the future numerical weather prediction (NWP) in Europe highlighted “the need for a much greater collaborative effort between National Meteorological and Hydrological Services (NMHS)”.
After discussion of this document by the EUMETNET Council, a “Vision Meeting” has been organised at the ECMWF (March 15th-17th, 2006), where a Working Group identified a set of relevant cooperation themes for implementation. The first theme on the list was “interoperability”.
The recommendation of the Working Group is to enhance the interoperability between the different modelling systems used in Europe as well as with users and suppliers outside of meteorology. The use of industry standards, the development of common formats and the harmonization of output parameter specifications would facilitate data exchange. Possible outcomes would include using each other NMHS’s data as backup and the creation of multi-model forecast ensembles.
The proposals of the Working Group for the theme “interoperability” are:
1. to agree on a common methodology for the provision of lateral boundary conditions taking into account scientific constraints
2. to define a common interface between NWP models and post-processing, to harmonise parameters and file formats, with an emphasis on multi-model ensembles
3. to foster a more widespread use of ECMWF software
4. to adopt a common framework for NWP system software that takes into account the developments by the climate community (PRISM).
The Vision Meeting also proposed to strengthen the SRNWP Programme by the creation of accompanying programmes. Three topics have been selected for which programme proposals should be initiated: interoperability, LAM-EPS ensemble (now termed "EurEPS"), and verification. Redaction Committees have been tasked to prepare the corresponding proposals.
This proposal deals with increasing interoperability between the European modelling systems – abbreviated to "Interoperability".
Within Europe, four different global models (GMs: IFS, Arpčge, UM global, GME) and four different types of limited area models (LAMs: ALADIN, UM_NAE, HIRLAM, COSMO-Model) are operated[1]. The four LAMs are developed and maintained by the corresponding Consortia. Each NMHS, which belongs to a specific Consortium, runs the corresponding LAM with a configuration specific to its own modelling system.
The “ability to increase … capability whilst at the same time maintaining a diversity of modelling approaches” has been identified by Dr. Rogers as one of the factors for the continuing success of the European NWP. But he also pointed out the “need for a much greater collaborative effort … to maintain, develop and run … high resolution models”. The term “Interoperability” was chosen for the ability of the different modelling systems to “work together”. Several aspects and proposals have been discussed, but no exact definition is yet available, nor is it clear, what can be realized for which costs.
“Interoperability” can mean many things. The need to define workable objectives lead to the following requirements:
Preliminary discussions suggested that: “Data Interoperability” deserves a higher priority than “Software Interoperability”, because it is quicker and cheaper to develop. The concept of “Interoperability” would be difficult to apply to all components of the NWP production suite, so several possible starting points (and levels of difficulty) have to be defined.
· “Interoperability” should aim to allow quasi real-time exchange of model output of observable weather parameters for model intercomparison, as well as collaboration in verification and validation.
· “Interoperability” needs standardization of data formats. This is necessary for the model output, but also for observational data, which is processed in verification software and data assimilation.
Contacts to the other SRNWP Programme redaction committees showed that the “Data Interoperability” development proposed here would help the "EurEPS ensemble" and "Verification" programmes. Especially the standardization of data formats and the ability to use output from all global models have been mentioned with a very high priority.
The practical expectation of this project is that:
1. It should be
possible to drive every LAM from any of the four global models.
This has a high priority. To define and implement a satisfactory
"conversion procedure" is a non trivial development that needs rather
sophisticated interpolation algorithms and physical computations, e.g. the
conversions of surface fields between the various soil models and grids need
special care. A further (but lower priority and longer term) development could
be LAM-to-LAM converters, in order to drive fine scale LAMs using any LAM
available on an encompassing domain.
2. Every LAM
should be able to start from the analysis of any other LAM.
This has a lower priority. In principle, the interpolation procedures could be
similar to the interpolation from the global models, the main difference being
in the input geometry.
3. Exchanged
model output and observational data should be usable at every NMHS without
effort.
This has a very high priority. It needs the specification of a standard format
for output data as well as for observational data. Rather than requiring every
LAM to write its output directly in the standard output format, conversion
procedures should transform every model output to the standard format.
For observational data, the situation is different. With the BUFR code, a
standard format for exchanging exists, but it cannot be used efficiently for
processing observational data in verification or data assimilation software.
The BUFR code has also the problem that people outside the NMHSs (like
Academia) usually cannot deal with it. Here, a standard format is desirable
that can efficiently be handled by the software, and that can be used for
exchange outside the NMHSs. A conversion procedure should transform
observational data from the BUFR code to the standard format for observational
data.
“Software Interoperability” is regarded as a lower priority option. Its goal (the exchange of software components e.g. as “plug compatible parameterizations”) would be very ambitious and difficult. The definition and development of common interfaces between model components, or, more generally, the development of a common software framework for NWP systems (similar to PRISM for the climate community) would need much time and resources. A possible way to start investigating these questions would be the coupling of different soil models to the atmospheric models. For this programme, which has a limited time frame, “Software Interoperability” has been explicitly excluded and therefore no task has been planned in the present proposal.
The objective of this "Interoperability" programme is to enable an increased collaboration between the modelling systems of ALADIN, HIRLAM, COSMO and the UM on an operational level for several activities. These activities include both following the SRNWP programmes:
· "EurEPS": usage of boundary conditions from different global models.
· "Verification and Model Intercomparison": use of a standard output format for input to verification and downstream post-processing software.
The outcome of this programme will be the provision and continued improvement of software tools, called adaptors, for conversion between different data formats and model grids.
In the future, this programme could be further expanded to work on the “Software Interoperability” issue.
The programme should start the 1st of January 2008 and is aimed at a duration of 2 years.
The programme is headed by a Programme Manager, who shall send short quarterly “Activity and Management Reports” to the SRNWP Programme Manager, reflecting the progress of the work and existing or foreseen problems. These reports will be part of the Quarterly Reports that the SRNWP Programme Manager shall deliver to the Advisory Committee and to the EUMETNET CO. Detailed yearly reports will be sent to the SRNWP Programme Manager and the EUMETNET CO.
Tasks:
1. "Communication":
Definition of working procedures with technical staff of the four consortia,
regarding
· access to documentation of preexisting model data representations practices and conversion software;
· identification of contact points in the consortia for future maintenance of the adaptor software, dissemination and validation of its releases (since all models are constantly evolving, keeping adaptor software in working order is going to be a significant job).
2. "Specification":
a. Definition of a standard model data exchange format and a set of processed parameters. The following issues have to be addressed:
· Choice of intermediate file format (e.g. GRIB-2), grid (2D vs. 3D, spectral vs. grid point, vertical discretization, geographical projection, ...)
· How to handle differences in orography and physiography (e.g. using a common orography)
· Definition of mandatory and optional parameters to be handled by the adaptors.
· Identification of the conversion methods to be used in the "development" task below.
b. Definition of a standard format for observational data that can be processed by verification and data assimilation software. This format should also be useable for data exchange with non-NMHS groups. The following issues have to be addressed:
· Choice of intermediate file format. Recommended is NetCDF, which is already widely used within the academic community.
3. "Development ":
a. Software tools, called “adaptors”, that transform relevant model parameters from each LAM output to the standard exchange format.
b. Enhancement of existing software tools that transform relevant model parameters from each global model to each LAM boundary coupling (and possibly, analysis) format. In a second step, further enhancement to also process data from the LAMs.
These subtasks include the definition, implementation, testing and documentation of the conversion algorithm for every parameter. The following issues have to be addressed:
· Is horizontal interpolation enough?
· Does a parameter need vertical adaptation?
· Is re-calculation of a parameter necessary? The treatment of soil variables will require special attention.
· Are other special treatments necessary?
c. A software
tool that transforms observational data from BUFR code to NetCDF (called
BUFR-to-NETCDF).
Comments on Task 3b:
l The Multi-Model Ensemble Prediction System, which is implemented at INM in Spain, already uses IFS, UM_global and GME as global models and HIRLAM, UM (regional version) and the COSMO-Model as regional models. Most of the work necessary for Task 3b therefore has already been addressed and a close collaboration with the colleagues in Spain is recommended.
l If the global-to-LAM conversion involves the use of the standard LAM format as an intermediate step, it would open extra possibilities (for comparison between global and LAM model output, and for interoperability of the LAM-to-LAM coupling). However, the use of an intermediate step may degrade the quality of the conversion.
l More generally, for the adaptor software to be successful, the physical quality of the conversion between all GMs and LAMs should be comparable. Poorly implemented adaptors may wrongly give a poor image of some GMs or LAMs, which would harm the scientific communication between the consortia.
Milestones:
· M1: Definition of a set of common output parameters that have to be exchanged for verification and other purposes. Inventory of existing model output formats, conversion tools and contact points (i.e. Tasks 1 and 2a).
· M2: Definition of the standard output format (completion of Task 2a).
· M3: Definition of the standard observation data format (completion of Task 2b).
· M4: For every LAM: Provision of an adaptor that transforms the output of this LAM to the standard output format (Task 3a).
· M5: For every LAM: Extension of existing software tool(s) to process data from all global models (Task 3b).
· M6: For every LAM: Extension of the software tools beyond Task 3b to process data also from other LAMs.
Deliverables:
These are the deliverables of the programme:
· D1: A report documenting the standard output format and including a list of parameters for which the standard output format is applied.
· D2: A report documenting the standard observational data format.
·
D3:
Requirements and Specifications for the adaptor software:
This document includes the identification of the methods that can be used for
implementing the adaptors, and for maintenance of the software in connection
with the consortia. It must be agreed by all groups involved.
· D4: Four adaptors that transform the output from every LAM to the standard output format. This includes the software as well as the documentation.
· D5: Enhancements to existing software tools, that enable all LAMs to process data from the four available GMs. This includes the software as well as the documentation.
· D6: Enhancements to existing software tools, that enable all LAMs to process data from the other LAMs. This includes the software as well as the documentation.
Tasks 1 and 2 and their deliverables (D1, D2, D3) are a necessary requirement for the start of the developments of Task 3. The corresponding work should be headed by a scientist located at the Responsible Member, but needs substantial support from the groups involved in the programme. It is important that everybody agrees on the common data formats and the adaptor methods.
The development of the adaptors in 3a can be treated by a scientist (located at the Responsible Member) but again needs substantial support from the consortia representing the different models. Later maintenance of the adaptors has to be done by the consortia.
The situation for Task
3b is different to Task 3a. Software tools already exist, which transform the
data from at least one global model to the LAM and these tools have to be
enhanced. Every consortium that wants to take part in this programme (or in the
EurEPS programme) shall do these developments on its own, supported by the
Responsible Member. It is again noted, that most of this work has already been
done for the Multi-Model EPS in Spain.
After the definition of a standard format for observational data, the
preliminary development work for the BUFR-to-NETCDF converter can be handled by
a NMHS to demonstrate the chosen approach.
The Budget of this programme is estimated as follows (per year)
Costs for a full time equivalent scientist working by the Responsible Member:
€ 90,000.00
Costs for the working time of the Programme Manager (25% of € 90,000.00)
€ 22,500.00
Travel expenses for Responsible Member (Programme Manager and Scientist)
€ 6,000.00
Travel expenses for participating NMHSs
€ 5,000.00
Total cost per year: € 123,500.00
[1] Several LAM variants are used in each consortium, e.g. ALADIN has several flavours of ALADIN models, ALARO, AROME, etc... We shall assume that the consortia already maintain converters between their own models, so we only need to focus here on the more difficult inter-consortium interoperability.