1.1 The Expert Meeting on Implementation of GDPS Centres was held at the kind invitation of the United States of America National Weather Service, held at the National Centres for Environmental Prediction headquarters located within the NOAA Science Centre in Camp Spring, Maryland , from 1 to 5 June 1998. Ms Cynthia Nelson (NCEP) welcomed the participants, indicated facilities available for the meeting and gave an overview of the NCEP facilities housed within the science centre as the Central Operation which link all National centres together, Environmental Modelling Centre, Hydrometeorological Prediction Centre, Marine Prediction Centre, and Climate Prediction Centre and those located else where are the Aviation Weather Centre, Kansas City Missouri; Storm Prediction Center, Norman, Oklahoma; Tropical Prediction Centre, Miami, Florida and Space Environment Centre, Boulder, Colorado.

1.2 Mr Morrison Mlaki, Chief Data Processing Division, WWW Department of WMO, thanked and expressed appreciation to the US National weather Service for hosting the meeting, welcomed the participants. He noted that in recent years ocean observational facilities and improvements of numerical models of the atmosphere and ocean have reached a state of the art good enough for use in coupled forecast systems. Mr Mlaki recalled that CBS-XI 1996 adopted regulatory provisions covering GDPS functions for the prevision of long-range forecasts up to multi season scale and environmental monitoring and prediction products.

1.3 Mr Mlaki pointed out that the two major issues before the meeting relate to the provision of model out put as guidance to NMS on the occurrence of severe weather and provision and use of long-range forecasts up to multi season scale. Another issue of concern is the provision and use of environmental quality monitoring and prediction products such as UV indices etc. He noted that other issues before the meeting address tools to support implementation of the major issues and maintenance and enhancement of the capability of the GDPS to respond to current and envisaged requirements including the current issue of the Year 2000 compliance.

1.4 Mr Mlaki further noted that severe weather events which occurred around the globe as an impact or other wise of the 1997/98 ENSO event has brought into sharp focus the need to address as a matter of urgency these issues and supports the rationale of having an integrated World Weather Watch basic system GDPS for both short and long-term forecasting. Mr Mlaki noted also that the need to co-ordinate requirements with the Commission for Climatology (CCl) in particular the WCP/CLIPS programme and make use of available up to date Commission for Atmospheric Science (CAS) and related Research Institutes research results in these endeavours are recognised. For this reason he noted that the meeting would be joined by CCl/WCP/CLIPS experts and also would have an afternoon joint discussions on items 3 and 4.

2.1.1 Mr Al Kellie, Chairman of the Working Group on Data Processing was declared chairman of the meeting by acclamation.

2.2.1 The meeting adopted the agenda given in Appendix I.

2.3.1 The meeting agreed on its working hours , mechanism and work schedule.

2.3.2 There were 18 participants at the meeting as indicated in the list of participants given in Appendix II.

3.1 For the discussion on and items 3 and 4 the meeting welcomed the participation of several attendees from the WMO Science and Technology Retrospective Group that was meeting in Washington at the same time. Individuals attending were Dr. Mike Coughlan, Director of the WMO WCP department, Dr. Phil Arkin from IRI, Robert Livzey from CPC, Prof. Fred Semazzi University of Maryland), Dr. Raphael Okoola from the University of Nairobi, Mr. James Lever, Deputy Director, NCEP/CPC and Mr. W. Erb, IOC/UNESCO.

3.2 The meeting reviewed the current practices using numerical prediction models in the production of long range forecasts. It was noted that there had been considerable progress in the development of long range forecasting activities at a variety of centres. It was noted that while different approaches give different results in some cases, with different levels of success, no comprehensive documentation of skill levels is available. It was noted that assessments of the scientific quality of long range forecasts are not generally provided to users, apart from very simple measures of skill and warnings provided along with products on the Web from some issuing centres.

3.3 Some issues and concerns were noted about long range forecasting. More than one seasonal forecast is being issued by different centres and institutes and made available in the public domain via Internet for the same areas. These forecasts were noted to differ substantially, due to the inherent low skill. The meeting acknowledged the scientific merit of these differences and encouraged the different approaches as a means to spur on progress on the research fronts. However, concerns were raised that this situation tends to lead to confusion in the public, media interest frequently focuses and amplifies this confusion, and ultimately, reflecting back on the science behind the Long range Forecasts.

3.4 There was concern raised and considerable agreement on the need to have a more coherent approach to the Long Range Weather/Climate Forecasting. It was noted that this coherence was just beginning to take shape within the WMO and would of necessity involve CBS, CCl and CAS. It was noted that the members have invested heavily over the years to implement a robust operational global infrastructure over a long period of time, including observing systems and data collection (GOS), processing facilities (GDPS) and telecommunications (GTS) and that the emergence of Long range Forecasting was placing additional challenge on the WWW systems to adjust to broader requirements of CCl.

3.5 The meeting reached agreement to exchange products, to discuss and approve a suitable approach to verification of long range forecasts and to exchange such verifications in an intercomparison. IRI indicated a willingness to participate in verification intercomparison. Further treatment of this topic can be found under the discussion on item 4.

3.6 The meeting concluded that joint sessions are beneficial to establish broader personal networks to share common visions and issues. The climate representatives identified a desire for the GDPS to consider means by which the GDPS systems can be adjusted to serve climate requirements and influence other components of the WWW to support climate needs.

4.1.1 An increasing number of seasonal and long-range forecasts are appearing in the public domain. Some of these derive from national meteorological services and others do not. At present any assessment of the quality and utility of these forecasts is carried out locally and in a piece-meal version. Routine exchange of quality information is very limited. It is certainly a much lower level than the comparable exchange of NWP verification.

4.1.2 CBS-XI has given the Working Group on Data Processing the task of developing standard verification procedures and organising intercomparisons of such scores. The Commissions for Climatology and Atmospheric Science have also recognised the need for exchange of quality assessment information as a vital component of the prediction activity. In particular, the report of the Twelfth Session of the Commission for Climatology noted:

"The Commission considered that the need for information on assessing the types and quality of prediction methods was urgent, and that ways should be developed to enable users to be informed of the quality associated with a given method. The benefits and problems associated with the free flow of the predictions on the Internet was noted, and the Commission concluded that while there were no means or will to stop anyone from putting climate information on the Internet, it could be proposed that official predictions be accompanied by information on uncertainties and quality."

"The Commission further wished to emphasize the importance it attached to the concept of, and international standards for, indices for evaluation of quality of climate predictions. This was to allow for a comparison of performance of different models in various situations, which was essential for both users and scientists who developed these models." (CCl XII August 1997, Sections 7.4.1 and 7.4.2).

4.1.3 CAS trailed some verification measures at the WMO meeting on Dynamical Extended Range Forecasts (DERF) held in Toulouse in November 1997. A small Working Group has been established to refine their recommendations on the most appropriate verification scores to use, with a view to making recommendations to WMO on standard procedures and scores for routine exchange.

4.1.4 The meeting considered that its role is to develop some procedures for the exchange of verification scores building on the work of CAS with a particular focus on the practical details of producing and exchanging appropriate scores.

(a) CBS-XI proposed some standard procedures for the verification of long-range outlooks (see annex to paragraph 4.5.2.1). These included a variety of measures for different types of forecasts (categorical, probability, continuous).

(b) The CAS Working Group on Medium and Long Range Weather Forecasting Research is also attempting to develop a standardised reporting system for long-range forecasting. They sponsored the meeting on DERF held in Toulouse in November 1997 which included on its agenda the results of a trial intercomparison of verification scores. A protocol was issued to participants specifying the information requested and the method for computing the scores. It was acknowledged that the scores routinely exchanged for short- and medium-range NWP predictions may not be appropriate. The trial attempted to use measures that would allow cross-comparison of skill from different systems (empirical, dynamical), for different parts of the globe and include direct climate impacts. The trial provided useful information on the conduct of such an exchange of such scores and some of the difficulties and ambiguities.

• root mean square skill score (RMSSS)
• relative operating characteristic (ROC)

The former can only be used for categorical forecasts, whereas the latter can be used for both categorical and probabilistic forecasts.

4.4 It is recognised that it will take time to organise and implement systems for exchange of scores and the work needs to be done to establish procedures as much as the appropriate scores. Consequently, the meeting recommended that a pilot exchange of scores be commenced using a limited number of scores and measures, with highest priority being given to the two scores recommended by the CAS group, but with the CBS list as additional measures to be included if possible.

4.5.1 The question of extending the procedures dealing with the exchange of verification results among NWP centres to long range forecasting is recognised to be more than a simple matter of extending parameters and forecast ranges. This meeting therefore based its proposals on several contributions that have been solicited from different experts in the field of seasonal forecasting at two GDPS expert meetings (Geneva, Dec. 1995 and Washington, June 1998) and at the CAS DERF Workshop (Toulouse, Nov. 1997).

4.5.2 The discussions clearly showed that the problem of issuing commonly agreed verification statistics for long-range forecasting was twofold:

(a) one problem is to ensure that the seasonal forecasts are issued to the users with quantitative information on the skill that can be expected from past records of performance;

(b)    a different problem is to organise an exchange of verification results on the basis of commonly agreed sets of scores, areas, parameters and forecast ranges;

4.5.2.1 Both problems require some common agreement on a limited set of skill measures that are relevant in seasonal forecasting. CBS-XI indeed considered a fairly complete list of scores (see annex) that may be used depending on the type of forecast (categorical forecasts, probabilistic forecasts of categorical events, continuous forecasts). They should be taken as a list of commonly agreed measures of skill, a subset of which should always be provided to users of seasonal forecasting products as a measure of what can reasonably be expected in terms of skill of the current forecast (item a). As the time series of operational seasonal forecasts is often short compared to what is needed for statistical significance, provision of historical forecast scores should also be provided. A comprehensive scientific assessment of the performance of the systems is likely to require computations of further scores.

4.5.2.2 Item b) requires an even more pragmatic approach in order to gather reasonable chance of success. The exchange of scores has to provide results that are meaningful enough to allow useful interpretation, but the set has to be limited in order to get the participation of a large proportion of seasonal forecasts providers. In some countries, institutions other than the NHMSs have a role in long-range prediction (e.g. IRI in USA). The proposed arrangements need to be open to include participation from such agencies. In fact, their active cooperation needs to be sought.

4.7.1 Recommendations on the datasets to be used as climatological references will be determined in the future work. The WCRP sets of Global Precipitation fields and the GCOS network of surface observations have been mentioned as potential sources.

4.7.2 The common session with CCl/CLIPS representatives confirmed their interest in taking part in the exchange of verification results; for that reason, it is important not only to consider verification of fields but also over commonly agreed sets of stations to allow both dynamical and statistical seasonal prediction models to take part.

4.7.3 It was noted that a distinction should be made between forecasts (or predictions) that are uninterpreted results from dynamical/ statistical models and value-added forecasts that are resulting from the interpretation of possibly several sources of prediction, adapted to the special needs of well-defined categories of users.

• R. Livezey (NCEP/CPC) will further elaborate on this exchange of scores proposal in the coming weeks and will liaise with an informal GDPS subgroup (F. Lalaurette, ECMWF (Convenor); T. Hart, Australia;  N. Mannoji, Japan; G. Verner, Canada).
• Further progress is expected from discussions during the Seasonal Forecasting Expert Meeting convened at ECMWF, Reading 17-18 June 1998; input from further experts affiliated to IRI, CPTEC, Meteo-France and UKMO are expected on this occasion.
• As a follow-up of discussions at this meeting, a report on the outcome of the above ECMWF discussions should be made to the chairman of CBS Working Group on Data Processing.

5.1.1 The meeting noted that with the future global NWP models expected to run at resolutions of less than 20 km and the increasing power and falling cost of workstations, the implementation and operations of limited domain local NWP models was becoming a more viable option. It was noted that a number of countries have already benefited from this development as some major centres have made available their workstation versions of such models in particular the NCEP ETA regional model and RSM, The French ALADIN model and the DWD Deutschland model, for implementation and operation at various NMS Research Institutes and Universities. INPE Brazil also indicated that they have contracted a University to produce a parallel workstation version of their RSM.

5.1.2 The meeting identified the data assimilation processes as becoming extremely computationally expensive and represents a limiting factor on the quality of outputs coming from NWP models being run on workstations. In the area of Regional Climate modelling it was noted that this is still a topic of early research and development and had not yet overcome the limitations inherent in these techniques.

5.1.3 Since the implementation and operations of such workstation models require a critical mass of knowledgeable scientific and technical staff, as a strategy the major centres provide extensive training, the model code, documentation but do not normally provide for local adaptation or support.

5.1.4 It was noted that the strategy for obtaining boundary conditions is either to be through the GDPS product normally disseminated or through special bilateral arrangements with a major centre for tailored boundary conditions.

5.1.5 One of the options recognised for NMCs with limited resources is to concentrate workstation and human resources in the development and implementation of post-processing to produce value added products using imported global model products in GRIB from major centres .

5.1.6 In order to facilitate information dissemination to the GDPS community on progress in implementation of workstation version models, it was agreed that centres should report in the GDPS annual progress report on models running on workstations.

5.2.1 The meeting was informed on two models developed in DWD which allow local microclimate simulation with option to run on workstation. These two models are:

8 The Year 2000 Problem (Agenda item 8)

8.1 The meeting was informed that all major centers have numerous operational systems which they have identified as being critical to their overall operations and which they consider to be susceptible to possible failure arising from the Year 2000 problem. Of the identified systems, a significant proportion is directly involved in operational activities in support of Members’ international WWW commitments and obligations. A typical example of identified systems by the UKMO is given in the Annex to this paragraph.

8.2 The meetings was informed that most major centers have embarked on major programmes designed to ensure that all their critical systems are Year 2000 compliant. This major effort was estimated at 200 person years at cost of CAD 20 million and 122 person years by Canada and UKMO respectively. It was noted from the progress reports presented that each identified system may have to be validated by formal project-management- methods, and where appropriate set up a Risk Register identifying all risks to NMS activities, together with counter measures being taken to offset these risks. All aspects of the Risk Register should be kept under frequent review. Where a project falls behind schedule it may be necessary to concentrate effort in that area to try and get the system back on schedule or to work toward an adjusted target date for completion.

8.3 The meeting paid particular attention to risk associated with the availability of observations over the year 2000 and was concerned with the status of activities to assure Year 2000 compliance in this area.

8.4 The meeting noted that Members are making extensive efforts to safeguard the various components of their operational functions to other NMS, and one committed to completing all compliance actions well in advance of the critical time and should keep other Members informed, as appropriate through the WMO Year 2000 Web Site.

8.5 It was noted that the WMO Web site provides extensive information on Members focal points on the subject, compliance of manufacturers and also free software which can be down loaded for use in tests of PC compliance.

8.6 The meeting noted with appreciation the UKMO support and encouragement of the WMO Secretariat to be as proactive as possible in ensuring the problem is addressed. The indication of the UKMO that it was ready to contribute to a Trust Fund to help WMO Members who have an identified problem but lack the means of solving that problem was received with appreciation.

8.7 The meeting emphasized the need for continued education as many Members do not seem to understand the serious nature of the problem and its effects on all computer systems including PCs, micro- chips and related facilities in automated observing, communications and processing systems.

9.1 The meeting considered proposals to update Chapter 6, quality control procedures of the Guide on the GDPS, developed by two members of the Working Group Planning and Implementation of WWW in RA VI and endorsed by the Chairman of that Working Group. The meeting supported the proposal given in the annex to this paragraph.

9.2 Following an exchange of views on the details of the proposals, the meeting requested a small ad hoc group composed of Mr. Mannoji (JMA) as coordinator, Ms. Cynthia Nelson (NCEP), Terry Hart (Australia), Ms Simard (Canada) and Dr. Chen (China), Mr David Forrester (NIC) and Mr Fredéric Chavaux (France) to further review the proposal and recommend adjustment, where appropriate, to Chairman of the Working Group on Data Processing for his approval on behalf of the Working Group.

9.3 The ad hoc group should exchange views by E-mail and make its recommendation to the Chairman as soon as possible but preferably not later than the end of September 1998.

10.1 The meeting was closed on Friday 5 June 1998.

(Parameters to be verified: Climate anomalies of temperature and precipitation)

Climatology to be used: 1961 - 1990, to be updated every ten years

Categories:    above normal, normal, below normal

                     temperature limits: equal probability categories

                     Precipitation limits: equal probability categories*

Scores to be used:

A. Linear error in probability space for categorical forecasts (LEPSCAT)

B. Bias

C. Post agreement

D. Percent correct

Categories: above normal, below normal

Scores to be used:

A. Brier

B. Brier Skill score with respect to climatology

C. Reliability

D. Sharpness (measure to be decided)

Categories:    above normal, normal, below normal

Scores to be used:

A. Ranked probability score

B. Ranked probability skill score with respect to climatology

Scores to be used:

Murphy-Epstein Decomposition (phase error, amplitude error, bias error)

Scores to be used:

A. Mean-square-error

B. Correlation

C. Bias

----------------

* Arid and semi-arid areas will need special consideration.

6.3.1 Gross-error limit checks

Table 6.2

Fixed limits for parameters in coded form

(The parameters are considered to be erroneous outside the limits.

The denotations are the same as those used in the Manual on Codes, WMO-No. 306)

Table 6.3 - 6.9 (Limit values for parameters) - without any changes

The different parameters in SYNOP reports are checked against each other. In the description below, the suggested checking algorithms have been divided into areas where the physical parameters are closely connected:

(a) Wind dd/ff

The wind information is considered to be erroneous in the following cases:

(b) Visibility VV and weather ww (in case of indicator i_x = 1, 4)

The values for visibility and weather are considered suspect when:

(c) Visibility VV and cloud information

The values for visibility and cloud cover are considered suspect when:

(d) Cloud information

The values for cloud cover are considered erroneous when:

N = 0 and {C_H or C_M or C_L > 0};

N_h = 0 and {C _H = / or C _H = 0};

0 < N < 9 and N_h = C _L = C _M = C _H = 0;

C_L > 0 and {N_h = 0 or N_h = /};

C_M > 0 and {N_h = 0 or N_h = /};

C_L = 0 and C_M = /;

C_M = 0 and C_H = /;

C _L > 0 and C < 6 at 1^st layer;

{C _M = 1 or C _M = 2} and C = 3 at any layer;

N_s > N at any layer;

C _L = 0 and N_h < N_s for all groups 8N_sCh_sh_s with C from (3...5);

(e) Cloud information and weather ww (in case of indicator i_x = 1, 4)

Clouds and weather are considered suspect when:

N = 9 and {ww < 39 or ww = 40 or ww = 41 or ww = 42 or ww = 44 or ww = 46 or ww = 48

(f) Temperature T and weather ww (in case of indicator i_x = 1, 4)

Both elements are considered suspect when:

(g) Temperature T and dew-point temperature T_d

Both values are considered suspect when:

T_d > T;

(h) Pressure tendency appp

The values of a and ppp are considered erroneous when:

a = 4 and ppp > 0;

(i) Weather ww, W₁, W₂ (in case of indicator i_x = 1, 4)

The values of ww and W₁, W₂ are considered erroneous when:

(j) Past weather W₁ , W₂

The values of W₁, W₂ are considered erroneous when:

W₁ < W_2;

(k) Past weather W₂ and cloud cover N

The values of W₂ and cloud cover are considered suspect when:

(l) Weather ww and wind speed (in case of indicator i_x = 1, 4)

The values for ww and ff are considered suspect when:

(m) Temperature T and Minimum/Maximum temperature T_n / T_x

The values of T and T_n , T_x are considered erroneous when:

T < T_n ;

T_x < T;

(n) Precipitation RRR and past weather W₁ , W₂

The values of RRR and W₁ , W₂ are considered erroneous when:

(o) Indicator i_R and precipitation RRR

The values of i_R and RRR are considered erroneous when:

i_R = 0-2 and RRR = 000 or RRR = ///;

(p) Indicator i_x and weather ww, W₁ , W₂

The values of i_x and W₁ , W₂ are considered erroneous when:

i_x = 2, 3, 5, 6 and ww = 00 - 99 or W₁ = 0 - 9 or W₂ = 0 - 9;

(q) Check on station pressure reduction

The paragraph: without any changes.

1. OPENING OF THE MEETING

2. ORGANIZATION OF THE MEETING

2.1 Election of a chairman

2.2 Approval of the agenda

2.3 Working arrangements for the meeting

3. REVIEW OF DEVELOPMENTS IN NWP AND IN LONG-RANGE FORECASTS UP TO MULTI SEASON SCALE

4. DEVELOPMENT AND IMPLEMENTATION OF A VERIFICATION SYSTEM FOR LONG-RANGE FORECASTS

5. DEVELOPMENT OF GUIDANCE ON IMPLEMENTATION AND APPLICATIONS OF RESULTS OF NWP LOCAL MODELS AND LOCAL CLIMATE SIMULATIONS USING WORKSTATION RESOURCES AT NMCs

6. PROVISION OF MODEL OUTPUT PRODUCTS AS FORECAST GUIDANCE ON THE OCCURRENCE OF SEVERE WEATHER

7. GENERATION, PROVISION AND USE OF ENVIRONMENTAL QUALITY MONITORING AND PREDICTION PRODUCTS AND LONG-RANGE FORECASTS

8. YEAR 2000 PROBLEM

9. UPDATE TO THE GUIDE ON THE GLOBAL DATA-PROCESSING SYSTEM; WMO-No 305

10. CLOSURE OF THE MEETING