Teleconference on GISC synchronization (in preparation of WIS functionality demo to CBS)

Wednesday 18th of August 8:00am GMT, WMO WEBEX system

List of participants:

Lipeng, Jiang CMA
Markus Heene, DWD (sick for the meeting, participated in preparations)
Siegfried Fechner, DWD
Heinrich Knottenberg, DWD
Jose Mauro de Rezende, INMET
Eizi Toyoda, JMA
Timo Proescholdt, WMO

Agenda

1. Opening
2. Discussion of testplan
3. Discussion of options for catalogue synchronization
4. Discussion of steps needed to ready GISC software for synching
5. Timeline
6. Any other business
7. Closure

Minutes

1. The participants met in WMO’s new WEBEX system. After checking that everybody was connected, Timo Pröscholdt from WMO welcomed everybody. The participants were informed that Markus Heene from DWD had been instrumental in making the preparations for the meeting but could not attend today due to sickness and that Heinrich Knottenberg would attend instead of Markus. Since WEBEX cannot be used from DWD’s internal system, the DWD participants were connected as telephone-only participants, with no access to the chat. (a transcript of which is attached to this document)
2. The participants noted that the aim of the teleconference was to synchronize the metadata catalogue of four or more GISCs over a period of 4 weeks, in preparation of the technical demonstration to CBS and find out about potential long-term issues. For this a testmedata set would have to be generated and shared. The testing consists of making changes to metadata records and checking that the changes are correctly propagated to the other GISCs. They agreed that for this the following procedure would be used:
   • Each participating GISC generates a set of 100 metadata records. The records can be copies of “real” metadata, but the citation Authority fileIdentifier has to be int.wmo.wis.test (e.G urn:x-wmo:md:int.wmo.wis.test::FCUK31EGRR) to avoid interference with operational systems. It will help during the testing if the origin of the metadata record can be inferred from the fileIdentifier.
   • The WMO core profile 1.1 current IPET-MDI recommendations should be used as metadata standard, although slight deviations from it would be ok if there were problems with it. Further discussion during follow up call when everybody has generated the test metadata.
   • The metadata set is shared using OAI and the union of these sets is the test catalogue.
   • Each centre changes their metadata frequently using their favourite in-house method, and monitors if the changes propagate to the other centres. If an automated procedure is used for verification, SRU should be used. (to avoid checking the functioning of OAI with OAI).
   • The result of the changes are written down and shared with the group. The minimum information to capture for each change is: fileIdentifier, date of change, nature of change (update,deletion,insertion), the actual change, information if the change has successfully propagated to all participating GISCs on the following day. More detailed info such as logfiles were welcomed.
   • The participants noted that an agreement still had to be reached on how the results of the individual GICSs would be shared. Options include a page on the WISWIKI, google groups or email. It was noted that google docs cannot be used by DWD. A tentative WIKI page can be found here.
   • Note from Timo: we should maybe not use all the 100 records initially and leave some over for insertion of metadata during the tests. (or we generate more than 100 records).
3. The teleconference then discussed the technical details of the synchronization and agreed on the following procedure. Each GISC will create two OAI sets. The first one, called WISTEST-#GISCNAME# (e.G for Brazil WISTEST-INMET), contains the 100 testmetadata records of the GISC. The other one, called WISTEST-CATALOGUE, is the union of all the WISTEST-#GISCNAME# sets of all participating GISCs (if there are 4 participating GISCs it contains 400 records). The GISCs point their harvesters to the WISTEST-#GISCNAME# OAI sets at the other GISCs and put the harvested metadata into their set WISTEST-CATALOGUE. The contents of their own set are also added to WISTEST-CATALOGUE
4. See 3). The participants noted that configuration changes could have to be made to OAI harvesters and providers, but that the changes are rather small and feasible for everybody. DWD noted that they have already created the necessary sets.
5. The participants agreed to have another teleconference Monday August 30th. Till then each participating GISC should have generated the 100 metadata test records and created the needed OAI sets. Slight changes needed to the metadata records for validation could then be discussed. The test is planned to take place mostly during September, so that the results can be used for the CBS in November.
6. There was no other business
7. The meeting was closed at 9:11 GMT with thanks to all participants, especially for the ones having stayed late or gotten up early and a remark that this encounter was a further pioneer step in the implementation of WIS.

Action items

• Create 100 metadata records
• Create OAI sets WISTEST-#GISCNAME# and WISTEST-CATALOGUE
• Check if harvesting procedures can insert harvested data into OAI sets.
• Create in-house test procedure to modify testrecords and check propagation to other GISCs. (manually or automated)
• Discuss a way to share the synchronization results between us

Annex: transcript of the WEBEX chat:

from TOYODA Eizi to Everyone:
is siegfried coming?
from TOYODA Eizi to Everyone:
okay talk to siegfried
from TOYODA Eizi to Everyone:
bottom line
from TOYODA Eizi to Everyone:
please type
from TOYODA Eizi to Everyone:
i see
from TOYODA Eizi to Everyone:
who's coming instead of siegfried?
from TOYODA Eizi to Everyone:
okay so all online now?
from WMO Webex 5 to Everyone:
Siegfried will be there instead
from WMO Webex 5 to Everyone:
Jose can you hear us?
from JMauro to Everyone:
OK, Understood
from TOYODA Eizi to Everyone:
Who wants v 1.1 of Profile?
from TOYODA Eizi to Everyone:
or anybody trouble with latest IPET-MDI recommendation?
from JMauro to Everyone:
ok
from TOYODA Eizi to Everyone:
the Zip file siegfried is talking abouott is what I've sent.
from TOYODA Eizi to Everyone:
I made it 100% compatible with IPET-MDI-I recommendation.
from WMO Webex 5 to Everyone:
ok
from TOYODA Eizi to Everyone:
unfortunately there was some typos and all validation errors are coming from them.
from TOYODA Eizi to Everyone:
yes I agree w/siegfried Each node should generate metadata.
from TOYODA Eizi to Everyone:
and that shoud be automatically updated
from TOYODA Eizi to Everyone:
for testing synchronization.
from JMauro to Everyone:
I will try to synchronize my metadata with Eizi...
from TOYODA Eizi to Everyone:
so
from Lipeng to Everyone:
int.wmo.gts.cccc.test.*
from TOYODA Eizi to Everyone:
no
from TOYODA Eizi to Everyone:
urn
from TOYODA Eizi to Everyone:
our system does not depend on specificc scheme
from TOYODA Eizi to Everyone:
IPET-MDI-I style is like
from TOYODA Eizi to Everyone:
urn:x-wmo:md:int.wmo.wis::FCUK31EGRR
from WMO Webex 5 to Everyone:
urn:x-wmo:md:int.wmo.wis.test::FCUK31EGRR
from TOYODA Eizi to Everyone:
shoule be okay
from Lipeng to Everyone:
ok for cma
from WMO Webex 5 to Everyone:
thanks
from TOYODA Eizi to Everyone:
glad to hear everybody is flexible on that
from TOYODA Eizi to Everyone:
how about using google groups or yahoo groups
from TOYODA Eizi to Everyone:
log file?
from TOYODA Eizi to Everyone:
I've long thinkng on why WIS Wiki doesnt work well collecting information.
from TOYODA Eizi to Everyone:
I feel like we need casual conversation more than hastely persuing central place.
from TOYODA Eizi to Everyone:
I mean I want Google GROUPS, not Google Docs.
from TOYODA Eizi to Everyone:
the history of conversation is itself useful.
from TOYODA Eizi to Everyone:
change the topic ----
from TOYODA Eizi to Everyone:
how to collect history of log.
from TOYODA Eizi to Everyone:
sorry history of harvesting
from TOYODA Eizi to Everyone:
Look at our RSS of harvesting log
from TOYODA Eizi to Everyone:
http://www.gisc.kishou.go.jp/cgi-bin/oaisync-news.cgi
from JMauro to Everyone:
ok.
from TOYODA Eizi to Everyone:
that should be enough compact information
from WMO Webex 5 to Everyone:
did you hear heinrich?
from TOYODA Eizi to Everyone:
Heinrich?
from WMO Webex 5 to Everyone:
yes.. he joined siegfried
from WMO Webex 5 to Everyone:
Knottenberg
from TOYODA Eizi to Everyone:
Got it but sound is no so good for me
from WMO Webex 5 to Everyone:
what we need to know is how we keep track of the updaes
from WMO Webex 5 to Everyone:
if you modify a metadata record
from WMO Webex 5 to Everyone:
how would you know if it has been propagated sucessfully
from JMauro to Everyone:
yes
from JMauro to Everyone:
I can hear eveyone...
from TOYODA Eizi to Everyone:
you can harvest metadata from every GISCs to check your new metadata is propagated?
from TOYODA Eizi to Everyone:
doesn't have to be SRU. Ex. we have OAI spool separated from SRU, so we test two systems if we use SRU for testing, while we test only OAI system if we use OAI for testing.
from TOYODA Eizi to Everyone:
= agree w/timo
from TOYODA Eizi to Everyone:
yes
from JMauro to Everyone:
yes
from Lipeng to Everyone:
the name of OAI set
from JMauro to Everyone:
no problen
from TOYODA Eizi to Everyone:
using hyphen?
from TOYODA Eizi to Everyone:
I think it's okay
from TOYODA Eizi to Everyone:
thanks
from TOYODA Eizi to Everyone:
single metadata fall in two sets both DWD and CATALOGUE?
from WMO Webex 5 to Everyone:
WISTEST-INMET, WISTEST-JMA , WISTEST-DWD ,
from WMO Webex 5 to Everyone:
WISTEST-CMA
from WMO Webex 5 to Everyone:
and the ful catalogue: WISTEST-CATALOGUE
from JMauro to Everyone:
That´s good...
from TOYODA Eizi to Everyone:
I think i understand but I'd appreciate very much if you Timo could repeat the setttings of experiment in meeting memo
from WMO Webex 5 to Everyone:
of course
from JMauro to Everyone:
I have no idea yet... But if available I will participate..
from TOYODA Eizi to Everyone:
30 okay for me, how bout CMA?
from WMO Webex 5 to Everyone:
thanks Jose
from WMO Webex 5 to Everyone:
30th of August
from JMauro to Everyone:
ok
from TOYODA Eizi to Everyone:
good : around same time
from JMauro to Everyone:
No prolem
from JMauro to Everyone:
Sorry, my microphone did not worked...
from JMauro to Everyone:
Sure
from TOYODA Eizi to Everyone:
thank you
from JMauro to Everyone:
ok thank you.
from JMauro to Everyone:
bye
from Lipeng to Everyone:
bye
from WMO Webex 5 to Everyone:
bye
from WMO Webex 5 to Everyone:
I?m out of here, too... cu you around and thanks!