Download www.met.rdg.ac.uk

COL: data acquisition, storage
and preservation
Roger Brugge
[email protected]
www.met.reading.ac.uk/~brugge
Climatological
Observers Link
Department of Meteorology
University of Reading
Transmission, presentation and archiving of meteorological data, 5 Oct 2011
Outline
• What is COL?
– What the data are used for
• Data gathering
– Daily versus monthly data
– Old and new methods
• Data quality – station gradings
• Data storage
– What is stored?
– How is it stored?
• Data preservation
– For use by COL
– For use by others
What is COL?
• June 1950 - An advert in Weather in June 1950 by Tom Suttie
– enquiring about setting up an organisation for the exchange of weather
data by amateurs
• 1969 - A chance sighting of this advert by Eric Bonsor
– there followed an exchange of letters, and
• January 1970 - first edition of the Directory of Amateur Observers
published
– Site details, instrumentation and observing routines
• June 1970 - first issue of the Bulletin
– 48 stations reporting, describing the weather of May 1970.
– Nowadays more than 330 sites each month
• Aim of COL - to publish the monthly bulletin at the earliest
opportunity
– Achieved for over 41 years now
– The COL Bulletin is now a rare source of comprehensive UK monthly
weather summaries.
– Membership details added to the Directory of Amateur Observers
– Almost 1300 stations are now included, together with photographs of
stations
Data gathering
• Monthly data only
– A few exceptions for a daily data table
– COL cannot handle the volume of daily data that would
otherwise accrue
• Paper
– The original method (15% of returns)
• Email
– Also used as a reminding mechanism to most members (15%)
• Web (70% of returns)
–
–
–
–
Underpinned by a database
PHP5 MyAdmin and SQL >archiving
Provides members with an early snapshot
The way of the future?
Paper form –
main data
features only
shown
Data gathering
• Monthly data only
– A few exceptions for a daily data table
– The Bulletin
• Paper
– The original method (15% of returns)
• Email
– Also used as a reminding mechanism to most members (15%)
• Web (70% of returns)
–
–
–
–
Underpinned by a database
PHP5 MyAdmin and SQL > archiving
Provides members with an early snapshot
The way of the future?
Email reply form – the
start of a message for
August 2011
Data gathering
• Monthly data only
– A few exceptions for a daily data table
– The Bulletin
• Paper
– The original method (15% of returns)
• Email
– Also used as a reminding mechanism to most members (15%)
• Web (70% of returns)
–
–
–
–
Underpinned by a database
PHP5 MyAdmin and SQL > archiving
Provides members with an early snapshot
The way of the future?
Data quality
• COL welcomes observations from everyone
– Multiple stations per town
– Wide variety of instrumentation and exposure
– Airports, universities, institutions and the hobbyist
• Instrumentation
– Originally based around Stevenson screen, mercury-in-glass
thermometers and Snowdon raingauges
– Many good quality sites
– Increasing use being made of AWS equipment
• Varying costs, and also performance
• Need a way to distinguish between stations that (even in
the same locality) might report widely differing obs
– > COL station grading system (Stephen Burt)
Buxton, Norfolk
Stratfield Mortimer
Elderslie
Tivington
Red Wharf Bay,
Anglesey
Compton Martin
Data quality
• COL welcomes observations from everyone
– Multiple stations per town
– Wide variety of instrumentation and exposure
– Airports, universities, institutions and the hobbyist
• Instrumentation
– Originally based around Stevenson screen, mercury-in-glass
thermometers and Snowdon raingauges
– Increasing use being made of AWS equipment
• Varying costs, and also performance
• Need a way to distinguish between stations that (even in
the same locality) might report widely differing obs
– > COL station grading system (Stephen Burt)
Station grading system
(as used in 2011)
Grading system development over the
years has been led by Stephen Burt
COL station grades
U
15%
A
27%
Incl. SYNOP
and CLIMAT
sites
C2
33%
C1
8%
B
17%
• Station grade + photograph + station
details = a reasonable idea of data quality
Data storage
• What is stored?
• Monthly summaries
– A subset of the data; coverage 1940s onwards
• Column-aligned text files
• Easily manipulated by Excel and Fortran, etc.
• Used to make quick calculations and climatological values
– e.g. recently published 1981-2010 averages
• All monthly data, numbers and text
– Database; coverage 2006 onwards
• Generated by SQL/webpage
• Little-used
• All back issues of the bulletin available electronically
– 40th anniversary CD
Members observations – where
do/could they go?
•
•
•
Lots of high-quality COL observations (especially grade A stations)
Effectively unused or un-archived simply because they are not reporting to
UKMO
Data from these sites are likely to be lost unless special arrangements are
made. Currently paper copies
– traditionally,
• sometimes saved in local history libraries
• donated to an observer living close by
• sent to the local rubbish tip (the majority)
– now: they can be deposited with the Chiltern Observatory Trust
•
Electronic data
– Monthly – some/all retained by COL
– Daily – ‘delete’ key
– Either way – file formats soon become outdated
•
•
A proper remote archiving/backup facility is preferable.
Might BADC have a role here?
•
Two suggested solutions…
Preserving members observations
– Possible solution (1)
• 2011: Many amateur sites are fully/partial AWS
• Might consumer-AWS instrument manufacturers agree a
common ‘archive file’ export standard with BADC?
– to simplify the archiving process
– ensure amateur data is not lost when a disk crashes or the
observer dies.
• If this was a file that could be exported then uploaded to
a BADC site, say annually, it would take only minutes.
• A mandatory metadata section should be required.
• Might Davis…
– who are the market leaders
– who currently provide an export file format for NOAA
• …take a lead on this?
Preserving members observations
– Possible solution (2)
• BADC to propose a standard Excel spreadsheet-type
template for amateur observers to use.
– This would greatly simplify the transmission and storage of
amateur observations to BADC.
– A mandatory metadata section should be required, perhaps
including site plan/s and photographs from cardinal points.
– Perhaps pilot
• using the existing ‘COL archive’ template materials (hourly, daily,
monthly templates have been created)
• and with a voluntary participation of a handful of A-grade COL
stations in 2012?
• Issues:
– There are as many observer-owned spreadsheet formats as
observers
– However, given a lead by BADC, COL members might be
persuaded to modify or convert their formats
More information
• This presentation –
– To be placed at www.met.reading.ac.uk/~brugge
• Website
– www.met.reading.ac.uk/~brugge/col.html
• Weather article
– Brugge, R., 2010. Forty years of the Climatological
Observers Link. Weather, 65, 139-143.
• Email the author
– [email protected]
• Bulletin – some copies available today
• ‘Processed data’: an example
– Burt, S.D. and R. Brugge, 2011. Climatological Averages
for 1981-2010 and 2001-2010 for stations appearing in the
monthly bulletin of the Climatological Observers Link.
ISBN 9780956948502.