Download PPT

Data quality and surveys for
SEEAW
Training Course on Water Accounting
Amman, Jordan
10-13 March 2008
Michael Vardon
United Nations Statistics Division
1
Outline
•
•
•
•
Data quality
Data sources
Surveys options
Surveys
• Examples of surveys from Australia
Data quality
•
•
•
•
•
•
Relevance
Accuracy
Timeliness
Accessibility
Interpretability
Coherence
Statistics Canada – Quality Guidelines 4th Edition 2003
http://www.statcan.ca/english/freepub/12-539-XIE/12-539XIE03001.pdf
Relevance
“The relevance of statistical information reflects the
degree to which it meets the real needs of clients. It is
concerned with whether the available information
sheds light on the issues that are important to users.
Assessing relevance is subjective and depends upon
the varying needs of users. The Agency’s challenge is
to weigh and balance the conflicting needs of current
and potential users to produce a program that goes as
far as possible in satisfying the most important needs
within given resource constraints.”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 6
Accuracy
“The accuracy of statistical information is the degree to
which the information correctly describes the
phenomena it was designed to measure. It is usually
characterized in terms of error in statistical estimates
and is traditionally decomposed into bias (systematic
error) and variance (random error) components. It
may also be described in terms of the major sources
of error that potentially cause inaccuracy (e.g.,
coverage, sampling, non response, response).”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 6-7
Timeliness
“The timeliness of statistical information refers to the
delay between the reference point (or the end of
the reference period) to which the information
pertains, and the date on which the information
becomes available. It is typically involved in a
trade-off against accuracy. The timeliness of
information will influence its relevance.”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 7
Accessibility
“The accessibility of statistical information refers to
the ease with which it can be obtained from the
Agency. This includes the ease with which the
existence of information can be ascertained, as
well as the suitability of the form or medium
through which the information can be accessed.
The cost of the information may also be an aspect
of accessibility for some users.”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 7
Interpretability
“The interpretability of statistical information
reflects the availability of the supplementary
information and metadata necessary to interpret
and utilize it appropriately. This information
normally includes the underlying concepts,
variables and classifications used, the
methodology of data collection and processing,
and indications or measures of the accuracy of the
statistical information.”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 7
Coherence
“The coherence of statistical information reflects the
degree to which it can be successfully brought
together with other statistical information within a
broad analytic framework and over time. The use
of standard concepts, classifications and target
populations promotes coherence, as does the use
of common methodology across surveys.
Coherence does not necessarily imply full
numerical consistency.”
Statistics Canada – Quality Guidelines 4th Edition 2003 p. 7
Data sources
• Administrative
• e.g. Licensing data bases
• Company reports
• Many companies have reports that include
information on the environment (e.g. water and
energy use, pollution control measures)
• Academic sources
• Can be government or non-government
• Surveys
Surveys
Data
• Physical data
• Monetary data
Survey options
• Adding questions to existing surveys
• Running specialized surveys
Survey methodology
• Mail-out/mail-back, telephone interview, personal
interview
• Mail-out/mail-back the cheapest and hence most often
used
• Regardless of methods it is critical to test forms in the
field prior to the survey.
• Sample design
Survey testing regime
Good questionnaire design will help minimize response
biases.
• Forms should attractive and easy to understand and
fill in.
• Develop a draft of form based on data requirements
and knowledge of survey respondents, using terms
and concepts familiar to survey designers and survey
respondents. Test on a small number via personal
interviews (10)
• Develop 2nd draft of form and test on a larger number
of respondents (30-60) via personal interviews and
mail-out/mail-back (if this is the method to be used)
Specialized Surveys
Advantage
• Can collect a range of data need for the
accounts as well as additional data to aid
interpretation and understanding
• Can select sample to represent total water
use (not total economic activity)
Disadvantage
• Costly to develop and run
Adding questions to existing
surveys
Advantage
• Direct link to the other data collected in the survey
• Cheaper than running specialized surveys
Disadvantage
• Financial officers or business accountants who
typically fill in business survey forms may not be
familiar with environmental data
• Sample selection may not be ideal
Sample selection may not be ideal
For example, in business surveys sample may be
selected to estimate total economic activity, not
total water use or supply
As such large water using industries (food and
beverage manufacturing, metal manufacturing,
paper manufacturing, electricity generation) may
be under-represented in sample design. This
problem can be overcome by adding sample in
these industries
Example questionnaires from
Australia
Specialized surveys
• Water Supply and Sewerage Survey
• Water Survey (supplement to Agricultural Survey)
Adding questions to existing surveys
• Mining Survey
• Manufacturing Survey
• Agricultural Census
All surveys mail-out /mail-back
ABS Surveys
– Water Supply and Sewerage Industries Survey
ABS Surveys –
Water Survey, Agricultural
ABS Surveys –
Agricultural Census
ABS Surveys –
Economic Activity Survey (Mining)
ABS Surveys –
Economic Activity Survey
(Manufacturing)
Intensive follow-up, 1
• Survey forms will often be filled in by a
financial officer or business accountant who
may not have easy access to the data
required
• Intensive follow-up required and should be
targeted at large water using industries. For
example:
• Pulp and paper manufacturing,
• Food and beverage manufacturing
• Metals manufacturing
• Electricity generation
Intensive follow-up, 2
Steps for intensive follow-up in Australia
• 1st Reminder letter
• 2nd Reminder letter
• Phone call - important to carefully train staff. Also
it may be possible to collect key data items over
the phone
• Final reminder
• Notice of Direction (fill in the form or be
summoned to court)
Data checking / input editing
• Identification of outliers or improbable (that
is very large or very small numbers) are
identified for each industry (or region).
• Double check very large numbers with the
data supplier (often units are wrongly
reported (e.g. m3 reported not 1,000 m3 or
acres not hectares)
• Check blank cells – they may be zero
Contact details
Michael Vardon
Adviser on Environmental-Economic Accounting
United Nations Statistics Division
New York 10017 USA
Room DC2 1532
Phone: +1 917 367 5391
Fax: +1 917 363 1374
Email: [email protected]