Download mm/h - H-SAF

The Precipitation Product error structure
Silvia Puca, Emanuela Campione, Corrado DeRosa
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
UniFe and DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS
(Turkey)
Dipartimento della Protezione Civile Italiana
outlines
• PP Validation group;
• data used;
Puca,
Emanuela
Campione,
CorradoValidation);
DeRosa
• validation Silvia
approach
(Common
and
Institute Specific
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
• precipitation
classes;
UniFe and
DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS
(Turkey)
• statistical scores;
•Common Validation results;
Dipartimento della Protezione Civile Italiana
•Validation Results publication (web-page);
• Next steps;
Developer need: Any product has to be related to information on its error
structure, necessary for its correct use in the application
‘Calibration and validation is a difficult activity in the case of precipitation, due
to the natural space-time variability of the precipitation field and the
problematic error structure of the ground truth measurements. ‘
The calibration and validation activity will accompany all steps of the Development
Phase and also will be routinely carried out during the Operational Phase:
Aims:
•
•
To improve the accuracy and the applicability of the products delivered
during the Development phase:
–
supporting the calibration and algorithm tuning,
–
generate the information on error structure to accompany the data,
–
quantify improvements stemming from the progressive implementation of new
developments.
To monitor data quality and provide feedback for progressive quality
improvement during the Operational phase.
Product development
calibration
To assess the
accuracy:
Tuning of the
algorithm to
maximase the
accuracy
validation
Difference from the
measured value and
the “ground truth”
outlines
• PP Validation group;
• data used;
Puca,
Emanuela
Campione,
CorradoValidation);
DeRosa
• validation Silvia
approach
(Common
and
Institute Specific
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
• precipitation
classes;
UniFe and
DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS
(Turkey)
• statistical scores;
•Common Validation results;
Dipartimento della Protezione Civile Italiana
•Validation Results publication (web-page);
• Next steps;
PP Validation group
WP-2300
Precipitation validation
Italy (DPC)
WP-2310
Philosophy
DPC
WP-2320
in Belgium
IRM
WP-2330
in Germany
BfG
WP-2340
in
Hungary
OMSZ
WP-2350
in Italy
UniFerrar
a
WP-2360
in Italy
DPC
WP-2370
in Poland
IMWM
WP-2380
in Slovakia
SHMÚ
WP-2300
Silvia Puca (Team leader)
[email protected]
WP-2310
Silvia Puca
[email protected]
WP-2320
Emmanuel Roulin (+ Angelo Rinollo)
[email protected] (+ [email protected])
WP-2330
Peer Helmke
[email protected]
WP-2340
Eszter Lábó
[email protected]
WP-2350
Federico Porcù
[email protected] (+ [email protected])
WP-2360
Silvia Puca
[email protected]
WP-2370
Bozena Lapeta
[email protected]
WP-2380
Ján Kaňák
[email protected]
WP-2390
Ibrhaim Sonmez + Ahmet Öztopal
[email protected] (+ [email protected])
WP-2390
in Turkey
ITU
PPV Raingauge network is composed by 4100 stations:
Data Sources
raingauges
Instrument
characteristics
Telemetric and
mechanic
time domain
Near real time, case
(near real time/
studies
case studies)
time resolution
(15 min, 30 min)
10 – 30 min
(telemetric),
3 – 24 h
(mechanic)
spatial
distribution
(whole national
territory/ limited
area)
Whole national
territory
~390 mechanic
number of station
(RMI) + 12
(please attach a telemetric (RMI) +
map)
4160 telemetric
(SETHY)
Operational (RMI) +
operational/ for
research (other
research only
networks)
data quality
check
Telemetric:
automatically
checked /
mechanic: autom. +
manually checked
PPV Radar network is composed by 40 C-band and 1 Ka-band:
We have now radars in Turkey
Data Sources
radars
Beam width ~1°,
max range ~150
Instrument
Km, 250m,
characteristics
C-band, single
polarization, Doppler
polarimetric
time domain
near real time/ case
studies
time resolution
5 min, 15 min, 30
min, 1h, 24h
spatial
distribution
Whole national
territory
number of
station
33 C band +1 Ka
band
operational/ for
research only
Operational
data quality
check
Permanent ground
clutter removed;
monitoring of
electronic calibration
validation approach
1)
For the Common Validation activity all Institutes:
- use rain gauges and/or radar data,
- comparisons (sat vs obs) are evaluated on Satellite native grid: same upscaling techniques ;
- evaluate the same monthly statistical scores (Multi-categorical and
Continuous statistics) for the defined precipitation classes;
2) In addition to the common validation each Institute has developed an
Institute Specific Validation activity based on its own knowledge and
experience:
- case studies;
- also lightning data, numerical weather prediction and nowcasting products;
The Common Validation is based on
• Continuous verification statistics: calculating Mean absolute
error, root mean square error, correlation coefficient, standard
deviation.
• Multi-Categorical statistics: calculating the contingency table
(which allows for evaluation of false alarm rate, probability of
detection, equitable threat score, Heidke skill score, etc ).
Continuous
This means that the statistics are calculated using the numeric value of the
satellite precipitation estimation (SPE) and observation at each point.
Categorical
This means that the statistics are calculated from a contingency table, where
each SPE-observation pair is tabulated in the appropriate precipitation
classes. This results in a contingency table. Because most of the
categorical scores are actually computed for "threshold" intervals
(wherein an event occurrence means observed or SPE was equal to or
greater than the threshold value), entries in the table are appropriately
combined to form a 2x2 table for each threshold.
scores evaluated for multi-categorical and continuous
statistics:
MC statistic:
– ACCURACY
– POD
– FAR
– BIAS
– ETS
CS statistic:
- Mean error
- Multiplicative bias
- Mean absolute error
- Root mean square error
- correlation coefficient
- Standard deviation
Plots:
- Scatter plot
- Probability density function
Continuous Score
Mean Absolute Error (MAE)
This score is the mean of the absolute differences between the observations and PSE in the interval. The
score provides a good measure of the accuracy. The closer the MAE is to zero the better the accuracy.
Root Mean Square Error (RMSE)
This score is the square root of the mean of the squared differences between the observations and SPE in
the interval. The score provides a good measure of the accuracy while giving a greater weight to the larger
differences than the MAE does. The closer the RMSE is to zero the better the accuracy.
Mean Error (ME) (bias)
This score is the mean of the arithmetic differences between the observations and SPE in the interval. The
score is a measure of SPE bias, where positive values denote overforecasting, negative values denote
underforecasting, and zero indicates no bias.
Standard Deviation (StD)
This score shows how much variation there is from the "average" (mean). It may be thought of as the
average difference of the scores from the mean of distribution, how far they are away from the mean. A
low standard deviation indicates that the data points tend to be very close to the mean, whereas high
standard deviation indicates that the data are spread out over a large range of values.
Correlation Coefficient
This score is a good measure of linear association or phase error. Visually, the correlation measures how
close the points of a scatter plot are to a straight line. Does not take SPE bias into account -- it is possible
for a SPE with large errors to still have a good correlation coefficient with the observations. Sensitive to
outliers.
Categorical Scores
Equitable Threat Score (ETS)
This score measures the fraction of observed and/or forecast events that were correctly predicted,
adjusted for hits associated with random chance (for example, it is easier to correctly forecast rain
occurrence in a wet climate than in a dry climate). Sensitive to hits. Because it penalises both misses
and false alarms in the same way, it does not distinguish the source of SPE error.
Probability of Detection (POD)
This score is the fraction of the observed area of a threshold precipitation amount that was correctly
forecast. A Satellite product with a perfect POD have a value of one, and forecast with the worst
possible POD have a value of zero.
False Alarm Rate (FAR)
This score is the fraction of the forecast of a threshold precipitation amount that were incorrect. The
worst is one the best is zero. Sensitive to false alarms, but ignores misses. Very sensitive to the
climatological frequency of the event. Should be used in conjunction with the probability of detection
Bias (Bias)
This score is the ratio of the number of forecasts to the number of observations given the threshold
amount. Forecast with perfect bias have a value of one, overforecasting results in bias greater than
one, and underforecasting results in bias less than one.
Accuracy (Acc)
Simple, intuitive. Can be misleading since it is heavily influenced by the most common category,
usually "no event" in the case of rare weather.
Plots
•
Scatter plot - Plots the SPE values against the observed values.
•
This score is good first look at correspondence between SPE and observations. An accurate SPEt will have
points on or near the diagonal.
•
Probability Density Function plot
The comparisons (Sat vs obs) on Satellite native grid:
Up-scaling techniques
The radar and rain gauge data were up-scaled taking into account that the
product follows the scanning geometry and IFOV resolution of AMSU-B scan and
SSMI. Radar and rain gauge instruments provide many measurements within a
single AMSU-B pixel, those measurements were averaged following the AMSU-B
antenna pattern shown and SSMI.
All institutes involved in PP validation activity uses the same up-scaling
technique which was indicated by CNR-ISAC. The codes were developed by
University of Ferrara and RMI.
Precipitation classes
PR-OBS5: 3, 6, 12 and 24
hours accumulated precipitation:
PR-OBS1, PR-OBS2 and PR-OBS3:
Validation Prec. Classes
URD Prec. Classes
Class #
AP [mm]
Class #
PR [mm/h]
Class #
PR [mm/h]
1
PR<0.25
1
AP<1
1
PR<1
2
0.25≤PR<0.50
2
1≤AP<2
2
1≤PR<10
3
0.50≤PR<1.00
3
2≤AP<4
3
10≤PR
4
1.00≤PR<2.00
4
4≤AP<8
5
6
7
PRECIPITATION CLASSES for PR-OBS1, PR-OBS2 5and PR-OBS3:
8≤AP<16
2.00≤PR<4.00
4.00≤PR< 8.00
10.00≤PR<16.00
8
16.00≤PR<32.00
9
32.00≤PR<64.00
10
64.00≤PR
PR= PRECIPITATION RATE
0,25 mm/h is the threshold for
precipitation/no-precipitation.
6
16≤AP<32
7
32≤AP<64
8
64≤AP<128
9
128≤AP<256
10
256≤AP
AP= ACCUMULATED PRECIPITATION
1.00 mm is the threshold for
precipitation/no-precipitation.
H01 continuous statistic: radar data and rain gauge LAND
Period: September 2008 – June 2009
H01 Land: ST DEV (radar, rain gauge)
H01 Land: MEAN ERROR (radar, rain gauge)
1
10
0
gi
u09
ag
-0
9
m
ap
r09
ar
-0
9
m
-0
9
fe
b
ge
n09
di
c08
no
v08
ot
t-0
8
se
t-0
8
9
8
-2
7
-3
6
ST DEV (mm/h)
ME (mm/h)
-1
-4
-5
-6
5
4
3
2
-7
1
-8
H01 Land: MEAN ABSOLUTE ERROR (radar, rain gauge)
9
9
8
8
7
7
6
6
5
4
gi
u09
ag
-0
9
ap
r09
4
3
2
2
1
1
0
ar
-0
9
5
3
H02 New version
gi
u09
-1
ag
-0
9
09
ugi
m
m
9
-0
ag
ap
r09
9
r-0
ap
ar
-0
9
9
m
m
-0
ar
-0
9
09
bfe
fe
b
09
nge
ge
n09
08
cdi
di
c08
08
vno
no
v08
8
t-0
ot
ot
t-0
8
0
8
t-0
se
se
t-0
8
-1
H01 Land: RMSE (radar, rain gauge)
10
RMSE (mm/h)
MAE (mm/h)
10
m
-10
m
-0
9
fe
b
ge
n09
di
c08
no
v08
ot
t-0
8
-1
se
t-0
8
0
-9
H01 continuous statistic: radar and rain gauge
Period: September 2008 – December 2008
N RAD
[mm/h]
0.25≤PR<1
8797
1≤PR<10
ME
[mm/h]
StD
[mm/h]
MAE
[mm/h]
-0.41
0.23
0.48
RMSE
[mm/h]
H01
0.51
URDrmse
[%]
N RG
[mm/h]
ME
[mm/h]
StD
[mm/h]
MAE
[mm/h]
RMSE
[mm/h]
URDrmse
[%]
105.09
17501
-0.56
0.02
0.67
0.66
109.83
6138
-7.22
0.09
7.32
3.03
147.13
1047
-19.45
1.05
19.45
14.26
104.95
24686
-2.65
0.06
2.75
1.52
119.57
MAE
[mm/h]
RMSE
[mm/h]
URDrmse
[%]
3222
-2.43
0.74
2.48
2.66
(mm/
h) 98.90
215
-15.55
2.25
15.55
12.75
100.00
12234
-1.20
0.40
1.27
1.29
INDEX:
10.00≤PR
MEAN ERROR (mm/h)
MEAN
0,04
MULTI BIAS
3,39
MEAN ABS. ERR
(mm/h)
N RAD
[mm/h]
ME
[mm/h]
Period:0,13
Jenuary 2009 – June 2009
MAE
[mm/h]
RMSE
[mm/h]
StD
[mm/h]
0.73
2.05
CORRELATION
COEF. 6.15
10.00≤PR
602
-10.94
ROOT MEAN SQUARE
163116
-0.04
1.16
ERROR
(mm/h)
0.25≤PR<1
1≤PR<10
MEAN
69283
233001
103.37
-0.87
-0.31
1.44
URDrmse
[%]
N RG
[mm/h]
ME
[mm/h]
StD
[mm/h]
1.24
251.73
0,30
112635
0.05
1.60
0.86
1.62
289.68
1.84
2.39
117.34
45549
-0.42
3.01
2.16
3.22
145.89
12.14
13.03
0,2577.05
261
-12.16
5.68
12.86
13.70
74.47
1.09
1.61
211.32
158445
-0.12
2.03
1.27
2.13
246.37
STANDARD DEVIATION
ME=ERROR
Mean Error,
SD=Standard Deviation, MAE =Mean
(mm/h)
0,34Aboslute Error,
RMSE= Root Mean Square Error; URD RMSE= Root Mean Square Rrror defined in URD doc.
•There is an evident increase of the errors in the higher precipitation class;
H01: Multi-category statistic
PR<0.25
0.25<PR
tot RD
PR<0.25
0,96
0,04
139695
0.25<PR
0,82
0,18
7383
140049
7029
147078
tot SAT
PR<0.25
0.25<PR<1
POD(rain/Norain)
0,95
FAR(rainNorain)
0,04
1<PR<10
10<PR
tot RD
PR<0.25
0,960
0,023
0,017
0,000
139695
0.25<PR<1
0,879
0,052
0,066
0,003
5713
1<PR<10
0,614
0,073
0,306
0,007
1668
10<PR
1,000
0,000
0,000
0,000
2
tot SAT
140049
3601
3334
94
147078
•Good value of POD and FAR for rain/no-rain;
•Clear underestimation of the precipitation.
coast/land analysis H01
University of Ferrara: F. Porcù
coast/land analysis H01
University of Ferrara: F. Porcù
H02 continuous statistic: radar data and rain gauge LAND
Period: September 2008 – June 2009
H01 H02
Land:
ST DEV
(radar,
rain gauge)
Land:
ST DEV
(radar,
rain gauge)
H02 Land: MEAN ERROR (radar, rain gauge)
1
10 10
9 9
39
63
8
-3
6 6
4 4
H02 Land:
Land: MEAN
MEAN ABSOLUTE
ABSOLUTEERROR
ERROR(radar,
(radar,rain
raingauge)
gauge)
H01
1010
9 9
7 7
7 7
6 6
6
MAE (mm/h)
(mm/h)
MAE
5 5
4 4
3 3
RMSE (mm/h)
8 8
RMSE (mm/h)
8 8
5
4
3
2 2
2
1 1
1
6
5
4
3
2
1
gi
u09
ag
-0
9
m
ap
r09
ar
-0
9
-1
26
40
-1
-0
9
u
gi
9
-0
m
m
9
-0
ag
63
8
9
r-0
ap
fe
b
m
9
-0
ar
ge
n09
b
fe
9
-0
di
c08
39
09
nge
no
v08
d
08
ic-
ot
t-0
8
84
08
vno
39
63
8
8
t-0
ot
26
84
40
31
17
2
0
8
t-0
se
se
t-0
8
31
17
2
0
0 0
-1
Land:
RMSE
(radar,
gauge)
H01 H02
Land:
RMSE
(radar,
rain rain
gauge)
10 10
9 9
-1
gi
u09
0 0
-1 -1
se
t-0
31 8
17
2
-9
-10
ag
-0
9
1 1
m
2 2
-8
ap
r09
-7
ar
-0
9
3 3
m
-6
-0
9
-5
5 5
fe
b
-4
26
di
c08
39
63
8
ge
n09
ST DEV (mm/h)
7 7
ot
t-0
8
84
40
no
v08
26
8 8
-2
ST DEV (mm/h)
Mean error (mm/h)
-1
84
40
31
17
2
0
H02 continuous statistic: radar and rain gauge
Period: September 2008 – December 2008
N RAD
[mm/h]
ME
[mm/h]
StD
[mm/h]
MAE
[mm/h]
RMSE
URDrmse
[mm/h]
H01 [%]
N RAD
[mm/h]
ME
[mm/h]
StD
[mm/h]
MAE
[mm/h]
RMSE
[mm/h]
URDrmse
[%]
31172
-0.21
0.48
0.51
0.68
(mm/
h) 135.33
39683
-0.44
0.85
0.59
0.95
208.09
INDEX: 8440
1≤PR<10
-0.57
1.12
1.32
1.63
95.24
43260
-2.38
2.05
2.90
3.28
106.03
ERROR
(mm/h)
26
-7.14
1.20
7.20
7.32
0,04
62.50
5250
-9.90
6.29
13.72
14.11
91.50
0.69
0.89
126.74
3,39
88193
-0.86
1.11
1.09
1.45
186.28
MAE
[mm/h]
RMSE
[mm/h]
URDrmse
[%]
0.25≤PR<1
MEAN
10.00≤PR
MEDIA
39638
MULTI BIAS
-0.29
0.62
MEAN ABS. ERR
(mm/h)
Period: 0,13
Jenuary 2009 – June 2009
ROOT MEAN SQUARE
N RAD
ME
StD
ERROR
(mm/h)
[mm/h]
[mm/h]
[mm/h]
MAE
[mm/h]
RMSE
[mm/h]
URDrmse
0,30
[%]
CORRELATION
COEF. 0.82
0.25≤PR<1
134299
-0.15
0.58
1.00
0,25
185.89
69358
-0.22
0.97
0.60
0.98
217.35
1≤PR<10
63441
STANDARD
1.75
2.33
111.99
27808
-1.10
2.24
1.90
3.41
149.45
11.42
12.54
0,34
69.59
167
-10.49
5.58
11.40
13.11
81.68
0.99
1.46
161.89
97333
-0.54
1.39
1.05
1.62
195.21
10.00≤PR
MEDIA
-0.88
1.73
DEVIATION
ERROR
624 (mm/h)
-8.72
6.59
198364
-0.41
1.13
N RAD
[mm/h]
ME
[mm/h]
StD
[mm/h]
ME= Mean Error, SD=Standard Deviation, MAE =Mean Aboslute Error,
RMSE= Root Mean Square Error; URD RMSE= Root Mean Square Rrror defined in URD doc.
•There is an evident increase of the errors in the higher precipitation class;
H02: Multi-category statistic
H02: Validation Precipitation classes for LAND
areas. Data used: RADAR. Period: 10 2008
PR<0.25
0.25<PR
tot RD
PR<0.25
0,989
0,009
292709
0.25<PR<1
0,719
0,281
17407
POD(rain/Norain)=
0,958
302293
7823
310116
FAR(rainNorain)=
0,003
tot SAT
H02: Validation Precipitation classes for LAND areas. Data used: RADAR. Period: 10 2008
PR<0.25
0.25<PR<1
1<PR<10
10<PR
tot RD
PR<0.25
0,989
0,006
0,003
0
292709
0.25<PR<1
0,818
0,093
0,086
0,001
9097
1<PR<10
0,626
0,131
0,237
0,004
7601
10<PR
0,437
0,158
0,379
0,025
709
tot SAT
302293
3821
3863
139
310116
•Good value of POD and FAR for rain/no-rain;
•Clear underestimation of the precipitation but more capacity to
discriminate the precipitation than H01.
H03 continuous statistic: radar data and rain gauge LAND
Period: September 2008 – June 2009
H01 Land: ST DEV (radar, rain gauge)
10
9
H03 Land: MEAN ERROR (radar, rain gauge)
1
10
9
3
1.
1E
+0
7
19
06
03
5
8
ST DEV (mm/h)
25
47
83
74
04
5
0
-1
-3
-4
-5
65
5
4
4
3
-6
7
76
ST DEV (mm/h)
-2
ME (mm/h)
H03 Land: ST DEV (radar, rain gauge)
8
3
22
-7
10
Land:MEAN
MEAN ABSOLUTE
ERROR
(radar, rain
gauge)rain gauge)
H01 H03
Land:
ABSOLUTE
ERROR
(radar,
10
9
9
7
8
8
6
7
7
5
6
gi
u09
ag
-0
9
m
ap
r09
ar
-0
9
m
-0
9
fe
b
ge
n09
43
3
H01 Land:
RMSE
(radar,
gauge)
H03 Land:
RMSE
(radar, rain
rain gauge)
10
9
di 1
c-0 9
0840
1
-1
no
v- 19
08 06
-10
-0
8
-1
2
0
83
se 7404
t
5
0
-9
ot 5 47
t-0 03
8 5
11
-8
2
10
94
01
43
0
09
ugi
-1
gi
u09
m
9
-0
ag
ag
-0
9
9
r-0
ap
m
9
ap
r09
m
-0
ar
ar
-0
9
09
bfe
m
09
nge
-0
9
08
cdi
fe
b
08
vno
ge
n09
8
t-0
ot
di
c08
0
8
t-0
se
no
v08
-1
1
ot
t-0
8
1
83
74
04
5
2
-1
19
06
3
3
0
25
47
03
5
3
10
94
01
43
4
1
19
06
3
4
5
25
47
03
5
5
6
se
t-0
8
2
RMSE
RMSE (mm/h)
3
MAE (mm/h)
4
83
74
04
5
MAE (mm/h)
8
H03 continuous statistic: radar and rain gauge
Period: September 2008 – December 2008
N
N RAD
RAD
[mm/h]
[mm/h]
StD
StD
[mm/h]
[mm/h]
MAE
MAE
[mm/h]
[mm/h]
-0.21
-0.31
-0.57
-1.65
0.48
0.53
1.12
0.94
0.51
0.52
1.32
1.88
0.68
0.74
1.63
2.07
ERROR
(mm/h)
26
-7.14
1.20
19063
-15.94
3.91
7.20
15.96
0.69
0.87
0.25≤PR<1
31172
0.25≤PR<1
8374045
INDEX: 8440
1≤PR<10
1≤PR<10
2547035
MEAN
10.00≤PR
10.00≤PR
MEDIA
39638
BIAS
MEANMULTI 10940143
-0.29
-0.65
0.62
0.63
MEAN ABS. ERR
(mm/h)
[mm/h]
[mm/h]
MEME
StDStD
[mm/h]
[mm/h]
[mm/h]
[mm/h]
URDrm
MAE
RMSE
MAE
RMSE URDrmse
se
[mm/h]
[mm/h]
[%]
[mm/h]
[mm/h]
[%]
95.24
105.93
39683
803197
43260
391313
-0.44
-1.18
-2.38
-13.22
0.85
0.01
2.05
0.09
0.59
1.33
2.90
13.46
0.95
0.94
3.28
5.54
208.09
189.18
106.03
204.57
7.32
16.59
0,04
62.50
94.47
5250
1147818
-9.90
-14.73
6.29
2.10
13.72
14.76
14.11
14.06
91.50
97.49
0.89
1.08
126.74
3,39
160.81
88193
2342328
-0.86
-4.01
1.11
0.03
1.09
4.18
1.45
2.03
186.28
192.61
MAE
MAE
[mm/
[mm/h]
h]
0.58
RMSE
RMSE
[mm/h]
[mm/h]
1.00
URDrm
URDrmse
0,30
se
[%]
[%]
0,25
185.89
0.50
1.75
0.58
2.33
0.58
6.59
1.75
11.42
[mm/h]
CORRELATION
COEF. 0.82
0.25≤PR<1
134299
-0.15
0.25≤PR<
1
14767131
-0.44
0.29
1≤PR<10
63441 DEVIATION
-0.88
1.73
STANDARD
4178286
-1.70
ERROR
624 (mm/h)
-8.72
(mm/
135.33
h) 177.65
N NRAD
RAD
[mm/h]
[mm/h]
Period: 0,13
Jenuary 2009 – June 2009
ROOT MEAN SQUARE
RAD
ME
StD
NNRAD
ME
StD
ERROR
(mm/h)
[mm/h]
[mm/h]
[mm/h]
1≤PR<10
10.00≤PR
URDr
RMSE
RMSE URDrmse
mse
[mm/h]
[%]
H01
[mm/h]
[%]
ME
ME
[mm/h]
[mm/h]
URDrm
URDrmse
se
[%]
[%]
217.35
NNRAD
RAD
[mm/h]
[mm/h]
ME
ME
[mm/h]
[mm/h]
StD
StD
[mm/h]
[mm/h]
MAE
MAE
[mm/h]
[mm/h]
RMSE
RMSE
[mm/h]
[mm/h]
69358
-0.22
0.97
0.60
0.98
108.48
111.99
69358
27808
-0.43
-1.10
0.35
2.24
0.53
1.90
0.60
3.41
131.65
149.45
1.85
12.54
96.41
0,34
69.59
27808
167
-1.56
-10.49
0.68
5.58
1.68
11.40
1.76
13.11
110.80
81.68
161.89
95.05
97333
167
-0.54
-14.26
1.39
2.69
1.05
14.69
1.62
15.21
195.21
83.15
97333
-0.69
0.42
0.79
0.87
MEDIA
10.00≤PR
198364
15838
-0.41
-14.51
1.13
3.94
0.99
17.20
1.46
18.05
MEAN
18961255
-0.73
0.36
0.79
0.87
105.81
ME= Mean Error, SD=Standard Deviation, MAE =Mean Aboslute Error,
RMSE= Root Mean Square Error; URD RMSE= Root Mean Square Rrror defined in URD doc.
•There is an evident increase of the errors in the higher precipitation class;
127.02
H03: Multi-category statistic
PR<0.25
0.25<PR
tot RD
PR<0.25
0,957
0,040
51880730
0.25<PR<1
0,723
0,276
2324380
POD(rain/Norain)
0,967
51298933
2906177
54205110
FAR(rainNorain)
0,040
0.25<PR<1
1<PR<10
tot SAT
PR<0.25
10<PR
tot RD
PR<0.25
0,957
0,026
0,013
0,001
51880730
0.25<PR<1
0,740
0,167
0,088
0,003
1727431
1<PR<10
0,673
0,153
0,162
0,010
588126
10<PR
0,647
0,065
0,273
0,028
8823
tot SAT
51298933
1871293
997318
37566
54205110
•Good value of POD and FAR for rain/no-rain;
•Clear underestimation of the precipitation but several cases of
overestimation of precipitation area;
coast/land analysis H03
University of Ferrara: F. Porcù
coast/land analysis H03
University of Ferrara: F. Porcù
H05: Continuous statistic
[mm]
# RADAR
ME[mm]
SD[mm]
MAE[mm]
RMSE[mm]
URDrmse
AP< 8
13920871
-0,360
1,363
1,063
1,609
127%
8 ≤AP<32
3562429
-1,418
3,006
3,680
4,396
86%
32 ≤AP< 64
162626
-5,133
3,465
6,041
6,546
86%
64 ≤AP< 128
170027
-7,948
4,326
8,408
9,257
92%
25758
-14,42
4,997
14,421
15,308
87%
17841711
-0,707
1,744
1,721
2,303
107%
ME[mm]
SD[mm]
MAE[mm]
RMSE[mm]
URDrmse
489652,3
-5,927
0,413
0,011
469%
149317,7
-48,705
6,286
0,033
2410%
37553,23
-159,432
56,892
0,014
6882%
12010,45
-356,61
238,898
0,061
11699%
7381,739
-298,066
52,913
0,129
23309%
838385,3
-19,631
4,449
0,016
10546%
128 ≤AP
MEAN
[mm]
# RAIN
GAUGE
AP< 8
4943648
8 ≤AP<32
3029460
32 ≤AP< 64
601979
64 ≤AP< 128
170098
128 ≤AP
MEAN
29611
8774796
ME= Mean Error, SD=Standard Deviation, MAE =Mean Aboslute Error,
RMSE= Root Mean Square Error; URD RMSE= Root Mean Square Rrror defined in URD doc.
It is necessary a verification of rain gauge validation results!
Some conclusions
• All the PP were validated by comparison with both radar and rain gauge data by
7 countries,
• Multi category and continuous statistical scores were evaluated;
• All the statistical scores evaluated and the case studies analysed are available in
AM ftp server;
*H01:
-the majority of the precipitation is estimated less than 0.25 mm/h by H01;
-there is a general under-estimation of the precipitation estimation;
-No strong seasonal component is present;
-there is an evident increase of the errors in the lower classes respect the previus
version.
Some conclusions
*H02:
-There is a general underestimation but more capacity to discriminate precipitation
greater than 0.25 mm/h;
- Seasonal component is present;
-there is an evident increase of the errors in the higher precipitation class;
-problem with Noaa16:replacment of a channel. (noise effect)
*H03:
-There is a general underestimation of precipitation rate and an overestimation of
precipitation area;
-Seasonal component is present;
-There is an evident increase of the errors in the higher precipitation class;
-heavy convective precipitation events were underestimated;
-Moderate and light convective precipitation events were often overestimated;
*H05:
-Same not realistic value of precipitation;
-Not enough results;
Validation Results publication
• Rep 3: collects all the results of the PP Validation activity: It is
a rolling document.
Silvia Puca,
Emanuela Campione,
DeRosa
• User Requirement
Documents:
summarise Corrado
the PP validation
results;
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
UniFe and DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS
• Web Page: all the results are in
the H-saf web-page in the
(Turkey)
validation section.
Dipartimento della Protezione Civile Italiana
Next steps
• Rep 3: collects all the results of the PP Validation activity: It is
a rolling document.
Silvia Puca,
Emanuela Campione,
DeRosa
• User Requirement
Documents:
summarise Corrado
the PP validation
results;
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
UniFe and DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS
• Web Page: all the results are in
the H-saf web-page in the
(Turkey)
validation section.
Dipartimento della Protezione Civile Italiana
THANK YOU!