Download **** 1 - UFL MAE

Paper Helicopter Project
Chanyoung Park
Raphael T. Haftka
Problem1: Conservative estimate of the fall time
 Estimating the 5th percentile of the fall time of one helicopter
 Estimating the 5th percentile to
compensate the variability in the fall time
(aleatory uncertainty)
 The sampling error
(epistemic uncertainty)
mt,P
Sampling
 Estimating the sampling uncertainty
in the mean and the STD
 Obtaining a distribution of the
5th percentile
 Taking the 5th percentile of the 5th
percentile distribution to compensate
the sampling error
st,P
tP
Sampling
t0.05,P
2/25
Structural & Multidisciplinary Optimization Group
Problem1: Conservative estimate of the fall time
 Estimating the 5th percentile of the fall time of first helicopter
(mean 3.78, std 0.37)
15000
 100,000 5th percentiles of fall time
 Helicopter 1 of the dataset 3
10000
 Height: 148.5 in
 2.88 (sec) is the 5th percentile
of the histogram
(a conservative estimate of
the 5th percentile of the fall
time for 95% confidence)
5000
0
2
2.5
3
5th percentile
3.5
4
3/25
Structural & Multidisciplinary Optimization Group
Problem2: Predicted variability using prior
 Calculating predicted variability in the fall time
 We assume that the variability in the fall time is caused by the
variability in the CD
 The variability in the fall time is predicted using the computational
model (quadratic model) and the distribution of the CD
Height at time t
h  t   Vs t 


log e 2Vs ct  1  log 2
g
Steady state speed Vs 
0.5 CD A / m
c
where
c  0.5CD A / m
 The prior distribution represents our initial guess for the distribution of
the CD
4/25
Structural & Multidisciplinary Optimization Group
Problem2: Comparing predicted variability and
observed variability using prior
 Area metric with the prior
 Data set 3
Area metric
Area metric
1
0.8
0.8
0.8
0.6
0.4
0.2
0
2.5
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric
1
0.6
0.4
0.2
3
3.5
4
Fall time (sec)
4.5
0.29
5
0
2.5
0.6
0.4
0.2
3
3.5
4
Fall time (sec)
4.5
5
0
2.5
0.34
CD from the fall time data Helicopter1
3
3.5
4
Fall time (sec)
5
0.49
Helicopter2
Helicopter3
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
Prior
4.5
Mean of CD = 1 / STD of CD = 0.28
5/25
Structural & Multidisciplinary Optimization Group
Problem3: Calibration: Posterior distribution of
mean and standard deviation
 Estimating parameters of the CD distribution
 We assume that CD of each helicopter follows the normal distribution
CD,test ~ N  CD , s test 
 The parameters, CD and σtest are estimated using 10 data
 Posterior distribution is obtained based on 10 fall time data
p  CD , s test | C
   p C
10
1
D ,test
,..., C
10
D ,test
i 1
i
D ,test
| CD , s test  p  CD  p s test 
 Non informative distribution is used for the standard deviation
0.5
0.5
0.5
0.45
0.45
0.45
After 1 update
0.4
0.35
After 5 updates
0.4
0.35
0.4
0.35
0.3
0.3
0.3
0.25
0.25
0.25
0.2
0.2
0.2
0.15
0.15
0.15
0.1
0.1
0.1
0.05
0.05
0.05
0
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
0
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
0
0.5
After 10 updates
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
6/25
Structural & Multidisciplinary Optimization Group
Problem3: Comparing predicted variability
using posterior and observed variability
 Comparing MLE and sampling statistics
CD from the fall time data Helicopter1
Helicopter2
Helicopter3
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
MLE of the CD mean
0.896
0.843
0.783
MLE of the CD STD
0.182
0.146
0.081
 MCMC sampling
 10,000 pairs of the CD and the STD of CD are generated using
Metropolis-Hastings algorithm
 An independent bivariate normal distribution is used as a proposal
distribution
 MLE of the posterior distribution is used as a starting point
7/25
Structural & Multidisciplinary Optimization Group
Problem3: Comparing predicted variability
using posterior and observed variability
 Handling the epistemic uncertainty due to finite sample
 How to handle epistemic uncertainty in the CD and the test standard
deviation estimates?
 Comparing the posterior predictive distribution of the fall time and the
empirical CDF of tests (combining epistemic and aleatory
uncertainties)
 Using p-box with 95% confidence interval of epistemic uncertainty
(separating epistemic and alreatory uncertainties)
8/25
Structural & Multidisciplinary Optimization Group
Problem3: Comparing predicted variability
using posterior and observed variability
 Area metric of the posterior predictive distribution of CD
1
0.8
0.8
0.8
0.6
0.4
0.2
0
-2
CDF of fall time
1
1
CDF of fall time
CDF of fall time
Area metric
Area metric
Area metric
0.6
0.4
2
Fall time (sec)
4
6
0
0
0.4
0.2
0.2
0
0.6
1
2
3
4
Fall time (sec)
0.15
5
6
0
1
2
0.11
3
Fall time (sec)
4
5
0.07
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
2 clips (ref)
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
9/25
Structural & Multidisciplinary Optimization Group
Problem3: Comparing predicted variability
using posterior and observed variability
 Area metric of the posterior predictive distribution of CD
1
0.8
0.8
0.8
0.6
0.4
0.6
0.4
1
2
3
4
Fall time (sec)
5
6
0
0
0.6
0.4
0.2
0.2
0.2
0
0
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric with p-box
Area metric with p-box
Area metric with p-box
1
1
2
3
4
Fall time (sec)
0.01
5
6
0
1
2
0.00
3
Fall time (sec)
4
5
0.00
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
2 clips (ref)
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
10/25
Structural & Multidisciplinary Optimization Group
Problem4: Predictive validation for the same
height and different weight
 Area metric of the posterior predictive distribution of CD
Area metric
Area metric
1
0.8
0.8
0.8
0.6
0.4
0.6
0.4
0
2
Fall time (sec)
4
6
0
0
0.6
0.4
0.2
0.2
0.2
0
-2
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric
1
1
0.19
2
3
4
Fall time (sec)
5
6
0
2
2.5
0.34
3
3.5
4
Fall time (sec)
4.5
5
0.38
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
1 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample STD of CDs
0.916
0.147
0.992
0.069
0.968
0.049
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
11/25
Structural & Multidisciplinary Optimization Group
Problem4: Predictive validation for the same
height and different weight
 Area metric of the distribution of CD with p-box
Area metric with p-box
Area metric with p-box
1
0.8
0.8
0.8
0.6
0.4
0.6
0.4
0.2
0.2
0
0
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric with p-box
1
2
4
Fall time (sec)
6
8
0
0
0.6
0.4
0.2
1
0.01
2
3
4
Fall time (sec)
5
6
0
2
3
0.11
4
Fall time (sec)
5
6
0.22
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
1 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample STD of CDs
0.916
0.147
0.992
0.069
0.968
0.049
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
12/25
Structural & Multidisciplinary Optimization Group
Problem6: Predictive validation for different
height and the same weight
 Area metric of the posterior predictive distribution of CD
1
0.8
0.8
0.8
0.6
0.4
0.6
0.4
0
2
4
Fall time (sec)
6
8
0
0
0.6
0.4
0.2
0.2
0.2
0
-2
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric
Area metric
Area metric
1
2
4
Fall time (sec)
0.34
6
8
0
2
3
0.19
4
Fall time (sec)
5
6
0.09
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
181.25 in
2 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample STD of CDs
0.886
0.079
0.866
0.119
0.816
0.066
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
13/25
Structural & Multidisciplinary Optimization Group
Problem6: Predictive validation for different
height and the same weight
 Area metric of the distribution of CD with p-box
Area metric with p-box
Area metric with p-box
1
0.8
0.8
0.8
0.6
0.4
0.2
0
0
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric with p-box
1
0.6
0.4
4
Fall time (sec)
6
8
0
0
0.4
0.2
0.2
2
0.6
2
0.05
4
Fall time (sec)
6
8
0
2
3
0.00
4
Fall time (sec)
5
6
0.01
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
181.25 in
2 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample STD of CDs
0.886
0.079
0.866
0.119
0.816
0.066
Sample mean of CDs
0.896
0.842
0.783
Sample STD of CDs
0.190
0.154
0.084
14/25
Structural & Multidisciplinary Optimization Group
Problem5: Linear model
 Area metric with the prior
 Data set 3
1
0.8
0.8
0.8
0.6
0.4
CDF of fall time
1
1
CDF of fall time
CDF of fall time
Area metric
Area metric
Area metric
0.6
0.4
0
2
3
4
Fall time (sec)
5
6
0.65
0
2
3
4
Fall time (sec)
Sample STD of CDs
Prior
5
6
0
2
0.68
CD from the fall time data Helicopter1
Sample mean of CDs
0.4
0.2
0.2
0.2
0.6
0.978
0.102
3
4
Fall time (sec)
5
6
0.81
Helicopter2
0.949
0.091
Helicopter3
0.916
0.050
Mean of CD = 1 / STD of CD = 0.28
15/25
Structural & Multidisciplinary Optimization Group
Comparison to predictive validation
 Area metric of the posterior predictive distribution of CD
 The predictive validation with the linear model is not as successful as
that with the quadratic model
Helicopter1
Helicopter2
Helicopter3
148.5 in / 1 clips
0.44
0.30
0.14
148.5 in / 2 clips (ref)
0.13
0.11
0.07
 Area metric with p-box
 Area metric with p-box tries to capture the extreme discrepancy
between the predicted variability and the observed variability
Helicopter1
Helicopter2
Helicopter3
148.5 in / 1 clips
0.13
0.06
0.02
148.5 in / 2 clips (ref)
0.01
0.00
0.00
16/25
Structural & Multidisciplinary Optimization Group
Problem5: Linear model
 Area metric of the posterior predictive distribution of CD
Area metric
Area metric
1
0.8
0.8
0.8
0.6
0.4
0.2
0
1
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric
1
0.6
0.4
0.2
2
3
4
5
Fall time (sec)
6
7
0
1
0.6
0.4
0.2
2
0.13
3
4
Fall time (sec)
5
6
0
2
2.5
0.11
3
3.5
4
Fall time (sec)
4.5
5
0.07
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
2 clips (ref)
Sample mean of CDs
Sample STD of CDs
0.978
0.102
0.949
0.091
0.916
0.050
17/25
Structural & Multidisciplinary Optimization Group
Problem5: Linear model
 Area metric of the distribution of CD with p-box
Area metric with p-box
Area metric with p-box
1
0.8
0.8
0.8
0.6
0.4
0.2
0
1
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric with p-box
1
0.6
0.4
3
4
5
Fall time (sec)
6
7
0
1
0.4
0.2
0.2
2
0.6
2
3
4
5
Fall time (sec)
0.01
6
7
0
2
2.5
0.00
3
3.5
4
Fall time (sec)
4.5
5
0.00
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
2 clips (ref)
Sample mean of CDs
Sample STD of CDs
0.978
0.102
0.949
0.091
0.916
0.050
18/25
Structural & Multidisciplinary Optimization Group
Problem5: Linear model with one clip
 Area metric of the posterior predictive distribution of CD
Area metric
Area metric
1
0.8
0.8
0.8
0.6
0.4
0.2
0
0
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric
1
0.6
0.4
0.2
2
4
Fall time (sec)
6
8
0
2
0.6
0.4
0.2
3
0.44
4
5
Fall time (sec)
6
7
0
2
0.30
3
4
Fall time (sec)
5
6
0.14
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
1 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample mean of CDs
Sample STD of CDs
Sample STD of CDs
0.882
0.078
0.978
0.102
0.923
0.033
0.949
0.091
0.911
0.025
0.916
0.050
19/25
Structural & Multidisciplinary Optimization Group
Problem5: Linear model with one clip
 Area metric of the distribution of CD with p-box
Area metric with p-box
Area metric with p-box
1
0.8
0.8
0.8
0.6
0.4
0.2
0
0
CDF of fall time
1
CDF of fall time
CDF of fall time
Area metric with p-box
1
0.6
0.4
0.2
2
4
Fall time (sec)
6
8
0
0
0.6
0.4
0.2
2
0.13
4
Fall time (sec)
6
8
0
2
3
0.06
4
Fall time (sec)
5
6
0.02
CD from the fall time data Helicopter1 Helicopter2 Helicopter3
148.5 in
1 clips
Sample mean of CDs
148.5 in
2 clips (ref)
Sample mean of CDs
Sample STD of CDs
Sample STD of CDs
0.882
0.078
0.978
0.102
0.923
0.033
0.949
0.091
0.911
0.025
0.916
0.050
20/25
Structural & Multidisciplinary Optimization Group
Comparison between quadratic and linear models
 Area metric of the posterior predictive distribution of CD
148.5 in / 1 clip
148.5 in / 2
clips (ref)
Helicopter
1
Helicopter
2
Helicopter
3
Linear
0.19
0.34
0.38
Quadratic
0.44
0.30
0.14
Linear
0.13
0.11
0.07
Quadratic
0.15
0.11
0.07
 Area metric of the distribution of CD with p-box
148.5 in / 1 clip
148.5 in / 2
clips (ref)
Helicopter
1
Helicopter
2
Helicopter
3
Linear
0.13
0.06
0.02
Quadratic
0.01
0.11
0.22
Linear
0.01
0.00
0.00
Quadratic
0.01
0.00
0.00
21/25
Structural & Multidisciplinary Optimization Group
Concluding remarks
 Predictive validation for both quadratic and linear models
 The predictive validation for different mass is a partially success
 The predictive validation for different height is a success but the
assumption of constant CD is not clearly proven
 Comparison between models
 Cannot conclude
 Overall
 Reason for the differences in the area metric is not clear
 The effect of the manufacturing uncertainty is significant
(i.e. very different area metrics for the same test condition)
22/25
Structural & Multidisciplinary Optimization Group
Kaitlin Harris,
VVUQ Fall 2013
Comments:
•Chose to maintain uniform
distribution for calibration
parameter based on
histogram results (vs
normal)
Conclusions:
• Best models: calibrated
quadratic at both heights
and calibrated linear with 2
paper clips
• Worst models: uncalibrated linear with 2
paper clips and uncalibrated linear with 1 clip
for data set 1
Area Metric
Paper
Clips
Height
2
1
no
quadratic
0.5376
0.8397
0.5717
0.3004
0.3706
0.5299
2
1
yes
quadratic
0.143
0.442
0.19
0.305
0.267
0.0946
1
2
yes
quadratic
0.6272
0.833
0.6503
0.3927
0.593
0.5442
2
1
no
linear
0.9639
1.0202
1.0062
0.6385
0.5737
0.7388
2
1
yes
linear
0.1073
0.5003
0.2587
0.3571
0.3104
0.1398
1
1
yes
linear
1.3216
1.5398
1.3457
0.213
0.207
0.168
2
2
yes
quadratic
0.169
0.214
0.284
0.461
0.424
0.269
Calibrated?
Model
Set 1 Model 1 Set 1 Model 2 Set 1 Model 3 Set 3 Model 1 Set 3 Model 2 Set 3 Model 3
Validation of analytical model used to predict fall time for
Paper Helicopter
By Nikhil Londhe
Comparison of Analytical Cdf and Empirical Cdf
Data Set 5, H=18.832, No. of Pins=2 Helicopter 1 Helicopter 2 Helicopter 3
0.7309
0.6932
0.6169
Validation Area Before Calibration
Metric
After Calibration
0.2809
0.0906
0.142
Calibration Results
Helicopter 1 Helicopter 2 Helicopter 3
Maximum Likelihood
Estimate of Cd
Standard deviation in
Posterior pdf of Cd
0.7889
0.8965
0.9111
0.046
0.0191
0.0238
Predictive Validation for Data Set 5, No. of Pins =1
H=18.832ft.
Helicopter 1 Helicopter 2 Helicopter 3
Validation Area
Metric
0.1869
0.2453
0.192
Validation Area Metric for Linear Dependence Model
Data Set 5
Helicopter 1
Helicopter 2 Helicopter 3
Validation Area
Metric
0.4435
0.2606
0.3506
Validation of Cd is constant at different height, h=11.482ft
Data Set 5, Pins = 2 Helicopter 1
Helicopter 2 Helicopter 3
Validation Area Metric
0.4186
0.1592
0.1513
*Calibrated Analytical Model is validated to
represent experimental data
*Quadratic dependence is valid assumption between
drag force and speed
*For given difference in fall height, Cd can be assumed to be constant
Course Project: Validation of Drag Coefficient
Validation based on 1 set of data:
Quadratic Dependence
2 clips
Prior VS. Posterior Dist.
Quadratic Dependence
1 clip
Predictive Validation.
Linear Dependence
2 clips & 1clip
Prior VS. Posterior Dist.
Quadratic Dependence
2 clips
Different height.
-Yiming Zhang
Validation based on 3 set of data:
Prior Area
Metric:0.4042
Prior Area
Metric:0.4920
Post Area
Post Area
Metric:0.1695
Metric:0.2
95% Confidence
0.83±0.11
95% Confidence
0.77±0.05
Validation Area
Metric:0.1923
Validation Area
Metric:0.1267
If just use this set
to calculate
posterior.95%
Confidence
0.86±0.03
If just use this set
to calculate
posterior.95%
Confidence
0.82±0.08
2 clips:
Posterior Area
Metric:0.1729
95% Confidence
0.91±0.06
1 clip:
Validation Area
Metric:0.1960
95% Confidence
0.82 ± 0.04
 Summary:
2 clips:
Two confi. interval of
Cd don’t coincide.
Linear dependence
seems inaccurate.
Posterior Area
Metric:0.1987
95% Confidence
0.88±0.03
Validation Area
Metric:0.0970
If just use this set
to calculate
posterior.95%
Confidence
0.79±0.04
1 clip:
Validation Area
Metric: 0.1253
95% Confidence
0.84 ± 0.02
 Summary:
Seems reasonable,
but not accurate
Validation Area
Metric:0.1394
If just use this set
to calculate
posterior.95%
Confidence
0.76±0.03
Summary: (1) Quadratic dependence seems accurate using one set; Quadratic and linear dependence both don’t match well using 3 sets;
(2) SRQ is required to be fall time. If use Cd as SRQ, comparison could be more consistent and clear;
(3) 0.8 seems a reasonable estimation of Cd. This estimation would be more accurate while introducing more sets of data.
Backup Slides
Problems
 Problem1: Conservative estimate of the fall time
 Problem2: Comparing predicted variability and observed
variability using prior
 Problem3: Comparing predicted variability and observed
variability using posterior
 Problem4: Predictive validation for the same height and different
weight
 Problem5: Comparing the quadratic and linear models
 Problem6: Predictive validation for different height and the same
weight (proving the assumption of constant CD)
27/25
Structural & Multidisciplinary Optimization Group
Problem1: Conservative estimate of the fall time
 Estimating the 5th percentile of the fall time of one helicopter
 Since fall time follows a normal distribution, estimating the 5th
percentile is based on estimating the mean and standard deviation
(STD) of the fall time distribution
 The mean and STD are estimated based on 10 samples
 There is epistemic uncertainty in the estimated mean and STD due to a
finite number of samples
 To compensate the epistemic uncertainty, a conservative measure to
compensate the epistemic uncertainty is required
 Estimating the 5th percentile with 95% confidence level
28/25
Structural & Multidisciplinary Optimization Group