Download Experimental Errors and Uncertainty: An Introduction Prepared for

Experimental Errors and Uncertainty:
An Introduction
Prepared for students in AE 3051
by J. M. Seitzman
adapted from material made available
by J. Craig
JMS090800-1
AE3051
The Measurement System
• Measurements
– Direct/Indirect comparison (rulers, balance scale, interferometer
– Calibrated system (odometer, spring scale, pressure gage)
• Measurement System
Physical
Measurand
DetectorSensor
Transducer
Signal
Conditioner
Indicator,
Recorder,
Controller,
Computer,
etc.
• Issues:
– Detector/Sensor: device which detects and responds to measurand
– Transducer: converts measurand to an analog more easily
measured (force-displacement-resistance-voltage)
– Signal Cond.: amplify, filter, integrate, differentiate, convert freq. to
voltage, etc.
– Computer: widely used today
JMS090800-2
AE3051
Example of Measurement System
readout
readout
transducer
detector
detector
transducer
Tire Pressure Gage
•
•
Bourdon Type Pressure Gage
These are simple mechanical systems
Issues
–
–
–
–
JMS090800-3
mechanical vs electrical output
analog vs digital
calibration
accuracy, precision, resolution, sensitivity, linearity, drift, backlash?
AE3051
Computer Readout Systems
Multichannel Data Acquisition (sequential sampling)
Volt
sample
Chan #1
Voltages
MUX
S/H
ADC
digital words
to computer
Chan #2
Computer
Control
•
•
Each channel is read in sequence (10 ms to 1 ms per reading).
ADC may output a reading that is from 8 to 16 bits in size.
skew
Readings are not
recorded simultaneously
– 8 bits & bipolar range yields readings from –128 to +127 (28=256) corresponding
to –Full Scale and + Full Scale
– 16 bits & bipolar range yields reading from –32768 to +32767 (216=65,536) so
resolution is about 4½ digits or about 0.01% of Full Scale (not of reading)
– With ±1v range, an 8 bit ADC has a 7.9 mv resoultion (=1/127)
– With ±10v range, a 16 bit ADC has a 0.31 mv resolution (=10/32767)
Key:
MUX = multiplexer (switch)
S/H = sample & hold (hold voltage while ADC reads)
ADC = analog to digital converter (voltmeter)
JMS090800-4
t
AE3051
Experimental Error
• Error: all measurements have some uncertainty
error = ε = xmeas − xexact
• Objectives
1. Minimize error so that uncertainty, u, is:
-u ≤ ε ≤ +u at N:1 certainty (a statistical confidence)
or
xmeas − u ≤ xexact ≤ xmeas + u
2. Estimate error to
determine reliability, meaningfulness of data
∆
uexact
JMS090800-5
ε
umeas
AE3051
Accuracy and Precision
• Accuracy: also called systematic or bias error
– denotes something repeatably “wrong” with the
measurement or experiment
• Precision: also called random error or noise
– denotes errors that change randomly each time you
try to repeat experiment
Good Accuracy
Good Precision
JMS090800-6
Good Precision
Poor Accuracy
(can calibrate)
Good Accuracy
Poor Precision
(can average)
Poor Accuracy
Poor Precision
AE3051
Other Related Terms
• Resolution
– Smallest increment of change in a system or
property that a measurement device can reliably
capture
• Sensitivity
– Change in a measurement device’s output for a
unit change in the measured (input) quantity, e.g.,
volts/Torr for the Baratron
• Dynamic Range
– Maximum output of a measurement device
divided by its resolution
JMS090800-7
AE3051
Accuracy/Systematic Errors
• Sources
– Measuring system errors
• difference between model of measuring system
and reality
• could be corrected, e.g., with better model of
measurement
– Measured system “errors”
• influence of uncontrolled or unaccounted for
variables in the experiment
• the measured data may be “correct”, but may lead
to an incorrect model of the object/process being
studied
JMS090800-8
AE3051
Some Systematic Errors of Measuring Systems
•u=u(x)
backlash
u
Model
u
u(x=const)
Actual
Actual
Model
Model
Actual
hysteresis
x
Nonzero offset - Background
x
Backlash & Hysteresis
u
Systematic errors can be
eliminated/removed if
they are known
Model
time
Drift (e.g., offset changing in
systematic way with time)
u
Model
Actual
Nonlinearity
x
JMS090800-9
Quantization Error
(digitized data –
impacts resolution)
Actual
x
AE3051
Some Random Errors of Measuring Systems
u
u
u(x=const)
Actual
Model
Model
time
Background “Noise” (offset
changing randomly with time)
0
Actual
x
x
Detector “Noise” (random
change in sensitivity of device)
Disturbances – “Noise”
(e.g., pickup electrical
signals from other sources
After data acquired, nearly
impossible to separate random error
(noise) sources
JMS090800-10
AE3051
Uncertainty: Statistical Approaches
• Probability & statistics provide a way to deal with uncertainty
– we will cover only a VERY limited introduction
precision (random) error
Bias (systematic) error
total error
Frequency
Bias error: systematic errors that could be
removed by proper calibration
theoretical distribution
number of times
reading falls in
this range of x
Precision error: random error - not directly
controllable
xm
Xexact
Bias:
calibration
consistent human error
background
JMS090800-11
Xavg Xm
Precision:
disturbances
noise
variable conditions
Blunders:
human error
software!
Either:
backlash, hysteresis, friction
damping
drift
variations in test procedure
AE3051
Basic Concepts in Probability
• Sample Space: all possible events or outcomes of an
experiment; also can be referred to as a population.
• Event: subset of sample space
• Sample: finite number taken from population (e.g., 15
turbine blades taken from 8600 produced for XX-300 engine)
– sample must be randomly selected from population
– sample may or may not be returned to population before
resampling
• Probability: likelihood of an event (measured as % of
successful events in sample space or population). We can
often analytically compute this likelihood based on
mathematics applied to the population. 0 ≤ P(event) ≤ 1.
• There are many rules for computing probabilities for different
kinds of events; these are the subject of courses and books
on probability theory.
JMS090800-12
AE3051
Numbers on one die
1
2
•
3 4 5
Number
6
Sum of numbers on 2 dice
6
1
2
3
4
5
6 7 8 9 10 11 12
Number (sum of dice)
Heads in 100
coin tosses
Number of Events
1
Number of Events
Number of Events
Examples
0
50
Number
100
These happen to be binomial distributions (repeat experiment N
times; each trial is independent of others; each trial is
successful or not; probability of success, p, is same for each
trial).
 p x ( 1 − p ) N − x
P( x successes in N trials ) =  N
x
N!
 N  =
 x  x! ( N − x )!
•
JMS090800-13
These are discrete distributions but there are also continuous
distributions that are defined by CDF and PDF curves (next)
AE3051
Properties of Probability Distribution Functions
1.0
PDF
0.6
CDF
Area = 0.6
f(x)
x
x
x1
CDF = Cumulative Distribution Function = P(x ≤ x1)
x1
PDF = Probability Distribution Function = f(x )
P( x ≤ x1 ) = 0.6
also
P( x ≤ ∞) = 1.0
P( x ≤ −∞) = 0
d
(CDF )
dx
CDF = f ( x ) dx
f(x)=
or
∫
So P( x ≤ x1 ) =
x1
∫ f ( x ) dx
−∞
and P( x1 ≤ x ≤ x2 ) =
x2
∫ f ( x ) dx
x1
JMS090800-14
AE3051
Normal/Gaussian Probability Distribution
PDF(x)
1
σ 2π
µ
x
 (x − µ )2 
1
f ( x) =
exp −
2σ 2 
σ 2π

µ = mean
σ2 = variance
σ = standard deviation
• Uses:
– When N→∞ for a binomial
distribution (e.g., for very
large populations)
– For events that are made up
from many independent
events each with any kind of
distribution
– For properties of samples
when # samples is very
large (e.g., sample means)
– others…
You
You can
can use
use aa
spreadsheet!
spreadsheet!
JMS090800-15
AE3051
Normal Distributions - Confidence Levels
One Sigma:
Two Sigma:
Pr ob ( µ − σ ≤ x ≤ µ + σ ) =
µ +σ
+1
−
−1
f (ξ ) dξ = ∫ f ( z )dz = 0.683
∫
µ σ
Pr ob ( µ − 2σ ≤ x ≤ µ + 2σ ) =
µ + 2σ
+2
−2
−2
f (ξ )dξ = ∫ f ( z )dz = 0.954
∫
µ σ
Three Sigma: Pr ob ( µ − 3σ ≤ x ≤ µ + 3σ ) = 0.997
Error Level Name
Error Level
Probable Error
One Sigma
90% error
“Two” Sigma
Three Sigma
Maximum Error
Four Sigma
Six Sigma
± 0.67 σ
±σ
± 1.65 σ
± 1.96 σ
±3σ
± 3.29 σ
±4σ
±6σ
JMS090800-16
Prob. that
Error is
Smaller
50%
68%
90%
95%
99.7%
99.9
99.994%
99.9999999%
Prob. that Error
is Larger
1:2
~ 1:3
1:10
1:20
1:370
1:1000
1:16000
1:1.01e9
Six Sigma is used for
many electronic
manufacturing processes
AE3051
Sample Statistics
• What if µ and σ are unknown (as is often the case)?
– use estimates from measurements,x and s
Mean Square
N
x
Sample mean = x = ∑ i
i =1 N
N
Sample variance = s = ∑
2
i =1
Square of mean
 N 2

x
i
 N
( xi − x )2  ∑
2
i =1
=
−x 
N −1
 N
 N −1




– use (N-1) to compute average in s2 because we have N
independent xi but if we also know xmean then we only
need to know (N-1) xi to compute last remaining xi !
⇒We call (N-1) the “degrees of freedom” for this calculation
JMS090800-17
AE3051
Central Limit Theorem
• One of the most important statistical theorems
• Basis for most statistical methods commonly applied to
measurements
• Example: Suppose we measure a pressure 100 times and
the avg. (mean) is 75 psi. Repeat test with 100 more p
measurements and get 78 psi. Repeat many times (N).
– Question: what is distribution for mean of average
values?
σ
– Answer: Gaussian (normal) with σ x =
N
where σ is std dev of actual
distribution of variable x.
• This is also the distribution for any random variable that is
the result of many independent random variables - no
matter what the underlying distributions may be
JMS090800-18
AE3051
Application: Confidence Intervals
• Question: If one takes N readings and computes the
sample average, how confident can you be that the
average is really close to the true mean (µ)?
• Confidence intervals are way to describe this; we know
that the probability distribution of the sample mean for
many different samples (N large) will be normal.
Prob = c% that µ lies in shaded area defined by
Example : c = 95%,
x = µ ± ac σ x
ac = 1.96, x = µ ± 1.96 σ x
Answer 1 : 95% of x readings fall in µ ± 1.96 σ x
µ − acσ x
µ
µ + acσ x
Answer 2 : With 95% confidence, the true µ
will fall within x ± 1.96 σ x
JMS090800-19
AE3051
Application: Confidence Intervals - cont’d (1)
• Since σ x = σ N where σ = true population standard
deviation, we can write:
Answer 3: 95% confidence limits : x ± 1.96 σ
N
• For large sample sizes (large N), we can approximate
σ with sx so that we have:
Answer 4 : 95% confidence limits : x ± 1.96 sx
N
EXAMPLE
Find 95% confidence limits for x = 75 psi when Sx=8.3 psi for N = 50 samples.
Answer : limits = x ± 1.96 S x
8.3
N = 75 ±1.96
= 75 ± 2.3 psi
50
Find 99.9% confidence limits for previous case (use previous table):
Answer : limits = x ± 3.29 S x
JMS090800-20
N = 75 ± 3.29
8.3
= 75 ± 3.9 psi
50
AE3051
Application: Confidence Intervals - cont’d (2)
EXAMPLE (cont’d)
How many N are required to assume that mean is in x ± 5% with 95% confidence?
8.3
= 75 ± (75 × 0.05) psi
N
or : N = 18.8 ≈ 19 samples
Answer : limits = 75 ± 1.96
• Remarks
– Above works only for N=large, or when σ is known
– When N=small, then we cannot approximate σ by sx (due to N-1 in
the denominator), and must treat σ as unknown
• This leads to replacing the normal distribution with the
“Student T distribution”, a subject beyond this introduction
JMS090800-21
AE3051
Propagation of Uncertainty
•
•
Many times experimental results are the result of several independent
measurements combined using a theoretical formula. For mass flowrate
p
through a pipe: m& = ρuA =
uπD 2
RT
How do random and bias uncertainties in each variable contribute to
whole?
For the case where y=y(x1, x2, … xN) is a linear function, a statistical theorem states that:
2
2
 ∂y
 ∂y

  ∂y
σ xN
σ x2  + L
σ x1  + 
σ y = 
x
x
∂
x
∂
∂
 1

  2
 N

2 1 / 2

 
 
For uncertainties, ui, that are small compared to xi we can use a Taylor Series expansion in ui:
y( x1 + u1 , x2 + u2 ,... x N + u N ) = y( x1 , x2 ,...x N ) +
∂y
∂y
∂y
u1 +
u2 + ...
uN
∂x N
∂x1
∂x2
Now, y is a linear function of the uncertainties and we can use the first equation to yield:
 ∂y   ∂y 
 ∂y
 
u y = 
u1  + 
u2  + L
u N  
 ∂x1   ∂x2 
 ∂x N
 
2
JMS090800-22
2
2
1/ 2
AE3051
Examples of Uncertainty Propagation
• Suppose the output is an additive combination:
Then
y = Ax1 + Bx2
∂y
∂y
= A and
=B
∂x1
∂x2
1/ 2
and
2
 ∂y 2  ∂y
 
u y = 
u x1  + 
ux2  
 ∂x1   ∂x2  
1/ 2
2
2
=  A2ux1 + B 2ux2 
m
• Suppose that the output is a multiplicative combination:
m −1
Then
and
n
∂y
mx1 x2
m x1 x2
y
=A
= A
=m
k
k
∂x1
x1
x1
x3
x3
n
m
n
∂y
y
∂y
y
=n
and
= −k
x2
x3
∂x2
∂x2
2
2
u y  u x1   u x2   ux3
=  m  +  n
 +  −k
y  x1   x2  
x3

JMS090800-23
x x
y = A 1 k2
x3


 
1/ 2
AE3051
Bias and Precision Uncertainties
•
•
•
•
•
We noted earlier that errors will include both bias (systematic) and
precision (random) components
These can usually be treated as independent and therefore the
uncertainties for each can be combined into a total:
[
]
2 1/ 2
u y −total = u y − precision + u y −bias
Note: if they are not independent, other combining in other ways may
be necessary.
SINGLE SAMPLE: if we are considering a single measurement and
calculation of the dependent result (y), we must assume N=1 so that:
uy
σ
y = y ± ac
= y ± ac
= y ± ac u y
N
1
This affects only the precision error which is purely random so:
2
u y − single sample = ac u y = 1.96 u y
[
u y −total − single sample = (ac u y − precision ) + u y −bias
•
2
]
2 1/ 2
Note: above based on 95% confidence interval (1.96 σ), ac=1.96
JMS090800-24
AE3051
Example #1: Uncertainty Calculation
•
L
Consider determining the
mass flowrate through a
round pipe
p Lπ
p π
v D =
D
m& = ρvA =
RT ∆t 4
RT 4
2
•
D
2
The fractional uncertainty in m
& can then be developed
using the previous formula for multiplicative combinations
to be:
u  u   u   u   u   u  
=   +   +   +   +  2  
m&  p   T   L   ∆t   D  
2
m&
JMS090800-25
p
2
T
2
L
2
t
2
1/ 2
D
AE3051
Example #1: Uncertainty Calculation - cont’d
Assume the following uncertainties (ux-precision= sx)
Variable
Bias (ux/x)
Precision (ux/x)
p
0.55%
0.1%
T
0.55%
0.4%
∆t
0.01%
2%
L
0.1%

D
1%

Summed (Σu2)1/2
1.3%
2.0%
Single-Sample
1.3%
4.0%
Notes
High precision pressure transducer, with
7.5-bit digitizer
Medium precision temp. transducer, with
7.5-bit digitizer
Accurate clock, but starting/stopping
uncertainty of 0.01 sec in 0.5 sec
measurement
Only measured once with ruler having
maximum 0.5 mm reading error over 0.5 m
pipe length
Only measured once with with ruler having
maximum 0.5 mm reading error over 50
mm diameter
∆t meas. dominates precision error
D meas. dominates bias error
Precision error dominates
Total uncertainty in single measurement of m& is then:
um
2
2 1/ 2
= u precision + ubias  = 4.2%
m&
&
JMS090800-26
AE3051
Example #2: Uncertainty Calculation
•
Strain gage
Consider a windtunnel drag
transducer made by attaching a
strain gage at the root of a
cantilever beam as shown. A
tip force, P, will produce a
bending strain, εx, as shown.
b
h
L
εx =
•
•
P
Assume the electrical output of
the strain gage circuit is
e=Kεxe0 where e0 is the
excitation voltage and K is the
calib. factor to be determined.
σx
M y
6 P L
= b
=
E
EI
b h2 E
e
e b h2 E
K=
=
ε e0 e0 6 P L
The uncertainty in K is then:
u K  ue   ue0   ub   uh   u E   u P   u L 
=   +   +   +  2  +   +   +  
K  e   e0   b   h   E   P   L 

2
JMS090800-27
2
2
2
2
2
2



1/ 2
AE3051
Example #2: Uncertainty Calculation - cont’d
Assume the following uncertainties:
Variable
e
e0
b
h
E
P
L
Uncertainty
Single sample
Bias (ux/x)
1%
0.5%
0.5%
0.5%
2%
1%
Precision (ux/x)
0.2%
0
0
0
0
2%
0.2
2.65%
2.65%
0
2.01%
3.94%
Notes
Digital DVM yields good precision
Only single initial measurement made
same
same
same
Bias reflects calibration errors while
Precision is random error in readings
Only single initial measurement made
Total uncertainty in the measurement of K is then:
[
]
uK
2
2 1/ 2
= u precision + ubias
= 4.75%
K
JMS090800-28
AE3051
Wrap-up on Uncertainty Analysis
•
•
Uncertainty analysis should always be a part of the design of any
experiment
– Avoid measurement processes that lead to greater uncertainty
– Estimate the uncertainties in all measured and computed results
Consider numerical precision in computational results (since even the
best calculator doesn’t have infinite precision…)
– Example: avoid taking differences between very large numbers
– Example: s = ∑ ( x − x ) =  1 ∑ x − x  N
2
N
2
i
i =1
•
N −1
N
N

i =1
2
i
2
 N −1

Difference
Difference between
between
large
large numbers
numbers
Better
Better
BE CONSISTENT:
– If the uncertainty in a computed result is 0.1%, then DO NOT include
digits beyond 0.1% of the value!
– Example: if computed result is 13,451.8793 with uncertainty of 0.1%,
the number provided in the report should be: 13340.
JMS090800-29
AE3051