Download Uncertainty due to Finite Resolution Measurements

Uncertainty due to Finite Resolution Measurements
S.D. Phillips, B. Tolman, T.W. Estler
National Institute of Standards and Technology
Gaithersburg, MD 20899
[email protected]
Abstract
We investigate the influence of finite resolution on measurement uncertainty from a perspective
of the Guide to the Expression of Uncertainty in Measurement (GUM). Finite resolution in the
presence of Gaussian noise yields a distribution of results that strongly depends on the relative
location of the true value to the resolution increment. We show that although there is no simple
expression relating the standard deviation of the distribution to the uncertainty at a specified
level of confidence, there is an analytic relation between the mean value and the standard
deviation. We further investigate the conflict between the GUM and ISO 14253-2 regarding the
method of evaluating the standard uncertainty due to finite resolution and show that, on average,
the GUM method is superior, but still approximate.
1 Introduction
The issue of measurement uncertainty is becoming increasingly relevant to both calibration
laboratories and factory floor metrology. In some cases measurement uncertainty has significant
economic impact on the cost of a product. For example, the ISO standard 14253-1 [1] (default
condition) requires the expanded uncertainty to be subtracted from both ends of the product
tolerance yielding a smaller “conformance zone.” A measurement result must lie in this zone in
order for the manufacturer to distribute the product. Avoiding over estimation of the
measurement uncertainty can result in a larger conformance zone and hence lower product cost.
In some situations the finite resolution of a measuring instrument is a significant contributor to
the uncertainty statement. For example, the manufacturers of hand held digital calipers typically
set the resolution of the instrument such that repeated measurements of the same quantity yields
nearly the same result, within one or two units of the least count (i.e. resolution) of the
instrument. (In principle the caliper would be more accurate with an additional display digit,
however, customers perceive quality as the ability of the instrument to yield the same value for
repeated measurements.) The uncertainty budget of such hand held instruments typically
contains only a few significant influence quantities since most quantities such as the accuracy of
the calibration standards (e.g. gauge blocks) and thermal effects are usually small compared to
the instrument resolution. Therefore the inclusion (or not) of the uncertainty associated with the
finite resolution of the display can significantly affect the magnitude of the uncertainty
statement. Similar issues may arise with the finite resolution introduced by analog to digital
conversion electronics where repeated sampling of a signal differs by only one or two bits. In
this paper we first examine some general properties of the probability distribution associated
with finite resolution and then examine how it impacts the uncertainty evaluation under different
measurement scenarios.
The Guide to the Expression of Uncertainty in Measurement (GUM) [2] identifies the finite
resolution of a measuring instrument as a contributor to measurement uncertainty and suggests
that it should be evaluated by a (Type B) uniform distribution with a full width of one resolution
unit resulting in a standard uncertainty of
1
12
in units of the resolution (GUM F.2.2.1). The
GUM also suggests that this contribution is uncorrelated to other uncertainty sources and that it
should be added in a root-sum-of-squares (RSS) manner in the uncertainty statement (e.g., see
GUM H.6).
Alternatively, ISO 14253-2 [3] recommends examining the standard uncertainty of the resolution
relative to the standard deviation of repeated measurements and then including the larger of the
two in the uncertainty evaluation and discarding the lesser of the two. The rational of this
method is that the resolution is already intertwined with the standard deviation of repeated
measurements since that data is recorded with the resolution of the instrument.
We summarize these two procedures as:
Rule 1: RSS the standard uncertainty of the resolution (Type B via a uniform distribution)
with the uncertainty associated with the standard deviation of repeated
measurements in the uncertainty evaluation.
Rule2: Include the larger of the following two: the standard uncertainty of the resolution
(Type B via a uniform distribution) and the standard deviation of repeated
measurements in the uncertainty evaluation, and discard the lesser of the two.
Clearly Rule 1 and Rule 2 cannot both simultaneously be the best estimate of the uncertainty for
all measurement cases.
There are two measurement scenarios we will consider. The “special test” scenario involves
constructing an uncertainty statement for one specific measurement quantity. Typically this will
involve repeated observations of the quantity, each recorded with finite resolution. The best
estimate of the measurand is considered as the mean of the repeated observations (after all
corrections are applied) and the uncertainty statement will be associated with this mean value.
The “measurement process” scenario involves constructing an uncertainty statement that will be
applicable to a series of future measurement results. This is typical of commercial metrology
where a large number of nearly identical artifacts or workpieces are produced and only a single
measurement, having finite resolution, is recorded. In this scenario, the uncertainty statement is
developed once and then associated with each future (single observation) measurement result.
The uncertainty evaluation process will involve, among other things, repeated observations
recorded with finite resolution and used to characterize the measurement variation.
Without loss of generality we consider the case where a resolution increment has the value of
unity, and the infinite resolution value of the measurement, µ, lies between zero and one-half (all
other cases are modulo this problem). We consider the case where the measurement is corrupted
by noise from a Gaussian distribution with standard deviation σ. The situation of interest will be
for σ < 1 since large σ is equivalent to infinite resolution. Our approach will be to examine both
the special test scenario and the measurement process scenario in the limit of large sample size.
2 Probability Distributions
In the limit of large sample size, statistics can be reasonably estimated by their corresponding
expectation values. Finite resolution requires that only integer values can be observed. Hence a
continuous probability distribution function (pdf) becomes a discrete pdf that can be written:
Discrete pdf ( x) = w( µ , σ , n) δ ( x − n) n : − ∞,..., −3, −2, −1, 0,1, 2,3,..., +∞
n+
with w( µ , σ , n) =
1
2
∫ pdf (µ ,σ , x)dx = Φ(µ , σ , n +
n−
1
2
(1)
) − Φ ( µ , σ , n − 12 )
1
2
where w is a weight function, δ (x - n) is a delta function, and Φ is the cumulative probability
distribution function. Figure 1 illustrates the quantization (using equation 1) of a Gaussian
distribution with µ = 0.4 and σ = 1.
-3
-2
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
-1
1
2
3
-3
-2
-1
1
2
3
Figure 1. A Gaussian distribution with µ = 0.4 and σ = 1 is converted into a discrete distribution.
3 Descriptive Statistics: Mean and Standard Deviation
Using the discrete probability distribution, expectation values can be calculated. The expected
value of the sample mean, x , and the sample standard deviation, s, can be calculated from:
x (µ ,σ ) =
+∞
∑ w(µ ,σ , n) n
n =−∞
and s ( µ , σ ) =
+∞
∑ w(µ ,σ , n) (n − x )
2
(2)
n =−∞
Figure 2 displays plots of x and s for relevant values of µ and σ. The plot for x shows the
expected step function (due to the discrete resolution) at µ = 0.5 when σ → 0, and the expected
behavior x → µ for large σ. Similarly, the plot for s displays the expected rapid rise when µ →
0.5 and σ is small and the expected behavior s → σ for large σ.
(b)
(a)
1
0.75
0.5 __
x
0.25
0
0.8
0.4
σ
0.6
0.2
0.4
0.2
0.8
0.4
0.2
µ
0
0.6
σ
0.4
0.2
0.6
0.4
0.2
0
µ
0
Figure 2. Plots of (a) the sample mean, x , and (b) sample standard deviation, s, as a function of
µ and σ for data recorded with unit resolution.
An interesting feature of Figure 2 can be more easily seen in Figure 3 which shows the sample
mean x as a function of σ for various values of µ. For any given value of x ≤ 0.5, which is a
horizontal line on Figure 3, the possible values of µ are always greater than x . Hence the
observed sample mean is always biased toward the closest resolution increment regardless of the
value of σ (for large σ this bias becomes insignificant). For small σ, e.g. σ < 0.5, it is incorrect
to believe that uncertainty associated with the sample mean approaches zero as a result of
averaging a large sample; rather it approaches a fixed value resulting in a systematic error.
__
x
Observed Mean as a Function of µ and σ
1
µ = 0
0.8
µ = 0.1
µ = 0.2
0.6
µ = 0.3
µ = 0.4
0.4
µ = 0.5
0.2
µ = 0.6
0.1
0.2
0.3
0.4
0.5
σ
Figure 3. Plot of x vs. σ for various values of µ, note that for any particular x the possible
values of µ are always greater than x resulting in a bias of x toward the resolution unit.
Figure 4 shows the dependence of s on σ for various values of µ. As σ becomes large, s → σ,
approaching it from above. The slight over estimation of σ by s is well known and the difference
s
is called Sheppard’s correction [4]. We further point out that s ≥ σ when s > 1 regardless of
12
µ. Similarly, 1 ≥ σ when s < 1 regardless of µ; hence Max[ 1 , s] ≥ σ for all µ; a fact we
12
12
12
will make use of later.
s
0.8
s=σ
µ = 0
0.6
µ = 0.1
µ = 0.2
0.4
1
s=
12
0.2
µ = 0.3
µ = 0.4
µ = 0.5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
σ
Figure 4: Plot of the standard deviation of the finite resolution population as a function of the
standard deviation of the Gaussian noise.
What is not obvious from examining the previous plots is the interaction of x and s. In
particular, there are combinations of x and s that are forbidden. Figure 5(a) shows a plot of the
x , s space generated (numerically, with a sample size of 10,000) by allowing µ and σ to vary
over the domain 0 ≤ µ ≤ 0.5 and 0 ≤ σ ≤ 0.5, where the Z dimension is the systematic (i.e.
expected) error x − µ. As previously described, the expected error is always biased toward the
resolution unit, i.e. the bias is negative in Figure 5(a). Additionally, there is a disk with a radius
of one-half resolution unit, as described in equation (3), inside which no value of x and s can
occur. This disk is more clearly seen in the 2D plot of Figure 5(b) where only the x , s
coordinates are shown. The dense set of x , s coordinates that occur on the disk boundary are
understood by considering values of n where the equality in equation (3) holds. For the discrete
pdf given by equation (1) the equality holds when the weight functions w(0,µ,σ) + w(1,µ,σ) = 1.
That is, for any µ (0 ≤ µ ≤ 0.5) and σ such that Φ ( µ , σ ,1 12 ) − Φ ( µ , σ , − 12 ) = 1 then all of the
probability is contained in the weight functions w(0,µ,σ) and w(1,µ,σ) and this forces the
equality of equation (3) and consequently the value of x and s lie on a circular arc of radius one
half resolution unit. (Since all Gaussian distributions have some probability, albeit infinitesimal,
to extend to infinity, the disk radius is actually a limiting value.) As seen in Figure 5(a), it is
2
2
only the values where  1 − x  + s 2 =  1  that are significantly biased while other values of x , s
2

 2
away from this disk edge rapidly become an unbiased estimator of µ.
2
2
1

 1  n =+∞
1
2
2
−
+
=
x
s


  + ∑ w ( n, µ , σ ) ( n − n ) ≥  
2

 2  n =−∞
 2
s 0.4
(a)
0.6
(b)
2
∀ n, µ , σ
(3)
s
0.6
0.2
0.5
0
0
0.4
x̄ −µ
-0.2
0.3
-0.4
0.2
0
0.1
0.2
x̄
0.4
0.1
0.2
0.3
0.4
0.5
x̄
Figure 5: (a) the value of x , s, and x − µ for all combinations of 0 ≤ µ ≤ 0.5 and 0 ≤ σ ≤ 0.5; (b)
the 2D plot of the x , s coordinates shown in (a).
4 Errors and Level of Confidence
For the discrete pdf given by equation (1) with standard deviation given by equation (2) we can
explore the relationship between the standard deviation and the level of confidence, i.e. the
coverage factor. Although we are considering random Gaussian noise corrupting our
measurement result, the subsequent rounding due to finite resolution significantly changes the
level of confidence for a coverage factor of k = 2, particularly when σ < 0.5. We can gain some
insight into this effect by examining the magnitude of the error at the 95th percentile in relation to
the magnitude of the standard deviation. The magnitude of the error that occurs at the 95th
percentile is given by the cumulative probability distribution associated with equation (1) and
evaluated to determine the integer, n95%, and the corresponding error, ε95%, by increasing the
number of terms in equation (4) until the inequality just holds.
w(0, µ , σ ) + w(1, µ , σ ) + w(−1, µ , σ ) + w(2, µ , σ ) + … + w(n95% , µ , σ ) ≥ 0.95
ε 95% = n95% − µ
(4)
Clearly ε95% will depend on the particular values of µ and σ and we expect that ε95% will also
display the discreteness of the underlying discrete pdf. For example, as σ increases with µ fixed,
ε95% will remain fixed until the inequality in equation (4) is violated forcing the addition of
another term to the cumulative probability and a corresponding jump in the value of ε95%. Figure
6(a) shows the ε95% value over the region: 0 ≤ µ ≤ 0.5 and 0 ≤ σ ≤ 1. Figure 6(b) displays the
required coverage factor to achieve a 95% level of confidence; i.e. k = ε95% / s. For µ small but
nonzero and σ small, the coverage factor approaches infinity since all the measured values of x
yield zero resulting in s → 0 while x − µ remains finite. One method to avoid infinite coverage
factors is to establish a finite lower bound for the standard uncertainty; this is employed by Rules
1 and 2.
(b)
(a)
Figure 6
» ε95 % » 1.5
2
1
1
0.5
0
0
k
0.8
0.6
0.4
0.2
µ
σ
3
2
1
0
0
1
0.8
0.6
0.4
0.2
0.2
µ
0.4
σ
0.2
0.4
Figure 6: (a) The 95th percentile error (in units of resolution) as a function of the true value (µ)
and the standard deviation of the Gaussian noise (σ); (b) the corresponding coverage factor
required to include the 95th percentile error
In the special test scenario many measurements are taken on the object under consideration and
the average value (which in general will not be an integer) is computed as the best estimate of the
measurand. The uncertainty of the mean value is of interest, however the mean value has a
systemic error, i.e. the expectation value of (x − µ), that is biased toward the resolution unit for
values of x , s that lie on the forbidden disk edge (see Figure 5), the details of this bias is shown
in Figure 7(a). Also shown in Figure 7(a) is the lower bound of the expected error given by x −
½, note that the range of errors for a given x result from different µ, σ points mapping to the
same value of x , but having different systematic errors. Figure 7(b) shows the points in µ, σ
space that map to values of x , s that lie on the disk boundary; this represents a significant region
of µ, σ space and thus gives rise to the density of x , s points on the disk boundary in shown in
Figure 5(b). The bold points shown in Figure 7(b) all map very close to the point x = 0 , s = 0.
Error
(a)
-0.1
-0.2
0.1
0.2
0.3
0.4
0.5
x̄
σ
(b)
0.25
0.2
0.15
-0.3
0.1
0.05
-0.4
0.1
0.2
0.3
0.4
µ
0.5
-0.5
Figure 7: (a) the error ( x − µ) for points on the disk boundary shown in Figure 5. (b) the points
in µ, σ space that map to the disk boundary; the bold points map to x = 0 , s = 0.
It is tempting to apply a correction for the average systematic error (as a function of x ) to obtain
a better estimate of µ. However, since the bias is nearly a step function that rapidly approaches
zero away from the disk edge, applying such a correction based on the proximity of a x , s
coordinate to the disk edge is subject to misapplication due to statistical variations expected in
any finite sample size. A simpler but cruder approximation would be to use the uncorrected
value of x and consider the systematic error to be bounded between zero and the line x - 0.5
whenever s ≤ 0.6. We will consider an uncertainty rule based on this approximation for the
special test scenario.
5 Measurement Process Scenario
In the measurement process scenario an uncertainty evaluation is performed on a measurement
system and then a future (single observation) measurement result, x, which represents the best
estimate of the measurand is assigned an uncertainty based on the prior evaluation. Since x is a
single reading from an instrument having resolution R, and in this paper we take R = 1, hence x
must be an integer. One complicating factor in this scenario is that the reproducibility evaluation
is conducted on a reference artifact having some value µRef while the uncertainty statement is
relevant to a future measurement having a different (infinite resolution) value µ; and the two
values are independent. Hence all of the measurements used in the uncertainty evaluation to
determine s, (based on a large sample of measurements) will depend on the particular value of
µRef, and different values of µRef will yield different values of s; see Figure 2b. (Note that the
best estimate of the reference artifact, x R e f , is discarded since it provides no information about
the value of the future measurement result). In this paper we will examine only two values for
µRef. The case µRef = 0 is common in practice since reference artifacts used in uncertainty
evaluations usually have a value that is exactly on a unit of resolution. For example, a digital
caliper reading in units of inches that is evaluated by examining repeated measurements on
gauge blocks from an English (inch based) set. The caliper may have a resolution of 0.001 inch
and the gauge blocks will be within a few microinches of some multiple of 0.001 inch. (Note for
this condition to hold the systematic error of the instrument must be small with respect to the
resolution.) Alternatively, we examine the case where µRef is uniformly distributed between 0
and 0.5, denoted Σµ. This could occur when repeated measurements are made on several
different metric gauge blocks using an inch based caliper and the results are pooled to obtain the
standard deviation.
We consider three rules to evaluate the standard uncertainty due to finite resolution and
repeatability in the measurement process scenario given in equation (5);
2
1 
 R
Rule 1: u ( x) =  2  + s ( µ Ref ) 2
 3


1

2R

Rule 2: u ( x) = Max 
, s ( µ Ref ) 
 3



2
1  
1

R 
R



2
2
Rule 3: u ( x) = 
, s ( µ Ref )  
 +  Max 
 3  
 3
 

 


(5)
2
where R is the resolution (in this paper R = 1), and s(µRef) is the standard deviation of a large
sample of measurements on an artifact with a (infinite resolution) value µRef.
Rules 1 and 2 are based on the GUM and ISO 14253-2 respectively as previously described.
Rule 3 includes the observation that Max[ 1 , s] ≥ σ. All three rules contain two quantities, one
12
dependent on R and the other on s, and a prescription to combine them. The first quantity (with
R = 1) has the value 1 and represents the standard uncertainty associated with a uniform
12
distribution with limits ± ½ representing that the single (future) observation x could be located
anywhere in the resolution interval. The second quantity is an estimate of the underlying
population variance σ that gives rise to the observed variation in x. That is, the first quantity
accounts for the ambiguity due to the resolution and the second quantity accounts for the
reproducibility of the measurement result. Rule 1 always uses the calculated standard deviation
as an estimate for σ. However, when σ is small, s can underestimate σ, e.g. if µ = 0 and σ = 0.2
nearly all of the repeated observations will be zero and hence s ≈ 0. Rule 2 also estimates σ
using s and then selects the larger of s or the standard uncertainty of the resolution. We can
expect that by omitting one of the uncertainty sources that Rule 2 will underestimate the
uncertainty associated with the measurand. We introduce Rule 3 that estimates σ by selecting a
quantity that is an upper bound for σ, as previously noted in section 3.
6 Testing the Uncertainty Rules for the Measurement Process Scenario
We can estimate how effective these rules are in containing the reasonable values that can be
attributed to the measurand. In the measurement process scenario, the single measurement
result, x, is determined by the value of µ (which is a property of the object but is unknown),
corrupted by a random perturbation arising from σ, and rounded to unit resolution. We can
imagine that this measurement is repeated a large number of times and then ask what fraction of
the errors are contained in the uncertainty statement, where the error is ε = x - µ, and is due to the
random Gaussian perturbation and the rounding to an integer value. We will adopt this
viewpoint since in commercial metrology, it is typical to construct an expanded uncertainty using
the coverage factor k = 2 and then expect that 95 % of the potential measurement errors to be
contained within this interval.
One complicating factor in the measurement process scenario is that the probability of a future
measurement result is contained within the expanded (k = 2) uncertainty interval is a function of
three population variables: µRef, µ, and σ. Our approach will be to fix the population variables,
evaluate Uk=2, and then draw a large number of samples (each representing a future
measurement) and determine what fraction are contained within the expanded uncertainty
interval. We will then select another set of population variables and continue this methodology
until we have examined the set of population variables of interest. Figure 8 displays the
probability that a future measurement result is contained within the expanded uncertainty interval
evaluated of Rule 1 (the GUM rule) with µRef = 0 over the domain 0 ≤ µ ≤ 0.5 and 0 < σ ≤ 1. As
seen in the figure, the surface consists of a relatively flat mesa having a containment probability
near unity, interrupted by abrupt canyons of significantly lower containment probability. The
corresponding plots using Rule 2 and Rule 3 are similar, but with broader - deeper canyons, and
narrower - shallower canyons, respectively. We note that Figure 8, and the associated three
statistics in Table 1, that the containment probability only addresses the question of what fraction
of errors (for a particular µ, σ) are contained in the uncertainty interval, it does not address their
relative position within the interval. Hence the same containment probability would be assigned
to two different distributions having the same fraction of uncontained errors, despite that one
distribution might have the uncontained errors only slightly away from the uncertainty interval
and the other may have its uncontained errors grossly away from the uncertainty interval.
P H x ≤ UH R1, µ Ref =0 LL
1
1
0.9
0.8
0.8
0.6
0
0.4
0.2
µ
σ
0.2
0.4
Figure 8: A plot of the probability that the measurement error of a single observation (having
value µ and corrupted by Gaussian noise of standard deviation σ) will be contained within the
expanded uncertainty of Rule 1 evaluated with µRef = 0.
Table 1 attempts to summarize these results. Min P is the minimum average containment
probability over the domain of population variables examined. That is, over the domain of µ, σ
values considered, find the particular value of µ, σ that has the smallest containment probability
and report that value as Min P. For example, when Rule 1 is evaluated with µRef = 0 the
minimum average containment probability occurs at σ = 0.15 and µ = 0.4; that is, for this set of
values a (randomly chosen) future measurement result has only a 0.75 probability of being
contained in the expanded uncertainty interval of this rule. The Min P value tells us something
about the lowest point on the surface shown in Figure 8.
Avg P is the average probability of a measurement error containment over all values of µ, σ
considered; for the previous example of Rule 1 with µRef = 0, Avg P(x ≤ U) = 0.97, hence the
probability of error containment for most values of µ and σ is near unity.
Most metrologists expect approximately 95% of the potential measurement errors to be
contained in the expanded (k = 2) uncertainty statement. However, even for the infinite
resolution case of a Gaussian population 95% is the expected fraction of error containment,
hence any finite sample size will have slightly more or less than 95% of the deviations from the
mean contained within 2 s. As another measure of the effectiveness of an uncertainty rule we
introduce the percentage of the surface of Figure 8 that does not include at least a 94%
containment probability, this is denoted % P ≤ 0.94; the value of 0.94 was selected since, for
these sample sizes, the corresponding infinite resolution case always included at least this
fraction of the errors. Note both the % P ≤ 0.94 and Avg P(x ≤ U) depend on the size µ, σ
domain under consideration, hence they are only relative metrics that are useful when comparing
rules over the same µ, σ domain.
Table 1 Measurement Process Scenario
Rule 1 (GUM)
Min P(x ≤ U)
Avg P(x ≤ U)
% P ≤ 0.94
Avg ( U - |e95| )
Max ( U - |e95| )
Min ( U - |e95| )
µRef = 0
0.75
0.97
13 %
0.23
0.69
-0.13
µRef = Σµ
0.84
0.98
7%
0.29
0.89
-0.25
Rule 2 (ISO)
µRef = 0
0.65
0.95
30 %
0.12
0.58
-0.38
µRef = Σµ
0.74
0.97
19 %
0.16
0.76
-0.48
Rule 3
µRef = 0
0.85
0.98
10 %
0.27
0.81
-0.15
µRef = Σµ
0.91
0.98
5%
0.30
0.89
-0.14
∞ Resolution
U=2s
0.94
0.95
0.00
0.02
0.07
0.00
From Table 1 we see that none of the three (ad hoc) rules guarantee 95% error containment for
all values of µ and σ, with Rule 2 failing this criteria 30 % of the time for the µRef = 0 case. All
of the three rules benefited from computing the standard deviation of the measurement process
using many reference standards with different values and pooling the results, i.e. the µRef = Σµ
case. Indeed, for the µRef = Σµ case, Rule 1 (GUM) performance approached that of Rule 3
without the need to compare the relative size of s to the standard uncertainty of the resolution, as
required in both Rule 2 and Rule 3.
Also shown in Table 1 are three statistics that describe the difference between the expanded
uncertainty and the magnitude of the 95th percentile error, |ε95%|. This tells us something about
the amount of the over (or under) estimation of the uncertainty interval. As might be expected
for simple ad-hoc uncertainty rules, none of the three rules adapt themselves well to the
discontinuous nature of the discrete pdf, e.g. see Figure 6, and at some values of µ, σ
significantly overestimate the magnitude of |ε95%| while at other points they underestimate its
magnitude. These statistics will be of more value for the special test measurement scenario.
7 Special Test Scenario
In the special test scenario, a large sample of repeated observations is used to calculate both x
and s. Hence, in this scenario the number of observations used to calculate x is the same as the
number of observations used to calculate the standard deviation s. In the special test scenario the
GUM rule (Rule 1) combines the uncertainty of the resolution unit with the standard deviation of
the sample mean. For large samples, the uncertainty associated with the resolution unit will
clearly dominant since the standard deviation of the mean scales as 1/ N . Similarly, the ISO
rule (Rule 2) selects the larger of the two terms; hence we expect these two rules to be similar
since the uncertainty of the resolution unit dominates the standard deviation of the mean in both
cases. Consequently, it is clear that for large sample sizes both these rules overestimate the
uncertainty associated with x .
For Rule 3 we use our observation that near the forbidden disk region (rather crudely described
by s ≤ 0.6) that x is a biased estimator of µ and the value of µ is contained in the interval x ≤ µ
≤ 0.5 as seen in Figure 3. While in principle a correction should be applied to account for this
bias, in practice the typical user of a hand held instrument is unlikely to accommodate such an
inconvenience. Consequently we describe the ignorance in the location of µ by a uniform
distribution as seen in Rule 3 of equation (7). Furthermore, Rule 3 includes the observation that
Max[ 1 , s] ≥ σ; outside the disk region Rule 3 treats the uncertainty evaluation as the infinite
12
resolution case.
2
1 
2
 2 R  (s)
Rule 1: u ( x ) = 
 +
Nx
 3


1
 R
s
Rule 2: u ( x ) = Max  2 ,
Nx
 3







1


 2 R 
, s 
 Max 
2
3 
1
 


 2 R − x  


u
x
(
)
=
+
Rule 3:


Nx
3 



s
=
Nx
(7)
2
if s ≤ 0.6 R
if s > 0.6 R
8 Testing the Uncertainty Rules for the Special Test Scenario
In the case of a large sample size both the value of x and s become relatively stable and
consequently the magnitude of the systematic error | x - µ| and the uncertainty rule become
stable. Hence we can consider U - | x - µ| over a region of µ, σ values to examine if a
particular rule contains (or not) the systematic error.
Table 2 displays the information regarding the containment probability for the three rules as
applied to the special test scenario. Both Rule 1 and Rule 2 greatly overestimate the uncertainty
associated with the mean and hence always contain 100 % of the errors. Rule 3 more closely
approaches the 95% error containment interpretation with a minimum containment probability of
0.94 and an average (over the µ,σ values considered) of 0.98.
To consider the magnitude of the overestimation we examine the expected values of the
difference between the expanded uncertainty and |ε95%|. As previously suggested, Rule1 and
Rule 2 behave similarly and both typically overestimate |ε95%| by one-half a resolution unit. Rule
3 does significantly better by more closely matching the expanded uncertainty to the |ε95%| value,
with an average overestimation of 0.15 resolution units.
Table 2: Special Test Scenario
Rule 1 (GUM)
Rule 2 (ISO)
Min P(x ≤ U)
Avg P(x ≤ U)
% P ≤ 0.94
Avg U - |e95|
Max U - |e95|
Min U - |e95|
1.0
1.0
0
0.51
0.58
0.18
1.0
1.0
0
0.50
0.58
0.18
Rule 3
0.94
0.98
1%
0.15
0.58
-0.01
In the special test scenario with a large sample size, Rule 1 and Rule 2 become identical with
1
−x
1
2
u( x ) →
, and Rule 3 yields u ( x ) →
if s < 0.6 and u ( x ) → 0 if s ≥ 06. Elster [5]
12
3
has examined the general solution for the special test scenerio using Bayesian calculations. He
also considered the special case of data with an equal number of ones and zeros (hence x = 0.5
and s = 0.25), and showed for large samples that the best estimate of µ was the mean x = ½ and
1
.
u( x ) → 0; this result is given by Rule 3 but not by Rule 1 or Rule 2 which yield u ( x ) →
12
9 Conclusions
We have examined the statistics associated with data recorded with finite resolution in the limit
2
of large sample size. Due to finite resolution, there exists a disk of defined by  1 − x  + s 2 =  1 
2

2
 2
inside which the values of x , s are forbidden. Furthermore, values of x that occur on the
boundary of the disk are always (except for the µ = 0 and µ = 0.5 case) systematically biased
toward the resolution unit. Away from the disk boundary x rapidly becomes an unbiased
estimator of µ.
We have also considered various simple ad-hoc rules for evaluating the uncertainty due to finite
resolution including those recommended for the by the GUM and ISO 14253-2. For the
measurement process scenario and using the criteria that the (k = 2) expanded uncertainty should
include approximately 95 % of the potential errors, we found that the ISO 14253-2 rule
underestimates the uncertainty for roughly 30 % of the µ, σ values, while the GUM rule and
Rule 3 do significantly better. Additionally, we pointed out that evaluating the standard
deviation of repeated measurements using several different reference artifacts, with values that
are not increments of the resolution unit, improves the containment probability all of the
uncertainty rules examined.
For the special test scenario, where a large number of measurements are available to compute the
mean value and the standard deviation, both the GUM and ISO 14253-2 rules greatly
overestimate the uncertainty of the mean for large sample sizes. This overestimation results
from setting a lower limit for the uncertainty equal to 1 regardless of sample size or the value
12
of the standard deviation. Rule 3 more closely matches the expanded uncertainty to the 95th
percentile error while still maintaining a 95% error containment probability.
10 References
1 International Standard, "Geometrical Product Specifications (GPS) - Inspection by measurement of workpieces
and measuring instruments -- Part 1: Decision rules for providing conformance or non-conformance with
specification," ISO 14253-1 (1998).
2 International Organization for Standardization, "Guide to the Expression of Uncertainty in Measurement,"
Geneva, Switzerland, 1993. (corrected and reprinted 1995). This document is also available as a U.S. National
Standard: NCSL Z540-2-1997.
3 International Standard, "Geometrical Product Specifications (GPS) - Inspection by measurement of workpieces
and measuring instruments -- Part 2: Guide to the Estimation of Uncertainty in GPS Measurement in Calibration of
Measuring Equipment and in Product Verification,” ISO 14253-2 (1999).
4 Stuart, A. and Ord, J.K., Kendall’s Advanced Theory of Statistics: Distribution Theory, 6th edition, New York:
Oxford University Press, 1998.
5 Elster C., Evaluation of Measurement Uncertainty in the Presence of Combined Random and Analogue to Digital
Conversion Errors, Measurement Science Technology 11, 1359-1363, (2000).