Download 303-1 Guidelines for the Use of Significant Figures and Rounding

STP 303-1
Standard Test
Procedures Manual
1.
Section:
STATISTICAL QUALITY CONTROL PRINCIPLES
Subject:
GUIDELINES FOR THE USE OF SIGNIFICANT
FIGURES AND ROUNDING
SCOPE
1.1.
Description of Procedure
This Procedure provides a process for the rounding of significant figures used in reporting
observed or calculated values and comparing them to the specification limits.
1.2
Application of Test
The following applies to all specified limits in the specification. For the purposes of
determining conformance with the specification limits, an observed or calculated value
shall be rounded to the nearest unit in the last right hand digit used in expressing the
specification limit. The observed or calculated value shall be rounded in accordance with
the method described in Section 3 of this Standard Test Procedure.
2.
TERMINOLOGY
2.1.
Significant Digit “n”
A significant digit is any of the figures 0 through 9, except leading zeros and some
trailing zeros, which is used with its place to denote a numerical quantity to some
desired approximation.
2.2
Use of Zeros “0”
The digit zero may either indicate a specific value or indicate place only in accordance
with the following guidelines.
2.2.1
2.2.2
2.2.3
Date: 1996 04 09
Zeros leading the first non-zero digit of a number indicate order of magnitude
only and are not significant digits. For example, the number “0.0054” has two
significant digits.
Zeros trailing the last non-zero digit for numbers represented with a decimal
point are significant digits. For example, “1270.” and “32.00” both have four
significant digits.
The significance of trailing zeros for numbers represented without the use of a
decimal point can only be identified from knowledge of the source of the value.
For example, a modulus strength of “140,000 Pa”, may have as many as 6 and as
Page 1 of 6
Standard Test Procedures Manual
Section:
STP 303-1
Subject:
STATISTICAL QUALITY CONTROL PRINCIPLES
GUIDELINES FOR THE USE OF SIGNIFICANT FIGURES
AND ROUNDING
few as 2 significant digits. To eliminate ambiguity, exponential notation may be
used. Thus “1.40 x 105 Pa” indicates that the modulus is reported to the nearest
“0.01 x 105 Pa” or “1,000 Pa”. The use of SI prefixes is recommended to reduce
the need for trailing zeros of uncertain significance. Thus, “140 kPa” and “0.140
Mpa” each indicate that the modulus is reported to the nearest “1 kPa” or “1,000
Pa” while “140 kPa” may have two or three significant digits.
3.
PROCEDURE
3.1.
Rounding Procedure
The actual rounding procedure shall be as follows:
3.1.1
3.1.2
3.1.3
3.1.4
3.1.5
3.1.6
Date: 1996 04 09
When the next digit beyond the last figure to be retained is less than five, do not
change the digit in the last place to be retained. For example, if a test result was
found to be “5.14%” and it was to be reported to the nearest “0.1%”, the “4” is
less than “5”, so the test result would be reported as “5.1%”.
When the next digit beyond the last figure to be retained is greater than five,
increase by one the digit in the last place to be retained. For example, if a test
result was found to be “5.16%” and it was to be reported to the nearest “0.1%”,
the “6” is greater than “5”, so the test result would be reported as “5.2%”.
When the next digit beyond the last figure to be retained is five, and there are no
digits beyond this five, or only zeros, increase by one the digit in the last place to
be retained if it is odd, if it is even leave the digit unchanged. For example, if a
test result was found to be “5.150%” and it was to be reported to the nearest
“0.1%”, the “1” is an odd number, so the test result would be reported as “5.2%”.
If the test result was found to be “5.250%” and it was to be reported to the nearest
“0.1%”, the “2” is an even number so the test result would be reported as “5.2%”.
When the next digit beyond the last figure to be retained is five, and there are
digits beyond this five, increase by one the digit in the last place to be retained.
For example, if a test result was found to be “5.151%” and it was to be reported to
the nearest “0.1%”, there are non-zero digits beyond the five so the test result
would be reported as “5.2%”
The rounded value should be obtained in one step by direct rounding of the most
precise value available and not in two or more successive roundings. For
example, “89,490 Pa” rounded to the nearest “1,000 Pa” is “89,000 Pa”. It is
incorrect to first round to the nearest “100 Pa” giving “89,500 Pa” then to round
to the nearest “1,000 Pa” giving “90,000 Pa”.
If the standard indicates that values are to be rounded to the nearest 50, 5, 0.5,
0.005, etc. the recorded or observed value should be doubled, then rounded off to
the nearest 100, 10, 1, 0.1, etc. in accordance with the procedures listed in Section
Page: 2 of 6
Standard Test Procedures Manual
Section:
STP 303-1
Subject:
STATISTICAL QUALITY CONTROL PRINCIPLES
GUIDELINES FOR THE USE OF SIGNIFICANT FIGURES
AND ROUNDING
3.1.1 to 3.1.5. The rounded number would then be divided by two and reported.
For example, in rounding “6,025” to the nearest “50”, “6,025” is doubled giving
“12,050”. This number becomes “12,000” when rounded to the nearest “100”
(using Section 3.1.3). When “12,000” is divided by 2, the resulting number,
“6,000” is the rounded value of “6,025”. In rounding “6,075” to the nearest “50”,
“6,075” is doubled giving “12,150” (using Section 3.1.3) which becomes
“12,200” when rounded to the nearest “100”. When “12,200” is divided by 2, the
resulting number, “6,100”, is the rounded value of “6,075”.
3.2
Recording Test Data
Unless otherwise specified in a Standard Test Procedure, the following shall apply when
recording direct measurements on a graduated cylinder, dial, or ruler. All digits known
exactly from the measurement device plus one digit, which is uncertain due to estimation,
should be recorded. For example, if a graduated cylinder is graduated in increments of 0.1
mL, then an observation would be recorded as 9.76 mL where it is observed between the
9.7 and 9.8 marks on the cylinder and estimated to be about six tenths of the way between
those marks. When the measuring device has a vernier scale, the last digit recorded shall
be the one from the vernier.
3.3
Calculations of Test Results from Test Data
When calculating a test result from test data, avoid rounding of intermediate quantities.
As far as is practical with the calculating device or form used, carry out calculations with
the test data exactly and round only the final result.
For example:
Given:
1.
2.
3.
4.
5.
Calculated Asphalt Content (ASPH)
Specification Limit for Asphalt Content
Bulk Specific Gravity of Mix (BSGM)
Bulk Specific Gravity of Aggregate (BSG)
Specification Limit for Voids in Mineral Aggregate
(VMA)
=
=
=
=
=
5.142 %
5.3% ± 0.3%
2.186
2.6359
14.5 % Minimum
The asphalt content has been calculated to be exactly 5.142 %. In order to compare this
value against the specification limit which is significant to the nearest 0.1 % (i.e. 5.3 %),
the calculated asphalt content would be rounded to the nearest 0.1 % and be reported as
5.1 % (Section 1.2 and Section 3.1.1).
Date: 1996 04 09
Page: 3 of 6
Standard Test Procedures Manual
Section:
STP 303-1
Subject:
STATISTICAL QUALITY CONTROL PRINCIPLES
GUIDELINES FOR THE USE OF SIGNIFICANT FIGURES
AND ROUNDING








BSGM
2.186



 = 21.12397906%
= 100x 1 −
VMA = 100 x 1   ASPH 

  5.142 

 x BSG 
 x2.6359 
 1 +
 1 +
100 
100 
 

 

The VMA has been exactly calculated to be 21.12397906%. In order to compare this
value against the specification limit which is significant to the nearest 0.1% (i.e.
14.5%), the calculated VMA would be rounded to the nearest 0.1% and reported as
21.1% (Section 1.2 and Section 3.1.1). The exact asphalt content value was used in the
calculation of VMA not the rounded value. This is because the calculation of Asphalt
Content is an intermediate calculation in the computation of the VMA. In accordance
with Section 3.3, the intermediate values should not be rounded, the final result is
rounded and compared to the specification limits.
3.4
Rounding and Significant Figures (No Indication of Significant Figures)
The following rules will be used to derive the number and place of significant digits to
be reported if the number and place of significant digits are not defined.
3.4.1
When adding or subtracting test data the result shall contain no significant
digits beyond the place of the last significant digit of any datum.
Examples:
3.4.2
Date: 1996 04 09
1.
11.24 + 9.3 + 6.32 = 26.9, because the number “9.3” is significant to
only one decimal place. The value “26.9” is obtained by rounding the
exact sum of “26.86” to one decimal place (Section 3.1.2).
2.
926 – 923.4 = 3, because the last significant digit of “926” is the last
digit or the nearest “1”. The value “3” is obtained by rounding the
exact difference of “2.6” to the nearest “1” (Section 3.1.2).
3.
140,000 + 91,460 = 231,000, because the number “140,000” is known
to have been recorded to the nearest thousand. The value “231,000” is
obtained by rounding the exact value of “231,460” to the nearest
thousand (Section 3.1.1).
When multiplying or dividing the result shall contain no more significant
figures than the value with the least number of significant digits.
Page: 4 of 6
Standard Test Procedures Manual
Section:
STP 303-1
Subject:
STATISTICAL QUALITY CONTROL PRINCIPLES
GUIDELINES FOR THE USE OF SIGNIFICANT FIGURES
AND ROUNDING
Examples:
1.
2.
3.4.3
11.38 x 4.3 = 49, since the factor “4.3” has two significant digits the
exact product, 48.934, is rounded and reported to two significant
digits (Section 3.1.2).
(926 − 923.4 ) = 0.6 , in this case only one figure is significant because
4.3
the numerator difference has only one significant figure (Section 3.4.1
Example 2). As a result, the exact quotient, “0.6046511628” is
rounded and reported to one significant digit (Section 3.1.1).
When using logarithms the digits of ln(x) or log10(x) are significant through
the nth place after the decimal when x has n significant digits. When using
exponents the number of significant digits of ex or 10x is equal to the place of
the last significant digit in x after the decimal.
Examples:
3.4.4
1.
ln(3.46) = 1.241, since “3.46” has 3 significant digits, the exact value,
“1.241268589”, is rounded and reported to 3 places after the decimal
(Section 3.1.1).
2.
103.46 = 2,900, since “3.46” is significant to two places after the
decimal, the exact value, “2,884.031503”, is rounded and reported to
two significant digits (Section 3.1.2).
The rule for numbers representing exact counts or mathematical constants is
that they are to be treated as having an infinite number of significant digits.
Examples:
1.
2.
Date: 1996 04 09
x
0.23
= 1−
= 0.88 , in this case “1” and “2” are exact constants
2
2
with two significant digits and “0.23” is a measured quantity with two
significant digits. There will be two significant digits in the quotient
(Section 3.4.2 Example 2) and in the difference (Section 3.4.1
Example 2). As a result, the exact value, “0.885”, is rounded and
reported to two significant digits (Section 3.1.3).
1−
A count of 50 pieces times a measured thickness of 0.124 mm is 50 x
0.124 = 6.20 mm, since the factor of 0.124 has three significant digits
Page: 5 of 6
Standard Test Procedures Manual
Section:
STP 303-1
Subject:
STATISTICAL QUALITY CONTROL PRINCIPLES
GUIDELINES FOR THE USE OF SIGNIFICANT FIGURES
AND ROUNDING
and the exact count of 50 has an infinite number of significant digits,
the exact product, “6.2”, is rounded and reported to three significant
digits (Section 3.4.2 Example 1).
3.
4.
A measurement of 1.634 inches is to be converted to millimetres. The
result is 1.634 x 25.4 = 41.50 mm, since the factor of “1.634” has four
significant digits and the constant value “25.4” has an infinite number
of significant digits, the exact product, “41.5036”, is rounded and
reported to four significant digits (Section 3.1.1 and Section 3.4.2
Example 1).
REPEATABILITY
4.1
Sources of Error
Rounding test results avoids a misleading impression of precision while preventing
loss of information due to coarse resolution. Any approach to retaining only the
necessary significant figures involves some information loss. As a result, the level
of rounding is carefully selected by considering the planned and potential uses for
the data. The number of significant digits must be adequate for comparison against
specification limits. For some purposes, such as where calculations involve
differences of measurements that are close in magnitude and in some statistical
calculations, such as Students t-tests and autocorrelations, data should be reported to
at least two more significant figures than required by the specifications limits.
5.
ADDED INFORMATION
5.1
References
ASTM E29-93a “Standard Practice for Using Significant Digits in Test Data to
Determine Acceptance with Specifications.”
Date: 1996 04 09
Page: 6 of 6