Download erickson_projectThesis_041311

DESIGN OF A PARAMETRIC OUTLIER
DETECTION SYSTEM
By
Ronald J. Erickson
B.S., Colorado Technical University, 2001
A project thesis submitted to the Graduate Faculty of the
University of Colorado at Colorado Springs
In partial fulfillment for the degree
Masters of Engineering
Department of Computer Science
2011
©Copyright By Ronald J. Erickson 2011
All Rights Reserved
ii
This Project Thesis for Masters of Engineering Degree by
Ronald J. Erickson
has been approved for the
Department of Computer Science
by
Dr. C.H. Edward Chow
Dr. Xiaobo Zhou
Dr. Chuan Yue
______________________________
Date
iii
This Project Thesis for Masters of Engineering Degree by
Ronald J. Erickson
has been approved for the
Department of Computer Science
by
Dr. C.H. Edward Chow
Dr. Xiaobo Zhou
Dr. Chuan Yue
______________________________
Date
iii
Erickson, Ronald J. (M.E., Software Engineering)
Design of a Parametric Outlier Detection System
Project Thesis Directed by Professor C.H. Edward Chow
Integrated circuit defects are only discovered during the electrical test of
the integrated circuit. Electrical testing can be performed at the wafer level,
package level or both. This type of testing produces a data-log that contains
device specific electrical characteristic data known as parametric data. The
testing of the integrated circuit has become increasing expensive in an ultra
competitive market. The traditional method of setting upper and lower limits to
discover non-compliant circuit defects has become only a portion of the criteria
required to discover potential defects as the geometries of these integrated
circuits continue to decrease.
Integrated circuit manufacturers are now increasingly employing additional
methods of failure detection by adding test structures on the wafer. Other
techniques employ high percentage failure mode screening on these integrated
circuits to discover device marginality. However, marginality testing can only be
accomplished after extensive characterization of the circuit on numerous wafer
lots over standard process corners to establish device trends. Another technique
includes a design for manufacturing structure that detects a single parametric
measurement that is potentially out of family when compared to the device
simulation. All of these standard types of failure detection work well with an
environment that builds many wafer lots and thousands of a single circuit.
iv
Commercial circuit manufacturers will typically lower the FIT “failures in time” rate
requirements of their specific integrated circuit. This is especially true when the
integrated circuits of the potential amount of returned merchandise are lower
than the costs of more rigorous electrical testing. Currently there is no universally
industry adopted way to detect out of family circuit parametrics within a rigorous
FIT requirement. In addition, these circuits have an end use that is used in a
mission critical environment and where the product quantities are small.
This thesis paper and accompanying project is an attempt to address the
opportunities and challenges of creating a more rigorous parametric outlier
detection system to detect out of family measurements on embedded circuits
with very high FIT requirements, very small sample sizes, used in mission critical
devices.
v
For my family Crystal, Chelsea, and Isabel thank you for your support
To my parents Gerry and Janice for always compelling me to continue my
education
vi
Acknowledgements
Anyone who has ever gone back to a university to obtain a high level
degree knows the effort that is involved in the creation of a project of this
magnitude. This project thesis was no exception. While I have many people to
thank for helping me throughout this project, I would like to pay special mention
to a number of individuals.
First, I would like to sincerely thank Professor C.H. Edward Chow.
Dr. Chow you are an excellent instructor and I really enjoyed the time you have
taken mentoring me throughout the tedious graduate school process. Your time
is very valuable, and I thank you for sharing a portion of that time over the last
couple semesters. This project thesis would not have been successful without
your direct guidance.
To the members of my project thesis committee, Dr. Xiaobo Zhou and Dr.
Chuan Yue, thank you for your feedback, support, and of course your
encouragement throughout this process.
And finally to Mr. Troy Noble: Thank you for being a great motivator and
intuitive in asking and answering my questions. In addition, thank you for the
personal mentoring in electrical engineering and software development.
vi
vii
TABLE OF CONTENTS
Chapter
I.
INTRODUCTION....………………..……………………………………1
Background…………...…………………………………………2
General Circuit Manufacturing Techniques …………………7
Design for Manufacturing Architectures..……………………8
Motivation……………...…………….…………………………10
Approach Overview ………………………….……………….12
II.
SOFTWARE ENGINEERING.……………………………………….13
Software Engineering Basic Stages………………….……..14
Capability Maturity Model Integration………………………..16
Software Engineering Advanced Stages…………………...20
III.
STATISTICAL ANALYSIS……………………………………………23
Goodness of Fit...……………………………………….……..25
The Null Hypothesis…...….…………………………………..27
Anderson Darling……………………………………….……..28
IV.
SYSTEM DESIGN & ARCHITECTURE......………………………..30
Database………………………………………………………..31
Automatic Data Extraction…………………………………….32
Automatic Statistical Modeling ...…………………………….33
Testing and Evaluation of Prototype......…….……………...36
Analysis of Results………………........…….………………...38
viii
V.
CONCLUSIONS.............................................…….………………..44
Lessons Learned......…….…………………………………...46
BIBLOGRAPHY..........................................…………….…...……………..50
APPENDICIES
A. Data Set Parametric Outlier Application Dataset 1………..…..53
Normal Distribution Test Case
B. Data Set Parametric Outlier Application Dataset 2………..…..54
Normal Distribution Test Case
C. Data Set Parametric Outlier Application Dataset 3………..…..55
Normal Distribution Test Case
D. Data Set Parametric Outlier Application Dataset 4………..…..56
Bimodal Distribution Test Case
E. Data Set Parametric Outlier Application Dataset 5………..…..57
Bimodal Distribution Test Case
F. Data Set Parametric Outlier Application Dataset 6………..…..58
Normal Distribution with Negatives Test Case
G. Data Set Parametric Outlier Application Dataset 7………..…..59
Extreme Outlier with Multiple Ties Test Case
H. Data Set Parametric Outlier Application Dataset 8………..…..60
Multiple Ties Normal Distribution Test Case
I. Data Set Performance Graded Binder Sample 195……………61
Large Data-Set Non-Normal Distribution Test Case
J. Data Set Performance Graded Binder Sample 196……………62
Large Data-Set Non-Normal Distribution Test Case
ix
LIST OF TABLES
Table
1. Critical Level Table……………..….……..………………………………..…28
x
LIST OF FIGURES
Figure
1. Wafer Dice with Street Intact …….……..……………………..……………...3
2. Wafer Dice with Street Sawn…….……..……………………………………...4
3. Astronaut Repairing the Hubble Telescope…………………………………10
4. Software Engineering Basic Stages………………. …….…….…………...13
5. Software Engineering Advanced Stages………... …..….…….…………...21
6. Normal Distribution Curve………………………………….…….…………...34
7. Bimodal Distribution Curve………..……………………….…….…………...36
8. Cluster Algorithm Flowchart……....……………………….…….…………...41
xi
CHAPTER 1
INTRODUCTION
Integrated circuits are designed and simulated by designers and then sent
off to a wafer fabrication to be built on wafers. Unfortunately wafer fabrication
manufacturing and design defects are only discovered during the electrical test of
the integrated circuit. This testing of the circuit electrically can be performed at
the wafer level, package level or both. Electrical testing produces a data-log that
contains specific device characteristic data known as parametric data. Testing of
the integrated circuit has become increasing expensive in an ultra competitive
market and circuit manufactures are exploring new methods of discovering
failures.
Traditionally circuit manufactures would use a method that would set an
upper and lower limit on a parametric measurement to detect circuit failures. This
methodology has become only a portion of the criteria that is now required to
discover potential defects of small geometry integrated circuits. Integrated circuit
manufacturers are now employing additional methods of failure detection
including wafer test structures and specialized BIST (built in self test) circuitry.
After extensive characterization of the circuit on many devices, a high percentage
failure mode trending on these devices can be used to detect lower parametric
marginality. Design for manufacturing simulations is limited in that they only
employ detecting a limited parametric measurement to the original design
simulation.
2
All of these failure detection schemes work well with environments that
produce thousands and thousands of the single integrated products. Most
commercial non-mission critical integrated circuit companies will typically lower
the FIT rate requirements of their specific integrated circuits. This is because the
end users of the product are within a commercial market. Another reason
commercial manufactures can lower the FIT requirements is because the
potential returned merchandise costs are usually lower than the costs of more
rigorous testing of the integrated circuit. There currently is a need to create a
more rigorous, trusted automated parametric outlier detection system on
embedded circuits with special requirements. These special requirements include
high FIT requirements, small sample sizes, and high reliability on life and death
critical systems. These systems can be airport scanners, avionics circuitry,
military and satellite circuitry.
Background
As global markets become increasingly competitive some companies are
utilizing a fab-less wafer manufacturing system. These companies design,
assemble, and test the integrated circuits applications like aerospace, highaltitude avionics, medical, networking and other standard consumer electronics in
their facility. Then pay to have the wafers manufactured at an outside facility. In
this type of business model the company does not own a wafer fabrication
facility.
3
To stay price competitive companies must use many wafer manufacturing
facilities throughout the world, depending on the design geometries of the
transistor structures being manufactured within the integrated circuit design.
Wafer Dice with Street Intact
To mitigate a portion of the risk using differing wafer fabs, most companies
have begun inserting standardized transistor cell structures in the streets of the
wafer. (Marinissen, E. Vermeulen, B., Madge, R. Kessler, M., and Muller, M,
2003) The wafers streets are the areas between the die and is illustrated above.
However to save silicon the streets are the area of where the saw cuts the wafer
producing individual die. These test structures are only available for testing while
the wafer is intact at wafer probe. The saw processes destroys these test
structures and are no longer available for testing after sawing, see illustration
below.
4
Wafer Dice with Street Sawn
High reliability products are electrically tested multiple times prior to
shipment to ensure compliance to the device specification. In some cases this
testing is tailored to the specific industry for which they are designed (Marinissen,
E., Vermeulen, B., Madge, R. Kessler, M., and Muller, M, 2003). Companies
must consistently monitor large amounts of test data at pre-defined electrical test
steps during the manufacturing process to ensure compliance (Perez, R.,
Lilkendey, J., and Koh, S.1994). Contained within this test data are device
specific electrical parametric readings that are composed of DC and AC electrical
parameters pertaining to the individual device performance.
5
To maintain a high yielding product as the geometries continue to
decrease, the need to monitor device parametric data has become an
increasingly important tool in discovering discrete product defects (Moran, D.,
Dooling, D., Wilkins, T., Williams, R., and Ditlow, G 1999).
Naturally the business model of the fab-less microelectronic company
introduces even more risk to the development of quality wafers than a company
that owns their own fab and can tailor the process to their needs. The wafer fabs
write contracts on less restrictive wafer parametric acceptance. A less restrictive
wafer acceptance criterion is geared to increase fab starts and deliverables on a
is more positive for the fab. This philosophy leads to the potential of large
deviations in material parametrics that must be screened for out of family
measurements. Some consumer products companies have resorted to only
electrical testing at the wafer level. (Bhattachatya, S., and Chatterjee, A 2005).
Generally because it cheaper to replace the device than it is to electrically test
the device at package level. Most commercial manufacturers have been willing to
trade reliability for economics. Designers of high reliability systems cannot accept
this trade-off for the following reasons. Mission critical systems require high
reliability because replacement of faulty parts is difficult or impossible after
deployment, or component failures may compromise national security.
Furthermore, typical service life tends to be longer for military systems and
aerospace applications, so high reliability and mission critical devices must be
tested at higher level at package to ensure compliance.
6
Wafer fab minor deviations within the doping of the transistor can create
very different wafer structure parametric data (Singh, A., Mani, M., and
Orshansky, M 2005) (Veelenturf, K 2000). These types of deviations can be
easily normalized and thus predicted if there are many parametric wafer
samples. Deviations are much harder to predict on small lot quantities that can
have months or years between lot manufacturing.
Radiation tolerant circuits typically are only built in small quantities and
sold after a lengthy well documented qualification process. Some companies
have developed a manufacturing process to produce a range of specific
radiation-hard electronic products using rad-hard by design techniques or by
adding rad-hard doped trenches. These processes are somewhat different from
the ones used in commercial foundries because they include a modified process
that produces a circuit with greater radiation tolerance. The processes are
generally developed between the manufacturing fab and the fab-less circuit
design facility. Commercial foundries typically use standard material for all
material manufactured in the facility. By contrast radiation tolerant circuits require
nonstandard materials, incorporating epitaxial layers, insulating substrates;
doped trenches in the silicon that enhance radiation tolerance. The use of nonstandard materials makes out of family detection much more difficult due to the
inherent parametric leakage these techniques introduce. This is where better
predictability to detect outliers without many samples on non-standard processes
is driving the requirement to build a statistically based robust detection tool.
7
General Circuit Manufacturing Techniques
All integrated circuit defects are discovered through the electrical test of the
integrated circuit at the wafer level. The general flow of high reliability devices is
to re-test at the packaged device level to detect assembly, environmental and
accelerated life-time infant mortality failures.
A portion of these defects that affect integrated circuits throughout the
process of manufacture at a wafer fabrication facility, through the assembly,
environmental aspects of tests are listed below

Wafer fabrication defects
o transistor doping, oxide breakdown, interconnect and poly stringers
o opens, shorts, resistive metals, bridging materials
o metallization stability issues, dissimilar metals like AL & CU.

Potential device assembly defects
o wafer back-grind, wafer saw, die preparation
o device package defects including trace length, & integrity, package
leads
o solder ball integrity, die attach, wire bond, hermetic lid seal

Environmental defects on the package and die are caused by simulating
o temperature cycle, temperature shock, mechanical vibration
o centrifuge, high humidity, salt environment
o accelerated and dynamic electrical burn-in causing infant mortality
8
o

total ionizing dose (TID), gamma radiation effects on gate oxides
As a minimum the digital circuit DC parametric tests performed at
electrical test are
o Continuity, power shorts, input leakage, output tri-state leakage
o quiescent power, active power, input and output thresholds
o minimum voltage operation

However, these digital circuit measurements usually include the AC
electrical parameters
o data propagation, input set-up, output/input hold
o input rise, input fall, clock duty cycle
Design for Manufacturing Architectures
All high reliability IC manufacturers must electrically test their product at
the wafer level to ensure compliance to the product specification and save
package and assembly costs. Design for manufacturing techniques (DfM) imply
that device parametric data can potentially discover the mismatches between the
manufacturing process of the circuit and the circuit designer simulation within test
structures in street locations of a die reticle (Gattiker, A 2008) These parametric
defect techniques only detect limited parameters of quiescent current device
sensitivity and are limited to a very specific device function (Wang, W., and
Orshansky, M 2006). Test structures capable of testing interconnect circuit
dependencies, or the race conditions that can occur between circuit blocks
9
MARYL (Manufacturability Assessment and Rapid Yield Learning)
minimizes the risk of starting high volume manufacturing in a waferfab. Process
engineers consistently study the relationships between the process parameters
within the company owned wafer fabrication facility. Then modify the process and
evaluate the electrical parameters of the integrated circuits and predict a final
yield of the ongoing process (Veelenturf, K 2000). Some other DfM tools utilize
the circuit from a many building blocks approach. This type of DfM tool takes the
knowledge of smaller building block circuits in an attempt to solve a part of a
larger problem through simulations. This type of simulation on a real world
integrated circuit is based upon the performance or execution time of a particular
product. To include a specific product line in a specific wafer fab and is only
utilized as a simulation and not real silicon. (Moran, D., Dooling, D., Wilkins, T.,
Williams, R., and Ditlow, G 1999).
Standard DfM techniques work well within the commercial market because
these designs do not contain rad-hard non-standard materials or manufacturing
requirements.
Essentially
a
mature
process
wafer
test
structure
will
deterministically behave as predicted without the rad-hard circuit requirements.
However within a small fab-less business model the wafer manufacturer will
assume a no-liability stance on non-standard material due to the complexity and
variance from one wafer lot to another. Add in the fact that manufacturing these
wafers is generally more risky to yield significant die.
10
Motivation
A small fab-less rad-hard circuit company needs a tool that can process
relatively small amounts of individual device data based on small lots of the
electrical characteristic data. This small amount of data can be skewed by the
implied marginality effects of a Commercial Radiation Hardened (CRH) process
within the transistor structure on a wafer to wafer variability. These specialized
circuits require nonstandard starting materials, incorporating epitaxial layers,
insulating substrates; doped trenches in the silicon that enhance radiation
tolerance, but introduce marginality into the circuits with increased sub threshold
leakage effects. To compound the issue, this data being analyzed within these
small lots must be used to guarantee a very high FIT rate. On highly reliable
products that provide service within a mission critical sub system.
Astronaut Repairing the Hubble Telescope
11
To complicate matters if these devices are placed in service on a system
that is in a high altitude or a space application? It is becomes unbelievably
expensive to send an astronaut to replace a small faulty integrated circuit or
subsystem in space. For instance the 2009 repair of the Hubble space telescope
was valued at 1.1 billion dollars. It is highly visible when a space system fails, the
publicity involved in a failure of this magnitude, and the potential government
fines can easily drive a small company into bankruptcy and out of business.
Typically rad-hard device lot quantities are small and the product offering
are very broad to cover simple logic devices, to memories of DRAM, Flash, and
SRAM. Additional devices types could be protocols of spacewire, European
CAN, multiple clock generator chips, or single and multiple processors. All of
these products could be built of differing technologies from 0.600µm to 90n
meters by differing wafer fabrication facilities throughout the world.
Radiation tolerant methodologies increase the sub-threshold leakage of
the integrated circuit by design. These additional leakage paths provide pathway
to slough off excessive charge that could potentially damage the gate oxide of
the circuit transistors. These structures are required when experiencing an event
of total ionizing dose, single effect upset, and single effect latch-up on the
integrated circuit. The need to analyze the validity of the parametric data being
captured in real time during the testing and manufacture of these circuits is of
great importance.
12
Approach Overview
This automatic parametric outlier detection system prototype will utilize an
existing algorithm named Anderson-Darling to test for normal distribution on the
dataset. This test is a general normality test designed to detect all departures
from normality within a dataset, and is believed to be the most powerful test for
normality. The importance of checking the data-set for normality is very important
or the consequential methodology for choosing outliers may produce incorrect
results. If a dataset is deemed a normal Gaussian distribution by the automated
Anderson Darling test The prototype will perform an inner 75% quartile outlier
detection technique on the dataset and present the user with a histogram
distribution of the data, and the Anderson-Darling calculated value. The lower
12.5% calculated value, and the upper 87.5% value are also displayed for
comparison to the data-set for outlier identification. If a dataset is deemed nonnormal by Anderson Darling testing, the dataset will be presented in a histogram
format requiring user intervention. The user will be required to visually inspect the
data and set the upper and lower fences for a normal distribution. Once the user
has placed the upper and lower fences on the data distribution the remaining
steps for normal and non-normal user intervention datasets will be the same. The
prototype will perform an inner 75% quartile outlier detection technique. Create
the histogram, Anderson-Darling non-normal calculated value, the lower 12.5%
value, and the upper 87.5% value are then displayed for comparison to the dataset for outlier identification.
CHAPTER 2
SOFTWARE ENGINEERING
Software engineering is dedicated to the design, implementation, and
maintenance of software code, software applications, and software systems.
Software engineering integrates significant mathematics, computer science and
general best engineering practices. High quality, more affordable, easy to
maintain software is generated with a methodic approach to the key stages of the
software. This chapter defines the basics of creating software engineering
procedures and methodologies that produce reliable operating software.
Software Engineering Basic Stages
14
Software Engineering Basic Stages
The basic stages within a software engineering life-cycle are the software
requirements, design, construction, testing, and maintenance.
A software requirement is defined as a property that must be exhibited in
the software product being created that will solve a real-world problem.
Requirements documents identify the project problem that the software will solve.
This is completed through the elicitation of stakeholders, documenting these
requirements, and analyzing the requirements by the people and organizations
that are affected by the software. When an agreement on each of the
requirements, and priorities for these requirements are set on the software to be
created the project can move to the next stage; design.
Software design is the process of defining the software architecture, the
user and system interfaces. It also defines other characteristics of a system and
any component that affects the software product to the requirements that were
documented in the requirements stage. The use of tools similar to UML unified
modeling language produces class diagrams or other upper level coding
representation diagrams. These diagrams describe the static design of the
software objects and the relationships these objects have with to each other in
the software application. These diagrams are generally the basis used to
describe a software design, and complete the design stage.
15
Software construction refers to the methodical detailed creation of
operating, defect free, user friendly software through the combination of writing
and debugging the code. The construction follows the design foundation built
upon the class diagrams or representation diagrams defined in the design stage.
The construction of the unit tests that ensure individual units of code are fit for
use in the system. In addition to the construction of the integration tests that will
verify all the source code with the final system are built in the construction stage.
Software testing consists of the dynamic validation of the requirements
documentation while operating in the system environment. Generally the quality
of the software under test performs the process of executing the program or
application with the intent of finding software defects or errors. The software
verification ensures the software operates correctly. Unit testing ensures
individual units of code are fit for use in the system. Integration testing will verify
the source code operates within the final system environment.
Software maintenance is the most expensive stage in the software life
cycle as it covers the operational lifetime through the eventual retirement of the
software. When the software is delivered or placed in operation and errors are
discovered a software engineer must fix the issues for continued use. But errors
are not the only activity associated with software maintenance. In addition
operating environments can change, operating platforms can change, or new
requirements and functionality may be necessary? (Abrain, A. 2004)
16
Capability Maturity Model Integration (CMMI)
Capability Maturity Model Integration is a measure of the software process
maturity framework within an organization. The CMMI assess the process
maturity of software engineering company by setting specific benchmarks within
the software creation process. The CMMI contains five descriptive benchmark
levels to measure a company’s ability to produce quality software, these levels
are described below:
Maturity Level 1 Initial
Maturity Level 2 Managed
Maturity Level 3 Defined
Maturity Level 4 Quantitatively Managed
Maturity Level 5 Optimizing
Maturity Level 1: The initial level is characterized by processes and projects
that are usually unplanned from project management perspective. These projects
and processes are very informal, subject to last minute changes, and based on
improvised poorly documented requirements that eventually lead to project scope
issues. Companies and the projects that software engineer to this maturity level
are usually small start-up companies that tend to take short cuts when schedules
are due. The companies are also prone to over commit, over promise, underdeliver, and don’t have repeatability on past successes with new projects.
17
Maturity levels above level 1 have specific goals and requirements for the
software engineering process. These goals and requirements must be met for a
company to be certified at a higher level. As these levels progress higher from a
level 1 to level 5 implies the higher level contains the requirements of the lower
levels and the requirements of the higher level
Maturity Level 2: The managed level has requirements that are documented
and managed within the project. All processes have controlled documented
planning stages and are measured at major milestones. The status of these
processes and the software products are also reviewed at major milestones and
the completion of major tasks. Requirements management identifies the problem
the project will solve through the elicitation of stakeholders, documenting
requirements, and analyzing requirements. These requirements are then agreed
upon and priorities set through content management. Project planning identifies
the scope of the project; estimates the work involved, and creates a project
schedule based on the scope and work estimates. Project monitoring and control
is handled through the use of a program manager that will hold regular meetings
with project staff. This program manager will capture meeting notes and update
the schedule to current work completed. Then communicate these project results
to management and customers to manage expectations. Supplier agreement
management handles all project subsystems, hardware, software, manuals,
documentation, parts and materials that will be used on the project.
18
Measurement and analysis involves gathering quantitative data about the
software project and analyzing that data to influence project plans and effort
expended on the project. Process and product quality assurance involve formal
audits by a separate quality assurance individuals or groups. Technical peer
reviews are required at different levels within the project, and in-depth reviews of
all work performed. Configuration management outlines how the project team will
control and monitor changes to the project code as the project progresses are
required for a company to maintain maturity level 2.
Maturity Level 3: Now defined processes are understood and characterized
these processes are described in documented standards, procedures, and tools.
Processes performed across the organization are consistent except for the
specific variances to the standard guidelines that contained within a process
tailoring document. Additional requirements for level three from the previous level
include additional requirements analysis. Additional technical solutions that will
design, develop, and implement solutions to requirements document.
Project verification that ensures the software operates correctly and validation
that ensures that the correct product was built per the requirements
documentation. The addition of an organizational process improvement with a
focus on initiating process improvements on processes and process assets.
Organizational process definitions that establish and maintain a usable set of
process assets and work environment organizational standards, and then
completes training to these standards
19
Integrated project management that tracks the activities related to
maintaining the project plan, monitoring plan progress, managing commitments,
and taking corrective actions. Risk management that will assign risk items to the
project before and during the project as they are recognized. The project team
will have to work to mitigate or remove risk altogether as the project progresses.
Integrated teaming involves the teaming of software engineering to system
engineering. The integrated supplier management within the project is related to
management reporting and management. Decision analysis and resolution are
comprised of decisions that may move the project timeline, could have a major
impact on system performance, or that could have legal implications.
Organizational environment for integration will integrate product and process
development. Product integration will assemble the product and ensure that the
product functions properly when integrated in the system for maturity level three.
Maturity Level 4: The addition of quantitatively managed objectives of
quality and process performance. These objectives are then used as the criteria
in managing processes. Quality and process performance is controlled using
statistical and other quantitative techniques, and is now quantitatively predictable
based on these techniques. The organizational process performance views how
well the processes are performing, these performance matrices can be
parametrics of effort, cycle time, and defect rate. Quantitative project
management is the project’s defined process and the ability to statistically
achieve the project’s established quality and process-performance objectives.
20
Maturity Level 5: Finally optimizes these processes by continually
improving on quantitative understanding of processes variation causes. Through
both incremental and innovative technological improvements level 5 constantly
tries to improve process performance. Organizational innovation and deployment
innovative and incremental improvements that improve the software processes
and technologies of the company. Causal analysis and resolution that trace
defects to the root cause and then modifies the processes at fault to reduce or
eliminate those defects. (CMMI, 2010)
Software Engineering Advanced Stages
Software configuration management identifies the configuration of
software at distinct points during the software engineering lifecycle. With only one
purpose to control and track changes to the software configuration and
completed through the systematic and methodical use of version control software
tools. The use of configuration management will maintain the integrity and
traceability of the software configuration for the life cycle including benchmarks.
Software engineering management is the project management and
measurement of software at various stages with the software lifecycle by
monitoring plan progress, managing commitments, and taking corrective actions.
21
Software Engineering Advanced Stages
Software engineering process defines the implementation, assessment,
measurement, management, modification, and eventual improvement of all the
processes used in software engineering.
Software engineering tools and methods are the use of tools that provide
the assistance to the software developer in the development of software project.
The purpose of the software engineering tools and methods are to decrease the
software development time, increase the quality, reduce errors and software
defects during the construction and maintenance stages of a software lifecycle.
22
Software quality identifies the software quality considerations which occur
during the software life cycle. Through formal audits by quality assurance group
of the process and product, these audits provide a quality measure of how well
the software performs and conforms to the original design and requirements. It
also manages and controls the content management systems, then defines the
rules for the use of these systems.
Knowledge areas of related disciplines include computer engineering that
is a merge of electrical engineering and computer science. Project management
plans, organizes, secures, and manages all project resources in an effort to
successfully complete a software project. Computer science is the study and
implementation of techniques to perform automated actions on computer and
server systems. Quality management seeks to improve product quality and
performance that will meet or exceed requirements documentation for the
software product. Management software ergonomics deals with the safety and
compliance of the software systems to rules and regulations within the company,
state, and environment. Mathematics is the learning or studying of patterns of
numbers, counting, measurement, and calculations of quantity, space, and
change. Systems engineering is effort required to integrate the software to the
system platform that it will run. (Abrain, A. 2004) A combination of all of these
disciplines is required on every software engineering project. It is these skill set
requirements that are important to the success of any software engineering
project but usually require the resources of many experts.
CHAPTER 3
STATISTICAL ANALYSIS
An outlier is defined as an observation point that is inconsistent with other
observations in the data set. A data point within the data-set, an outlier has a
lower probability that it originated from the same statistical distribution as the
other observations in the data set. An extreme outlier is an observation that will
have an even low probability of occurrence than the rest of the data. Outliers can
provide useful information about the process or the designed product. An outlier
can be created by a shift in the mean of the data-set or the variability of
parameters in the process. Outliers provide a numerical gauge of measure into
the quality, yield, measurement errors, and process deviations of a product.
Though an observation in a particular sample might be a candidate as an
outlier, the process might have shifted and caused this outlier observation.
Sometimes the erroneous result is a gross recording error or an un-calibrated
measurement error. Measurement systems should be shown to be capable for
the process they measure to correct significant digits and calibrated for accuracy.
Outliers also occur with incorrect specifications that were based on the wrong
distributional assumptions at the time the specifications are generated. Incorrect
specifications can also occur if the sample size used was not large enough, or
from diverse samples of the process at differing times of manufacture.
24
Once an observation is identified by means of mathematical or graphical
inspection for potential outliers, root cause analysis should begin to determine
whether an assignable cause can be found for the outlier result. If the root cause
cannot be determined, and a retest cannot be justified due to equipment issues,
the potential outlier should be recorded for future reference and quality control
standards. These recorded outliers are then used for process improvement at
future time within the process flow. Outlier processing of a data-set can
potentially introduce risk in the evaluation if the assumption that the data-set
comes from a normal distribution; when actually the data-set is from a nonnormal distribution. To ensure that errors are not introduced into the outlier
analysis, the importance of testing the distribution for normality in the distribution
before performing outlier analysis is critical. Because the characteristic property
of the normal distribution is that 68% of all of its observations fall within a range
of ±1 standard deviation from the mean, and a range of ±2 standard deviations
includes 95% of the data points. If the outlier calculations are not calculated on
normal distributions the mathematical properties and assumptions made begin to
deteriorate and add error to the results. Outlier calculations using the standard
deviation, quartile, or CPK analysis only show the variation of the data-set from
the mean by usually ignoring the tails of the data-set and can misleading. Small
data-sets present the unique problem that they essentially can be easily viewed
by the user quickly as appearing to not be from a normal distribution. Only after
additional analysis using tools on the distribution can these data-sets be proven
to be from or not from a normal distribution.
25
This project utilizes the Anderson-Darling test for normality; and was
developed by two mathematicians with statistical backgrounds, Theodore
Anderson and Donald Darling in 1952. When used to test if a normal distribution
adequately describes a set of data, it is one of the more powerful statistical tools
for detecting most non normality distributions. The equation can also be used as
an estimation of the parameter for the basis of a form of minimum distance
calculation, but was not used for this purpose in this project. Statistically it is an
evidence test performed on the data set to give confidence it from non normal
distribution. The given probability distribution then identifies the probability for
each value of a random discrete variable within a given data set. Then provides a
single p-value for quick evaluation of the normalcy contained with a data-set
Goodness of Fit
The most commonly used quantitative goodness-of-fit techniques for
normality within a distribution are the Shapiro-Wilks, and the Anderson-Darling
tests. The Anderson-Darling is commonly the most popular and widely used
because it the accuracy of the test. Because of this fact the original requirements
defined that Anderson-Darling would be used exclusively on this project. The
Anderson Darling test is merely a modification of the Kolmogorov-Smirnov test
but gives more weight to the tails.
26
The Kolmogorov-Smirnov test is distribution free because the critical
values do not depend on the specific distribution being tested; this is the
difference with the Anderson Darling test which makes use of the specific
distribution
of
the
data
set
in
calculating
the
critical
values.
Because of these facts the Anderson Darling creates a more sensitive normality
test but the critical values must be calculated for each distribution. (Romeu J
2003)
Both Shapiro-Wilks and the Anderson-Darling tests use a confidence level
to evaluate the data set for normality, this value is referred to as a p-value or it is
also known as an alpha level. A lower p-value limit correlates to a higher value of
confidence and proves greater evidence that the data did not come from the
selected distribution and therefore is not a normalized distribution. This double
negative of not rejecting the null hypothesis is somewhat counter intuitive. Simply
put the Anderson Darling test only allows the data reviewer to have a greater
level of confidence as the p-value increases that no significant departure from
normality was found. A unique explanation or an interpretation of the null
hypothesis is below.
Just as a dry sidewalk is evidence that it didn't rain, a wet sidewalk might
be caused by rain or by the sprinkler system. So a wet sidewalk can't
prove that it rained, while a not-wet one is evidence that it did not rain.1
1Annis,C.StatisticalEngineering
[email protected], 2009
27
The Null Hypothesis
The definition of the null hypotheses is: H0: that the data-set follows a
specified distribution. This hypothesis regarding the distributional form of the
data-set is rejected at the chosen (α) significance level and if the AndersonDarling A2 test statistic is greater than the critical value obtained from the critical
value table below. The fixed values of α (0.01, 0.05, 00.25, 0.10.) are generally
only used to evaluate the null hypothesis (H0) at various significance levels. The
(α) significance value or sometimes called the p-value of 0.05 is typically used for
most applications, however, in some critical instances; a lower (α) value may be
applied. This thesis project used the (α) significance value limit of 0.05 as the
limit of p-value comparison to the application output when checking a data-set for
normal distribution. It is this value that is outputted by the application that is used
for the basis of user involvement on distributions that do not have a p-value
greater than 0.05.The critical values of the Anderson-Darling test statistic depend
on the specific distribution. The tables of critical values for many of the
distributions are not normally published, the exception are the most widely used
ones like normal, log-normal, and Weibull distributions. The Anderson-Darling
quantitative goodness-of-fit techniques for normality tests are typically used to
test if the data is from a normal, log-normal, Weibull, exponential, or from logistic
distributions. The assumption is made that data-sets tested against this
application for normalcy fall into the category of published critical value
distributions.
28
Due to the vast differing data distributions that can be analyzed by an
Anderson-Darling test, it was chosen as the best generic test for normality for this
thesis project application. The published confidence levels for normal, lognormal, Weibull distributions are published below in the critical values table. The
confidence levels are calculated from the p-value in the form of 100*(1 - pvalue).
Confidence Level
α-level
Critical Value A2
99.0%
0.01
0.631
95.0%
0.05
0.752
92.5%
0.025
0.873
90.0%
0.10
1.035
Critical Level Table
Anderson–Darling
Because the Anderson-Darling statistic belongs to the class of quadratic
empirical distribution that are based on the empirical distribution function
statistical model. Essentially a step function that jumps for 1/n at each of the n
data points, The Anderson Darling test statistic A, and the A2 equations are
shown below these for the normal, lognormal, and Weibull distributions.
29
A  n 
2i  1
ln F Yi   ln 1  F Yn1i 
n
n is the sample size of a continuous probability distribution. The equation
is well suited for smaller sample data set size calculations. The result is
Anderson Darling test statistic A;
 0.75 2.25 
A 2  A 2 1 +
+ 2 
n
n 

The A2 value calculated is then compared to an appropriate critical value
from the table showing confidence level, α-value, and critical value of the null
hypothesis from the critical value table shown above. In summary for simplicity of
this thesis project the Anderson-Darling test for a normal distribution proves the
data is from a normal distribution if the p –value > α-value and if A2 -value <
Critical-Value from the critical value table shown above. In the real world the
need to make decisions and then get on to other business matters is incredibly
important. The normal distribution function using Anderson-Darling will produce
quick estimates that are useful for such decisions. This is true even if the dataset is small and appears to be non-normal. It can still be assumed a normal
distribution if the histogram curve looks non-normal, providing the p-value of the
goodness-of-fit statistic shows the distribution is a normal Gaussian bell curve p
> 0.05
.
29
CHAPTER 3
SYSTEM DESIGN & ARCHITECTURE
The requirements documents of this project was to take a distribution of data,
analyze the data for a normal Gaussian bell curve distribution, perform outlier
analysis by some means and then indentify potential outliers within that data set.
The data will then be presented to user as a histogram and outliers are shown. If
the data-set did not come from a normal distribution then present the data-set in
a histogram format so the user can set the lower and upper fences of the dataset before applying 75% percentile analysis.
This project investigated the design of a prototype design that will utilize
multiple existing algorithms like an Anderson-Darling, statistical based outlier
analysis in a user friendly automated system. The goal of this design was to
combine various existing outlier detection techniques and make necessary
improvements so that the small company outlier detection system used on small
data-sets would operate similar to other elaborate expensive architectures in
making better business decisions. Specific software requirements included the
platform base code that would be C# used to as the prototype test vehicle. This
requirement was changed in the project due to unforeseen small company
infrastructure issues after an agreement with stakeholders was completed. This
requirement change added a significant amount of coding that had to be
converted to the C# from the original base code.
31
Database
This project did will not focus on the database design, however the
automated system inputs and future capability learning criteria was an original
requirement that accessed a database. SQL server a widely-used and supported
database can be utilized in this post project design relatively easily because of
the hooks built-in to the .NET framework and the C# code. The basis of
connection and implementation is enabled through the use of the SqlConnection
class in the C# program to initiate a connection to the SQL database containing
parametric data. The SqlConnection class is used in a resource acquisition
statement in the C# language and must have the Open method called; this allows
the query of the database with SqlCommand’s. The usage of this SqlConnection
in a using type resource acquisition statement for use in Sql for the C# language.
The SqlConnection has a constructor that requires a string reference pointing to
the connection string character data. This connection string is often autogenerated by the dialogs in Visual Studio that the automated system must
include in the SqlConnection code before the use of the SqlCommand object to
perform a database query. By using the using statement construct in the C# it
will lead to a more reliable, stabile C# database project. This application could
not gain access to the database in a remote laptop atmosphere due to potential
ITAR International Traffic in Arms regulations. Therefore, for demonstration
purposes the project data-sets were hard coded into varying arrays and tested
for functionality against test cases documented in a cited paper appendices.
32
Automatic Data Extraction
The automatic data extraction tool can overlap any parametric data-set to
complete the viewing of any data-set from more than one viewpoint. The
parametric historical data based on wafer manufactures like ON Semiconductor
wafer fab 0.6um products or TSMC Semiconductor wafer fab 0.25um products
are easily implemented within the parametric selections that are called and
analyzed for outlier detection. For instance, the tool can view parametric
historical data based on the PIC product identification code. 0.25um products like
SRAM synchronous random access memory that statically does not need to be
periodically refreshed. And DRAM dynamic random access memory that stores
each bit of data in a separate capacitor within an integrated circuit. Both of these
memories store the data in a distinctly different way and therefore have some
differing data based on the architecture of the integrated circuit. However some
parametric values will correlate between both memories built at the same wafer
foundry, and can used to detect parametric trending within the wafer fab. This
trending could be as simple as lower resistivity metal lines that exhibit faster
access timing ac’s between both devices. These trends can be used to validate
transistor design models, and help designers match the design simulations to
real world silicon. This application will provide an important statistical data
loopback to the original design models for verification. All data-sets were hand
generated or based on cited works due to potential ITAR regulations. Real device
parametric data was not used for this thesis project.
33
Automatic Statistical Modeling
Automated data modeling and goodness-of-fit tests are contained within the
normalcy test of Anderson-Darling. The Shapiro-Wilk test is a slight modification
of the Anderson Darling test but not as powerful, the Skewness-Kurtosis tests are
better predicting larger data-sets and therefore were not chose over the
functionality of the Anderson Darling. The only test implemented in this project
was Anderson-Darling test for the null hypothesis. Through this test the user can
gain a greater confidence that the data-set being analyzed is from a normal
distribution and will provide greater confidence in to the outlier analysis being
performed will be correct. This project implemented a 75% percentile analysis of
the normalized data identified by the Anderson-Darling test described throughout
this document as the basic default operation. The implementation followed a
screening for the bottom 12.5% of the data or visually the left side of the bell
curve. Then the tool screens for outliers on the upper 12.5% beginning at 87.5%
by calculating these values. The histogram is then configured to show the
distribution with the new lower and upper fences set. The toolset has other
statistical modeling abilities including standard deviation, inter quartile analysis
as options for the user to modify if needed but the default automated operation is
the inner 75% percentile analysis.
34
.
Normal Distribution Curve
If the data-set is determined by Anderson-Darling analysis to be bimodal or a
non-normal distribution, it will be shown in a histogram format requiring user
intervention. Bimodal distributions are a continuous probability distribution with
two different modes; these modes appear as distinct peaks in the distribution
diagram. The user is required to pick the upper and lower fence of the data to be
analyzed for 75% percentile analysis, in the case of a bimodal distribution the
user will have to pick one of the distinct peaks, or remove the entire population.
This is the only non-automated portion of the project that requires user
intervention by setting distribution of the data-set. Once the toolset knows what
data in the data-set needs be analyzed for outliers the toolset can continue to
perform the modeling on the data-set to locate potential outliers
35
Future enhancements to the outlier detection system for non-normal data sets
will utilize machine learning. All user defined changes to data-sets will be
captured into the database. These user defined changes will specifically capture
the user, the parametric data type, data-set, test equipment, lower, and upper
fence chose by the user. Then based on the decisions made by engineers on the
automated non-normal distributions, a future learning algorithm can start to be
developed. However this project and tool-set is launched as a beta version
prototype model for evaluation. Based on the evaluation of the beta version
additional requirements may be realized and required in the released version.
The small company requirements for this toolset deem the immediate release of
a beta version for wafer test quiescent sub threshold leakage current
measurements for out of family devices. (Gama, J., Rodrigues, P., and
Sebastiao, R 2003)
The effect of this software development methodology will be realized in the
maintenance stage of this design. A well documented fact that the maintenance
stage is the longest and the most expensive to the software project. Every
attempt will have to be made to understand the costs involved with future
enhancements and requirements. Releasing an incomplete beta version of the
software for user evaluation, changes in scope, the need for future requirements
and enhancements will certainly be a possibility as the known problem domain is
still probably unknown?
36
Bimodal Distribution Curve
Testing and Prototype Evaluation
The outlier detection system design implementation utilized small author
generated data-sets as the basis for the test cases used on the automated
system implemented in C#. The exception of two data-sets described in the
Asphalt Study cases 195 and 196. (Holsinger, R., Fisher, A., and Spellerberg, P
2005) These data-sets were listed in the document cited within paper as
Precision Estimates for AASHTO Test Method T308 and the Test Methods for
Performance-Graded Asphalt Binder in AASHTO Specification M320. For
specific testing of the Anderson-Darling implementation, there are extensively
documented test cases in the appendices of this document. The use of a nonautomated Excel spread sheet with Anderson-Darling test statistics A and A2
calculations for the Anderson-Darling Normality Test Calculator. (Otto, K.2005)
37
Identified data-sets are random representations of real data-sets but still are
limited to utilize only a specific single device parameter within these randomly
created data-sets, with the exception of the asphalt study case 195 and 196.
These data-sets were not pre-loaded into a database, but instead hard coded
into the application for ease of use and testing criterion for initial prototype
correct functionality. Outlier analysis is then performed on a default inner 75%
percentile analysis. The inner 75% percentile involves indentifying data values
that fall with the first 12.5% of the data and the last 12.5% of the data from the
mean as outliers with the normal distribution. The application can also filter for
detecting outliers and extreme values based on interquartile ranges.
These equations are in the form Q1 = 25% quartile, Q3 = 75% quartile,
IQR = Interquartile Range difference between Q1 and Q3
OF = Outlier Factor
EVF = Extreme Value Factor
Mild value detection:
Outliers: Q3 + OF*IQR < x <= Q3 + EVF*IQR
Outliers Q1 - EVF*IQR <= x < Q1 - OF*IQR
Extreme value detection:
x > Q3 + EVF*IQR
and
x < Q1 - EVF*IQR
38
Also presented in a histogram format of the standard sigma deviation from
the mean, the user can pick any number of sigma or standard deviations from the
mean data of the histogram. The application also presents the data in an all data
within the data-set pass histogram format. This mode is used if the user needs to
see a histogram format of all data points within the data-set. The all data format
histogram is presented to the user with the ability to manually set the fences for
upper and lower limits. This functionality allows the user to manually pick the
outliers based on an upper and lower fence choice. It can also manipulate any
data-set outlier to the user requirements and will be tracked and saved in the
data-base as described in section Automatic Statistical Modeling section of this
paper to eventually become part of a machine learning update version.
Analysis of Results
The Anderson-Darling test for normality are one of three general normality
tests designed to detect all departures from normality, and is sometimes touted
as the most powerful test available. The Anderson-Darling test makes use of the
specific distribution in calculating critical values, because the critical values do
depend on the specific distribution being tested. This has the advantage of
allowing a more sensitive test and the disadvantage that critical values must be
calculated for each distribution or data-set that it is tested against.
39
The Anderson-Darling test while having excellent theoretical properties
does appear to have limitations when applied to real world data. The AndersonDarling mathematical computation is severely affected by ties in the data due to
poor precision. Ties are data points or values that match exactly other data
points within the data-set. When a significant number of ties exist, the AndersonDarling will frequently reject the data as non-normal, regardless of how well the
data fits the normal distribution. The prototype application will still present the
data in a histogram format for the user to custom set the upper and lower limits of
the data-set. After a short analysis the user will notice the data-set is normally
distributed due to the histogram being presented by the application. The user can
still perform the 75% inner percentile analysis, or any other available in the
application. While potentially creating a need for the user to evaluate more datasets that have multiple ties of the same data, it by no means detracts from the
functionality of the application, validation to the original requirements were still
met. Additional conditional statements were hardcoded into the application that
dealt with the mathematical calculation of Anderson-Darling A and A2 test statistic
by checking for a unique divide by zero calculation. When computing these
values the standard deviation is tested to be greater than zero or the application
will set the value Anderson-Darling value to positive infinity. The calculations for
the value of Anderson-Darling A and A2 test statistic by dividing by the standard
deviation would result in a calculation that produces an application crash.
Similar to the multiple ties limitation of the equation more serious
mathematical errors occur when all the data values are the same resulting in a
40
standard deviation of zero. Adjustments were not made to the base AndersonDarling equation. Although the results of the check for normality on modified
versions of the equation are documented within formal papers that adjust the
base equation for a specific fit. These papers take the equation and modify the
integral then verify the functionality by comparison by the use of some other
normality test, like of Chi-Squares goodness of fit tests for instance. But even
after equation modification the original limitation is still left intact with the inability
to deal with multiple ties within the data-set. (Rahman, M., Pearson, M., Heien,
C.2006)
Investigated but not completed in this project, the added functionality to
cluster the data by using a hybrid format with the standard deviation to overcome
the limitation of Anderson-Darling on data ties. (Jain, A., Murty, M., and Flynn, P
1999) The K-Means Clustering algorithm based on data clustering methodology
was developed by J. MacQueen in 1967 and then again by J.A.Hartigan and
M.A. Wong in 1975. It is an algorithm to classify or to group objects based on
attributes or features into a variable. The number of the group, K is a positive
integer number and the grouping is done by minimizing the sum of squares of
distances between data and the corresponding cluster called a centroid. The Kmean clustering only purpose is to classify the data into groups or “clusters” of
data. (Technomo, K 2006)
41
Clustering Algorithm Flowchart
Then by using these clusters created in the K means clustering groups by
passing in a hybrid format to verify against the standard deviation of the normal
distribution. The data can then be automatically reviewed to verify if the clusters
of data are within the standard deviation set for the distribution. If these clusters
of data reside within the standard deviation set by the application usually a value
of one standard deviation. Then the assumption can be made that the data within
the distribution is from a normal distribution.
42
If the data had too many ties within the data-set and would fail the
standard Anderson-Darling p-value tests if the data-set were still normal. By
contrast it would not fail this hybrid cluster with Anderson-Darling test and would
not need user intervention to automatically to perform a 75% inner percentile
analysis for outliers. The k-means algorithm can be used as an automated
function when the Anderson-Darling p-value test fails on a data-set, as an
auxiliary test. If the k-means clustering test also fails then the data-set would
presented to the user in histogram format requiring user intervention as a nonnormal distribution.
Another mathematical equation the k-sample Anderson-Darling test for
removing large number non-automated data-set histograms from this prototype.
While the potential for this test to being statically the same as the k-means hybrid
solution for Anderson-Darling standard normalcy test. The details and the
equation of the test are described below and are not trivial by any means. This
test for normal distributions is a nonparametric statistical procedure that tests a
hypothesis. The hypothesis is that the populations from two or more groups of
data were analyzed are identical. In this equation the test data can be either
presented in groups or ungrouped because the Anderson-Darling k-sample test
will verify if a grouped data-set can modeled as ungrouped data-set, grouped
data-sets will look like bimodal distributions and this fail this test.
43
The k-sample Anderson-Darling test is shown below, note that variable L
is less than n if there are tied data-point observations
hj = the number of data values in the combined data-set sample equal to zj
Hj = the number of data-values in the combined data-set sample less than zj plus
one half the number of values in the combined data-set sample equal to zj
Fij = the number of values in the i-th group which are less than zj plus one half
the number of values in this group which are equal to zj where k is the number of
groups, ni is the number of hits in group i, xij is the j-th hits in the ith group
z1, z2 ..., zL are the distinct values in the combined data set ordered from
smallest to largest Implemented in pseudo code for k-means Anderson-Darling to
appreciate the complexity of this algorithm
This prototype application validated all requirements in a loosely written
requirements document. However after data analysis the chance that the
Anderson-Darling test for normalcy would fail normally distributed data-sets
became an unforeseen issue while meeting documented requirements. The
issue arose that additional user intervention was required to process potentially
normal data-sets. If additional time were available at the time of this write-up, a
more elegant algorithm would be implemented and tested for a more automated
version of the prototype. This elegant algorithm envisioned would certainly have
to implement some portion of a clustering methodology like k-means, or a hybrid
of k-means clustering or k-sample Anderson-Darling testing.
44
CHAPTER 4
CONCLUSIONS
Integrated circuits are designed and simulated by designers and then sent
off to a wafer fabrication to be built on wafers. Unfortunately wafer fabrication
manufacturing and design defects of integrated circuits are only discovered
during the electrical test of the device. Circuit manufactures traditionally use a
method that would set an upper and lower limit on a parametric measurement to
detect circuit failures. This methodology has become only a portion of the criteria
that is now required to discover potential defects of small geometry integrated
circuits. Integrated circuit manufacturers are now employing additional methods
of failure detection including wafer test structures. These methods include high
percentage failure mode testing, design for manufacturing simulations, including
lowering the FIT requirement because the potential returned merchandise costs
are usually lower than the costs of more rigorous testing of the integrated circuit.
Circuit design and manufacturing has become increasingly competitive.
More companies are now utilizing a fab-less wafer manufacturing system. These
companies design, assemble, and test the integrated circuits in their facility. Then
pay to have the wafers manufactured at an outside facility where the company
does not own the wafer fabrication facility. Being a fab-less microelectronic
company introduces even more risk to the development of quality product
45
because specialized flows and tailoring are not available in a non company
owned facility; the benefit is reduced overhead capital expenses.
Radiation tolerant circuits typically are only built in small quantities within a
well documented qualification process. Some companies have developed a
manufacturing process to produce a range of specific radiation-hard electronic
products using rad-hard by design techniques or by adding rad-hard doped
trenches. Commercial foundries typically use standard material for all material
manufactured in the facility. By contrast radiation tolerant circuits require
nonstandard materials, incorporating epitaxial layers, insulating substrates;
doped trenches in the silicon that enhance radiation tolerance. The use of nonstandard materials makes out of family detection much more difficult due to the
inherent parametric leakage these techniques introduce and the use of nonstandard materials and processes.
A small fab-less rad-hard circuit company needs a tool that can process
relatively small amounts of individual device data based on small lots of the
electrical characteristic data. This small amount of data is subject to marginality
effects of a rad-hard circuit effects that cause wafer to wafer variability. These
specialized circuits require nonstandard starting materials, incorporating epitaxial
layers, insulating substrates; doped trenches in the silicon that enhance radiation
tolerance, but introduce marginality into the circuits with increased sub threshold
leakage effects. To compound the issue, this data being analyzed within these
46
small lots must be used to guarantee a very high FIT rate. Then used on
potentially high reliable products that provide service within a mission critical sub
system.
Lessons Learned
This automatic parametric outlier detection system prototype will utilized
an existing algorithm Anderson-Darling to test for normal distribution on the
dataset. This test is a general normality test designed to detect all departures
from normality within a dataset, and is generally believed to be the most powerful
test for normality. The importance of checking the data-set for normality was very
important or the consequential methodology for choosing outliers may produce
incorrect results. If a dataset was deemed a normal Gaussian distribution by the
automated Anderson Darling test? The prototype would then perform an inner
75% percentile outlier detection technique on the dataset, and present the user
with a histogram distribution of the data. Non-normal datasets were presented in
a histogram format requiring user intervention. The user was required to visually
inspect the data and set the upper and lower fences for a normal distribution. The
remaining steps are the same operations as those discussed in normal
distributions.
47
The Anderson-Darling test for normality is one of three general normality
tests designed to detect all departures from normality, and is sometimes touted
as the most powerful test available. The Anderson-Darling test, while having
excellent theoretical properties, does have limitations when applied to real world
data when a data-point in the set ties or matches another data-point .When this
happens the Anderson-Darling mathematical computation is severely affected by
due to poor precision analysis of the data-set tails. Therefore when a significant
number of ties existed within a data-set the Anderson-Darling frequently rejected
the data as a non-normal distribution. This was done regardless of how well the
data fits the normal distribution. This action required user intervention on a false
negative condition, the automated functionality was then compromised b y the
additional user intervention.
Validation to the original requirements was completed at project end to the
original requirements. The requirements phase of this project evidenced a lack of
appreciation for the true complexity of the project. This was especially true as
certain aspects of the functionality of the Anderson-Darling test for normality
within the system automation features from the application to be less than
optimal and require more user intervention than anticipated. This lack of
appreciation for the complexity of evaluating a normal Gaussian distribution on
every data-set presented to the prototype system is simple in theory, but became
a large part of this project. Eventually became a larger portion of the project and
required additional research to indentify future enhancements to the system.
48
The problem domain was documented as a set of requirements by
managerial stakeholders only to produce a robust beta-release outlier detection
system. The additional requirement of detection within small data-sets was
identified in the requirements documentation. This requirement evaluated small
dataset distributions using mathematical equations for normalcy required
significant hours of research. The requirements also included evaluation of the
dataset into a histogram format on non normal distributions. After user
intervention a 75% percentile analysis was always a requirement, but as the
project entered the testing phase this default situation occurred more often than
desired in an automated sub-system, although it still met validation requirements.
During the design phase a significant amount of additional methods that
became more complex to implement as important algorithms of the system. This
became especially true when passing data back and forth from complex
mathematical equations within numerous arrays’s of the data as it was calculated
and maintaining these values. The use of case diagrams would have helped in
the design phase as the design was mainly. This of course equated to additional
time being required within the test phase because of the complexity of the
Anderson-Darling mathematical equation to check for normal distributions. These
deterministic equations easily produced divide by zero errors that were not taken
into account during the design phase, and the number of ties within data-sets
cause the tool to have false normalcy failures.
49
Post prototype validation analysis of the requirements revealed minor
departures of the software system within certain data-sets, this analysis revealed
potential non-automated limitations of the software system. Therefore additional
research was started in the final testing stage of the parametric outlier design to
solve these non-automated issues. Clustering methodologies created in the kmeans clustering groups could be passed in a hybrid format to verify against the
standard deviation of the normal distribution. This data can then be reviewed to
verify if the clusters of data are within the standard deviation of the distribution. If
these clusters of data reside within the standard deviation set by the application
usually a value of three standard deviations. Then the assumption can be made
that the data within the distribution is from a normal distribution.
The other promising methodology would be the use of the k-sample
Anderson-Darling test a nonparametric statistical procedure that tests the
hypothesis that the data from a data-set of two or more groups of data is identical
and therefore from a normalized distribution. The k-means Anderson-Darling
appears to be the most promising solution, however the mathematics required to
implement, verify correct operation, and validate to requirements would be a
significant task. Future enhancements could evaluate both methodologies of
clustering for processing and computational accuracy. The methodologies
certainly require additional research before implementation, in addition to more
robust test modules for testing these complex algorithms.
50
BIBLOGRAPHY
Abrain, A.: SWEBOK, Guide to Software Engineering Body of Knowledge, IEEE
Computer Society, 2004.
Bhattachatya, S., and Chatterjee, A. Optimized Wafer-Probe and Assembled
Package Test Design for Analog Circuits. ACM: Transactions on Design
Automation of Electronic Systems (TODAES), Volume 10 Issue 2, April
2005.
CMMI - Capability Maturity Model Integration Version 1.3, Nov 2010
Dy, J, and Brodley,C.. Feature Selection for Unsupervised Learning. JMLR.org:
The Journal of Machine Learning Research, Volume 5, December 2004.
Gama,J., Rodrigues,P., and Sebastiao,R.: Evaluating Algorithms that Learn from
Data Streams. ACM: SAC '09: Proceedings of the 2009 ACM symposium on
Applied Computing, March 2009.
Gattiker, A.: Using Test Data to Improve IC Quality and Yield. IEEE Press:
ICCAD '08: IEEE/ACM International Conference on Computer-Aided
Design, November, 2008
Jain, A., Murty, M., and Flynn, P. Data clustering: a review. ACM Computing
Surveys: Volume 31, Issue 3, Pages: 264 – 323, September 1999.
Jebara, T. Wang, J., and Chang, S.: Graph Construction and b-Matching for
semi-Supervised Learning. ACM: Proceedings of the 17th ACM international
conference on Multimedia, October 2009.
Holsinger, R., Fisher, A., and Spellerberg, P., Precision Estimates for AASHTO
Test Method T308 and the Test Methods for Performance-Graded Asphalt
Binder in AASHTO Specification M320. National Cooperative Highway
Research Program, AASHTO Materials Reference Laboratory,
Gaithersburg, Maryland, February, 2005
51
Marinissen, E. Vermeulen, B., Madge, R. Kessler, M., and Muller, M.: Creating
Value through Test: DATE '03: Proceedings of the conference on Design,
Automation and Test in Europe - Volume 1, March 2003.
Moran, D., Dooling, D., Wilkins, T., Williams, R., and Ditlow, G.:Integrated
Manufacturing and Development (IMaD). ACM: Supercomputing '99:
Proceedings of the 1999 ACM/IEEE conference on Supercomputing, Jan
1999.
Otto, K.: Anderson-Darling Normality Test Calculator,
http://www.kevinotto.com/RSS/templates/Anderson-Darling Normality Test
Calculator.xls, Version 6.0, 2005.
Perez, R., Lilkendey, J., and Koh, S, Machine Learning for a Dynamic
manufacturing Environment. ACM: SIGICE Bulletin, Volume 19, Issue 3 ,
February 1994.
Pyzdek, T.: Histograms: When to Use Your Eyeballs, When Not, Looks can be
deceiving. Quality Digest Magazine, Volume 31, Issue 3, March 2011.
Rahman, M., Pearson, L., and Heien, H.: A Modified Anderson-Darling Test for
Uniformity. BULLETIN of the Malaysian Mathematical Sciences Society (2)
29(1) (2006), 11–16
Romeu J.: Selected Topics in Reliability and Assurance Related Technologies,
Reliability Analysis Center, Volume 10, Number 5, May 2005
Saab, K., Ben-Hamida, N., and Kaminska, B.: Parametric Fault Simulation and
Test Vector Generation. ACM: Proceedings of the conference on Design,
automation and test in Europe, January 2000.
Singh, A., Mani, M., and Orshansky, M.: Statistical Technology Mapping for
Parametric Yield. IEEE Computer Society: ICCAD '05: Proceedings of the
2005 IEEE/ACM International conference on Computer-aided design, May
2005.
52
Technomo, K.: K-Means Clustering Tutorial, pdf file 2006 with reference to
website, http://people.revoledu.com/kardi/tutorial/kMean/index.html, 2006.
Veelenturf, K.: The Road to better Reliability and Yield Embedded DfM tools.
ACM: Proceedings of the conference on Design, automation and test in
Europe, January 2000.
Wang, W., and Orshansky, M.: Robust Estimation of Parametric Yield under
Limited Descriptions of Uncertainty. ACM: ICCAD '06: Proceedings of the
2006 IEEE/ACM international conference on Computer-aided design,
November 2006.
Walfish, S.: A Review of Statistical Outlier Methods: PharmTech - A Review of
Statistical Outlier Methods, Pharmaceutical Technology, August 2006
53
APPENDIX A
Data Set Parametric Outlier Dataset 1
54
APPENDIX B
Data Set Parametric Outlier Dataset 2
55
APPENDIX C
Data Set Parametric Outlier Dataset 3
56
APPENDIX D
Data Set Parametric Outlier Dataset 4
57
APPENDIX E
Data Set Parametric Outlier Dataset 5
58
APPENDIX F
Data Set Parametric Outlier Dataset 6
59
APPENDIX G
Data Set Parametric Outlier Dataset 7
60
APPENDIX H
Data Set Parametric Outlier Dataset 8
61
APPENDIX I
Data Set Performance Graded Binder Sample 195
62
APPENDIX J
Data Set Performance Graded Binder Sample 196