Download Verification, Validation and Evaluation of Expert Systems in Order to

Verification, Validation and Evaluation of Expert Systems in Order to
Develop a Safe Support in the Process of Decision Making
Ph.D. Prof. Adrian Cojocariu, Timişoara, Romania
Ph.D.Lect. Alin Munteanu, Timişoara, Romania, [email protected]
Prep. Ofelia Sofran, Timişoara, Romania, [email protected]
1. Introduction
Well known software researchers have given different definitions of expert
systems, but there was one thing they all agreed on: that knowledge based systems
are meant to function as systems able to solve specific intelligent activities that
human experts used to do. There was also confusion about the terms “expert
systems” and “knowledge based expert systems” that refer to one and the same
entity, artificial intelligence or high level knowledge based systems, similar to
competent specialists that may be able to achieve the same thinking and intuition
performances as human experts. The main factor of intelligent processing consists in
artificial judgment able to imitate human judgment.
Expert systems are being used to replace human experts in order to support the
process of decision making in various activity domains, being based on a number of
computational rules and a very well built knowledge base. Knowledge engineers
play an important role in the process of knowledge acquisitions, as the development
of an efficient expert system requires a high level expertise from the most competent
person in that very domain, and they will “compile” the knowledge that the human
expert has shared. Knowledge engineers are nowadays the highest qualified
personnel among computer specialists provided by domains as economics and
informatics. In order to build an expert system, the knowledge engineer will not only
have to be able to simulate the human expert’s activity in finding solutions for
different issues but also to perfectly reproduce the high performance of the human
expert. The role that knowledge engineers play in the field of expert system
developing is as important as the role of the software engineer.
It is a great deal for the software engineers and knowledge engineers to build a
complete expert system, with a solid knowledge base, as real-world knowledge
bases contain a very large number of rules that lead to a computational paths through
an expert system.
We shall present in Figure 1 a recently designed expert system development life
cycle model, very well adapted on the existing standards for conventional software
engineering [Awa 96].
PROBLEM IDENTIFICATION
PRELIMINARY REQUIREMENTS
ANALYSIS AND KNOWLEDGE
ACQUISITION
DEVELOPMENT TOOLS SELECTION
KNOWLEDGE AQUISITION
PROTOTYPE DEVELOPMENT
VERIFICATION AND VALIDATION
IMPLEMENTATION
OPERATING AND MAINTENANCE
Figure 1 Expert systems development life cycle model
It is important to have in mind that verification and validation are the most
crucial phases in building knowledge based systems.
2. Expert Systems Verification, Validation and Evaluation
Researchers and practitioners in the field of artificial intelligence applications
generally agreed that knowledge base in any intelligent system must be adequately
verified and validated. The lack of adequate and precise knowledge is a limiting
factor to build an efficient artificial intelligence application. When intelligent
methods are used in real applications, it is an important factor to be able to check the
consistency of knowledge and the correctness of the system’s reasoning. As with
any software system or application, attention to quality and safety must be paid
throughout the development of knowledge based systems.
To a better understanding of the difference between validation and verification,
there’s a very simple definition of the two: validation answers the question “are we
building the right system?” and verification answers the question “are we building
the system right” [Boe 84]. This means that validation is a process designed to
increase our confidence that a program works as intended and verification is a
formal or informal argument that a program works on all possible inputs.
Validation and verification can be done through testing. Testing is the process
of running on a set of test cases and comparing the actual results with expected
results. Testing involves considerably more than finding and fixing syntax errors. It
covers the verification of individual relationships, validation of program
performance and evaluation of the utility of the software package. Testing guides
reformulation of concepts, redesign of representations and other refinements.
Verification and validation must occur during the entire development process.
Verification proves that the models within the program are true relationships. It
ensures that the knowledge is accurately mimicked by having the domain expert
operate the program for all possible contingencies. The following figure shows the
testing phases for expert systems, according to American theory [Awa 96]:
TESTING
Certifying
VERIFICATION
Representation
VALIDATION
Implementation
Operating
Figure 2 Expert systems testing phases
Perhaps the most difficult aspect of testing is accurately handling the
uncertainty that is incorporated in most expert systems in one way or another.
Certainty factors are one of the most common methods for handling uncertainty.
Verification of the certainty factors assigned to the knowledge base is largely a
process of trial and error, refining the initial estimates by the domain expert until the
program consistently provides recommendations at a level of certainty that satisfies
the expert. To ensure program accuracy, all possible solution paths must be
rigorously evaluated.
Researchers have generally admitted that testing is the most challenging activity
in expert systems developing, as it follows the performance, efficiency, knowledge
base quality control.
In order to verify the knowledge base, there are two possible activities:
structural verification and functional verification. Structural verification of the
knowledge base refers to verifying each knowledge part, circular rules and
redundancy, also checking out the consistency, correctness and integrity of the
knowledge base. The functional verification deals with assuring the functionality
and reliability of the knowledge base.
Validation is an ongoing process requiring the output recommendations to be
accurate for a specific user's case. Validation is enhanced by allowing others to
review critically and recommend improvements. A formal project evaluation is
helpful to establish whether the system meets the intended original goal. The
evaluation process focuses on uncovering problems with the credibility,
acceptability and utility. This can be determined from the program accuracy that is
determined from comparisons with the real-world environment. Included are the
understanding and flexibility of the program, ease of use, adaptability of the design
and the correctness of solutions.
Evaluation is the process whereby we ensure the usability, quality and utility of
the expert systems. A complete testing cycle is performed in iterations through
which the expert system is updated and refined. The evaluation step is to assess the
quality, usability, and utility of the ES from the point of view of human experts
other than the domain expert, who participate in the system development.
3. Conclusions
It is vital in expert systems development management to realize the importance
of validation, verification and evaluation, especially when the expert system is being
used in important domains, such as medicine or management strategies. Through
high competition condition on the software market, there has been a tendency of the
software project managers to hurry the launching of the systems without a very
accurately testing, and the results have been almost disastrous. The reliability and
functionality of the systems were doubtful and the counted success was lacking.
Nowadays the importance of quality in software development, and especially in
expert systems development has greatly increased, and the risk of taking a wrong
decision by using a faulty decision making support has decreased, but it is not
inexistent yet.
Reference
[Awa 96]
Awad, E.M., Building Expert Systems, West Pub. Co., 1996
[And 01]
Andone, I.I., Mockler, R.J., Dologite, D.G., Ţugui, A.A.,
Dezvoltarea sistemelor inteligente în economie, Ed. Economică,
2001
[Boe 84]
Boehm, B.W., Verifying ans validating software requierements
and design specifications, IEEE Software, 1884
[Som 00]
Sommerville, I., Software Engineering (6 th Edition), AddisonWesley, 2000