Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Projecto em Informática e Gestão de Empresas TESTING METHODOLOGY Version 2.2 Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published DISTRIBUTION LIST To: Mário Romão CC: Ana Violante Francisco Mineiro REVISION HISTORY Revision Date Revision Author(s) / Summary of Change Additional Distribution Revision No. 23-Dec-2005 Joao Alves / First written 0.1 27-Dec-2005 Francisco Mineiro / Revision 0.2 3-Jan-2006 Ana Violante / Revision 0.3 10-Jan-2006 Joao Alves / Modified 0.4 14-Jan-2006 Joao Alves / Modified 0.5 19-Jan-2006 Ana Violante / Revision 0.6 23-Jan-2006 Joao Alves / Published 1.0 10-Feb-2006 Joao Alves / Modified 1.1 8-Mar-2006 Francisco Mineiro / Revision 1.2 10-Mar-2006 Joao Alves / Published 2.0 22-Mar-2006 Joao Alves, Ana Violante / Modified 2.1 22-Jun-2006 Francisco Mineiro, Joao Alves / Revision 2.2 Page 2 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published TABLE OF CONTENTS 1. INTRODUCTION ....................................................................................................................................... 7 1.1. DOCUMENT OBJECTIVE ...................................................................................................................... 7 1.2. DOCUMENT OVERVIEW ....................................................................................................................... 7 2. TESTING CONCEPTS .............................................................................................................................. 8 2.1. W HAT IS SOFTWARE TESTING? ........................................................................................................... 8 2.2. OVERVIEW ......................................................................................................................................... 8 2.3. TEST CASES, SUITES, SCRIPTS, AND SCENARIOS ................................................................................ 9 2.4. A SAMPLE TESTING CYCLE ................................................................................................................. 9 2.5. TEST TYPES..................................................................................................................................... 10 2.5.1. Functional Tests .................................................................................................................. 11 2.5.2. Regression Tests ................................................................................................................. 11 2.5.3. Performance Tests .............................................................................................................. 11 2.5.3.1. Load Tests ............................................................................................................................ 13 2.5.3.2. Stress Tests.......................................................................................................................... 13 3. MANUAL VS AUTOMATED TESTING................................................................................................... 14 3.1. AUTOMATED TESTING METHODOLOGIES ............................................................................................ 14 3.1.1. What is “Automated testing”? .............................................................................................. 14 3.1.2. Cost-Effective Automated Testing ....................................................................................... 14 3.1.3. The Record/Playback Myth ................................................................................................. 15 3.1.4. Viable Automated Testing Methodologies ........................................................................... 15 3.1.4.1. The “Functional Decomposition” Method .............................................................................. 15 3.1.4.2. The “Key-Word Driven” or “Test Plan Driven” Method .......................................................... 17 3.1.5. Preparation is the Key ......................................................................................................... 18 3.1.6. Managing Resistance to Change ........................................................................................ 19 3.1.7. Staffing Requirements ......................................................................................................... 19 3.1.8. Summary ............................................................................................................................. 20 4. TEST MANAGEMENT ............................................................................................................................ 21 4.1. W HAT IS TEST MANAGEMENT?.......................................................................................................... 21 4.2. PRINCIPLES ..................................................................................................................................... 21 4.3. IMPROVING THE PROCESS ................................................................................................................. 22 Page 3 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 4.4. REQUIREMENTS MANAGEMENT ......................................................................................................... 23 4.5. PLANNING AND RUNNING TESTS........................................................................................................ 23 4.5.1. Planning Tests ..................................................................................................................... 23 4.5.2. Running Tests ...................................................................................................................... 24 4.6. ISSUES AND DEFECTS TRACKING ...................................................................................................... 25 5. PROCESS MODELS ............................................................................................................................... 27 5.1. THE V-MODEL.................................................................................................................................. 27 5.1.1. Development Process.......................................................................................................... 27 5.1.1.1. Requirements ....................................................................................................................... 27 5.1.1.2. System Specification ............................................................................................................ 27 5.1.1.3. System Design ..................................................................................................................... 28 5.1.1.4. Detailed Design .................................................................................................................... 28 5.1.1.5. Coding .................................................................................................................................. 28 5.1.2. Testing Process ................................................................................................................... 29 5.1.2.1. Unit Testing .......................................................................................................................... 30 5.1.2.2. Integration Testing ................................................................................................................ 30 5.1.2.3. System Testing ..................................................................................................................... 31 5.1.2.4. Acceptance Testing .............................................................................................................. 31 5.2. RATIONAL UNIFIED PROCESS ............................................................................................................ 32 5.2.1. Overview .............................................................................................................................. 32 5.2.1.1. The Inception Phase ............................................................................................................. 33 5.2.1.2. The Elaboration Phase ......................................................................................................... 33 5.2.1.3. The Construction Phase ....................................................................................................... 33 5.2.1.4. The Transition Phase............................................................................................................ 34 5.2.1.5. Milestones ............................................................................................................................ 34 5.2.1.6. Iterations ............................................................................................................................... 34 5.2.2. Best Practices of RUP ......................................................................................................... 35 5.2.2.1. Develop software iteratively .................................................................................................. 35 5.2.2.2. Manage requirements ........................................................................................................... 35 5.2.2.3. Use component-based architecture ...................................................................................... 36 5.2.2.4. Visually model software ........................................................................................................ 36 5.2.2.5. Verify software quality........................................................................................................... 36 5.2.2.6. Control changes to software ................................................................................................. 36 5.3. CONCERT ........................................................................................................................................ 37 Page 4 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 5.3.1. Basic Principles ................................................................................................................... 38 5.3.2. Concert Phases ................................................................................................................... 39 6. TESTING PROCEDURES ....................................................................................................................... 40 6.1. TESTING PROCEDURE FOR THE V-MODEL .......................................................................................... 41 6.2. TESTING PROCEDURE FOR RATIONAL UNIFIED PROCESS (RUP) ......................................................... 43 6.3. TEST MANAGEMENT ......................................................................................................................... 44 6.3.1. Test Strategy and Plan ........................................................................................................ 44 6.3.2. Management ........................................................................................................................ 44 6.3.3. Test Team Organization ...................................................................................................... 45 6.3.4. Test Metrics ......................................................................................................................... 45 6.3.5. Risk Assessment ................................................................................................................. 45 6.3.6. Defect Tracking and Severity Levels ................................................................................... 46 6.3.7. Entry/Exit Criteria ................................................................................................................. 46 6.4. TESTING ACTIVITIES ......................................................................................................................... 47 6.4.1. Prepare Overall Test Strategy ............................................................................................. 47 6.4.1.1. Tasks .................................................................................................................................... 47 6.4.1.2. Components ......................................................................................................................... 49 6.4.2. Prepare Overall Test Plan ................................................................................................... 50 6.4.2.1. Tasks .................................................................................................................................... 50 6.4.2.2. Components ......................................................................................................................... 53 6.4.3. Prepare Unit Test Plan ........................................................................................................ 53 6.4.3.1. Tasks .................................................................................................................................... 54 6.4.3.2. Components ......................................................................................................................... 56 6.4.4. Prepare Integration Test Plan .............................................................................................. 56 6.4.4.1. Tasks .................................................................................................................................... 57 6.4.4.2. Components ......................................................................................................................... 59 6.4.5. Prepare System Test Plan ................................................................................................... 59 6.4.5.1. Tasks .................................................................................................................................... 60 6.4.5.2. Components ......................................................................................................................... 62 6.4.6. Prepare Acceptance Test Plan ............................................................................................ 62 6.4.6.1. Tasks .................................................................................................................................... 62 6.4.6.2. Components ......................................................................................................................... 64 6.4.7. Perform Unit Tests ............................................................................................................... 65 6.4.7.1. Tasks .................................................................................................................................... 65 Page 5 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.7.2. Components ......................................................................................................................... 68 6.4.8. Perform Integration Tests .................................................................................................... 68 6.4.8.1. Tasks .................................................................................................................................... 68 6.4.8.2. Components ......................................................................................................................... 72 6.4.9. Perform System Tests ......................................................................................................... 73 6.4.9.1. Tasks .................................................................................................................................... 73 6.4.9.2. Components ......................................................................................................................... 76 6.4.10. Perform Acceptance Tests ......................................................................................... 76 6.4.10.1. Tasks .................................................................................................................................... 76 6.4.10.2. Components ......................................................................................................................... 79 7. GLOSSARY ............................................................................................................................................. 80 8. REFERENCES ........................................................................................................................................ 81 APPENDIX 1 – AN EXAMPLE FOR “FUNCTIONAL DECOMPOSITION METHOD” .................................. 84 APPENDIX 2 – AN EXAMPLE FOR “KEY-WORD DRIVEN METHOD” ...................................................... 86 APPENDIX 3 – CONCERT METHODOLOGY OVERVIEW ........................................................................... 88 APPENDIX 4 – TEST TEAM ROLES AND RESPONSIBILITIES.................................................................. 99 APPENDIX 5 – DOCUMENT TEMPLATES ................................................................................................. 101 TABLE OF FIGURES Figure 1 - The V-Model ................................................................................................................................... 29 Figure 2 - Testing Phases ............................................................................................................................... 30 Figure 3 - The Rational Unified Process ......................................................................................................... 32 Figure 4 - CONCERT System Development and Related Process ................................................................ 38 Figure 5 - Testing Process in CONCERT ....................................................................................................... 40 Figure 6 - Testing Process in the V-Model ...................................................................................................... 41 Figure 7 - Testing in RUP ................................................................................................................................ 43 Figure 8 - Test Management Areas ................................................................................................................ 44 Page 6 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 1. INTRODUCTION 1.1. Document Objective This documented intends to provide CGI Portugal project team members with a complete test process within a standard software development model. Besides this it also provides with some concepts on software testing. 1.2. Document Overview The document is organized into the following sections: Section 2 – Testing concepts provide several concepts on software testing. Section 3 – This section intends to provide a comparison between manual and automated testing, as well as help the decision on whether to do, or not to do, automated testing, and how to do it. Section 4 – In this section will be provided an overview on effective test management process. Section 5 – This section presents the software development processes which will be the base for the testing procedure. Section 6 – This section shows the testing procedure to use at CGI Portugal, based on the processes introduced in the previous section. Section 7 – A glossary for this document Section 8 – The references used for writing this document. Page 7 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 2. TESTING CONCEPTS 2.1. What is Software Testing? Software Testing is a process used to help identify the correctness, completeness and quality of developed computer software. With that in mind, testing can never completely establish the correctness of arbitrary computer software. In computability theory, a field of computer science, a mathematical proof exists which concludes that it is impossible to solve the halting problem, the question of whether an arbitrary computer program will enter an infinite loop, or halt and produce output. In other words, testing is nothing but criticism or comparison, which is comparing the actual value with the expected one. There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following rote procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are things the tester tries to do with the product, and the product answers with its behaviour in reaction to the probing of the tester. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is connoted to mean the dynamic analysis of the product - putting the product through its paces. The quality of the application can, and normally does, vary widely from system to system but some of the common quality attributes include reliability, stability, portability, maintainability and usability. Software Testing does not guarantee that errors do not occur, although there are some techniques which are highly reliable on achieving zero defects, but at the same time are very expensive. One of these techniques is the Cleanroom methodology, in which correctness verification replaces unit testing and debugging, and the software enters system testing immediately after coding is complete. All test errors are accounted for from the first execution of the program. Since this type of techniques have very high costs, it is more usual to use risk-based testing, where functionalities are prioritized by its degree of importance, thus focusing the whole testing effort only on the most important modules. 2.2. Overview In general, software engineers distinguish software faults and software failures. In case of a failure, the software does not do what the user expects. A fault is a programming error that may or may not be actually manifested as a failure. A fault can also be described as an error in the correctness of the semantic of a computer program. A fault will become a failure if the exact computation conditions are met, one of them being that the software is ported to a different hardware platform or a different compiler, or when the software gets extended. Regardless of the methods used or level of formality involved, the desired result of testing is a level of confidence in the software so that the developers are confident that the software has an acceptable defect rate. What constitutes an acceptable defect rate depends on the nature of the software. An arcade video game designed to simulate flying an airplane would presumably have a much higher tolerance for defects than software used to control an actual airliner. A problem with software testing is that the number of defects in a software product can be very large, and the number of configurations of the product larger still. Bugs that occur infrequently are difficult to find in testing. A rule of thumb is that a system that is expected to function without faults for a certain length of time must have already been tested for at least that length of time. This has severe consequences for projects to write long-lived reliable software. Page 8 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published A common practice of software testing is that it is performed by an independent group of testers after finishing the software product and before it is shipped to the customer. This practice often results in the testing phase being used as project buffer to compensate for project delays. Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes. Another common practice is for test suites to be developed during technical support escalation procedures. Such tests are then maintained in regression testing suites to ensure that future updates to the software don't repeat any of the known mistakes. It is commonly believed that the earlier a defect is found the cheaper it is to fix it. 2.3. Test Cases, Suites, Scripts, and Scenarios Black box1 testers usually write test cases for the majority of their testing activities. A test case is usually a single step, and its expected result, along with various additional pieces of information. It can occasionally be a series of steps but with one expected result or expected outcome. The optional fields are a test case ID, test step or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table. The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests. Collections of test cases are sometimes incorrectly termed a test plan. Most white box2 testers write and use test scripts in unit, system, and regression testing. A test script is a short program written in a programming language used to test part of the functionality of a software system. A test scenario is a test based on a hypothetical story used to help a person think through a complex problem or system. They can be as simple as a diagram for a testing environment or they could be a description written in prose. The ideal scenario test has five key characteristics: it is (a) a story that is (b) motivating, (c) credible, (d) complex, and (e) easy to evaluate. They are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Test suites and scenarios can be used in concert for complete system tests. 2.4. A Sample Testing Cycle Although testing varies between organizations, there is a cycle to testing: 1 Refer to Glossary for Black box testing 2 Refer to Glossary for White box testing Page 9 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 1. Requirements Analysis: Testing should begin in the requirements phase of the software development life cycle (SDLC). 2. Design Analysis: During the design phase, testers work with developers in determining what aspects of a design are testable and under what parameter those testers work. 3. Test Planning: Test Strategy, Test Plan(s), Test Bed creation. 4. Test Development: Test Procedures, Test Scenarios, Test Cases, and Test Scripts to use in testing software. 5. Test Execution: Testers execute the software based on the plans and tests and report any errors found to the development team. 6. Test Reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release. 7. Retesting the Defects Not all errors or defects reported must be fixed by a software development team. Some may be caused by errors in configuring the test software to match the development or production environment. Some defects can be handled by a workaround in the production environment. Others might be deferred to future releases of the software, or the deficiency might be accepted by the business user. There are yet other defects that may be rejected by the development team (of course, with due reason) if they deem it inappropriate to be called a defect. 2.5. Test Types There are several test types: Functional Tests Regression Tests Performance Tests, and its subsets: o Load (or Volume) Tests o Stress Tests Security Tests User Acceptance Tests Assembly Tests Operational Acceptance Tests, and its subsets: o Scheduling Tests o Recovery/Fail Over Tests Installation/Deployment Tests From these types, the most relevant for this document are Functional, Regression, and Performance Tests. Page 10 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 2.5.1. Functional Tests System functional testing, or Functional Testing, is a form of software testing that attempts to determine whether each function of the system works as specified. 2.5.2. Regression Tests Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes. Common methods of regression testing include re-running all previously run tests of all types and checking whether accepted functionality or previously-fixed faults have re-emerged. Experience has shown that as software is developed, this kind of re-emergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but just as often a fix for a problem will be "fragile" - if some other change is made to the program, the fix no longer works. Finally, it has often been the case that when some feature is redesigned, the same mistakes that were made in the original implementation of the feature will be made in the redesign. Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools, frequently, a 'test suite', software tools that allows the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. 2.5.3. Performance Tests In software engineering, performance testing is a test type that is performed to determine how fast some aspect of a system performs under a particular workload. Performance testing can serve different purposes: It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. It can measure what parts of the system or workload cause the system to perform badly. In the diagnostic case, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. In performance testing, it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use. Technology For example in CRM systems, performance testing technology employs one or more PCs to act as injectors – each emulating the presence or numbers of users and each running an automated sequence of interactions (recorded as a script, or as a series of scripts to emulate different types of Page 11 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published user interaction) with the host whose performance is being tested. Usually, a separate PC acts as a test conductor, coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. The usual sequence is to ramp up the load – starting with a small number of virtual users and increasing the number over a period to some maximum. The test result shows how the performance varies with the load, given as number of users vs. response time. Various tools are available to perform such tests. Tools in this category usually execute a suite of tests which will emulate real users against the system. Sometimes the results can reveal oddities, e.g., that while the average response time might be acceptable, there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries, etc. Performance testing can be combined with stress testing, in order to see what happens when an acceptable load is exceeded – does the system crash? Performance specifications Performance testing is frequently not performed against a specification, i.e. no one will have expressed what is the maximum acceptable response time for a given population of users. However, performance testing is frequently used as part of the process of performance profile tuning. The idea is to identify the “weakest link” – there is inevitably a part of the system which, if it is made to respond faster, will result in the overall system running faster. It is sometimes a difficult task to identify which part of the system represents this critical path, and some test tools come provided with (or can have add-ons that provide) instrumentation that runs on the server and reports transaction times, database access times, network overhead, etc. which can be analyzed together with the raw performance statistics. Without such instrumentation one might have to have someone crouched, for example, over Windows Task Manager at the server to see how much CPU load the performance tests are generating. There is an apocryphal story of a company that spent a large amount optimizing their software without having performed a proper analysis of the problem. They ended up rewriting the system’s ‘idle loop’, where they had found the system spent most of its time, but even having the most efficient idle loop in the world obviously didn’t improve overall performance one iota! Performance testing almost invariably identifies that it is parts of the software (rather than hardware) that contribute most to delays in processing users’ requests. Performance testing can be performed across the web, and even done in different parts of the country, since it is known that the response times of the internet itself vary regionally. It can also be done in-house, although routers would then need to be configured to introduce the lag what would typically occur on public networks. It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time, then an injector configuration could be used to test whether the proposed system met that specification. Tasks to undertake Tasks to perform such a test would include: analysis of the types of interaction that should be emulated and the production of scripts to do those emulations decision whether to use internal or external resources to perform the tests set up of a configuration of injectors/controller Page 12 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published set up of the test configuration (ideally identical hardware to the production platform), router configuration, quiet network (we don’t want results upset by other users), deployment of server instrumentation running the tests – probably repeatedly in order to see whether any unaccounted for factor might affect the results analyzing the results, either pass/fail, or investigation of critical path and recommendation of corrective action 2.5.3.1. Load Tests Load testing is the act of testing a system under load. In software engineering it is a blanket term that is used in many different ways across the professional software testing community. Load testing generally refers to the practice of modelling the expected usage of a software program by simulating multiple users or instances accessing the program's services concurrently. As such, this testing is most relevant for multi-user systems, often built using a client/server model, such as web servers. However, other types of software systems can be load-tested also. For example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although there is a gray area between the two domains and no clear boundary exists when an activity ceases to be a load test and becomes a stress test. There is little agreement on what the specific goals of load testing are. The term is often used synonymously with performance testing, reliability testing, and volume testing. 2.5.3.2. Stress Tests Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. For example, a web server may be stress tested using scripts, bots, and various denials of service tools to observe the performance of a web site during peak loads. Page 13 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3. MANUAL VS AUTOMATED TESTING3 3.1. Automated Testing Methodologies 3.1.1. What is “Automated testing”? Simply put, what is meant by “Automated Testing” is automating the manual testing process currently in use. This requires that a formalized “manual testing process” currently exists in the company or organization. Minimally, such a process includes: 1. Detailed test cases, including predictable “expected results”, which have been developed from Business Functional Specifications and Design documentation 2. A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application If the current testing process does not include the above points, one is never going to be able to make any effective use of an automated test tool. The real use and purpose of automated test tools is to automate regression testing. This means that one must have or must develop a database of detailed test cases that are repeatable, and this suite of tests is run every time there is a change to the application to ensure that the change does not produce unintended consequences. An “automated test script” is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development. 3.1.2. Cost-Effective Automated Testing Automated testing is expensive. It does not replace the need for manual testing or enable one to “down-size” its testing department. Automated testing is an addition to the testing process. It can take between 3 to 10 times as long (or longer) to develop, verify, and document an automated test case than to create and execute a manual test case. This is especially true if one selects to use the “record/playback” feature (contained in most test tools) as primary automated testing methodology. Record/Playback is the least cost-effective method of automating test cases. Automated testing can be made to be cost-effective, however, if some common sense is applied to the process: 3 Choose a test tool that best fits the testing requirements of the organization or company. Realize that it doesn’t make sense to automate some tests. Overly complex tests are often more trouble than they are worth to automate. Concentrate on automating the majority of your tests, which are probably fairly straightforward. Leave the overly complex tests for manual testing. Only automate tests that are going to be repeated. One-time tests are not worth automating. Avoid using “Record/Playback” as a method of automating testing. This method is fraught with problems, and is the most costly (time consuming) of all methods over the long term. From Totally Data-Driven Automated Testing – A White Paper by Keith Zambelich Page 14 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3.1.3. Adopt a data-driven automated testing methodology. This allows developing automated test scripts that are more “generic”, requiring only that the input and expected results be updated. The Record/Playback Myth Every automated tool vendor will tell that their tool is “easy to use” and that non-technical user-type testers can easily automate all of their tests by simply recording their actions, and then playing back the recorded scripts. This one statement alone is probably the most responsible for the majority of automated test tool software that is gathering dust on shelves in companies around the world. Here’s why it doesn’t work: The scripts resulting from this method contain hard-coded values which must change if anything at all changes in the application The costs associated with maintaining such scripts are astronomical, and unacceptable. These scripts are not reliable, even if the application has not changed, and often fail on replay (pop-up windows, messages, and other things can happen that did not happen when the test was recorded). If the tester makes an error entering data, etc., the test must be re-recorded. If the application changes, the test must be re-recorded. All that is tested are things that already work. Areas that have errors are encountered in the recording process (which is manual testing, after all). These bugs are reported, but a script cannot be recorded until the software is corrected. So what is being tested? 3.1.4. Viable Automated Testing Methodologies Some effective methodologies for automating functional or system testing for most business applications: 3.1.4.1. The “Functional Decomposition” Method The main concept behind the “Functional Decomposition” script development methodology is to reduce all test cases to their most fundamental tasks, and write User-Defined Functions, Business Function Scripts, and “Sub-routine” or “Utility” Scripts which perform these tasks independently of one another. In general, these fundamental areas include: 1. Navigation (e.g. “Access Payment Screen from Main Menu”) 2. Specific (Business) Function (e.g. “Post a Payment”) 3. Data Verification (e.g. “Verify Payment Updates Current Balance”) 4. Return Navigation (e.g. “Return to Main Menu”) In order to accomplish this, it is necessary to separate Data from Function. This allows an automated test script to be written for a Business Function, using data-files to provide both the input and the expected-results verification. A hierarchical architecture is employed, using a structured or modular design. The highest level is the Driver script, which is the engine of the test. The Driver Script contains a series of calls to one or more “Test Case” scripts. The “Test Case” scripts contain the test case Page 15 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published logic, calling the Business Function scripts necessary to perform the application testing. Utility scripts and functions are called as needed by Drivers, Main, and Business Function scripts. Driver Scripts: Perform initialization (if required), then call the Test Case Scripts in the desired order. Test Case Scripts: Perform the application test case logic using Business Function Scripts; Business Function Scripts: Perform specific Business Functions within the application; Subroutine Scripts: Perform application specific tasks required by two or more Business scripts; User-Defined Functions: General, Application-Specific, and Screen-Access Functions. Functions can be called from any of the above script types. An example of this can be found in Appendix 1. 3.1.4.1.1. Advantages Using a modular design, and files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts. Scripts may be developed while application development is still in progress. If functionality changes, only the specific “Business Function” script needs to be updated. Since scripts are written to perform and test individual Business Functions, they can easily be combined in a “higher level” test script in order to accommodate complex test scenarios. Data input/output and expected results are stored as easily maintainable text records. The user’s expected results are used for verification, which is a requirement for System Testing. Functions return “TRUE” or “FALSE” values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed “recovery” routine, enables “unattended” execution of test scripts. 3.1.4.1.2. Disadvantages Requires proficiency in the Scripting language used by the tool (technical personnel); Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This usually requires data-files to be kept in separate directories by Test Case. Tester must not only maintain the Detail Test Plan with specific data, but must also reenter this data in the various required data-files. If a simple “text editor” such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or script-processing errors will occur due to data-file format and/or content being incorrect. Page 16 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3.1.4.2. The “Key-Word Driven” or “Test Plan Driven” Method This method uses the actual Test Case document developed by the tester using a spreadsheet containing special “Key-Words”. This method preserves most of the advantages of the “Functional Decomposition” method, while eliminating most of the disadvantages. In this method, the entire process is data-driven, including functionality. The Key Words control the processing. An example can be found in Appendix 2. The architecture of the “Test Plan Driven” method appears similar to that of the “Functional Decomposition” method, but in fact, they are substantially different: Driver Script o Performs initialization, if required; o Calls the Application-Specific “Controller” Script, passing to it the file-names of the Test Cases (which have been saved from the spreadsheets as a “tabdelimited” files); The “Controller” Script o Reads and processes the file-name received from Driver; o Matches on “Key Words” contained in the input-file; o Builds a parameter-list from the records that follow; o Calls “Utility” scripts associated with the “Key Words”, passing the created parameter-list; Utility Scripts o Process input parameter-list received from the “Controller” script; o Perform specific tasks (e.g. press a key or button, enter data, verify data, etc.), calling “User Defined Functions” if required; o Report any errors to a Test Report for the test case; o Return to “Controller” script; User Defined Functions o General and Application-Specific functions may be called by any of the above script-types in order to perform specific-tasks; 3.1.4.2.1. Advantages This method has all the advantages of the “Functional Decomposition” method, as well as the following: The Detail Test Plan can be written in Spreadsheet format containing all input and verification data. Therefore the tester only needs to write this once, rather than, for example, writing it in Word, and then creating input and verification files as is required by the “Functional Decomposition” method. Page 17 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Test Plan does not necessarily have to be written using MS Excel. Any format can be used from which either “tab-delimited” or “comma-delimited” files can be saved (e.g. Access Database, etc.). If “utility” scripts can be created by someone proficient in the automated tool’s Scripting language prior to the Detail Test Plan being written, then the tester can use the Automated Test Tool immediately via the “spreadsheet-input” method, without needing to learn the Scripting language. If Detailed Test Cases already exists in some other format, it is not difficult to translate these into the “spreadsheet” format. After a number of “generic” Utility scripts have already been created for testing an application, one can usually re-use most of these if he/she needs to test another application. 3.1.4.2.2. Disadvantages Development of “customized” (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. Note that this is also true of the “Functional Decomposition” method, and, frankly of any method used including “Record/Playback”. If the application requires more than a few “customized” Utilities, this will require the tester to learn a number of “Key Words” and special formats. This can be timeconsuming, and may have an initial impact on Test Plan Development. 3.1.5. Preparation is the Key If adequate preparations have not been made, then the “ramp-up” time required is increased dramatically. What then, does an organization do to prepare itself for this effort? 1. An adequate Test Environment must exist that accurately replicates the Production Environment. 2. The Test Environment’s database must be able to be restored to a known baseline; otherwise test performed against this database will not be able to be repeated, as the data will have been altered. 3. Part of the Test Environment includes hardware. The automated scripts must have dedicated PCs on which to run. If one is developing scripts, then these scripts themselves must be tested to ensure that they work properly. If the company is purchasing say, 5 licenses for the test tool, then it would be prudent to assign at least 3 PCs to run the automated scripts on. The other 2 can be used for script development (assuming 2 developers). Since test tool software requires a lot of processing speed and memory to run correctly, it is imperative to get the biggest, fastest machines affordable to run this software. 4. Detailed Test Cases that are able to be converted to an automated format must exist. The test-tool is not a thinking entity. One must tell it exactly what to do, how to do it, and when to do it. Data to be entered and verified must be specific data, not “post a payment to an account, and verify the results”. 5. The person or persons who are going to be developing and maintaining the automated scripts must be recruited within the company (or in the worst case, hired) and trained. Page 18 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3.1.6. Managing Resistance to Change One of the main reasons organizations fail at implementing automated testing (apart from getting mired down in the “record/playback” quagmire) is that most testers do not welcome what they perceive as a fundamental change to the way they are going to have to approach their jobs. Let’s examine some concerns that might be expressed by testers, and some answers to these: The tool is going to replace the testers This is not even remotely true. The automated testing tool is just another tool that will allow testers to do their jobs better. The testers are still going to have to perform tests manually for specific application changes. It will take too long to train all of the testers to use the tool If the “test-plan-driven” method (described above) is used, the testers will not have to learn how to use the tool at all if they don’t want to. All they have to learn is a different method of documenting the detailed test cases, using the key-word/spreadsheet format. The tool will be too difficult for testers to use Perhaps, but as we have already discussed, they will not have to use it. What will be required is that a “Test Tool Specialist” will need to be hired and trained to use the tool. The “test-plan-driven” testing method will eliminate most of the testers’ concerns regarding automated testing. They will perform their jobs exactly as they do now. They will only need to learn a different method of documenting their test cases. 3.1.7. Staffing Requirements One area that organizations desiring to automate testing seem to consistently miss is the staffing issue. A “Test Tool Specialist” or “Automated Testing Engineer” or some such position must be created and staffed with at least one senior-level programmer. This person must be capable of designing, developing. Testing, debugging, and documenting code. This person must want to do this job – most programmers want nothing to do with the Testing Department. This is not going to be easy, but it is nonetheless absolutely critical, In addition to developing automated scripts and functions, this person must be responsible for: Developing standards and procedures for automated script/function development Developing change-management implementation Developing the details and infrastructure for a data-driven testing method Testing, implementing, and Managing the test scripts (spreadsheets) written by the testers Running the automated tests, and providing the testers with the results procedures for automated script/function It is worth noting that no special “status” should be granted to the automation tester(s). The nontechnical testers are just as important to the process, and favouritism toward one or the other is counter-productive and should be avoided. Page 19 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3.1.8. Summary Establish clear and reasonable expectations as to what can and what cannot be accomplished with automated testing in your organization. o Educate yourself on the subject of automated testing o Establish what percentage of your tests are good candidates for automation Get a clear understanding of the requirements which must be met in order to be successful with automated testing o Technical personnel are required to use the tool effectively o An effective manual testing process must exist before automation is possible. You should have: o Eliminate overly complex or one-of-a-kind tests as candidates Detailed, repeatable test cases, which contain exact expected results A standalone test environment with a restorable database You are probably going to require short-term assistance from an outside company which specializes in setting up automated testing or a contractor experienced in the test tool being used. Adopt a viable, cost-effective methodology o Record/Playback is too costly to maintain and is ineffective in the long term o Functional Decomposition method is workable, but is not as cost-effective as a totally data-driven method o Test Plan driven method is more cost-effective Select a tool that will allow you to implement automated testing in a way that conforms to your long-term testing strategy. Make sure the vendor can provide training and support. Page 20 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 4. TEST MANAGEMENT The place of testing in the application lifecycle is beginning to expand. Increasingly more organizations are starting to adopt processes where testing takes place in parallel with application development, starting as soon as the project has commenced. Testing requires a methodical building-block approach that includes requirements definition, planning, design, execution, and analysis – to ensure full coverage, consistency, and reusability of testing assets. Based on the project objectives, test managers can build a master test plan that will communicate the testing strategy, priorities, and objectives to the rest of the team. With the master plan in place, testers can then define test requirements and specific testing goals. Requirements should define exactly what needs to be tested and which objectives should be met – such as performance goals. The aim of a well-designed test management process is to create one central point of control that is accessible to all members of the testing team. 4.1. What is Test Management? Test Management is a method of organizing application test assets and artefacts – such as test requirements, test plans, test documentation, test scripts, and test results – to enable easy accessibility and reusability. Its aim is to deliver quality applications in less time. 4.2. Principles Most organizations don’t have a standard process for organizing, managing, and documenting their testing efforts. Often, testing is conducted as an ad-hoc activity, which changes with every new project. Without a standard foundation for test planning, execution, and defect tracking, testing efforts are nonrepeatable, non-reusable, and difficult to measure. Properly designed test management processes can enable an organization to: Conduct testing processes better, faster, and cheaper Testing without following any planning or design standards can result in the creation of tests, designs and plans that are not repeatable and therefore unable to be reused for future iterations of the test. Perform daily builds and smoke tests A well-defined, systematic approach to testing and a centralized repository for tests, plans, and execution results can significantly increase the accuracy of smoke tests and add value to having frequent builds. Smoke tests generally consist of a suite of tests that can be applied to a newly created build (or even better, be performed automatically by the build system). It should reveal fundamental failures or anything severe enough for testers to reject the release. Manage changing requirements Complete requirements-based testing is the best way to ensure that the finished system meets the user’s needs. Without a test management process that ties test plans to application functional requirements and allows organizations to track requirements changes to test cases Page 21 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published (and vice versa), it is nearly impossible to design tests to verify that the system contains specified functionality. Implement global testing Cost considerations, higher availability of qualified testers, and better retention of skilled employees force companies to turn to a distributed testing model. Without a clearly defined testing process and an easy-to-use, intuitive collaboration method, any attempt to set up a geographically distributed “virtual” testing team will likely bring more problems than benefits. Organize many tests, many projects Effective testing requires a systematic process. When a testing team is handling multiple projects – each with its own cycles and priorities – it is easy to lose focus. To keep track of multiple test cases, testers need a process that will allow them to manage multiple projects and clearly define the objectives of each one. Conduct more than bug tracking There is much more to the test process than recording bugs and passing them to Research & Development (R&D). Testing today focuses on verifying that the application’s design and functionality meet the business constraints and user requirements. To successfully achieve these goals, a testing process requires clearly defined system specifications and application business rules. Test early in the lifecycle Testing no longer fits between the end of development and the beginning of implementation. Early problem detection (which is also up to 10 times cheaper than finding issues at the end of the lifecycle) requires that testing and development begin simultaneously. However, for testing to be implemented earlier in the lifecycle, testers need a clearly defined set of goals and priorities to help them better understand system design, requirements, and testing objectives. Only a well-defined, structured, and intuitive test management process can help ensure that the testing process meets its goals and contributes to enhancing application quality. Provide organizational visibility into quality status With the increase in both application complexity and importance, quality has become of interest to not only the team or teams directly involved in application delivery, but throughout the organisations. Users, managers, executives, and even external customers all want visibility into the status of quality. These divergent groups need a single, easily accessible, easily digestible summary of information they can use to learn about quality status, and as a guide to necessary action. 4.3. Improving the process No matter what the system does, how it is written, or what platform it is running on, the basic principles of test management are the same. It begins with gathering and documenting the testing requirements and continues through designing and developing tests, running the tests – both manual and automated, functional and load – and analyzing application defects. The testing process is not linear, and naturally it differs depending on each organization’s practices and methodologies. The underlying principles of every testing process, however, are the same. Page 22 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 4.4. Requirements Management The testing process begins with defining clear, complete requirements that can be tested. Requirements development and management play a critical role in both the development and testing of software applications. All too often, requirements are neglected in the testing effort, leading to a chaotic process in which testers fix what they can and accept that certain functionality will not be verified. Quality can be achieved only by testing the system against its requirements. Although unit testing of individual components may be sufficient during the development process, only testing the entire system against its requirements will ensure that the applications functions as expected. Of the many types of requirements, functional and performance requirements are the ones that most often apply to the testing process. Functional requirements help validate that the system contains specified functionality and should be delivered directly from the use cases that developers use to build the system. Performance requirements cover any performance standards and specifications identified by the project team, such as transaction response times or maximum numbers of users. These requirements – also referred to as service-level objectives (SLOs) – are aimed at ensuring that the system can scale to the expected number of users and provide a positive user experience. Both functional and performance requirements are designed to give the testing team a clear, concise, and functional blueprint with which to develop test cases. It is impossible to test everything. Requirements-based testing is one way to help prioritize the testing effort. Requirements can be grouped according to how critical they are to mission success. The ones that affect the core functionality of the system must be extensively tested, while the not-so-critical requirements can be covered by minimal testing efforts or be tested later in a lifecycle. The key to selecting the appropriate tool is the functionality. Commercial tools designed for requirements management are a better choice for organizations that want to create a solid, flexible, requirementsbased testing process. Requirements-based testing will help keep the testing effort on track – even when priorities are shifting, resources become tight, or time for testing runs out. Requirements-based testing is the best way to measure quality against the end-user needs. 4.5. Planning and Running Tests 4.5.1. Planning Tests Comprehensive planning is critical, even with short testing cycles. Planning tests and designing applications simultaneously will ensure a complete set of tests that covers each function the system is designed to perform. If test planning is not addressed until later in the application lifecycle, much of the design knowledge will be lost and testers will have to return to the analysis stage in order to try to recreate what has already been done before. The planning phase is used to define which tests need to be performed, how these tests must be executed, and which risks and contingencies require planning. The following section lists some of the steps involved in the test-planning phase: 1. Set the Ground Rules Page 23 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This stage can be used for setting the ground rules for keeping test logs and documentation, assigning roles within the team, agreeing on the naming convention for tests and defects, and defining the procedure for tracking progress against the project goals. 2. Set Up the Test Environment A test environment needs to be set up to support all testing activities and all testing phases possible. 3. Define Test Procedures During the planning stage, testers determine if tests are going to be manual or automated. If the test is automated, testers need to understand which tools will be used for automation and estimate what techniques and skills will be needed to effectively use these tools. 4. Develop Test Design Test design may be represented as a sequence of steps that needs to be performed to execute a test (in case of a manual test), or a collection of algorithms and code to support more sophisticated, complex tests. During test planning, testers create a detailed description of each test, as well as identify what kind of data or ranges of data are required for the tests. 5. Map Test Data The testing team must understand what types of data need to be obtained to support each type of test, and how this data could be obtained or generated. 6. Design Test Architecture Test architecture helps plan for data dependencies, maps the workflow between tests, and identifies common scripts that can potentially be reused for future testing. 7. Communicate the Test Plan This way, more people in the organization will have visibility into the project and can add their input, questions, or comments before the actual testing begins. 4.5.2. Running Tests To test the system as a whole, testers need to perform various types of testing – functional, regression, load, unit and integration – each with its own set of requirements, schedules, and procedures. During the test-run phase, testers can set up groups of machines to most efficiently use their lab resources. Scheduling automated tests is another way to optimize the use of lab resources, as well as save testers time by running multiple tests at the same time across multiple machines on the network. Having an organized, documented process not only helps with automated tests, it also helps make manual test runs more accurately by providing testers with the clearly defined procedure that specifies the tasks that need to be performed at each step of manual testing. For both manual and automated tests, testers need to keep a complete history of all test runs, creating audit trails to help trace history of tests, test runs and results. Steps involved in running tests include: 1. Create Test Sets Page 24 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published A popular way to manage multiple tests is grouping them into test sets, according to the business process, environment, or feature. Individual tests (both manual and automated) can be assigned to test sets to help testers ensure thorough coverage of each functional area of the application. 2. Set Execution Logic In order to verify application functionality and usability, tests have to realistically emulate the end-user behaviour. To achieve this, test execution should follow the predefined logic, such as running certain tests after other tests have passed, failed, or have completed. The execution logic rules should be set prior to executing the actual tests. 3. Manage Test Resources After the test environment has been set up and configured, testers define which tests or groups of tests should be run on which machines. Managing test lab resources by assigning tests to individual machines helps ensure that hardware and network resources are being used most efficiently and effectively. 4. Run Manual Tests While some manual tests may be routine, they are an essential part of the testing procedure, allowing the tester to verify functionality and conditions that automated tools are unable to handle. 5. Schedule Automated Test Runs Scheduling is another way to avoid conflicts with hardware and system resources – tests can be scheduled to run unattended, overnight, or when the system is in least demand for other tasks. 6. Analyze Test-Run Results During the test-execution phase, testers will uncover application inconsistencies, broken functionality, missing features, and other problems commonly referred to as “bugs” or “defects”. The next step is to view the list of all failed tests and determine what caused the test to fail. If the tester determines that the test failed due to an application defect, this defect has to be reported into the defect tracking system for further investigation, correction, and re-test. 4.6. Issues and Defects Tracking Managing or “tracking” issues and defects is a critical step in the testing process. A well-defined method for defect management will benefit more than just the testing team. Developers, managers, customer support, Quality Assurance, and even Beta customers can effectively contribute to the testing process by having access to an open, easy-to-use, functional defect-tracking system. The key to making a good defect-reporting and resolution process is setting up the defect workflow and assigning permission rules. Extra time spent on documenting the defect and its history is often well rewarded by easier analysis, shorter resolution times, and better application quality. Analyzing defects is what essentially helps managers make the go/no-go decisions about application deployment. Effective management and tracking of issues should address the following steps: 1. Agree on the Naming Convention The key to effective defect management is communication among different parties involved in the process. Before reporting mechanisms can be put into place, the testing team needs to set Page 25 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published the ground rules, such as defining the severity of the bugs and agreeing on what information must be included in the defect report. 2. Establish the Reporting Procedure The key to making a good defect report is supplying developers with as much information as they will need to reproduce and fix the problem. It is critical that a defect report not only provides information describing the bug, but also includes all data necessary to help fix the problem. 3. Set the Permission Rules For each step in the management process, there must be different access privileges. This is especially true in the defect-management stage, since the defect information helps the management make a “go/no-go” decision on the application release. 4. Establish the Process Flow The next step is to have it reviewed by a developer who determines if a reported issue is indeed a defect, and if it is – assigns it a “new” status. Since few development organizations have the bandwidth to repair all known defects, a developer or project manager must prioritize. 5. Re-Test Repairs Whatever fixes or changes have been made to repair a known defect, the application needs to be re-tested to verify that the changes have taken effect and that the fix did not introduce additional problems and unexpected side effects. Besides this regression testing must also be made. 6. Analyze Defects Analyzing defects is the most critical part of the defect-tracking process. It allows testers to take a snapshot of the application under test and view the number of known defects, their status, severity, priority, age, etc. Based on defect analysis, management is then able to make an informed decision as to whether the application is ready to be deployed. Page 26 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 5. PROCESS MODELS Since there isn’t any specific software development process within CGI Portugal, we shall adopt two different approaches: one which is a non-evolutionary model, the V-Model, and a second one which is an evolutionary model, the Rational Unified Process (RUP). The reasons for this choice have to do primarily with the fact that these models enjoy a broad acceptance among software developers, as well as the fact that the models have reached a fair maturity level. Considering these two models, we will then base the testing methodology on a software development process that already exists within CGI, but it is not used at CGI Portugal. This methodology is called Concert, and it is described in the third part of this section. 5.1. The V-Model As you get involved in the development of a new system, a vast number of software tests appear to be required to prove the system. While they are consistent in all having the word “test” in them, their relative importance to each other is not clear. In this point an outline of the various phases of software testing will be given. The main software testing phases are: Unit Integration System Acceptance To put them all in context requires an outline of the development process. 5.1.1. Development Process The development process for a system is traditionally as a Waterfall Model where each step follows the next. This shows how various products produced at each step are used in the process following it. It does not imply that any of the steps in a process have to be completed, before the next step starts, or that prior steps will not have to be revisited later in development. It is just a useful model for seeing how each step works with each of the others. 5.1.1.1. Requirements The first step is to define a set of “Requirements”, which is a statement by the customer of what the system shall achieve in order to meet the need. These involve both functional and nonfunctional requirements. 5.1.1.2. System Specification “Requirements” are then passed to developers, who produce a “System Specification”. This changes the focus from what the system shall achieve it by defining it in computer terms, taking into account both functional and non-functional requirements. Page 27 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 5.1.1.3. System Design Other developers produce a “System Design” from the “System Specification”. This takes the features required and maps them to various components, and defines the relationships between these components. The whole design should result in a detailed system design that will achieve what is required by the “System Specification”. 5.1.1.4. Detailed Design Each piece of coding then has a “Detailed Design”, which describes in detail exactly how it will perform its piece of processing. 5.1.1.5. Coding Finally each piece of coding is built, and then is ready for the test process. The level of test is the primary focus of a system and derives from the way a software system is designed and built up. Conventionally this is known as the V-Model, which maps the types of test to each stage of development. Page 28 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Development Testing Acceptance Testing Requirements System Specification System Testing Integration Testing System Design Detailed Design Unit Testing Coding Figure 1 - The V-Model 5.1.2. Testing Process As you go from Unit Testing all the way up to Acceptance Testing, the detail level of testing decreases, and becomes more business oriented. Page 29 of 101 TEST MANAGEMENT PROJECT Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published System Unit Acceptance Title: Testing Methodology Integration Type: Deliverable Testing phase + Technical + Functional Detail Itemized Module High-level Complete business flow Figure 2 - Testing Phases 5.1.2.1. Unit Testing Starting from the bottom the first test level is “Unit Testing”. It involves checking that each feature specified in the “Detailed Design” has been implemented in the code. In theory an independent tester should do this, but in practice the developer usually does it, as he is the only person who understands how the code works. The problem is that pieces of code perform only a small part of the functionality of a system, and it relies on co-operating with other parts of the system, which may not have been built yet. To overcome this, the developer either builds, or uses special software to trick the code into believing it is working in a fully functional system. 5.1.2.2. Integration Testing As the pieces of code are constructed and tested they are then linked together to check if they work with each other. It is a fact that two pieces of code that have passed all their tests, when connected to each other produce one new piece of code full of potential faults. These tests can be done by specialists, or by the developers. Integration Testing is not focused on what the pieces of code are doing but on how they communicate with each other, as specified in the "System Design". The "System Design" defines relationships between pieces of code, and this involves stating: What a piece of code can expect from another piece of code in terms of services How these services will be asked for How they will be provided Page 30 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published How to handle non-standard conditions, i.e. errors Tests are constructed to deal with each of these aspects. The tests are organized to check all the integrations, until all the code has been built and integrated with each other producing the whole system. 5.1.2.3. System Testing Once the entire system has been built then it has to be tested against the "System Specification" to check if it delivers the features required. In this phase begins the work of system testers, who are responsible for the testing effort from this on. It is not required that system testers have knowledge on programming. System testing tends to be more of an investigatory testing phase, where the focus is to have almost a destructive attitude and test not only the design, but also the behaviour and even the believed expectations of the customer. System testing is intended to test up to and beyond the bounds defined in the software/hardware requirements specification(s). One could view System testing as the final destructive testing phase before Acceptance testing. System testing can involve a number of specialist types of test to see if all the functional and non-functional requirements have been met. In addition to functional requirements these may include the following types of testing for the non-functional requirements: Performance - Are the performance criteria met? Volume - Can large volumes of information be handled? Stress - Can peak volumes of information be handled? Robustness - Does the system remain stable under adverse circumstances? Regression – Does the system work after new changes have occurred? There are many others, the needs for which are dictated by how the system is supposed to perform. 5.1.2.4. Acceptance Testing Acceptance Testing checks the system against the "Requirements". It is similar to system testing in that the whole system is checked but the important difference is the change in focus: System Testing checks that the system that was specified has been delivered. Acceptance Testing checks that the system delivers what was requested. The customer, and not the developer, should always do acceptance testing. The customer knows what is required from the system to achieve value in the business and is the only person qualified to make that judgment. To help them, prior training should be available. Page 31 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 5.2. Rational Unified Process The Rational Unified Process (RUP) is an iterative software development process created by the Rational Software Corporation, now a division of IBM. The RUP is not a single concrete prescriptive process, but rather an adaptable process framework. As such, RUP describes how to develop software effectively using proven techniques. While the RUP encompasses a large number of different activities, it is also intended to be tailored, in the sense of selecting the development processes appropriate to a particular software project or development organization. The RUP is recognized as particularly applicable to larger software development teams working on large projects. Rational Software offers a product – the Rational Unified Process Product – that provides tools and technology for customizing and executing the process. 5.2.1. Overview Figure 3 - The Rational Unified Process Figure 3 shows a high level view of the Process. The chart identifies which disciplines are the most active during each phase of the process. For example, the red shape labelled Business Modelling shows heavy activity only in the Inception and Elaboration phases, whereas the blue shape representing Project Management shows a more graduated activity over the life of the Process, time and the dynamic aspects of the RUP. From this point of view the process is described in cycles, phases, iterations, and milestones. Using the RUP, software product lifecycles are broken into individual development cycles. These cycles are further broken into their main components, called phases. In RUP, these phases are: Inception Page 32 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Elaboration Construction Transition Phases are composed of iterations. Iterations are timeboxes; iterations have deadlines while phases have objectives. 5.2.1.1. The Inception Phase In this phase, the business case which includes business context, success factors (expected revenue, market recognition, etc.), and financial forecast is established. To complement the business case, a basic use case model, project plan, initial risk assessment and project description (the core project requirements, constraints and key features) are generated. After these are completed, the project is checked against the following criteria: Stakeholder concurrence on scope definition and cost/schedule estimates. Requirements understanding as evidenced by the fidelity of the primary use cases. Credibility of the cost/schedule estimates, priorities, risks, and development process. Depth and breadth of any architectural prototype that was developed. Actual expenditures versus planned expenditures. If the project does not pass this milestone, called the Lifecycle Objective Milestone, it can either be cancelled or it can repeat this phase after being redesigned to better meet the criteria. 5.2.1.2. The Elaboration Phase The Elaboration phase is where the project starts to take shape. In this phase, the problem domain analysis is made, and the architecture of the project gets its basic form. This phase must pass the Lifecycle Architecture Milestone by meeting the following criteria: A use-case model in which the use-cases and the actors have been identified and most of the use-case descriptions are developed. The use-case model should be 80% complete. A description of the software architecture in a software system development process. Architecture prototype, which can be executed. Business case and risk list which are revised. A development plan for the overall project. If the project cannot pass this milestone, there is still time for it to be cancelled or redesigned. After leaving this phase the project transits into a high-risk operation where changes are much more difficult and detrimental when made. 5.2.1.3. The Construction Phase In this phase, the main focus goes to the development of components and other features of the system being developed. This is the phase when the bulk of the coding takes place. Page 33 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This phase produces the first external release of the software. Its conclusion is marked by the Initial Operational Capability (IOC) milestone. 5.2.1.4. The Transition Phase In the transition phase the product has moved from the development organization to the end user. The activities of this phase include training of the end users and maintainers and beta testing of the system to validate it against the end users' expectations. The product is also checked against the quality level set in the Inception phase. If it does not meet this level, or the standards of the end users, the entire cycle in this phase begins again. If all objectives are met, the Product Release (PR) milestone is reached and the development cycle ends. 5.2.1.5. Milestones In RUP, there are four major milestones that correspond to the four phases. If the milestone criteria are not met, the project can be stopped or run in a new iteration to revisit the bottlenecks. This meta-model of a milestone emphasizes the links between phases, iterations and milestone completion. It is very important to reach the milestone. 5.2.1.6. Iterations A typical project using the RUP will go through several iterations. Dividing the project into iterations has advantages, such as risk mitigation, but it also needs more guidance and effort than the traditional sequential approach. The RUP defines a Project Management Discipline that guides the project manager through iteration management. Using iterations, a project will have one overall phase plan, but multiple iteration plans. In RUP all activities are organized into nine disciplines: Engineering Business Modelling Requirements Analysis & Design Implementation Test Deployment Supporting Configuration and Change Management Project Management Page 34 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Environment 5.2.2. Best Practices of RUP 1. Develop software iteratively 2. Manage requirements 3. Use component-based architecture 4. Visually model software 5. Verify software quality 6. Control changes to software 5.2.2.1. Develop software iteratively Given the time it takes to develop large, sophisticated, software systems, it is not possible to define the problem and build the solution in a single step. Requirements will often change throughout a project's development, due to architectural constraints, customer's needs or a greater understanding of the original problem. Iteration allows the project to be successively refined and addresses a project's highest risk items as the highest priority task. Ideally, each iteration ends up with an executable release – this helps reduce a project's risk profile, allows greater customer feedback and helps developers stay focused. The RUP uses iterative and incremental development for the following reasons: Integration is done step by step during the development process, limiting it to fewer elements. Integration is less complex, making it more cost effective. Parts are separately designed and/or implemented and can be easily identified for later reuse. Requirement changes are noted and can be accommodated. Risks are attacked early in development since each iteration gives the opportunity for more risks to be identified. Software architecture is improved by repeated scrutiny. 5.2.2.2. Manage requirements Requirements Management in RUP is concerned with meeting the needs of end users by identifying and specifying what they need and identifying when those needs change. Its benefits include the following: The correct requirements generate the correct product; the customer’s needs are met. Necessary features will be included, reducing post-development cost. Page 35 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 5.2.2.3. Use component-based architecture Component Based Architecture creates a system that is easily extensible, intuitively understandable and promotes software reuse. A component often relates to a set of objects in Object-oriented programming. Software Architecture is increasing in importance as systems are becoming larger and more complex. RUP focuses on producing the basic architecture in early iterations. This architecture then becomes a prototype in the initial development cycle. The architecture evolves with each iteration to become the final system architecture. RUP also asserts design rules and constraints to capture architectural rules. By developing iteratively it is possible to gradually identify components which can then be developed, bought or reused. These components are often assembled within existing infrastructures such as CORBA, COM, or J2EE. 5.2.2.4. Visually model software Abstracting your programming from its code and representing it using graphical building blocks is an effective way to get an overall picture of a solution. Using this representation, technical resources can determine how best to implement a given set of inter-related logics. It also builds an intermediary between the business process and actual code through information technology. A model in this context is a visualization and at the same time a simplification of a complex design. RUP specifies which models are necessary and why. The Unified Modelling Language (UML) can be used for modelling Use-Cases, Class diagrams and other objects. RUP also discusses other ways to build models. 5.2.2.5. Verify software quality Quality assessment is the most common failing point of all software projects, since it is often an afterthought and sometimes even handled by a different team. RUP assists in planning quality control and assessment by building it into the entire process and involving all members of a team. No worker is specifically assigned to quality; RUP assumed that each member of the team is responsible for quality. The process focuses on meeting the expected level of quality and provides test workflows to measure this level. 5.2.2.6. Control changes to software In all software projects, change is inevitable. RUP defines methods to control, track and monitor changes. RUP also defines secure workspaces, guaranteeing a software engineer's system will not be affected by changes in another system. This concept ties in heavily with component based architectures. With the iterative approach, the need for change management is even more necessary because of the sheer volume of artefacts developed. These artefacts will also need to be updated as the iterations evolve. The Change Management workflow in RUP deals with three specific areas: Configuration Management Change Request Management Status and Measurement Management Configuration Management Page 36 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Configuration management is responsible for the systematic structuring of the products. Artefacts such as documents and models need to be placed under version control and these changes must be visible. It also keeps track of dependencies between artefacts so all related articles are updated when changes are made. Change Request Management During the system development process many artefacts with several versions exist. CRM keeps track of the proposals for change. Status and Measurement Management Change requests have states such as new, logged, approved, assigned and complete. A change request also has attributes such as root cause, or nature (like defect and enhancement), priority, etc. These states and attributes are stored in a database, so useful reports about the progress of the project can be produced. 5.3. Concert The Concert methodology is a result of the reflection and professional experience of CGI Subject Matter Experts from all areas of system development and integration. It is also based on industry best practice concepts (ISO-12207, IEEE) and methodologies, and is structured to best support CGI’s approaches to systems development. Concert provides developers and managers with a life cycle model for IS/IT solution development and delivery, and it establishes a standard approach to executions of related activities. In addition, It outlines the objectives, the content and the result of each of the activities; It describes the framework within which the approach has to be positioned; It integrates and associates in a coherent ensemble, fundamental methodology and best practice development techniques. These include: o Recommendations, guides, best practice approaches, templates and other job aids to promote and support managing requirements in an effective and compliant manner; o Procedures and references to enable effective interfaces with: Project stakeholders; Other CGI process areas; Solution components and services provided by various CGI support groups. The methodology consists of a comprehensive set of processes. A project team or organization depending on its purpose, will select the appropriate subset as a toolkit to fulfil that purpose, project or application. For more on Concert, check the website on CGI’s intranet: (http://www.intranet.cgi.ca/Concert/index_en.htm?%1%2ConcertHome_en.htm). If you prefer, you can see an overview of Concert in Appendix 3. Page 37 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published System Development Methodology Preliminary Analysis Integration and System Test Analysis Client Acceptance Test Design Construction Implementation Deployment Synchronization Points Related Processes Project Management Quality Assurance Configuration Management Reuse Management Figure 4 - CONCERT System Development and Related Process 5.3.1. Basic Principles The life cycle model contained in Concert is based on principles that result from extensive experience in systems development and management, as well as industry best practices. These principles advocate: A phased approach to development: Each phase is designed so that the information concerning decisions to be made is produced at the right moment, and so as to ensure that the solution will be constructed effectively, without losing sight of compliance requirements and of organization and financial management issues. The need for a flexible, adaptable and scalable approach: The methodology contains a flexible structure easily adaptable to any particular context which becomes an integration tool based on a set of recognized principles while not imposing needless constraints. Independent of technology: The methodology should provide a framework for the full life cycle of systems development independent of technology, platforms, vendors, etc. Client participation: Client participation is essential for the successful development of an information system. Our approach requires that everyone concerned or affected, must participate actively in activities at the right/appropriate moment in time. Effective use of models: Models supply a framework integrating the use of specific techniques, tools and norms, such as fourth-generation languages, mechanized configuration management, and prototyping Page 38 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published tools. They can be adapted to each organization’s unique context, and can also evolve with the progress of technology and software engineering best practice. Clear identification of deliverables: An accurate identification of deliverables facilitates management and control system development projects. Templates are provided to facilitate more rapid and consistent and professional production of these components. Effective interfaces with related processes: In order to facilitate the work to be done in each process area, each activity in Concert includes identification of related impacts on project management, on quality assurance, on configuration management and reuse management. 5.3.2. Concert Phases The method for development and implementation of a system is divided into eight phases: Phases Objective Preliminary Analysis To recommend a viable information system solution based on the client’s requirements, constraints and compliance with both applicable business and technological directions. Analysis To translate requirements into specifications needed for the development and implementation of an IS/IT solution. To develop strategies to optimally deliver client requirements. Design To produce the detailed architecture and design for the IS/IT solution. Develop plans in accordance with the strategies established during the Analysis Phase. Construction To produce executable software components that properly reflect the design in accordance with the Construction Plan. Integration and System Test To construct the system by progressively adding increments and testing each resulting assembly to ensure it operates properly; On completion of the integration of required components, complete testing of all system components to verify that they execute properly, and interface properly among themselves and with related applications. Client Acceptance Test To demonstrate that the system meets all client acceptance criteria. Implementation To make the solution available to the end users and ensure they can assume ownership. Deployment To manage planning and execution of Implementation Phase activities to enable rollout to multiple sites. Page 39 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6. TESTING PROCEDURES As it was stated in the previous section, we shall base our testing procedure on the Concert methodology. For that we will map the testing process in Concert with a testing process for the V-Model and another for RUP. In order to do this, first we have to take a look at the testing process in Concert, which is shown in the figure below. Overall Test Strategy Overall Test Plan Detailed Test Plans (Unit, Integration, System, Client Acceptance*) Unit Test Report Preliminary Analysis Analysis Design Construction Integration and System Test Client Acceptance Test Implementation Deployment Integration Test Report System Test Report Client Acceptance Test Report* (* = involves client participation) Figure 5 - Testing Process in CONCERT Considering the figure, we then have the following table, which map the various phases to its testing results. Phases Testing Results Preliminary Analysis None Analysis Overall Test Strategy Design Overall Test Plan Unit Plan Integration Test Plan System Plan Client Acceptance Test Plan Construction Unit Test Report Integration and System Test Integration Test Report System Test Report Client Acceptance Test Client Acceptance Test Report Page 40 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Implementation None Deployment None 6.1. Testing procedure for the V-Model For the V-Model the testing process will be very similar to testing process in Concert. Comparing Figure 5 with the figure below we can see that, in both, the testing cycle goes side by side with the development cycle. Development Testing Acceptance Testing Requirements Overall Test Strategy System Specification Overall Test Plan Acceptance Test Plan System Test Plan Integration Test Plan System Testing Integration Testing System Design Unit Test Plan Detailed Design Unit Testing Acceptance Test Report System Test Report Integration Test Report Unit Test Report Coding Figure 6 - Testing Process in the V-Model Considering the figure, each phase of the V-Model will include the following testing activities: Page 41 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Phases Testing Activities Requirements Prepare Acceptance Test Plan (Specify Acceptance Test Cases) System Specification Prepare Overall Test Strategy Prepare System Test Plan (Specify System Test Cases) System Design Prepare Overall Test Plan Prepare Acceptance Test Plan (Refine specified test cases and perform all other tasks) Prepare System Test Plan (Refine specified test cases and perform all other tasks) Prepare Integration Test Plan Detailed Design Coding Prepare Unit Test Plan None Unit Testing Perform Unit Tests Integration Testing Perform Integration Tests System Testing Perform System Tests Acceptance Testing Perform Acceptance Tests About this table there are a few notes that must be taken into consideration. The Prepare Acceptance Test Plan and Prepare System Test Plan activities are divided into two phases. Thus, during the Requirements phase the first task of the Prepare Acceptance Test Plan activity – Specify Acceptance Test Cases – is made. This will then serve as input for the other tasks, as well as increase the quality of the corresponding deliverable, which is only written in the System Design phase. Similarly to this, the Prepare System Test Plan activity is divided into the System Specification phase, where the Specify System Test Cases task is performed, and into the System Design phase, with all other tasks. Page 42 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.2. Testing procedure for Rational Unified Process (RUP) If you are using an evolutionary model, such as RUP, the testing activities will be performed along with the interactions for each of the process’ phases. Again we will base the testing procedure on Concert, and map the activities with the four phases of the RUP. Thus, considering the testing activities in Figure 5 we can extrapolate the activities for RUP. Figure 7 - Testing in RUP Below we can find which activities are performed in each of the RUP’s phases. 6.3. Phases Testing Activities Inception None Elaboration Prepare Overall Test Strategy Prepare Overall Test Plan Prepare Unit Test Plan Prepare Integration Test Plan Prepare System Test Plan Prepare Acceptance Test Plan Construction Perform Unit Tests Perform Integration Tests Perform System Tests Transition Perform Acceptance Tests Page 43 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.3. Test Management This methodology supports the premise that testing must be managed as a project within the project, using standard project management disciplines. Thus, test management tasks for this methodology address: preparation of test strategy and plan, risk assessment, problem escalation procedures, organization of the test team, test metrics, defects tracking, and verification of entry/exit criteria for the test levels and sub-levels. Test Management Test Strategy and Plan Management Risk Assessment Test Team Organization Defect Tracking/ Severity Levels Test Metrics Entry / Exit Criteria Figure 8 - Test Management Areas 6.3.1. Test Strategy and Plan Before construction begins, an Overall Test Strategy and an Overall Test Plan are created. These two documents go hand in hand. The test strategy defines the approach to be used to achieve the test objectives. The test plan quantifies the strategy in terms of type, amount, and timing of resources (both human and machine) required. The Overall Test Strategy is a high level system-wide expression of major activities which achieve the overall desired result as expressed by the testing objectives. The Overall Test Plan addresses the overall project at a high level. 6.3.2. Management Test Management tasks provide guidelines on overall testing including "how-to" navigate testing and reporting processes through the development activities. Management tasks include also defining: how test scripts will be stored and accessed, the progress reporting process, and the severity levels of defects. Page 44 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.3.3. Test Team Organization The roles of the resources required to support test activities in this methodology include: Project Manager, Technical Leader, System Analyst, System Administrator, Tester, Developer and Client Representative. The roles and the responsibilities of the test team members are described in a table provided in Appendix 4. 6.3.4. Test Metrics Metrics allow the project manager to know the current level of quality and how much work remains to be done to achieve a given target level of quality. Metrics also allow the project manager to make decisions about whether to continue testing or not. Metrics are used to assess each level of testing (unit, integration, system and acceptance) during the project. Testing metrics referenced in this methodology are: Test progress: specifies the number of test cases planned versus executed; Test success: specifies the number of test cases passed versus failed; Test sub-level metrics: specify for each level of testing if target levels of quality of its sublevels are reached; Test variance impact: specifies the number and the rate of defects by status and severity reported during the execution of testing; Defect detection: graphically represented, it specifies the number of defects by severity reported over time during the execution of testing. The project manager should control and maintain, or delegate the responsibility of maintaining and collecting the testing metrics for the project. 6.3.5. Risk Assessment The risks associated with the Information System are assessed to: Identify high-risk applications so that more extensive testing will be planned and performed for these applications. Risks include new and immature technology, new and difficult to understand business requirements, complexity of business logic, complexity of technology, size of the project, and skill mix on the project. Identify the critical components and/or focus areas for testing that are most important for the system from the client standpoint. Critical areas include security, usability, maintainability, reliability, and performance. Effective testing will result in effective risk management. Managing risks follows the steps described below: Identify the risks; Evaluate the risks and the cost of failure; Identify and sequence test focus areas that address or minimize the identified risks; Document the considerations which led to the selection and priority of test focus areas. Page 45 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.3.6. Defect Tracking and Severity Levels The detected defects should be described clearly and precisely. A defect can exist in three states: Open: a defect that has been discovered but has not been fixed yet Fixed: a defect that has been opened and fixed but not yet re-tested Closed: a defect that has been opened, fixed and re-tested The resolution of defects is prioritized using the following three levels of importance or severity: Severity 1: a problem that must be fixed and for which no work around is available; Severity 2: a problem which must be resolved quickly and for which a work around is available; Severity 3: all other problems. 6.3.7. Entry/Exit Criteria The movement of an application from one test to another is controlled through the use of entry and exit criteria. Entry criteria are used to determine when the system components are ready to be tested and provide answers to the following: Has the component reached a stage where it can be tested? Is there enough information available to properly test it? Exit criteria are used to determine if the application has been successfully tested for a given level and if the component can proceed to the next test level. Exit criteria are usually different for each test level to ensure appropriate metrics are gathered and to make an informed business decision on the risk of proceeding to the next level of testing. Page 46 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4. Testing Activities In this section, all the activities that have to be preformed during the testing effort are described. Please note that the order in which these activities here appear do not correspond to the chronological order in which they are performed. For this please refer to sections 6.1 and 6.2. 6.4.1. Prepare Overall Test Strategy The objective of the Overall Test Strategy is to establish the high level strategy that will govern how the system will be tested. Overall Test Strategy is established to reduce risk, lower costs related to test activities, and ensure successful implementation. This activity allows the project manager to: Assess the risks involved in developing or modifying the application; Identify and sequence test focus areas of the application to address or minimize the identified risks; Establish target levels of quality that need to be met for each focus area of the application. The desired target levels of quality reflect the risk in each test focus area. The higher the target level of quality to be reached for a given focus area, the more testing needs to be performed in that area; Select the test levels to be used during the project; Select the sub-levels of unit, integration, system, and acceptance tests to achieve the established target levels of quality; Provide an initial estimate of the testing activities duration; Produce a document that includes the elements listed above. 6.4.1.1. Tasks Prioritize Test Focus Areas To assess the risks associated with the system, and to identify and sequence the application areas where the test should be focused to address the risks. Establish Target Levels of Quality To establish target levels of quality that need to be met to ensure successful implementation of the system. Select Test Sub-levels To identify the different test sub-levels that will be used during the project. Estimate Test Activities To provide an initial estimate of the duration of the testing activities. 6.4.1.1.1. Prioritize Test Focus Areas The objective for this task is to assess the risks associated with the Information System, and to identify and sequence the application areas where the test should be focused to address the risks. Page 47 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This task is performed to ensure that all resources assigned to the testing activities are allocated to achieve the best results. It also assesses risks to help form test focus areas. Some risks are: New and immature technology; New and difficult to understand business requirements; Complexity of business logic; Complexity of technology; Size of the project; Skill mix on the project. This task selects and prioritizes the areas where the testing effort should be focused, taking the following factors into account: Security; Usability; Maintainability; Reliability; Performance. Steps Identify the risks. Evaluate the risks and the cost of failure. Identify and sequence test focus areas that address or minimize the identified risks. Document the considerations that led to the selection and priority of test focus areas. Skill Profile Technical Leader 6.4.1.1.2. Establish Target Levels of Quality The objective for this task is to establish target levels of quality that need to be met to ensure a successful implementation of the system. The target levels of quality are used to determine the amount of testing needed for each focus area within the software metrics and quality management. The desired target level of quality reflects the risk in each test focus area. The higher target level of quality to be reached for a given focus area, the more testing needs to be performed in that area. Steps Establish a target level of quality for each test focus area. Skill Profile Technical Leader Page 48 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.1.1.3. Select Test Sub-levels The objective for this task is to identify the different test sub-levels that will be used during the project. Test sub-levels are used to structure and break down the test activities in a way that will ensure complete testing of the system. This task selects the test sub-levels to be used for unit, integration, system, and acceptance testing. Unit testing includes: sub-routine testing, coverage testing, path coverage, boundary value testing, control flow testing, statement coverage testing, branch coverage testing and equivalence partitioning. Integration testing includes functional and integration testing. System testing includes: database testing, interface testing, conversion testing, operational testing, existing function and regression testing, stress testing, usability testing, networked system testing, security testing, operability testing, compatibility testing, configuration testing, installation testing, procedure testing, storage testing, backup-recovery testing, cohabitation/reliability testing. Acceptance testing includes documentation testing and regression testing. Steps Identify the unit test sub-levels. Identify the integration test sub-levels. Identify the system test sub-levels. Identify the acceptance test sub-levels. Skill Profile Technical Leader 6.4.1.1.4. Estimate Test Activities The objective of this task is to provide an initial estimate of the duration of the testing activities. This task provides an initial estimate of the time required to do each test activity based on current estimation practices, e.g. historical data and experience. This estimate will be used to plan the test activities for the system. Steps Estimate the duration of each test activity. Skill Profile Project Manager 6.4.1.2. Components Input (Examples) System Analysis Output Overall Test Strategy (Template in Appendix 5) Page 49 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.2. Prepare Overall Test Plan The objective of the Overall Test Plan is to provide an overall plan for managing the testing activities of the system. This activity provides the project manager with a plan to manage the activities of each test level (unit, integration, system and acceptance). The following elements are included in the plan: Criteria that govern the movement of the application from one test level to another; Number of test cases required for each test level and sub-level to meet test objectives; Metrics that the project will produce; Problem escalation procedures to be used; Identification of the required resources for each sub-level within each test level; Identification of the personnel with the necessary skills to prepare the test site and to execute the test activities; Estimates of the duration of the testing activities for all participants; Overall schedule of testing activities. 6.4.2.1. Tasks Define Test Level Entry and Exit criteria To describe the conditions that determine when a component is ready to be tested and whether the component can proceed to the next level. Estimate Required Number of Test cases To provide an initial estimate of the number of test cases required for each test level and sub-level, and to define the metrics that the project will produce. Define Problem Escalation Procedures To define problem escalation procedures to be used during the test activities of the project. Identify Test Environment Requirements To specify the necessary characteristics and properties of the test environment. Refine Test Team Organization To identify the roles and responsibilities of those supporting the test activities. Produce Overall Test Schedule To specify a schedule, including milestones, for testing activities by refining duration estimates. 6.4.2.1.1. Define Test Level Entry and Exit Criteria The objective for this task is to describe the conditions that determine when a component is ready to be tested and whether the component can proceed to the next test level. Entry conditions are used to determine when the system components are ready to be tested and provide answers to the following: Page 50 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Has the component reached a stage where it can be tested? Is there enough information available to properly test it? Exit conditions are used to determine if the application has been successfully tested for a given level and if the component can proceed to the next test level. Exit conditions are usually different for each test level to ensure that appropriate metrics are gathered, and to make an informed business decision on the risk of proceeding to the next level of testing. Steps Define entry conditions for unit, integration, system and acceptance testing. Define exit conditions for unit, integration, system and acceptance testing. Skill Profile Technical Leader 6.4.2.1.2. Estimate Required Number of Test Cases The objective for this task is to provide an initial estimate of the number of test cases required for each test level and sub-level and to define the metrics that the project will produce. The estimate of the number of test cases required (including those for each test sub-level for unit, integration, system and acceptance testing) is used to establish the level of effort and activities needed to test the system. The metrics to assess each level of testing (unit, integration, system and acceptance) during the project consist of the following: Test progress - specifies the number of test cases planned and executed; Test success - specifies the number of test cases passed and failed; Test target levels of quality - specifies if the target level of quality is reached for each test sub-level; Test variance impact - specifies the number and the rate of defects by status and severity reported during the execution of testing; Defect detection - specifies, using a graphic illustration, the number of detected defects versus the number of executed test cases over time. Steps Estimate the number of test cases required for unit testing. Estimate the number of test cases required for each sub-level of unit testing. Estimate the number of test cases required for integration testing. Estimate the number of test cases required for each sub-level of integration testing. Estimate the number of test cases required for system testing. Estimate the number of test cases required for each sub-level of system testing. Estimate the number of test cases required for acceptance testing. Estimate the number of test cases required for each sub-level of acceptance testing. Page 51 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Define the metrics that will be produced for each test level. Skill Profile Technical Leader 6.4.2.1.3. Define Problem Escalation Procedures The objective for this task is to define problem escalation procedures to be used during the test activities of the project. Escalation procedures are used by those involved in test activities to determine the course of actions to follow when they discover anomalies or defects. These are normally linked with the Quality Assurance, Configuration Management and Project Management policies and procedures. Steps Define the escalation procedures to be followed when the test team encounters problems during testing. Skill Profile Technical Leader 6.4.2.1.4. Identify Test Environment Requirements The objective for this task is to specify the necessary characteristics and properties of the test environment. The physical characteristics of the test environments are required to plan test activities. These include hardware, communications, system software, mode of usage (e.g. standalone), and any other software or supplies needed to support testing. In addition, the following requirements must be identified: The level of security that must be provided for the test facilities, system software, and proprietary components such as software, data and hardware; Special testing tools, utilities and facilities needed, and any other items related to testing such as publications, office space, etc.; The source and provisioning activities needed to supply the testers with what they need. Steps Identify the following resources for unit, integration, system and acceptance testing: The required test facilities: hardware, software and network; The test tools and utilities that will be used. Skill Profile Technical Leader Page 52 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.2.1.5. Refine Test Team Organization The objective for this task is to identify the roles and responsibilities of those supporting the test activities. In order to schedule test activities, it is necessary to identify all members (end users, testers, developers, database support, etc.) required to support those activities. Steps Identify the groups responsible for preparing the test environment and facilities. Identify the groups responsible for performing the testing. Skill Profile Project Manager 6.4.2.1.6. Produce Overall Test Schedule To specify a schedule, including milestones, for testing activities by refining duration estimates. Using the information gathered during the Prepare Overall Test Plan activity, the schedule for test activities can be produced by: Refining the estimate of time required to perform each testing task; Defining test milestones that must be met; Specifying the initial schedule for each testing task and milestone; Specifying the periods during which each testing resource (i.e., facilities, tools, and staff) must be available. Steps Refine the duration estimate of each testing activity. Define the test milestones that must be met. Specify the schedule for each testing activity. Specify the schedule for each test milestone. Specify the roles and responsibilities associated with each test activity. Skill Profile Project Manager 6.4.2.2. Components Input (Examples) System Design Overall Test Strategy (Template in Appendix 5) Output Overall Test Plan (Template in Appendix 5) 6.4.3. Prepare Unit Test Plan The objective of the Unit Test Plan is to provide a detailed plan for performing unit test sub-levels. Page 53 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This activity provides the project manager with a plan to manage the execution of unit tests in accordance with the Overall Test Plan. Unit testing is the initial testing of code in a module or subroutine. Unit test sub-levels include tasks to the following: Internal logic; Internal design; Test path and condition coverage; Test exception conditions and error handling. Unit test sub-levels include: sub-routine testing, coverage testing, path coverage testing, control flow testing, boundary value testing, statement coverage testing, branch coverage testing, and equivalence partitioning. The purpose of the unit test sub-levels is to find and fix coding errors and coding omissions in software components, such as modules or subroutines, prior to their integration with other sub-components into the complete package. The plan will provide the following for each component test sub-level, if applicable: Test cases needed; The number of test cases; The test tools and facilities needed (hardware, software, network); Staff resources, their roles and responsibilities required to perform test activities (includes time for education and mentoring if required); A detailed test schedule, including a duration estimate for test activities; Unit testing metrics. The Unit Test Plan must be aligned with all other plans. 6.4.3.1. Tasks Specify Unit Test Cases To specify and sequence the test cases needed to meet the objectives of unit test sub-levels. Update Test Facility Requirements To review and update the test tools and facilities required for unit test sub-levels. Refine Detailed Test Team Organization To identify resources required to perform unit test sub-levels. Produce Detailed Test Schedule To produce a test schedule, including duration estimates for the activities of unit test sub-levels. 6.4.3.1.1. Specify Unit Test Cases The objective for this task is to specify and sequence the test cases needed to meet the objectives of the unit test sub-levels. Unit test cases are scenarios that demonstrate that a unit of code meets the clients' requirements as specified, and validate that the code fulfils its intended use. Page 54 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published It is recommended that test cases be designed when the code is specified, that is, prior to coding. The advantage is that test conditions and cases are: Designed more objectively; Not influenced by coding style; Not overlooked. Steps For each unit test sub-level, design test cases following these practical steps: Derive test scenarios from client requirements; Break down test scenarios into test conditions. Each test condition has a measurable objective and associated test input data. Each test condition normally requires the execution of several test cases; Specify the expected outputs associated with the execution of a test case; Select the test cases that should be included in the regression test package; Prioritize the test cases (high, medium and low). The assigned priority is based on the risk analysis and the test focus areas that were selected, as defined in the Overall Test Strategy. Test cases are run by priority; the highest priority test cases are run first. Once all the unit test case specifications are completed, they are attached to the Unit Test Plan for the execution of unit testing. Skill Profile Technical Leader 6.4.3.1.2. Update Test Facility Requirements The objective of this task is to review and update the test tools and facilities required for each test sub-level. This task allows the technical leader to expand the detail level of the tools and facilities requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps For each test sub-level: Review the requirements for testing tools and facilities identified in the Overall Test Plan; If necessary, update these requirements to reflect the current situation in the appropriate test plans (Unit, Integration, System and Acceptance Test Plans). Skill Profile Technical Leader 6.4.3.1.3. Refine Detailed Test Team Organization The objective for this task is to identify the resources required to perform each test sublevel. Page 55 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This task allows the project manager to expand the detail level of the staff resource requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps Refine the roles and responsibilities of the team members responsible for each test sublevel and review this information with them. Define the internal management structure of the test team to depict the lines of authority, responsibility and communication within the test team. Graphical devices such as hierarchical organization charts or matrix diagrams may be used to describe the organizational structure. Skill Profile Project Manager 6.4.3.1.4. Produce Detailed Test Schedule The objective for this task is to produce a test schedule, including duration estimates for the activities of test sub-levels. This task allows the project manager to expand the detail level of planning requirements identified in the Overall Test Plan for Unit, Integration, System and Acceptance Test Plans. When establishing the schedule, the following should be considered for each test sub-level: Dependencies between test activities; Key dates of test activities; Alignment with all other plans and schedules (initial corrective actions as required). Steps Produce a duration estimate for the activities of each test sub-level to determine the total duration of the appropriate test level. Skill Profile Project Manager 6.4.3.2. Components Input (Examples) Overall Test Strategy (Template in Appendix 5) Overall Test Plan (Template in Appendix 5) Detailed Design Output Unit Test Plan (Template in Appendix 5) 6.4.4. Prepare Integration Test Plan The objective of the Integration Test Plan is to provide a detailed plan for performing integration test sub-levels. This activity provides the project manager with a plan to manage the execution of integration test in accordance with the Overall Test Plan. Integration testing verifies the proper execution of application components that interface with other applications. The communication among software components Page 56 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published (e.g. modules) within the sub-system is tested in a controlled and isolated environment within the project. It does not require the execution of the entire application as a unit, but of individual subsystems within the application. Integration test sub-levels include functional and integration testing. The plan provides the following for each applicable integration test sub-level: Test cases needed; The number of test cases estimated; The test tools and facilities needed (hardware, software, network); Staff resources, their roles and responsibilities required to perform test activities (include time for education and mentoring if required); A detailed test schedule, including duration estimates for test activities; Integration testing metrics. The Integration Test Plan must be aligned with all other plans. 6.4.4.1. Tasks Specify Integration Test Cases To specify and sequence the test cases needed to meet the objectives of integration test sub-levels. Update Test Facility Requirements To review and update the test tools and facilities required for integration test sub-levels. Refine Detailed Test Team Organization To identify resources required to perform integration test sublevels. Produce Detailed Test Schedule To produce a test schedule, including duration estimates for the activities of integration test sub-levels. 6.4.4.1.1. Specify Integration Test Cases The objective for this task is to specify and sequence the test cases needed to meet the objectives of the integration test sub-levels. Integration test cases are created to verify that application components interfacing with other applications execute properly. Steps For each integration test sub-level, design test cases following these practical steps: Derive test scenarios from client requirements; Break down test scenarios into test conditions. Each test condition has a measurable objective and associated test input data. Each test condition normally requires the execution of several test cases; Specify the expected outputs associated with the execution of a test case; Select the test cases that should be included in the regression test package; Page 57 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Prioritize the test cases (high, medium and low). The assigned priority is based on the risk analysis and the test focus areas that were selected, as defined in the Overall Test Strategy. Test cases are run by priority; the highest priority test cases are run first. Once all the integration test case specifications are completed, they are attached to the Integration Test Plan for the execution of integration testing. Skill Profile Tester 6.4.4.1.2. Update Test Facility Requirements The objective of this task is to review and update the test tools and facilities required for each test sub-level. This task allows the technical leader to expand the detail level of the tools and facilities requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps For each test sub-level: Review the requirements for testing tools and facilities identified in the Overall Test Plan; If necessary, update these requirements to reflect the current situation in the appropriate test plans (Unit, Integration, System and Acceptance Test Plans). Skill Profile Technical Leader 6.4.4.1.3. Refine Detailed Test Team Organization The objective for this task is to identify the resources required to perform each test sublevel. This task allows the project manager to expand the detail level of the staff resource requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps Refine the roles and responsibilities of the team members responsible for each test sublevel and review this information with them. Define the internal management structure of the test team to depict the lines of authority, responsibility and communication within the test team. Graphical devices such as hierarchical organization charts or matrix diagrams may be used to describe the organizational structure. Skill Profile Project Manager Page 58 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.4.1.4. Produce Detailed Test Schedule The objective for this task is to produce a test schedule, including duration estimates for the activities of test sub-levels. This task allows the project manager to expand the detail level of planning requirements identified in the Overall Test Plan for Unit, Integration, System and Acceptance Test Plans. When establishing the schedule, the following should be considered for each test sub-level: Dependencies between test activities; Key dates of test activities; Alignment with all other plans and schedules (initial corrective actions as required). Steps Produce a duration estimate for the activities of each test sub-level to determine the total duration of the appropriate test level. Skill Profile Project Manager 6.4.4.2. Components Input (Examples) Overall Test Strategy (Template in Appendix 5) System Design Overall Test Plan (Template in Appendix 5) Output Integration Test Plan (Template in Appendix 5) 6.4.5. Prepare System Test Plan The objective of the System Test Plan is to provide a detailed plan for performing system test sublevels. This activity provides the project manager with a plan to manage the execution of system tests in accordance with the Overall Test Plan. System tests verify that all components (software, hardware and network) of the entire application execute properly. The tests also verify the system's ability to read interfaces from other applications and to generate interfaces to other applications. System test sub-levels include: database testing, interface testing, conversion testing, operational testing, existing function and regression testing, stress testing, usability testing, networked system testing, security testing, operability testing, compatibility testing, configuration testing, installation testing, procedure testing, storage testing, backup-recovery testing, cohabitation/reliability testing. The plan provides the following for each applicable system test sub-level: Test cases needed; The number of test cases; The test tools and facilities needed (hardware, software, network); Staff resources, their roles and responsibilities required to perform test activities (include time for education and mentoring if required); A detailed test schedule, including duration estimates for test activities; Page 59 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published System testing metrics. The System Test Plan must be aligned with all other plans. 6.4.5.1. Tasks Specify System Test Cases To specify and sequence the test cases needed to meet the objectives of system test sub-levels. Update Test Facility Requirements To review and update the test tools and facilities required for system test sub-levels. Refine Detailed Test Team Organization To identify resources required to perform system test sub-levels. Produce Detailed Test Schedule To produce a test schedule, including duration estimates for the activities of system test sub-levels. 6.4.5.1.1. Specify System Test Cases The objective for this task is to specify and sequence the test cases needed to meet the objectives of the system test sub-levels. System test cases are created to verify that all components (software, hardware and network) of the entire application execute properly. Steps For each system test sub-level, design test cases following these practical steps: Derive test scenarios from the client's requirements; Break down test scenarios into test conditions. Each test condition has a measurable objective and associated test input data. Each test condition normally requires the execution of several test cases; Specify the expected outputs associated with the execution of a test case; Select the test cases that should be included in the regression test package; Prioritize the test cases (high, medium and low). The assigned priority is based on the risk analysis and the test focus areas that were selected, as defined in the Overall Test Strategy. Test cases are run by priority; the highest priority test cases are run first. Once all the system test case specifications are completed, they are attached to the System Test Plan so they can be used for the execution of system testing. Skill Profile Tester 6.4.5.1.2. Update Test Facility Requirements The objective of this task is to review and update the test tools and facilities required for each test sub-level. Page 60 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published This task allows the technical leader to expand the detail level of the tools and facilities requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps For each test sub-level: Review the requirements for testing tools and facilities as identified in the Overall Test Plan. If necessary, update these requirements to reflect the current situation in the appropriate test plans (Unit, Integration, System and Acceptance Test Plans). Skill Profile Technical Leader 6.4.5.1.3. Refine Detailed Test Team Organization To identify the resources required to perform each test sub-level. This task allows the project manager to expand the detail level of the staff resources requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps Refine the roles and responsibilities of the team members responsible for each test sublevel and review them with the affected people. Define the internal management structure of the test team to depict the lines of authority, responsibility and communication within the test team (graphical devices such as hierarchical organization charts or matrix diagrams may be used to describe the organizational structure). Skill Profile Project Manager 6.4.5.1.4. Produce Detailed Test Schedule The objective for this task is to produce a test schedule, including duration estimates for the activities of test sub-levels. This task allows the project manager to expand the detail level of planning requirements identified in the Overall Test Plan for Unit, Integration, System and Acceptance Test Plans. When establishing the schedule, the following should be considered for each test sub-level: Dependencies between test activities; Key dates of test activities; Alignment with all other plans and schedules (initial corrective actions as required). Steps Produce a duration estimate for the activities of each test sub-level to determine the total duration of the appropriate test level. Page 61 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Skill Profile Project Manager 6.4.5.2. Components Input (Examples) Overall Test Strategy (Template in Appendix 5) System Analysis Overall Test Plan (Template in Appendix 5) Output System Test Plan (Template in Appendix 5) 6.4.6. Prepare Acceptance Test Plan The objective of the Acceptance Test Plan is to provide a detailed plan for performing the acceptance test sub-levels. This activity provides the project manager with a plan to manage the execution of acceptance testing in accordance with the Overall Test Plan. Acceptance tests verify that the system meets client requirements as specified, and validate that the system fulfils its intended use. Acceptance testing takes place in, or simulates the client’s operational environment, and includes performance and security testing. It demonstrates that the system performs by the sponsor and client expected, so that they may accept it. Acceptance test sub-levels include documentation testing and regression testing. The plan provides the following for each applicable integration test sub-level: Test cases needed; The number of test cases estimated; The test tools and facilities needed (hardware, software, network); Staff resources, their roles and responsibilities required to perform test activities (include time for education and mentoring if required); A detailed test schedule, including duration estimates for test activities; Acceptance testing metrics. The Acceptance Test Plan must be aligned with all other plans. 6.4.6.1. Tasks Specify Acceptance Test Cases Specify and sequence the test cases needed to meet the acceptance test objectives. Update Test Facility Requirements Review and update the test tools and facilities required for acceptance test sub-levels. Refine Detailed Test Team Organization Identify the resources required to perform acceptance tests. Produce Detailed Test Schedule Produce a test schedule, including duration estimates for the activities of acceptance test sub-levels. Page 62 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.6.1.1. Specify Acceptance Test Cases The objective for this task is to specify and sequence the test cases needed to meet the objectives of acceptance test sub-levels. Acceptance test cases are created to verify that the system meets client requirements as specified, and to validate that the system fulfils its intended use. Steps For each acceptance test sub-level, design test cases following these practical steps: Derive test scenarios from the client's requirements; Break down test scenarios into test conditions. Each test condition has a measurable objective and associated test input data. Each test condition normally requires the execution of several test cases; Specify the expected outputs associated with the execution of a test case; Select the test cases that should be included in the regression test package; Prioritize the test cases (high, medium and low). The assigned priority is based on the risk analysis and the test focus areas that were selected, as defined in the Overall Test Strategy. Test cases are run by priority; the highest priority test cases are run first. Once all the acceptance test case specifications are completed, they are attached to the Acceptance Test Plan for the execution of acceptance testing. Skill Profile Tester, Client Representative 6.4.6.1.2. Update Test Facility Requirements The objective of this task is to review and update the test tools and facilities required for each test sub-level. This task allows the technical leader to expand the detail level of the tools and facilities requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps For each test sub-level: Review the requirements for testing tools and facilities as identified in the Overall Test Plan. If necessary, update these requirements to reflect the current situation in the appropriate test plans (Unit, Integration, System and Acceptance Test Plans). Skill Profile Technical Leader Page 63 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.6.1.3. Refine Detailed Test Team Organization The objective for this task is to identify the resources required to perform each test sublevel. This task allows the project manager to expand the detail level of the staff resources requirements identified in the Overall Test Plan for the Unit, Integration, System and Acceptance Test Plans. Steps Refine the roles and responsibilities of the team members responsible for each test sublevel and review them with the affected people. Define the internal management structure of the test team to depict the lines of authority, responsibility and communication within the test team (graphical devices such as hierarchical organization charts or matrix diagrams may be used to describe the organizational structure). Skill Profile Project Manager 6.4.6.1.4. Produce Detailed Test Schedule The objective for this task is to produce a test schedule, including duration estimate for the activities of test sub-levels. This task allows the project manager to expand the detail level of planning requirements identified in the Overall Test Plan for Unit, Integration, System and Acceptance Test Plans. When establishing the schedule the following should be considered for each test sub-level: Dependencies between test activities; Test activities key dates; Alignment with all other plans and schedules (initial corrective actions as required). Steps Produce a duration estimate of the activities for each test sub-level to determine the total duration of the appropriate test level. Skill Profile Project Manager 6.4.6.2. Components Input (Examples) Overall Test Strategy (Template in Appendix 5) System Requirements Overall Test Plan (Template in Appendix 5) Output Acceptance Test Plan (Template in Appendix 5) Page 64 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.7. Perform Unit Tests The objective for this activity is to find, fix and document coding errors and omissions in program sub-components, such as modules or routines, prior to their integration and packaging with other sub-components. Unit testing allows the developer to verify that the unit specification has been correctly translated to the internal logic of the module or routine, and to ensure that the unit is ready for integration testing. Unit testing is the initial testing of new and changed code in a module or subroutine. Unit test sublevels include tasks to: Test the internal logic; Verify the internal design; Test the path and condition coverage; Test exception conditions and error handling. The results of unit testing are documented in the Unit Test Report for future metrics analysis. 6.4.7.1. Tasks Execute Unit Test Plan To confirm the Unit Test Plan and execute all unit test cases. Track defects To report and track the status of defects associated with the execution of each unit test sub-level. Analyze Test Results To analyze and validate the results for each unit test sub-level and document the required actions. Repair Defects and Retest To implement corrective actions documented during testing. 6.4.7.1.1. Execute Unit Test Plan The objective for this task is to confirm the Unit Test Plan and execute all unit test cases. The following describes a typical sequence of unit test activities: 1. Ensure entry conditions are met as defined in the Unit Test Plan; 2. Execute the unit test cases by priority; the highest priority test cases are run first; 3. Record the successful/unsuccessful (pass/fail) execution of unit test cases in the Test Case Log. Steps Review the Unit Test Plan and identify any required additional unit test cases. Complete unit test activities in accordance with the Unit Test Plan. Attach all the Test Case Logs to the Unit Test Report for metrics analysis as defined in the Unit Test Plan. Skill Profile Developer Page 65 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.7.1.2. Track Defects The objective for this task is to report and track the status of defects associated with the execution of each test sub-level. Describe each detected defect in clear, precise terms so that if persons are somewhat unfamiliar with the specific system, or significant time has passed, it can still be understood. If appropriate care is taken in documenting errors, valuable data will be available in the future for analysis which could identify improved methods of developing or maintaining the system. The defect can exist in three states. These are: Open - a defect that has been discovered but has not been fixed yet; Fixed - a defect that has been opened and fixed but not yet re-tested; Closed - a defect that has been opened, fixed and re-tested. All the defects detected during the execution of tests are recorded and attached to the appropriate Test Report. Steps Track defects by updating the Defect Log as follows: Describe each detected defect in clear and precise terms; Record the status of the defect; Record the test case number to which the defect is related. Attach the Defect Log to the appropriate Test Report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.7.1.3. Analyze Test Results The objective for this task is to analyze and validate the results for each test sub-level, and document the required actions. Defects can frequently be grouped into categories, which allows future data analysis of encountered errors. The best time to categorize defects is when they are resolved and the information is still fresh. Possible classifications for error categorization include: Defect type - requirements, specification, design, code, test, etc.; Defect frequency - recurring, non-recurring; Defect severity - the resolution of defects is prioritized according the following levels: o Severity 1: a problem which must be fixed and for which no work-around is available; o Severity 2: a problem which must be resolved quickly and for which a workaround is available; o Severity 3: all other problems. Page 66 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published The severity of the defect and the difficulty of correcting it should be documented to provide a basis for determining resource allocation and scheduling defect correction priorities. Errors are frequently detected faster than they can be resolved, and performing an initial defect analysis can provide valuable information to project management for establishing priorities. Steps Analyze the results for each test sub-level as follows: Compare the expected test results with the obtained test results; Analyze the defects; Prioritize the resolution of the defects using the severity levels; Identify the required actions. It is important to document the correction of the defect to maintain proper configuration accounting. The description of the correction should include: o A narrative description of the correction; o A list of program units affected; o The number, revision, and sections of all documents affected; o Any test procedures changed as a result of the correction. Record the defect severity and the actions in the Defect Log. Attach the Defect Log to the appropriate test level report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.7.1.4. Repair Defects and Retest The objective for this task is to implement corrective actions documented during testing. The resolution of the defects depends on the severity level of the defect (level 1 is resolved first, level 2 second, and level 3 last). Corrective actions usually involve: Adding or changing test cases according to the client authority; Changing or updating code. It is important to record when the implementation was corrected, to identify which error was corrected in which version of software. The description of the implementation should include: o The software version in which the correction was incorporated; o The authority for incorporating the correction. Regression testing is used to help verify the changes. Re-testing the affected function is necessary after the change is incorporated, since as many as 20 percent of all corrections result in additional errors. In the event that latent defects were introduced by the correction, one method of resolution would be to treat them as new errors and initiate a new error report. The description of regression testing should include: A list of test paragraphs/objectives to be re-tested; The version of software used to perform regression test; Page 67 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Indication of successful/unsuccessful accomplishment of test. Steps Change test cases, code or both. Re-test the affected components. Document the changes. Ensure exit conditions are met. Skill Profile Developer, Tester 6.4.7.2. Components Input Unit Test Plan (Template in Appendix 5) Output Unit Test Report (Template in Appendix 5) 6.4.8. Perform Integration Tests The objective of Integration Tests is to integrate software components and then find, fix and document defects related to how the modules (including sub-systems) interface, to ensure they are ready for system testing. As defined during a design phase, software components are integrated with each other to ensure they interact as expected. This activity involves: Performing the steps in the integration plan; Executing related integration tests after the addition of each increment. The integration tests allow the tester to verify the proper execution (correct and timely) of application components that interface with other applications. The communication between modules within the sub-system is tested in a controlled and isolated environment within the project. It does not require the execution of the entire application as a unit, but of individual sub-systems within the application. The results of integration testing are documented in the Integration Test Report. 6.4.8.1. Tasks Integrate Software Components To assemble software components incrementally in accordance with what was defined during System Design. Execute Integration Test Plan To confirm the Integration Test Plan and execute all integration test cases. Track defects To report and track the status of defects associated with the execution of each integration test sub-level. Analyze Test Results To analyze and validate the results for each integration test sublevel and document the required actions. Repair Defects and To implement corrective actions documented during testing. Page 68 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Retest 6.4.8.1.1. Integrate Software Components The objective for this task is to assemble software components incrementally in accordance with what was defined during a design phase. During a design phase, a detailed technical framework for the construction, integration and testing of the system should be provided. This framework: Provides details of required component development, modification and integration; Identifies the sources of components to be acquired or reused; Contains a detailed plan to integrate the contents of each release. This activity performs the steps in the Integration Test Plan. In most cases, i.e. when there is more than one element to be added incrementally to another, this task will be executed repeatedly along with the Execute Integration Test Plan task, until the last increment is added to the target release. Each assembled set of software must be identified in accordance with naming conventions established for the project. Steps For the framework: Review dependencies and sequences, and update if necessary. For each planned system integration unit: Confirm entry criteria for each component to be integrated; Add the components to the previously tested set; Create or modify the required sub-components; Compile and link all software components; Perform applicable integration tests for the system integration unit. For each planned version: Confirm entry criteria for each system integration unit set to be integrated; Add all tested system integration unit components to the previously tested set; Create or modify the required sub-components; Compile and link all software components; Perform applicable integration tests for the version. For the assembled release: Confirm entry criteria for each version set to be integrated; Add all tested version components to the previously tested set; Create or modify the required sub-components; Compile and link all software components; Page 69 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Perform applicable integration tests for the complete release. Skill Profile Developer 6.4.8.1.2. Execute Integration Test Plan The objective for this task is to confirm the Integration Test Plan and execute all integration test cases. The following describes a typical sequence of integration test activities: 1. Ensure entry conditions are met as defined in the Integration Test Plan; 2. Execute the integration test cases by priority; the highest priority test cases are run first; 3. Record the successful/unsuccessful (pass/fail) execution of integration test cases in the Test Case Log. Steps Review the Integration Test Plan and identify any required additional integration test cases. Complete integration test activities in accordance with the Integration Test Plan. Attach the Test Case Log to the Integration Test Report for metrics analysis as defined in the Integration Test Plan. Skill Profile Tester 6.4.8.1.3. Track Defects The objective for this task is to report and track the status of defects associated with the execution of each test sub-level. Describe each detected defect in clear, precise terms so that if persons are somewhat unfamiliar with the specific system, or significant time has passed, it can still be understood. If appropriate care is taken in documenting errors, valuable data will be available in the future for analysis which could identify improved methods of developing or maintaining the system. The defect can exist in three states. These are: Open - a defect that has been discovered but has not been fixed yet; Fixed - a defect that has been opened and fixed but not yet re-tested; Closed - a defect that has been opened, fixed and re-tested. All the defects detected during the execution of tests are recorded and attached to the appropriate Test Report. Steps Track defects by updating the Defect Log as follows: Describe each detected defect in clear and precise terms; Record the status of the defect; Page 70 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Record the test case number to which the defect is related. Attach the Defect Log to the appropriate Test Report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.8.1.4. Analyze Test Results The objective for this task is to analyze and validate the results for each test sub-level, and document the required actions. Defects can frequently be grouped into categories, which allow future data analysis of encountered errors. The best time to categorize defects is when they are resolved and the information is still fresh. Possible classifications for error categorization include: Defect type - requirements, specification, design, code, test, etc.; Defect frequency - recurring, non-recurring; Defect severity - the resolution of defects is prioritized according the following levels: o Severity 1: a problem which must be fixed and for which no work-around is available; o Severity 2: a problem which must be resolved quickly and for which a workaround is available; o Severity 3: all other problems. The severity of the defect and the difficulty of correcting it should be documented to provide a basis for determining resource allocation and scheduling defect correction priorities. Errors are frequently detected faster than they can be resolved, and performing an initial defect analysis can provide valuable information to project management for establishing priorities. Steps Analyze the results for each test sub-level as follows: Compare the expected test results with the obtained test results; Analyze the defects; Prioritize the resolution of the defects using the severity levels; Identify the required actions. It is important to document the correction of the defect to maintain proper configuration accounting. The description of the correction should include: o A narrative description of the correction; o A list of program units affected; o The number, revision, and sections of all documents affected; o Any test procedures changed as a result of the correction. Record the defect severity and the actions in the Defect Log. Attach the Defect Log to the appropriate test level report for metrics analysis as defined in the appropriate Test Plan. Page 71 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Skill Profile Developer, Tester 6.4.8.1.5. Repair Defects and Retest The objective for this task is to implement corrective actions documented during testing. The resolution of the defects depends on the severity level of the defect (level 1 is resolved first, level 2 second, and level 3 last). Corrective actions usually involve: Adding or changing test cases according to the client authority; Changing or updating code. It is important to record when the implementation was corrected, to identify which error was corrected in which version of software. The description of the implementation should include: o The software version in which the correction was incorporated; o The authority for incorporating the correction. Regression testing is used to help verify the changes. Re-testing the affected function is necessary after the change is incorporated, since as many as 20 percent of all corrections result in additional errors. In the event that latent defects were introduced by the correction, one method of resolution would be to treat them as new errors and initiate a new error report. The description of regression testing should include: A list of test paragraphs/objectives to be re-tested; The version of software used to perform regression test; Indication of successful/unsuccessful accomplishment of test. Steps Change test cases, code or both. Re-test the affected components. Document the changes. Ensure exit conditions are met. Skill Profile Developer, Tester 6.4.8.2. Components Input (Examples) Construction Plan Software Components Database Components Integration Test Plan (Template in Appendix 5) Integration Test Environments Output Integration Test Report (Template in Appendix 5) Page 72 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 6.4.9. Perform System Tests The objective of System Tests is to find, fix and document defects related to the performance of the system and its interfaces, and to ensure that the system is ready for acceptance testing. System tests allow the tester to verify the correct and timely execution of all components of the application including the ability (read, generate) to interface with other applications. The results of system testing are documented in the System Test Report for future metrics analysis. 6.4.9.1. Tasks Execute System Test Plan To confirm the System Test Plan and execute all system test cases. Track defects To report and track the status of defects associated with the execution of each system test sub-level. Analyze Test Results To analyze and validate the results for each system test sub-level and document the required actions. Repair Defects and Retest To implement corrective actions documented during testing. 6.4.9.1.1. Execute System Test Plan The objective for this task is to confirm the System Test Plan and execute all system test cases. The following describes a typical sequence of system test activities: 1. Ensure entry conditions are met as defined in the System Test Plan; 2. Execute the system test cases by priority; the highest priority test cases are run first; 3. Record the successful/unsuccessful (pass/fail) execution of system test cases in the Test Case Log. Steps Review the System Test Plan and identify any required additional system test cases. Complete system test activities in accordance with the System Test Plan. Attach the Test Case Log to the System Test Report for metrics analysis as defined in the System Test Plan. Skill Profile Tester 6.4.9.1.2. Track Defects The objective for this task is to report and track the status of defects associated with the execution of each test sub-level. Page 73 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Describe each detected defect in clear, precise terms so that if persons are somewhat unfamiliar with the specific system, or significant time has passed, it can still be understood. If appropriate care is taken in documenting errors, valuable data will be available in the future for analysis which could identify improved methods of developing or maintaining the system. The defect can exist in three states. These are: Open - a defect that has been discovered but has not been fixed yet; Fixed - a defect that has been opened and fixed but not yet re-tested; Closed - a defect that has been opened, fixed and re-tested. All the defects detected during the execution of tests are recorded and attached to the appropriate Test Report. Steps Track defects by updating the Defect Log as follows: Describe each detected defect in clear and precise terms; Record the status of the defect; Record the test case number to which the defect is related. Attach the Defect Log to the appropriate Test Report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.9.1.3. Analyze Test Results The objective for this task is to analyze and validate the results for each test sub-level, and document the required actions. Defects can frequently be grouped into categories, which allow future data analysis of encountered errors. The best time to categorize defects is when they are resolved and the information is still fresh. Possible classifications for error categorization include: Defect type - requirements, specification, design, code, test, etc.; Defect frequency - recurring, non-recurring; Defect severity - the resolution of defects is prioritized according the following levels: o Severity 1: a problem which must be fixed and for which no work-around is available; o Severity 2: a problem which must be resolved quickly and for which a workaround is available; o Severity 3: all other problems. The severity of the defect and the difficulty of correcting it should be documented to provide a basis for determining resource allocation and scheduling defect correction priorities. Errors are frequently detected faster than they can be resolved, and performing an initial defect analysis can provide valuable information to project management for establishing priorities. Steps Page 74 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Analyze the results for each test sub-level as follows: Compare the expected test results with the obtained test results; Analyze the defects; Prioritize the resolution of the defects using the severity levels; Identify the required actions. It is important to document the correction of the defect to maintain proper configuration accounting. The description of the correction should include: o A narrative description of the correction; o A list of program units affected; o The number, revision, and sections of all documents affected; o Any test procedures changed as a result of the correction. Record the defect severity and the actions in the Defect Log. Attach the Defect Log to the appropriate test level report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.9.1.4. Repair Defects and Retest The objective for this task is to implement corrective actions documented during testing. The resolution of the defects depends on the severity level of the defect (level 1 is resolved first, level 2 second, and level 3 last). Corrective actions usually involve: Adding or changing test cases according to the client authority; Changing or updating code. It is important to record when the implementation was corrected, to identify which error was corrected in which version of software. The description of the implementation should include: o The software version in which the correction was incorporated; o The authority for incorporating the correction. Regression testing is used to help verify the changes. Re-testing the affected function is necessary after the change is incorporated, since as many as 20 percent of all corrections result in additional errors. In the event that latent defects were introduced by the correction, one method of resolution would be to treat them as new errors and initiate a new error report. The description of regression testing should include: A list of test paragraphs/objectives to be re-tested; The version of software used to perform regression test; Indication of successful/unsuccessful accomplishment of test. Steps Change test cases, code or both. Re-test the affected components. Page 75 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Document the changes. Ensure exit conditions are met. Skill Profile Developer, Tester 6.4.9.2. Components Input (Examples) System Test Plan (Template in Appendix 5) System Test Environments Output System Test Report (Template in Appendix 5) 6.4.10. Perform Acceptance Tests The objective of Acceptance Tests is to demonstrate that the system meets all client acceptance criteria and to record acceptance test results for future analysis. Acceptance tests allow the tester to verify that the system meets the client requirements as specified, and validate that it fulfils its intended use. The acceptance tests are conducted in the client’s operational environment (or a simulated one), and typically include performance, security and documentation testing. The tests will demonstrate that the system performs to client expectations, so that it may be accepted. The results of acceptance testing are documented in the Acceptance Test Report for future metrics analysis. 6.4.10.1. Tasks Execute Acceptance Test Plan To confirm the Acceptance Test Plan and execute all acceptance test cases. Track defects To report and track the status of defects associated with the execution of each acceptance test sub-level. Analyze Test Results To analyze and validate the results for each acceptance test sublevel and document the required actions. Repair Defects and Retest To implement corrective actions documented during testing. 6.4.10.1.1. Execute Acceptance Test Plan The objective for this task is to confirm the Acceptance Test Plan and execute all acceptance test cases. The following describes a typical sequence of acceptance test activities: 1. Ensure entry conditions are met as defined in the Acceptance Test Plan; 2. Execute the acceptance test cases by priority; the highest priority test cases are run first; Page 76 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 3. Record the successful/unsuccessful (pass/fail) execution of acceptance test cases in the Test Case Log. Steps Review the Acceptance Test Plan and identify any required additional acceptance test cases. Complete acceptance test activities in accordance with the Acceptance Test Plan. Attach the Test Case Log to the Acceptance Test Report for metrics analysis as defined in the Acceptance Test Plan. Skill Profile Tester, Client Representative 6.4.10.1.2. Track Defects The objective for this task is to report and track the status of defects associated with the execution of each test sub-level. Describe each detected defect in clear, precise terms so that if persons are somewhat unfamiliar with the specific system, or significant time has passed, it can still be understood. If appropriate care is taken in documenting errors, valuable data will be available in the future for analysis which could identify improved methods of developing or maintaining the system. The defect can exist in three states. These are: Open - a defect that has been discovered but has not been fixed yet; Fixed - a defect that has been opened and fixed but not yet re-tested; Closed - a defect that has been opened, fixed and re-tested. All the defects detected during the execution of tests are recorded and attached to the appropriate Test Report. Steps Track defects by updating the Defect Log as follows: Describe each detected defect in clear and precise terms; Record the status of the defect; Record the test case number to which the defect is related. Attach the Defect Log to the appropriate Test Report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester 6.4.10.1.3. Analyze Test Results The objective for this task is to analyze and validate the results for each test sub-level, and document the required actions. Page 77 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Defects can frequently be grouped into categories, which allow future data analysis of encountered errors. The best time to categorize defects is when they are resolved and the information is still fresh. Possible classifications for error categorization include: Defect type - requirements, specification, design, code, test, etc.; Defect frequency - recurring, non-recurring; Defect severity - the resolution of defects is prioritized according the following levels: o Severity 1: a problem which must be fixed and for which no work-around is available; o Severity 2: a problem which must be resolved quickly and for which a workaround is available; o Severity 3: all other problems. The severity of the defect and the difficulty of correcting it should be documented to provide a basis for determining resource allocation and scheduling defect correction priorities. Errors are frequently detected faster than they can be resolved, and performing an initial defect analysis can provide valuable information to project management for establishing priorities. Steps Analyze the results for each test sub-level as follows: Compare the expected test results with the obtained test results; Analyze the defects; Prioritize the resolution of the defects using the severity levels; Identify the required actions. It is important to document the correction of the defect to maintain proper configuration accounting. The description of the correction should include: o A narrative description of the correction; o A list of program units affected; o The number, revision, and sections of all documents affected; o Any test procedures changed as a result of the correction. Record the defect severity and the actions in the Defect Log. Attach the Defect Log to the appropriate test level report for metrics analysis as defined in the appropriate Test Plan. Skill Profile Developer, Tester, Client Representative 6.4.10.1.4. Repair Defects and Retest The objective for this task is to implement corrective actions documented during testing. The resolution of the defects depends on the severity level of the defect (level 1 is resolved first, level 2 second, and level 3 last). Corrective actions usually involve: Adding or changing test cases according to the client authority; Page 78 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Changing or updating code. It is important to record when the implementation was corrected, to identify which error was corrected in which version of software. The description of the implementation should include: o The software version in which the correction was incorporated; o The authority for incorporating the correction. Regression testing is used to help verify the changes. Re-testing the affected function is necessary after the change is incorporated, since as many as 20 percent of all corrections result in additional errors. In the event that latent defects were introduced by the correction, one method of resolution would be to treat them as new errors and initiate a new error report. The description of regression testing should include: A list of test paragraphs/objectives to be re-tested; The version of software used to perform regression test; Indication of successful/unsuccessful accomplishment of test. Steps Change test cases, code or both. Re-test the affected components. Document the changes. Ensure exit conditions are met. Skill Profile Developer, Tester, Client Representative 6.4.10.2. Components Input (Examples) Acceptance Test Plan (Template in Appendix 5) Acceptance Test Environments Output Acceptance Test Report (Template in Appendix 5) Page 79 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 7. GLOSSARY Black box testing – It is used in computer programming, software engineering and software testing to check that the outputs of a program, given certain inputs, conform to the functional specification of the program; The term black box indicates that the internal implementation of the program being executed is not examined by the tester. For this reason black box testing is not normally carried out by the programmer. In most real-world engineering firms, one group does design work while a separate group does the testing. Also referred as concrete box or functional testing. Initial Operational Capability Milestone – third milestone in RUP, at the end of the construction phase; at this point one decides if the software, the sites, and the users are ready to go operational, without exposing the project to high risks. This release is often called a "beta" release. Lifecycle Architecture Milestone – second milestone in RUP, at the end of the elaboration phase; at this point one examines the detailed system objectives and scope, the choice of architecture, and the resolution of the major tasks. Lifecycle Objectives Milestone – first milestone in RUP, at the end of the inception phase. Product Release Milestone – fourth milestone in RUP, at the end of the transition phase; at this point, one decides if the objectives were met, and if another development cycle should start. In some cases, this milestone may coincide with the end of the inception phase for the next cycle. Test bed - A test bed is a platform for experimentation for large development projects. They allow rigorous testing of scientific theories and new technologies. The term is used across many disciplines to describe a development environment that is shielded from the hazards of testing in a live or production environment. UML – Unified Modelling Language; The Unified Modelling Language is a non-proprietary, object modelling and specification language used in software engineering. UML includes a standardized graphical notation that may be used to create an abstract model of a system: the UML model; while UML was designed to specify, visualize, construct, and document software-intensive systems, UML is not restricted to modelling software. UML has its strengths at higher, more architectural levels and has been used for modelling hardware (engineering systems) and is commonly used for business process modelling, systems engineering modelling, and representing organizational structure. White box testing – It is used in computer programming, software engineering and software testing to check that the outputs of a program, given certain inputs, conform to the structural specification of the program. The term white box indicates that testing is done with a knowledge of the code used to execute certain functionality. For this reason, a programmer is usually required to perform white box tests. Often, multiple programmers will write tests based on certain code, so as to gain varying perspectives on possible outcomes. Also referred to as glass box testing or structural testing. Page 80 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 8. REFERENCES ZAMBELICH, Keith, Totally Data-Driven Automated Testing – A White Paper, Automated Testing Specialists, Inc., 2000 SPILLNER, Andreas, The W-MODEL – Strengthening the Bond Between Development and Test, 2002 Testing Methodology – An Overview, SolutionNET, 2004 Implementing an Effective Test Management Process, Mercury, 2005 Rational Unified Process – Best Practices for Software Development Teams, Rational Software Corporation, 1998 Rational Unified Process Base Plug-in, Version 7.0, CD-ROM, IBM, 2005 Types of testing in the development process including the V model, Coley Consulting, <http://www.coleyconsulting.co.uk/testtype.htm> Cleanroom Software Engineering, The Data & Analysis Center for Software, <http://www.dacs.dtic.mil/databases/url/key.hts?keycode=64> Concert Methodology, CGI, 2000, <http://www.intranet.cgi.ca/Concert/index_en.htm?%1%2ConcertHome_en.htm> Wikipedia contributors, Acceptance test, Wikipedia, The Free Encyclopedia, 15-Dec-2005, <http://en.wikipedia.org/w/index.php?title=Acceptance_test&oldid=31421946> Wikipedia contributors, Black box testing, Wikipedia, The Free Encyclopedia, 29-Dec-2005, < http://en.wikipedia.org/w/index.php?title=Black_box_testing&oldid=33156338> Wikipedia contributors, Load test, Wikipedia, The Free Encyclopedia, 4-Nov-2005, <http://en.wikipedia.org/w/index.php?title=Load_testing&oldid=27326338> Wikipedia contributors, IEEE 829, Wikipedia, The Free Encyclopedia, 15-Dec-2005, <http://en.wikipedia.org/w/index.php?title=IEEE_829&oldid=31450519> Page 81 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Wikipedia contributors, Performance testing, Wikipedia, The Free Encyclopedia, 12-Dec-2005, <http://en.wikipedia.org/w/index.php?title=Performance_testing&oldid=31058312> Wikipedia contributors, Rational Unified Process, Wikipedia, The Free Encyclopedia, 5-Jan-2006, <http://en.wikipedia.org/w/index.php?title=Rational_Unified_Process&oldid=33947191> Wikipedia contributors, Regression testing, Wikipedia, The Free Encyclopedia, 21-Dec-2005, <http://en.wikipedia.org/w/index.php?title=Regression_testing&oldid=32232135> Wikipedia contributors, Smoke test, Wikipedia, The Free Encyclopedia, 27-Dec-2005, <http://en.wikipedia.org/w/index.php?title=Smoke_test&oldid=32831863> Wikipedia contributors, Software testing, Wikipedia, The Free Encyclopedia, 3-Jan-2006, <http://en.wikipedia.org/w/index.php?title=Software_testing&oldid=33717177> Wikipedia contributors, Stress testing, Wikipedia, The Free Encyclopedia, 30-Oct-2005, <http://en.wikipedia.org/w/index.php?title=Stress_testing&oldid=26850644> Wikipedia contributors, System functional testing, Wikipedia, The Free Encyclopedia, 6-Jul-2005, <http://en.wikipedia.org/w/index.php?title=System_functional_testing&oldid=18281463> Wikipedia contributors, System testing, Wikipedia, The Free Encyclopedia, 16-Nov-2005, <http://en.wikipedia.org/w/index.php?title=System_testing&oldid=28467956> Wikipedia contributors, Testbed, Wikipedia, The Free Encyclopedia, 22-Sep-2005, < http://en.wikipedia.org/w/index.php?title=Testbed&oldid=23721460> Wikipedia contributors, Test plan, Wikipedia, The Free Encyclopedia, 23-Dec-2005, <http://en.wikipedia.org/w/index.php?title=Test_plan&oldid=32481848> Wikipedia contributors, Test script, Wikipedia, The Free Encyclopedia, 11-Apr-2006, <http://en.wikipedia.org/w/index.php?title=Test_script&oldid=47995971> Wikipedia contributors, Unified Modeling Language, Wikipedia, The Free Encyclopedia, 12-Jan-2006, <http://en.wikipedia.org/w/index.php?title=Unified_Modeling_Language&oldid=34883276> Wikipedia contributors, V model, Wikipedia, The Free Encyclopedia, 18-Sep-2005, <http://en.wikipedia.org/w/index.php?title=V_model&oldid=23439967> Page 82 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Wikipedia contributors, White box testing, Wikipedia, The Free Encyclopedia, 3-Jan-2006, < http://en.wikipedia.org/w/index.php?title=White_box_testing&oldid=33714644> Page 83 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published APPENDIX 1 – AN EXAMPLE FOR “FUNCTIONAL DECOMPOSITION METHOD” The following steps could constitute a “Post a Payment” Test Case: 1. Access Payment Screen from Main Menu 2. Post a Payment 3. Verify Payment Updates Current Balance 4. Return to Main Menu 5. Access Account Summary Screen from Main Menu 6. Verify Account Summary Updates 7. Access Transaction History Screen from Account Summary 8. Verify Transaction History Updates 9. Return to Main Menu A “Business Function” script and a “Subroutine” script could be written as follows: Payment: Start at Main Menu Invoke a “Screen Navigation Function” to access the Payment Screen Read a data file containing specific data to enter for this test, and input the data Press the button or function-key required to Post the payment Read a data file containing specific expected results data Compare this data to the data which is currently displayed (actual results) Write any discrepancies to an Error Report Press button or key required to return to Main Menu or, if required, invoke a “Screen Navigation Function” to do this. Verify_Acct (Verify Account Summary & Transaction History): Start at Main Menu Invoke a “Screen Navigation Function” to access the Account Summary Read a data file containing specific expected results data Compare this data to the data which is currently displayed (actual results) Write any discrepancies to an Error Report Press button or key required to access Transaction History Read a data file containing specific expected results data Compare this data to the data which is currently displayed (actual results) Write any discrepancies to an Error Report Page 84 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Press button or key to return to Main Menu or, invoke a “Screen Navigation Function” The “Business Function” and “Subroutine” scripts invoke “User Defined Functions” to perform navigation. The “Test Case” script would call these two scripts, and the Driver Script would call this “Test Case” script some number of times required to perform all the required Test Cases of this kind. In each case, the only thing that changes are the data contained in the files that are read and processed by the “Business Function” and “Subroutine” scripts. Using this method, if we needed to process 50 different kinds of payments in order to verify all of the possible conditions, then we would need only 4 scripts which are re-usable for all 50 cases: 1. The “Driver” script 2. The “Test Case” (Post a Payment & Verify Results) script 3. The “Payment” Business Function script 4. The “Verify Account Summary & Transaction History” Subroutine script If we were using Record/Playback, we would now have 50 scripts, each containing hard-coded data, that would have to be maintained. This method, however, requires only that we add the data-files required for each test, and these can easily be updated/maintained using Notepad or some such text-editor. Note that updating these files does not require any knowledge of the automated tool, scripting, programming, etc. meaning that the non-technical testers can perform this function, while one “technical” tester can create and maintain the automated scripts. It should be noticed that the “Subroutine” script, which verifies the Account Summary and Transaction History, can also be used by other test cases and business functions (which is why it is classified as a “Subroutine” script rather than a “Business Function” script) – Payment reversals, for example. This means that if we also need to perform 50 “payment reversals”, we only need to develop three additional scripts. 1. The “Driver” script 2. The “Test Case” (Reverse a Payment & Verify Results) script 3. The “Payment Reversal” Business Function script Since we already had the original 4 scripts, we can quickly clone these three new scripts from the originals (which takes hardly any time at all). We can use the “Subroutine” script (Verify Account Summary & Transaction History) as-is without any modifications at all. If different accounts need to be used, then all we have to do is update the Data-Files, and not the actual scripts. It ought to be obvious that this is a much more cost-effective method than the Record/Playback method. Page 85 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published APPENDIX 2 – AN EXAMPLE FOR “KEY-WORD DRIVEN METHOD” Consider the following example of our previous “Post a Payment” Test Case: COLUMN 1 COLUMN 2 COLUMN 3 COLUMN 4 COLUMN 5 Key_Word Field/Screen Name Input/Verification Data Comment Pass/Fail Start_Test: Screen Main Menu Verify Starting Point Enter: Selection 3 Select Payment Option Action: Press_Key F4 Access Payment Screen Verify: Screen Payment Posting Verify Screen accessed Enter: Payment Amount 125.87 Enter Payment data Payment Method Check Action: Press_Key F9 Process Payment Verify: Screen Payment Screen Verify screen remains Verify_Data: Payment Amount $ 125.87 Verify updated data Current Balance $1,309.77 Status Message Payment Posted Action: Press_Key F12 Return to Main Menu Verify: Screen Main Menu Verify return to Menu Each of the “Key Words” in Column 1 causes a “Utility Script” to be called which processes the remaining columns as input parameters in order to perform specific functions. Note that this could also be run as a manual test. The test engineer must develop and document the test case anyway – why not create the automated test case at the same time? Page 86 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published The data in red indicates what would need to be changed if one were to copy this test case to create additional tests. How does this work? When a Key Word is encountered, a list is created using data from the remaining columns. This continues until a “null” (blank) column-2 is encountered. The “Controller” script then calls a Utility Script associated with the Key-Word, and passes the “list” as an input parameter. The Utility Script continues processing until “endof-list”, and then returns to the “Controller” script, which continues processing the file until “end-of-file” is reached. Page 87 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published APPENDIX 3 – CONCERT METHODOLOGY OVERVIEW 1. INTRODUCTION There is a critical need for a common framework that can be used by IS/IT practitioners to address the creation and management of systems in a consistent and effective manner. The Concert Methodology is a major component of the processes established by CGI to address this need. Concert provides developers and managers with a life cycle model for IS/IT solution development and delivery, and it establishes a standard approach to execution of related activities. The concert methodology facilitate synergy and effective communication among all the participants of a system development project (management, client representatives, and development team members), by giving them a common base of reference. 2. METHODOLOGY In order for systems to meet the needs of the Client while being developed efficiently, the active collaboration of all participants is essential. Problems must be analyzed and addressed in the correct order so that the necessary information to make decisions concerning directions, solutions, and installations is available when required. The structure of this methodology is detailed in the figure below. Figure 1 - Concert System Development and Related Processes Page 88 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published The method for development and implementation of a system is divided into eight phases. Phases Objective Preliminary Analysis To recommend a viable information system solution based on the client's requirements, constraints and compliance with both applicable business and technological directions. Analysis To translate requirements into specifications needed for the development and implementation of an IS/IT solution. To develop strategies to optimally deliver client requirements. Design To produce the detailed architecture and design of the IS/IT solution, Develop plans in accordance with the strategies established during the Analysis Phase. Construction To produce executable software components that properly reflect the design in accordance with the Construction Plan. Integration and System Test To construct the system by progressively adding increments and testing each resulting assembly to ensure it operates properly; On completion of the integration of required components, complete testing of all system components to verify that they execute properly, and interface properly among themselves and with related applications. Client Acceptance Test To demonstrate that the system meets all client acceptance criteria. Implementation To make the solution available to the end users and ensure they can assume ownership. Deployment To manage planning and execution of Implementation Phase activities to enable rollout to multiple sites. Table 1 – Methodology Phases Synchronization Points The development process, as well as each of the related processes has an associated life cycle, that is, they are established during project initiation in response to specified criteria and required activities are executed at appropriate points as the development project progresses. However, none of them can be executed in a completely stand-alone manner, and effective alignment of activities that are specified in physically separate frameworks can often be challenging and time consuming. Concert contains identification of the "appropriate points" where, in order to properly complete an activity, or to most effectively use the information provided by execution of an activity, it is necessary to execute other activities defined within one or more of the related processes. This enables developers responsible for performing any activity defined in Concert to effectively align related components of the Project Management (PM), Quality Assurance (QA), Configuration Management (CM) and Reuse Management (ReM) processes. Page 89 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published In short, Synchronization points are defined development life cycle checkpoints where corresponding activities within one or more of the other processes should take place. At each of these checkpoints, the methodology includes instructions referring to specific aspects of the other process that apply to that development activity. 2.1. Preliminary Analysis This phase is normally performed previous to investing significant effort in developing a system. It is intended to provide decision makers with sufficient information to increase the level of confidence they will have in determining whether to proceed further with the project. This is done by examining the following: The current situation which includes reference to the context and functionality of related systems, user needs/expectations, specific objectives to be achieved and constraints to be respected. Current and planned business/technological direction that impacts the methods and technology that can be considered or used when conceiving the information system solutions. System requirements and constraints as they pertain to the needs and expectations of the client. Type Entry Description Client Requirements Project Constraints and other compliance criteria Exit A set of system requirements that is based on an analysis of the specific intended use of the system, and matches the customer's stated and implied needs; A viable (conceptual) solution to the system requirements, along with estimates of (development and operational) costs and benefits. Table 2 – Preliminary Analysis’s Entry and exit conditions The objective of Preliminary Analysis phase is to recommend a viable information system solution based on the client's requirements, constraints and compliance with both applicable business and technological directions. The work performed in this phase will become valuable input for the subsequent phases and that the level of details needed to convey the information is dependent on the needs of the client's decision makers.~ 2.2. Analysis This phase is performed to refine the work completed during the Preliminary Analysis and will serve to establish the baseline by which the subsequent development phases will be defined. During this phase, the proposed functionality and features of the system are identified and translated into technical terms. Page 90 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published The scope of this phase always includes the development of a System Specification (a detailed specification of what the system will do), followed by the definition of a high level Systems Architecture and the descriptions of the environments needed for the development and implementation activities. Wherever feasible, prototypes are built to assist the client to: Clarify and refine requirements; Confirm selected features of the specified system; Provide immediate and informed feedback on system's usability (e.g. GUI design). Type Description Entry The client's general needs are clearly identified, and have been analyzed within the context of the operational environment associated with the proposed solution. Exit Requirements have been rigorously translated into specifications, and these have been validated with the client; The requirements allocated to all components of the system architecture, and their interfaces have been defined to match the customer's needs; Analyzed, correct and testable software requirements have been identified; The impact of defined software and other system requirements on the operational environment is understood; A relevant software release strategy that defines the priority for implementing software requirements has been developed. Table 3 – Analysis’s Entry and exit conditions The objective of this phase is to translate requirements into specifications needed for the design and construction of an information system solution and to analyze issues related to solution development and delivery (including operational impacts), and to develop appropriate strategies. 2.3. Design The design process enables the development of detailed engineering specifications for all aspects of the system. It addresses four distinct but interrelated sets of requirements: data design, architectural design, interface design, and procedural design. This phase: Completes the external design of the system, by fully and completely defining both interfaces and processes (this includes manual and automated ones), data elements, and network components of the system. When applicable, the work completed for the prototypes is refined into full detailed specifications. Generates detailed designs for: o components of each software item to be developed to ensure that all software requirements are allocated, interfaces external to software items and between software components; Page 91 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published o databases. Type Description System Specification (must be officially approved. Entry System Architecture Specification. The design is traceable to the analysis model, exhibits uniformity and integration and is compliant with applicable standards and requirements; Exit Feasibility of construction, integration and testing and future operation and maintenance has been established. Table 4 – Design’s Entry and Exit conditions This phase’s objective is to produce the detailed architecture and design of the system by performing design activities in accordance with the strategies established during the Analysis Phase and to produce detailed plans for the construction, integration and testing of system components. 2.4. Construction This phase consist in building the physical components (software code, environment setup and related documentation) that will be needed to implement the system. Since unit level testing is best performed by those who developed the code, Unit testing has been included as part of this phase. Typically this phase consists of the following activities: Creation of database and construction environment infrastructure; Coding, programming and debugging; Creation of system control, conversion, and backup/recovery processes; Unit testing, defect reporting and repairing; Technical writing for the supporting documentation. Type Entry Exit Description The System Design, Environment Specification, Operational Design, Construction Plan, and Unit Test Plan components are up to date and complete. Software code has been successfully evaluated for: Traceability and consistency with the requirements and design of the software item; Test coverage of units; Appropriateness of coding methods and standards used. Page 92 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Table 5 - Construction’s Entry and Exit conditions This phase’s objective is to produce system components that properly reflect their design and in accordance with the Construction Plan. 2.5. Integration and System Test This phase consists of two major sets of activities. The first set requires establishing appropriate component integration, and integration test environments, followed by iterative sets of tasks that integrate components and then perform applicable Integration Tests. After that process is completed for all components to be integrated, the next steps are to establish appropriate System Test environments, followed by performance of each specified System Test sublevel. The Construction Plan contains the dependencies and sequences for managing integration. The Integration Test Plan provides details for addressing required Integration Test sub-levels (including testing of new functions). The System Test Plan provides details for addressing required System Test sub-levels (including operability testing). Type Entry Description Software integration plans that include test requirements, procedures, data, responsibilities and schedules; System integration plans that include test requirements, procedures, data, responsibilities and schedules; Satisfactory completion of Unit Test activities; Confirmation that each component (purchased, built, modified, etc.) meets its Specification and Design criteria. Exit All anomalies identified during the testing processes scheduled for this phase have been satisfactorily addressed; An integrated system demonstrating compliance with the system requirements (functional, non-functional, operational and maintenance). Table 6 – Integration’s and system’s test Entry and Exit conditions The objective of Integrations and system Test phase is to construct the system by progressively adding increments and testing each resulting assembly to ensure it operates properly. For the complete testing of all system components it has to verify that they execute properly, and interface properly among themselves and with related applications; and also to verify that all functional, performance, operational and other requirements have been satisfactorily addressed. Page 93 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published 2.6. Client Acceptance Test Client Acceptance Tests are conducted in accordance with the Client Acceptance Test Plan (prepared during the Design phase) and in a manner that will increase the client's confidence level sufficiently to declare the system ready for implementation and deployment. The tests allow the client to verify the system meets the requirements as specified and validate that it fulfils its intended use. Special attention is given to conducting these tests in a manner that replicates the real conditions that will be encountered when the client uses the system. During this phase, the following is performed: Client test environment is established (this may include elements of the actual production environment or simulations of the target production environment); Performance, security and documentation testing to ensure the system meets pass/fail criteria (meets applicable specification, design, and operational requirements) and works as expected (intended use) in accordance with the Client Acceptance Test Plan. Test coverage normally includes software, hardware, training, guides, documentation, and manual procedures; Tracking and resolution of reported defects; Knowledge transfer to enable effective execution of applicable test sub-levels. Type Entry Description All anomalies identified during performance of Integration and System Test processes have been satisfactorily addressed. Client Acceptance criteria have been established to the satisfaction of each stakeholder in this process, and are reflected in an Client Acceptance Test Plan. Exit All anomalies identified during the Client Acceptance Test Plan process have been satisfactorily addressed. Table 7 - Client Acceptance’s Test Entry and Exit conditions This phase’s objective is to demonstrate that the system meets all client acceptance criteria. 2.7. Implementation This phase is performed in accordance with the Delivery Plan and includes activities related to: Releasing the deliverables to the user community; Converting existing data to make it available for the new system; Assisting the client in tuning the new system; Providing training and support for using the system; Installing support facilities to manage system performance requirements. Page 94 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Type Description All anomalies identified during the Client Acceptance Test have been satisfactorily addressed for the components that are being deployed. Entry Customer requirements have been defined to the satisfaction of each stakeholder in this process, and are reflected in the Delivery Plan. Roles and Responsibilities are assigned and understood and required facilities and resources are available. Completion of all tasks required to implement the delivery plan in the designated sites. Exit All anomalies identified during the Implementation Phase have been satisfactorily addressed for the components that are being deployed. Agreement between project stakeholders that development project responsibilities have been satisfactorily addressed. Table 8 – Implementation’s Entry and Exit conditions The objective of Implementation phase is to make the solution available to the client and ensure they can assume ownership. 2.8. Deployment The Implementation Phase made the system available to the first set of selected users. The successive rollout to additional sites, or additional sets of users, is the focus of Deployment Phase activities. It is important that project planning for deployment, explicitly tailors each of the Implementation Phase activities that are to be executed to support the implementation of the system at each site. The deployment must reflect what has been done already and what is required for the designated sites. As a result of the lessons learned from each site, the Delivery Plan is updated when needed. Type Entry Description All anomalies identified during the Client Acceptance Test process have been satisfactorily addressed for the components that are being deployed. All anomalies identified during previous executions of the Implementation process e.g. for other sites have been satisfactorily addressed. Customer requirements have been defined to the satisfaction of each stakeholder in this process, and are reflected in the Deployment Plan. Roles and Responsibilities are assigned and understood and required facilities and resources are available. Exit Completion of all tasks required to implement the deployment plan in the designated sites; Page 95 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Type Description Agreement between project stakeholders that related development responsibilities has been satisfactorily addressed. Table 9 – Deployment’s Entry and Exit conditions The Deployment’s phase objective is to manage planning and execution of Implementation Phase activities to enable rollout to multiple sites. 3. TESTING The testing methodology in Concert is based on Full Life-cycle Testing (FLT) which is a collection of best of sort testing practices being used in industry. It supports the premise that testing must be managed as a project within a project, using standard project management disciplines. In addition to calling for more attention to testing early in the project life cycle, the methodology advocates more use of testing automation throughout the project. It also calls for taking measurements and tracking quality throughout the project so that, at every step of the way, the risks of proceeding to the next phase are known. With its emphasis on disciplined management of the testing process, the testing methodology is fully integrated with Concert's development process activities and tasks. The testing process is spread across the different phases of Concert as shown in the figure below. Figure 2 – Testing Process in Concert Methodology Page 96 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Various components are produced in the different phases to successfully and efficiently manage and execute testing and to document the results. 4. SUMMARY As stated earlier, the most critical aspect of doing things right is having an understanding of what is required to be done. Every project team member is responsible for ensuring that his or her activities are planned and executed within an appropriate management and technical framework. In order to establish this framework, one must always start with an understanding of requirements. It is essential that developers, reviewers, and testers are aware of: Completion criteria for the activity being performed or the component to be produced (or reviewed); Requirements to be addressed i.e. what has to be done in order to complete an activity (or task); Component compliance requirements; Related roles and responsibilities for addressing various sets of requirements; Available tools e.g. templates and other job aids, examples, and reference material. It is therefore important to review and understand all relevant components of the methodology, and to use it as a base for executing all tasks. 5. ARCHITECTURE As thinking about a system progresses from the general to the particular, it should be guided by an architectural perspective to help ensure that all the parts continue to work together, and that nothing required is left out. Taking different perspectives on System Specification elements and relationships facilitates an orderly transition from Analysis through Design. With most non-trivial systems, it is difficult to capture all the desired architectural ideas in a single representation. This methodology therefore recommends constructing up to four different "views" of the architecture, to convey all the information required. We call these four views by the following names: Conceptual View; Process View; Development View; Physical View; Page 97 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published Figure 3 – View Model Each one of these views provides a different representation of a system, each valid and useful in its own right. The conceptual and development views show the static system structure while the process and physical views give us the dynamic or runtime system structure. The conceptual and development views, though very close, address very different concerns as described above. Further, there is no requirement or implication that these structures allow any topological resemblance to each other. While the views give different system structural perspectives they are not fully independent. Elements of one view will be "connected" to elements of other views, and one needs to reason about those connections. 6. GLOSSARY Expression Meaning IS/IT Information services / Information Technologies PM Project Manager QA Quality Assurance CM Configuration Management ReM Reuse Management FLT Full Life-cycle Testing Table 10 - Glossary Page 98 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published APPENDIX 4 – TEST TEAM ROLES AND RESPONSIBILITIES This appendix presents the roles and the responsibilities of the test team members. Roles of test team members Responsibilities of test team members Client Representative The client representative is the spokesperson assigned to represent the client’s interest (end users, operator, IT client, etc.) during the development of the system. They normally interact with the development team for validation and verification purpose. Developer The primary responsibility of this individual is related to the programming of the system. This person is involved with design, unit testing and programming activities. The developer is called to participate early in the development cycle in order to better understand the user needs and boundaries of the project. Maintaining a disciplined, versatile and creative approach, the developer combines functional and technical skills so that innovative, yet robust, solutions can be programmed. Project Manager As defined in the Project Management Framework, the project manager: Plans and oversees the project; Acts as the CGI project team' s contact with the client's internal project team; Ensures that the necessary management procedures are implemented; Keeps the project on time and on budget; Supervises the project team; Provides information on the project's status to those outside the project team. Note that the project manager is also responsible for certain reuse management activities. System Administrator The system administrator is responsible for providing technical support to the development team during the development and implementation of the system in the operating environment. Pro-active, autonomous and good technical skills are traits that describe good system administrators. Technical Leader The technical leader is primarily responsible for the technical directions followed by the other developers on the team. Having excellent technical skills, this person must understand the entire development life cycle and be able to interact with non-technical people. Tester The tester is primarily involved in the design, construction, execution of test cases and defect reporting. The is a typical list of activities the tester is required to perform: Translate client requirements to test scenarios Identify the test conditions Translate the test cases into machine-readable test cases Create the input, test beds and test scripts necessary to run the test Page 99 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published cases Run test cases Report the anomalies associated with the execution of test cases Analyse the test results and ensures all the requirements have been met Record the test results (defects and test cases) Retest all the components Page 100 of 101 TEST MANAGEMENT PROJECT Testing Methodology Type: Deliverable Title: Testing Methodology Author(s): Last Saved Date / Time Document No. / Revision No. Draft / Published Group 25: Joao Alves 22/06/2017 18:31:00 DOC-TM0004 / 2.2 Published APPENDIX 5 – DOCUMENT TEMPLATES Page 101 of 101 TEST MANAGEMENT PROJECT Testing Methodology