Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
The Impact of Software Reuse and Incremental Development on the Quality of Large Systems Parastoo Mohagheghi, Dept. Computer and Information Science (IDI), University of Science and Technology (NTNU), Trondheim [email protected] 1 Parastoo Mohagheghi 07 September 2004 What is this presentation about? • It presents results of a series of empirical studies at Ericsson in Grimstad in 2001-2004. • The studies are performed during a doctoral work in the INCO project and supervised by Prof. Reidar Conradi. • 11 students were involved in different studies, an example of university-industry cooperation. • For more details, see the PhD thesis at: http://www.idi.ntnu.no/grupper/su/publ/phd/moha gheghi-thesis-10jul04-final.pdf 2 Parastoo Mohagheghi 07 September 2004 Motivation • Systematic software reuse, product family development and incremental development seem be potent technologies to achieve benefits in productivity, quality and maintainability, and to reduce risks of changes. Quantifiable benefits, but few empirical studies in industry that can verify these benefits or possible other impacts. 3 Parastoo Mohagheghi 07 September 2004 Company background • Ericsson has developed two large-scale telecom systems (470 KSLOC or 1000 KSLOC in equivalent C each) that share – – – – Software architecture, Components and a component model, Development environment, Software process. • Systems are developed incrementally, and lots of data is gathered during 5 years on defects, changes, characteristics etc. 4 Parastoo Mohagheghi 07 September 2004 Overview of the Ericsson packetswitched core network- GPRS MSC/ VLR SMS-GMSC HLR SMS-IWMSC EIR Other networks IP network SGSN Backbone network GGSN BSC/ RNC Packet-switched core network Other SGSN BGW 5 Parastoo Mohagheghi 07 September 2004 Evolution of the GSN software architecture and the software process model The original architecture The hierarchical architecture Rev. & Re-eng. GPRS for WCDMA GPRS for GSM 3’ GPRS 4 for GSM 1’ GPRS for GSM 1 4 2 5 3 6’ 6 Wireless Packet Platform (WPP) 5’ 2 8 7 Application-specific layer Business-specific layer Common services Reused Layer (including component framework) Wireless Packet Platform (WPP) Adaptation of RUP Initial Process 6 Parastoo Mohagheghi Adapted RUP 07 September 2004 The GSN RUP 7 Parastoo Mohagheghi 07 September 2004 Research questions • RQ1. Why a reuse program is initiated and how is it implemented? • RQ2. What is the impact of software reuse, Component-Based Development (CBD) and incremental development on the quality? The impact of development approaches on product quality metrics and on project attributes such as schedule or effort are sought. • RQ3. How to improve the practice of incremental development of product families in some aspects? 8 Parastoo Mohagheghi 07 September 2004 Research methods • Quantitative studies by exploring (‘data mining’) company databases and assessing hypotheses. • Quantitative results are discussed with experienced personnel, and combined with qualitative feedbacks and studies of the process and practice to assess validity of the results. • Several case studies, a small survey and an experiment. • Combining results of several studies in the interpretation phase: – The impact of introducing reuse or incremental development is widespread. – Taking benefit of all available data, – Confirming results by other studies/data. 9 Parastoo Mohagheghi 07 September 2004 Qualitative study of RUP in the context of reuse • The approach to initiating a product family was an extractive one. • Software architecture has evolved to support reuse, while GSN RUP has not been adapted for reuse to the same degree. • Examples of proposals: – “Domain analysis” and “Make vs. Reuse vs. Buy” decisions in the inception phase. – “Update reuse documentation” for reusable components. – “Record reuse experience” in the conclusion phase. 10 Parastoo Mohagheghi 07 September 2004 Quantitative study of Trouble Reports (TRs) • For defects detected during integration testing, system testing, or later in maintenance. • For defects detected in previous releases. • For all types of defects (software, hardware, toolbox, and documentation). • We have ‘data mined’ databases: – 13000 TRs for several systems and releases were parsed, data was extracted, and inserted in a SQL database by a C# program. – Inconsistencies were resolved -> Data was not analyzed. • 11 The company provided us excel sheets on software size (software module level). Parastoo Mohagheghi 07 September 2004 Hypotheses: Software Reuse and Quality (Defect-density and Stability) 12 H01 Reused components have the same defect-density as non-reused ones. H02 There is no relation between number of defects and component size for all/reused/non-reused components. Not-rejected Not-rejected Rejected H03 There is no relation between defectdensity and component size for all/reused/non-reused components. H04 Reused and non-reused components are equally modified. Not-rejected Not-rejected Not-rejected Rejected Parastoo Mohagheghi Rejected 07 September 2004 H1: Reuse and defect-Density, Release 3 and blocks Defect-density of blocks Mean Median Variance #TRs/KLOC, Reused 1.32 0.76 1.70 #TRs/KLOC, Non-Reused 3.01 2.44 4.39 #TRs/MKLOC, Reused 3.50 1.78 21.26 #TRs/MKLOC, Non-Reused 5.69 3.73 21.76 •Reused components have lower defect density. •Reused components have proportionally more defects with Severity A (highest severity). •Reused components have proportionally less defects after delivery. 13 Parastoo Mohagheghi 07 September 2004 H2: Size vs. the number of Defects 180 160 R2 = 0,7698 140 Reused R2 =0,7213 120 #TR Non-reused Blocks 100 Linear (Non-reused Blocks) 80 Linear (Reused) R2 = 0,5876 R2 =0,573 60 Poly. (Reused) Poly. (Non-reused Blocks) 40 20 0 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 LOC 14 Parastoo Mohagheghi 07 September 2004 H3: Size vs. Defect-Density 8 7 #TRs/KLOC 6 5 Reused 4 3 Nonreused 2 1 0 0 5 10 15 20 25 30 35 40 45 KLOC 15 Parastoo Mohagheghi 07 September 2004 Discussion • Reuse benefits: Reused components are less defect-prone and more stable. • Why? Reused components are designed more thoroughly, and changed with care. – Confounding factors (conclusion validity): • Programming language. Not applicable. • Type of functionality. Not applicable for stability, but may be for defect-density. • Developers’ experience: Not applicable. • Internal validity: Missing data, but not systematic. • Construct validity: Are defect-density and stability indicators of quality? • External validity: At least inside the company for similar systems, or in the domain. 16 Parastoo Mohagheghi 07 September 2004 Contributions • Research: – Empirical verification of reuse benefits. No other studies on large industrial systems. – Combined with other studies in the company to assess development approaches and identifying metrics. • Company: – Evaluating the trouble reporting system. – Evaluating the measurement program: Granularity of component definition and data for incremental development. – Identifying defect-prone and unstable components. 17 Parastoo Mohagheghi 07 September 2004 Software evolution and incremental development • With incremental development, software changes: – during each release (development) – between successive releases (evolution) – and after delivery (maintenance). • Some changes are pre-planned (incremental development) and some are not (iterative improvement, adaptive changes etc.). • The IEEE Standard 1219 on software maintenance defines software maintenance as “The modification of a software product after delivery to correct faults (i.e. corrective), to improve performance or other attributes (i.e. perfective/preventive), or to adapt the product to a modified environment (adaptive)”. 18 Parastoo Mohagheghi 07 September 2004 Quantitative study of Change Requests (CRs) • Ericsson defines requirements for each release in an ARS (Application Requirement Specification). • Changes to ARS or previous deliveries (releases) are handled by issuing Change Requests (CRs). – CRs handle coarse-grained changes to add/delete functionality, improve performance or improve design. – CRs cover changes during developing each release as well as between releases (in addition to new requirements stated in the ARS). • How this data can be used in our research? – 169 CRs for 4 releases of one system were analyzed. – Exploratory study: Origin (internal vs. external), acceptance rate and classes of changes are studied. 19 Parastoo Mohagheghi 07 September 2004 Origin of CRs Perfective/ Functional No 40 Perfective/ Quality Attributes (QA) 74 Adaptive Preventive 35 30 Other (cost) 8 •18 CRs have indicated two reasons. •Most perfective CRs are to improve QA (also preventive CRs improve quality). •23 of 169 CRs are issued because of customer demands. These and adaptive CRs have external origin (34% of all CRs). 20 Parastoo Mohagheghi 07 September 2004 Acceptance Rate over Releases 100 % 80 % 60 % Rej ec t ed CRs (%) 40 % A c c ept ed CRs (%) 20 % 0% Release 1 Release 2 Release 3 Release 4 Acceptance rate is increasing (59% in total); i.e. the product is getting more change-prone. 21 Parastoo Mohagheghi 07 September 2004 Conclusions • Most changes originate from the project organization itself in order to improve quality and enhance functionality. The share of the first group is higher. • The practice indicates that the iterative realization and improvement of quality attributes has great impact on the evolution of the products. • We could not observe any significant difference between reused and non-reused components in the number of CRs per KLOC. 22 Parastoo Mohagheghi 07 September 2004 Conclusions (continued) • Most CRs are accepted and the acceptance rate can have impact on the project plans in terms of decreasing planning precision (observed). Incremental development opens for (more) changes. • Most CRs are issued pre-delivery, and especially in the short time right after requirement baseline. The organization possibly takes the decision to baseline requirements too early. 23 Parastoo Mohagheghi 07 September 2004 Overview of all studies Phase 1 P1 P7 Phase 2 C1 P8 Study of Trouble Reports 2003 Study of Reuse Practice 2002 C6b Experiment on Inspection P4 2002 Combining results in Phase 3 P11 Developing Data Mining Method 2004 C1 C2 P10 Study of Change Requests 2003 P2 P9 P6 Survey 2002 Study of Software Process & RUP 2001-2002 March 2001 24 P12 C3 Parastoo Mohagheghi P11 Assessing the Impact of Dev. Approaches & Identifying Metrics 2003-2004 Study of Effort 2003-2004 Developing Effort C3 Estimation Method 2003-2004 P13 Prototype Study of MDA P3 2003 Quantitative study Qualitative study C4 P5 C6a C5 July 2004 P Paper C Contribution Input 07 September 2004 Impact of dev. approaches and context Reuse Software evolution 25 Incr. dev. CBD Large systems Change rate Earlier rel. not evolved Granularity Improved incr. Long-lived Effort estimation ROI? Software No data for from a components previous rel., Incr. changes in requirements , Large CM Large CM and testing effort Metrics yes yes Different granularity Parastoo Mohagheghi yes 07 September 2004 Mining industrial databases Theory or Hypothesis confirmatory theoretical Literature Search Research Question exploratory Pre-Study Data Select/Define Theory or Hyp. Select Methods & tools preparation Sampling yes/no Explore Data Modify Data Model Assess execution Package Results conclusion Top-down theory search integrated with bottom-up data exploring. 26 Parastoo Mohagheghi 07 September 2004 Assessing development approaches Development Approaches Practices Product/process Quality Requirement Modification +/- Incremental & Iterative Development Planning Precision +/Incremental Delivery Success Solution Modification Needed Effort Software Reuse Incremental Planning Component Stability + Incremental Integration Component-Based Development + Reusable Artifacts/ Components + Reduced Defect-Density Changeability Leads to Development approaches should be seen as coherent systems of practices, be chosen depending on the attribute that should be optimized and may trade-off for other practices [IEEE software, MacCormack, 2003 ] 27 Parastoo Mohagheghi 07 September 2004 Main Contributions • Empirical verification of reuse benefits. • Increased understanding of the origin and type of changes in each release and between releases. • Identifying metrics for a combination of reuse and incremental development. • A data mining method for exploring industrial databases. • An effort estimation method for incremental development. 28 Parastoo Mohagheghi 07 September 2004 Discussion • Studies are performed in a real context. – Context-independent knowledge is rare in SE and generalization is difficult. • Studies combine different methods and data (triangulation). – Integration of the results is a challenge: physical and conceptual integration challenges. • Incremental development is combined with reuse of software in multiple products and CBD. – All aspects of software development need adaptation; e.g. estimation method, software process model, data collection routines etc. 29 Parastoo Mohagheghi 07 September 2004 Discussion (continued) • Own experience and access to internal data – Confidentiality of some results and other ethical issues should be respected. • Some studies are evaluation of others’ observations or methods in a new context (e.g. estimation method, reuse benefits or distribution of maintenance categories), while others contain new observations and methods. • Being exposed to organizational changes leaded to an emerging research design and changes in plans. • Working in the field needs flexibility and a suitably timeframe. 30 Parastoo Mohagheghi 07 September 2004 Why such studies are important for industry? • Methods should be tested for development in large. • Companies are increasingly using mainstream methods, tools and programming languages. • The studies analyzed data that was hardly studied before: evaluation of data collection routines and metrics for the company, in addition to concrete results. 31 Parastoo Mohagheghi 07 September 2004 Acknowledgements • Thanks to INCO, Simula and NTNU for financial support. • Thanks to my supervisor, Prof. Reidar Conradi, and colleagues at INCO and IDI. • Thanks to Ericsson for the permission to perform studies and to publish the results. • Thanks to the SPIKE seminar and the audience. 32 Parastoo Mohagheghi 07 September 2004