Download SMQL

Toward an Environment for
Comprehending Distributed
Systems
Reverse Engineering (Dynamic Analysis)
© SERG
Outline
•
•
•
•
•
•
Motivation
Software comprehension environment
Data gathering
Data repository
Preliminary results
Conclusions and future work
Reverse Engineering (Dynamic Analysis)
© SERG
Background
• Project
– Comprehending distributed component-based
systems
– Phase-I
• Development of a core toolset
– Phase-II (Iterative)
• Modeling and visualization
• Evaluation
Reverse Engineering (Dynamic Analysis)
© SERG
Characteristics of modern
software systems
•
•
•
•
Large and complex
Distributed
Written in multiple programming languages
Developed using component-models
Reverse Engineering (Dynamic Analysis)
© SERG
Software Comprehension
Challenges
• Unavailability of source code
– Vigder-2001, Weyuker-2001
• Physical distribution of components
– Bruegge-1993, Mendonca-1999
• Use of pre-built component
– Stroermer-2002, Vigder-2001, Weyuker-2001
• Multiple programming languages
– Hassan-2002, Chen-1995,
– Koschke-1998, Stroermer-2002
– Demeyer-1999, CDIF-1994, XML/XMI-2002
Reverse Engineering (Dynamic Analysis)
© SERG
Approach
• Software Comprehension Environment
– Technical issues
• Distributed profilers
• Single data repository
• Data management
• Extensible architecture
 Data gathering and Repository subsystems
– Modeling and visualization subsystem
– Empirical evaluation
• Survey/questionnaire
• Industrial-strength case studies – Cigna Corporation
Reverse Engineering (Dynamic Analysis)
© SERG
Software Comprehension
Environment
Repository Subsystem
Universal Schema
Analysis and Visualization Subsystem
Conceptual
Models
Data Gathering Subsystem
Software
Program(s)
Source
code
Execution
Traces
Static Analyzers
Data Model
Meta-data
Model
Dynamic Analyzers
(Debugger/Profiler)
Program Data
(Static &
Dynamic)
Meta-Data
Visualization
(Presentation)
Reverse Engineering (Dynamic Analysis)
Navigation
Human-Driven/
Feature-Driven
© SERG
Data Gathering subsystem
GDB
Profiler
Data
Collection
Adapter
Java Run-time
JVMPI
JVMDI
Profiler
Data
Collection
Adapter
Java Static
Analyzer
Data
Collection
Adapter
Java Source-code
or Byte-code
Data Collection Manager
Logical Time Server
Data Collection Server
To Data
Repository
Import/Export
GNU C/C++
Run-time
XML Documents
Win32 C/C++ and VB
Runtime
Win32
C/C++/VB
Profiler
Data
Collection
Adapter
Reverse Engineering (Dynamic Analysis)
© SERG
Data Gathering subsystem
• Distributed Profiler
– Local Profiler
– Logical Time Server
– Data collection manager
• Remote Interactions
– Network interceptor
– Communication Endpoint
• Program Facts/Data
– Static: Entities and Relations
– Dynamic: Entities, Relations and Events
Reverse Engineering (Dynamic Analysis)
© SERG
Data Gathering subsystem
• Endpoint entity
– Local and Remote host
– Local and Remote port numbers
– Time-stamp
• Connects Relation
Endpoint-A
Local Host
Remote Host
Local Port
Remote Port
Time
Delta < Threshold
Local Host
Remote Host
Local Port
Remote Port
Time+Delta
Endpoint-B
Reverse Engineering (Dynamic Analysis)
© SERG
UserClass
java.net
SokectOutputStream
write()
Java.net.
SokectInputStream
Logical Time
Example: Sequence Diagram
(Winsock)
Ws2_32.DLL
UserModule
EPTime(A)
EP(A)
recv()
EP(B)
EPTime(B)
send()
read()
Java Program
C/C++ Win32 Program
Reverse Engineering (Dynamic Analysis)
© SERG
Data Gathering subsystem
• Supported data collectors
–
–
–
–
WDBG Win32 Profiler
JVPROF Java Profiler
JSA Java static analyzer
Others, provided the conversion to XML
• Network-call interceptors
– WDBG and JVPROF
Reverse Engineering (Dynamic Analysis)
© SERG
Data Repository
• Logical Models
– Language definitions
– Program Data
– Used-defined views
• Implementation
– Relational database
• Query/Manipulation Language
– SQL
– SMQL
Reverse Engineering (Dynamic Analysis)
© SERG
Language Definitions
• Model(General) = Graph(Eg , Rg)
Eg: Set of entity types
Rg: Set of relation types
• Java, example:
• Model(Java) = Graph(Ejava , Rjava)
Ejava  Eg : Entity types supported by Java
Rjava  Rg : Relation types supported by Java
Such that the source and destination entity  Ejava
• C++, defining
Model(C++) = Graph(Ec++, Rc++)
Ec++ = Ejava  {template, struct, typedef, function}
- {interface, package}
Reverse Engineering (Dynamic Analysis)
© SERG
Java Language Example
connects
End-point
reads
declares
field
writes
method
defines
invokes
creates
object
instanceof
declares
interface
class
implements
contains
package
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL
• SMQL
– Software Modeling Query Language
• Set-based:
– Typed Set
• Entity Set
• Relation Set
• Event Set
– Generic Set
• Extensible using Java
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL
• Operations
– Union, intersection, Difference
• Functions
– Closure
Closure(Source, Target, RelationTypes)
Returns a Relation-Set
– Composition
• Builds a hierarchical model from flat set of entities into a
nested set of entities
– Output: Saves a typed-set as an XML document
– Output_dot: Saves a relation set as DOT graph
Reverse Engineering (Dynamic Analysis)
© SERG
Example: Deployment
Architecture
Linux
IP Address:
192.92.200.2
Windows 2000
IP Address:
192.92.200.3
Jetty WebServer
Tear.exe
Dynamic
(Java JVM 1.4.2)
jvprof
Data Collection
wdbg
Data collection
SMQL
Data Collection
Manager
SCDR
SQL
SQL Server 2000
Analysis
Environment
jsa
Data collection
Static
Jetty Source code
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: Defining Entity Set
Entity Identifier
{
[caption = String ;]
type = StringSet ;
(EntityFirstOrderFilter)+
}
EntityFirstOrderFilter ::
include ( EntityAttrib ) = StringSet ;
| exclude ( EntityAttrib ) = StringSet ;
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: Inheritance example
// Get Inheritance tree rooted at org.mortbay.http.HttpListener
// OR org.mortbay.http.HttpHandler
EntitySet Http
{
caption = "HttpListener" ;
type = {“interface”, “class”} ;
include (name) = {
"org.mortbay.http.HttpListener",
"org.mortbay.http.HttpHandler" } ;
include (project) = { "jetty" } ;
}
Inh_Http = closure(null, Http, { “implement”, “subclass” } ) ;
Output (“file.xml”, Inh_Http, Http) ;
Output_dot (“graph.dot”, Inh_Http) ;
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: Inheritance example
Reverse Engineering (Dynamic Analysis)
© SERG
Reverse Engineering (Dynamic Analysis)
© SERG
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: Events Example
// Events from Tear & Jetty
import
java:serg.sc.filter.CallGraph as CallGraph ;
EventSet Events
{
caption = "Event (Jetty & Tear) " ;
type = { "method-entry",
"method-exit",
"endpoint",
"thread-start",
"thread-end",
"module-load",
"process-start",
"process-end"} ;
include (project) = {"jetty-rt", "Tear"} ;
}
// Convert events trace into a CallGraph (Relation Set)
GC = CallGraph( events ) ;
// Save as results
output ("tear-jetty.xml", events);
Output_dot (“GC.dot”, GC) ;
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: Events Example
Reverse Engineering (Dynamic Analysis)
© SERG
Reverse Engineering (Dynamic Analysis)
© SERG
SMQL: User defined functions
• User-defined functions
– IFilter interface
Import java:Qualified-Java-Class-Name as FunctionName ;
• Example
Import java:serg.sc.MyClass as MyFunction
EntitySet E
{
caption = "HttpListener" ;
type = {“interface”, “class”} ;
include (name) = {
"org.mortbay.http.HttpListener” } ;
include (project) = { "jetty" } ;
}
X = MyFunction(E) ;
Reverse Engineering (Dynamic Analysis)
© SERG
Modeling and visualization
subsystem
Program
Data
Analysis
Model
Visual Style
Navigational
Style
SMQL
Visualization
Revised
Model
Reverse Engineering (Dynamic Analysis)
Software View
© SERG
Progress
• Repository implementation
– SQL Server & Hypersonic DB
• Data gathering
– WDBG, JVPROF, and JSA
• Tools
– SMQL
– Dbmgr - Database and SMQL front-end
– dbImport, dbexport
Reverse Engineering (Dynamic Analysis)
© SERG
Conclusions and Future Work
• Software Comprehension Environment
•
•
•
•
•
Dynamic analysis of distributed systems
Multi-language systems
Common repository
Distributed profiler
SMQL
• Future work
– Phase I
• Performance
• Support for Microsoft.Net, VB and COM+
– Phase II (Early stage)
• Construction and visualization of views
• Empirical evaluation in an industrial setting (at Cigna)
Reverse Engineering (Dynamic Analysis)
© SERG
References
Vigder-2001] M. Vigder. The evolution, maintenance, and management of
component-based systems. In G. Heineman and W. Councill, editors,
Component-Based Software Engineering: Putting the Pieces Together,
chapter 29, pages 527. 539. Addison-Wesley, 2001.
[Weyuker-2001] E. J. Weyuker. The trouble with testing components. In G.
Heineman and W. Councill, editors, Component-Based Software
Engineering: Putting the Pieces Together, chapter 28, pages 499.512.
Addison-Wesley, 2001.
[Stroermer-2002] C. Stroermer, L. O'Brien, and C. Verhoef. Practice
patterns for architecture reconstruction. In Proceedings of the 9th
Working Conference on Reverse Engineering (WCRE), Richmond, VA,
USA, October 2002.
[Hassan-2002] A. E. Hassan and R. C. Holt. Architecture recovery of web
applications. In 24th International Conference on Software Engineering
(ICSE), Orlando, Florida, USA, May 2002.
[Chen-1995] Y.-F. R. Chen, G. S. Fowler, E. Koutsoos, and R. S. Wallach.
Ciao: A graphical navigator for software and document repositories. In
Proceedings of International Conference on Software Maintenance
(ICSM), Nice, France, October 1995. IEEE.
Reverse Engineering (Dynamic Analysis)
© SERG
References
[Demeyer-1999] S. Demeyer, S. Tichelaar, and P. Steyaert. Famix 2.0 –
the famoos information exchange model. Technical report, University of
Berne, August 1999.
[CDIF-1994] Electronic Industries Alliance and International Standards
Organization (EIA/ISO). CDIF - Framework for Modeling and
Extensibility, 1994.
[XML/XMI-2002] Object Management Group. XML Metadata Interchange
(XMI), version 1.2 edition, January 2002.
[Koschke-1998] R. Koschke, J.-F. Girard, and M.Wurthner. An intermediate
representation for reverse engineering analysis. In Proceedings of the
5th Working Conference on Reverse Engineering (WCRE), Honolulu,
HI, USA, October 1998.
[Bruegge-1993] B. Bruegge, T. Gottschalk, and B. Luo. A framework for
dynamic program analysis. In Conference on Object Oriented
Programming Systems, Languages, and Applications (OOSLA93),
Washington, USA, September 1993.
[Mendonca-1999] N. C. Mendonca. Software Architecture Recovery for
Distributed Systems. PhD thesis, Department of University of London,
NovemberReverse
1999. Engineering (Dynamic Analysis)
© SERG