* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download WebSphere Application Server Enterprise Process Choreographer Enhanced Audit Log Data Analysis and
Survey
Document related concepts
Transcript
WebSphere Application Server Enterprise Process Choreographer Enhanced Audit Log Data Analysis and Query Frank Neumann, Anke Robeller January 2004 © Copyright IBM Corporation, 2004 Contact information The authors of this paper, Frank Neumann ([email protected]) and Anke Robeller ([email protected]) work both in the WebSphere Application Server Enterprise Process Choreographer development team. Table of Contents 1. Abstract ....................................................................................................................... 3 2. Setting up the Business Process for Auditing ............................................................. 3 3. Navigation and States of a Business Process.............................................................. 4 3.1 Process Instance Events ........................................................................................ 4 3.2 Activity Instance Events ....................................................................................... 6 3.3 Other Events.......................................................................................................... 7 4. Audit Log Data ........................................................................................................... 8 5. Accessing Audit Log Data in the Database ................................................................ 9 5.1 Timestamp Representation.................................................................................. 10 5.2 Sample SQL Statements ..................................................................................... 11 5.3 Performance Considerations ............................................................................... 13 5.4 Deleting Audit Log Data..................................................................................... 14 6. JDBC Program to Query Audit Log ......................................................................... 15 6.1 Accessing Audit Log Inside WebSphere Application Server Enterprise ............... 17 7. Query Process Choreographer's Views..................................................................... 17 8. Summary ................................................................................................................... 17 References..................................................................................................................... 18 Trademark attributions and copyrights ......................................................................... 18 Special attributions.................................................................................................... 18 Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 2 1. Abstract IBM WebSphere Application Server Enterprise Process Choreographer Version 5 writes business relevant information about navigated processes to a database audit log table. This information can be queried and analyzed using standard SQL statements for the underlying database system. This paper describes how audit log is set up, and the data which is written to the audit log table. It also provides information on the best practices for using this data to answer typical statistical and historical questions regarding business processes. The paper refers to WebSphere Application Server Enterprise Versions 5.0, 5.0.1 and 5.0.2. 2. Setting up the Business Process for Auditing The process engine generates audit events during the process lifetime when auditing is enabled. Each audit event leads to a new row in the audit log table (AUDIT_LOG_T). In contrast to other Process Choreographer database tables, the audit log entries are not deleted automatically at the end of a process execution or on a delete request; thus the audit log can be used to query information about currently active processes as well as to track historical data about processes that have run in the past. An audit event is associated with an entity of the process, these entities are: • process template • process instance • activity • control link • variable The audit log facility can be enabled during modeling time in WebSphere Studio Application Developer (Integration Edition). For processes this is done on the server tab of the business process editor, as illustrated in figure 1. This causes the process engine to log events related to variables, control links and the process instance itself. Please note, auditing can be enabled for non-interruptible and for interruptible processes. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 3 figure 1 -- enable auditing for a process Auditing can be enabled for each individual activity of a business process in the server settings of the property pages (see figure 2). You can turn on activity auditing independently of process level auditing, i.e. activities can be audited even if auditing for the process is turned off. figure 2 -- enable auditing for an activity In addition to that, when an enterprise application, which contains a business process, is installed or uninstalled, an audit log entry will be generated. This behavior can not be disabled. 3. Navigation and States of a Business Process An audit event is generated whenever something of significance to the business process happens. This is tightly coupled with process and activity state transitions. 3.1 Process Instance Events Figure 3 shows the state transitions of a business process and the audit events produced. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 4 When a process is started the state changes to running and a "process started" (21000) event is generated. If the process ends successfully -- this will lead to state "finished" – a "process completed" (21020) event is written. When the navigation hits on a fault the process changes its state to "failing" until all long-running activities are terminated. Finally, the process is in state "failed". A process can be terminated through a forceTerminate API request or due to a failure in the parent process. The termination of a process can span multiple navigation steps and thus multiple chained transactions. For example all active sub-processes inside the process need to be terminated in a separate step. In this interim phase the process state is "terminating" before it ends in state "terminated". figure 3 -- process state transition and related audit events Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 5 3.2 Activity Instance Events The states, state transitions, and generated events for activities depend on the activity kind. When the navigation hits on a non-staff activity -- i.e. empty, invoke, event, block, and loop activities -- its initial state is running and an "activity started" (21007) event is fired. Staff activities have an initial state of "ready". It can be claimed and completed by user interaction. An activity can end in states: "finished", "failed", "terminated", and "expired". In long-running processes, an activity failure can result to an activity state: "stopped". These state transitions and events are summarized in figure 4. figure 4 -- activity state transition and related audit events The activity state transition and audit events "activity failing" (42011) and "activity terminating" (42008) are not shown in the state diagram above to avoid confusion. A process activity shows the states of its sub-processes at any given time. In addition to events related to state transitions, an audit event can signal that an activity has been manipulated through an API request. These events are: Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 6 Event name activity output message set activity fault message set Event number 42012 42013 Description A claimed staff activity's output is set without completing the activity. A claimed staff activity's fault is set without completing the activity. 3.3 Other Events There are events to report a: • change to a process variable, • evaluation of a transition condition (Transition condition evaluation to false is not shown in the audit log for performance reasons), • installation or uninstallation of a process template. The events are summarized in the table below: Event name Event number Description variable updated 21090 The content of a variable has been changed, either in a java snippet or through the completion of an event, invoke of staff activity. control link evaluated to 21034 The transition condition has been evaluated true to true. process template 42006 The process template has been installed installed process template 42007 The process template has been uninstalled uninstalled Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 7 4. Audit Log Data Process Choreographer writes audit log data into the AUDIT_LOG_T table of the Process Choreographer database. This table has the following layout (please note that tables for DB2 on z/OS may have slightly different, shorter, names): Name process instance event x x x x activity instance event x x x x variable event ALID EVENT_TIME AUDIT_EVENT PTID process template event x x x x x x x x control link event x x x x PIID - x x x x PROCESS_TEMPL_NAME - x x x x PROCESS_INST_NAME - x x x x TOP_LEVEL_PI_NAME - x x x x TOP_LEVEL_PIID - x x x x PARENT_PI_NAME PARENT_PIID VALID_FROM - x x x x x x x x x x x x ACTIVITY NAME ACTIVITY_KIND - - x x - - ACTIVITY_STATE - - x - - Process Choreographer - Enhanced Audit Log Data Analysis and Query Description Internal identifier and primary key for a row. Timestamp of when the event occurred (in UTC format). For a list of audit event codes refer to the previous section. Identifier of the process template that is related to the current event. Identifier of the process instance that is related to the current event. Name of the process template that is related to the current event. Name of the process instance that is related to the current event. Name of the top-level process that is related to the current event. Identifier of the top-level process that is related to the current event. Name of the parent process instance. Identifier of the parent process, or null if no parent exists. Valid-from date of the process template that is related to the current event. Name of the activity on which the event occurred. Kind of the activity on which the activity occurred. It can have the following values: • sub process 1 • block or loop 2 • empty 3 • invoke or java snippet 6 • staff 8 • event 9 Note: These are the constants defined on ActivityInstanceData.KIND_* State of the activity that is related to the event: • inactive 1 • ready 2 • running 3 • skipped 4 • finished 5 • failed 6 • terminated 7 • claimed 8 • terminating 9 • failing 10 • waiting 11 • expired 12 Page 8 CONTROL_LINK_NAME - - - - x IMPL_NAME - - x - - PRINCIPAL - x x - - TERMINAL_NAME - - x - - VARIABLE_DATA EXCEPTION_TEXT - x x x - - DESCRIPTION - x x - - • stopped 13 Note: These are the constants defined on ActivityInstanceData.STATE_* Name of the control link that is related to the current control link event. Name of the activity implementation. This is only applicable for elemental activities. Name of the principal that requested the API-call related to the event, applicable for these events (but only if directly called over the API): • activity claimed • activity completed • activity failed • activity fault message set • activity output message set • process deleted • process started • process started • process terminated Name of the fault terminal that has been set, applicable for "activity set fault message". Data for variables for "variable updated" events. Exception message that caused an activity or process to fail. Applicable for: • process failed • activity failed. Description of activity or process, containing potentially resolved replacement variables. 5. Accessing Audit Log Data in the Database Depending on your objectives, you may want to choose different approaches to access the audit log data in the Process Choreographer database. • Direct SQL usage You can use your database runtime client to process SQL statements, directly accessing the AUDIT_LOG_T table. This provides quick and easy gathering of information from the audit log without the effort of writing application code. • Audit Application You can write your own application and use JDBC to access data in AUDIT_LOG_T. An application can gather data, post-process it and present it in a more readable form. • Offline analysis You can copy the contents of AUDIT_LOG_T to another table and then do offline analysis. This could be beneficial if you want to avoid a performance penalty when querying online data. In either case, you need to think about the SQL statements that return the desired results. The following chapter provides sample statements for commonly requested queries in the area of volume and response information. They can be used as-is or modified to meet individual needs. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 9 5.1 Timestamp Representation Process Chorographer uses the UTC timezone (UTC = Coordinated Universal Time, also called GMT = Greenwich Mean Time) to represent timestamp information internally and in database tables. While the Process Choreographer API (such as the query() API call, see [1] "WebSphere Application Server Enterprise Process Choreographer Work items and the query() API call" ) performs appropriate timezone conversions to match client settings, the timezone computation must be done explicitly when querying the database tables directly. If you run a DB2 Universal Database query such as: SELECT EVENT_TIME, AUDIT_EVENT FROM AUDIT_LOG_T to get the time and type of events logged in the audit log table, the resulting timestamp will be in UTC: EVENT_TIME AUDIT_EVENT -------------------------- ----------2003-11-29-18.00.00.000000 21000 2003-11-29-18.00.02.432100 21004 If you run the query in Rochester, Minnesota with Central Standard Time (CST), the timezone offset to UTC is -6 hours. Thus, to get to a time representation in the local timezone, modify the query as follows: SELECT EVENT_TIME -6 HOURS, AUDIT_EVENT FROM AUDIT_LOG_T to get the following result: 1 AUDIT_EVENT -------------------------- ----------2003-11-29-12.00.00.000000 21000 2003-11-29-12.00.02.432100 21004 Note, that you also have to consider daylight saving time. If you plan to implement audit log analysis in a (Java-) program you may want to check for the Timezone and DateTime class which provides ready-to-use timezone conversions. Important: If you run Oracle as the database system for Process Choreographer, note that Oracle 8i does not support the TIMESTAMP column type - the corresponding columns are defined using DATE instead. While Oracle 9i supports the TIMESTAMP column type, time stamp resolution is less that with other database systems. As a result, multiple sequential events may appear to have happened at the very same time. When timestamps are subtracted, the result is a timestamp duration. Most database systems support timestamp arithmetic, the following samples are again for DB2 Universal Database: Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 10 Given two timestamps, you are interested in the duration between them: SELECT TIMESTAMP('2003-11-29-12.10.20') – TIMESTAMP('2003-11-29-12.09.10') FROM SYSIBM.SYSDUMMY1 1 ---------------------110.000000 The resulting timestamp duration 110.000000 has to be interpreted in the format yyyyxxddhhmmss.zzzzzz where yyyy, xx, dd, hh, mm, ss and zzzzzz represent respectively the number of years, months, days, hours, minutes, seconds and microseconds. In the above example, the duration is one minute and ten seconds. However, timestamp duration doesn't go together with aggregation functions (such as AVG) because the result is computed based on the numerical value. For example, if your table has two rows - one resulting in a timestamp difference of 110.000000 (which is one minute and ten seconds) and another resulting in 230.000000 (which is two and a half minutes). The average would be one minute and 50 seconds while the average computation of the values 110 and 230 results in 170 (thus, one minute and 70 seconds, which is not what you want). The solution for aggregation of timestamp durations is to use TIMESTAMPDIFF, a built-in UDF (user defined function) that ships with DB2. The first argument determines resolution of the result (2 for seconds, 4 for minutes etc.), the second argument is the result of subtracting two timestamps and converting the result to CHAR: SELECT TIMESTAMPDIFF(2,CHAR(TIMESTAMP('2003-11-29-12.10.20') – TIMESTAMP('2003-11-29-12.09.10')) FROM SYSIBM.SYSDUMMY1 1 ----------70 5.2 Sample SQL Statements SQL samples in this paper are described for DB2 Universal Database V8.1. SQL syntax for other database systems may vary but the statements are easy to adapt. If you are running DB2 for z/OS, keep in mind that long column names are shortened in order to match the 18 character limit. If you want to know the average time (in seconds) for a business process to complete in your system, select the 'process started' (21000) and 'process completed' (21004) events of a process instance and let the database system compute the average difference between both events: Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 11 SELECT AVG(TIMESTAMPDIFF(2,CHAR(AL_COMPLETED.EVENT_TIME – AL_STARTED.EVENT_TIME))) FROM AUDIT_LOG_T AL_STARTED, AUDIT_LOG_T AL_COMPLETED WHERE AL_STARTED.AUDIT_EVENT = 21000 AND AL_COMPLETED.AUDIT_EVENT = 21004 AND AL_STARTED.PIID = AL_COMPLETED.PIID The names of processes that failed can be retrieved by searching for 'process failed' events (42001): SELECT PROCESS_INST_NAME AS NAME FROM AUDIT_LOG_T WHERE AUDIT_EVENT = 42001 The current number of processes that are still running - they have a 'process started' (21000) entry but none of the events that describe an 'end state' (finished, failed, terminate). SELECT COUNT(*) FROM AUDIT_LOG_T AL1 WHERE AL1.AUDIT_EVENT = 21000 AND NOT EXISTS (SELECT 1 FROM AUDIT_LOG_T AL2 WHERE AL1.PIID = AL2.PIID AND AL2.AUDIT_EVENT IN (21004, 21005, 42001) ) You want to know how many business processes completed regularly in November 2003. Note that times must be specified in a format known by the database system and that Process Choreographer stores timestamps in UTC: SELECT COUNT(*) FROM AUDIT_LOG_T WHERE AUDIT_EVENT = 21004 AND EVENT_TIME >= '2003-11-01 00:00:00' AND EVENT_TIME < '2003-12-01 00:00:00' Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 12 A user 'Frank' is interested in an overview of how often he initiated different business processes templates since January 2003: SELECT PROCESS_TEMPL_NAME AS NAME, COUNT(*) AS FREQUENCY FROM AUDIT_LOG_T WHERE AUDIT_EVENT = 21000 AND PRINCIPAL='Frank' AND EVENT_TIME >= '2003-01-01 00:00:00' GROUP BY PROCESS_TEMPL_NAME The result of this query displays the process template name in the first column and shows how often this process has been started in the second column. You are interested in the average time that staff activities needed to be completed. To figure out the right SQL statement, you have to know: • Entries for staff activities have ACTIVITY_KIND=8 • The event code for 'activity ready' is 21006 (note that staff activities don't go to state 'started') • The event code for 'activity completed' is 21011 • Two events belong to the same activity instance, if they have the same name (ACTIVITY_NAME) and belong to the same process instance (that is, have the same PIID) SELECT AVG(TIMESTAMPDIFF( 2, CHAR(AL1.EVENT_TIME - AL2.EVENT_TIME))) AS AVERAGE_TIME FROM AUDIT_LOG_T AL1, AUDIT_LOG_T AL2 WHERE AL1.ACTIVITY_KIND = 8 AND AL2.ACTIVITY_KIND=8 AND AL1.PIID = AL2.PIID AND AL2.AUDIT_EVENT = 21006 AND AL1.AUDIT_EVENT = 21011 AND AL1.ACTIVITY_NAME = AL2.ACTIVITY_NAME 5.3 Performance Considerations In a production environment the size of the audit log table can increase rapidly. The definition of indexes on AUDIT_LOG_T for the query predicates is required in order to run the SQL queries in reasonable time. But, even with the right indexes in place, the database system might have to read thousands of pages into the bufferpool to compute the result. Other database pages that are used for navigation of business processes might get displaced. As a result, Process Choreographer performance may degrade. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 13 The data for the AUDIT_LOG_T should reside in a separate tablespace and should be stored on a separate physical disc to improve the performance of writing audit log entries. Note that writing audit log data can easily become the bottleneck of the entire system when large amounts of data are written and the system is not configured appropriately. The Process Choreographer default configuration already places AUDIT_LOG_T in a separate tablespace (for database systems supporting tablespaces). In order to avoid affecting the runtime system performance with undesired page displacing, data for the audit Log table should also be associated with a separate bufferpool. While indexes are the right choice to support queries, insert performance suffers, in particular for many, large indexes. Thus, you have to find a balance between quick response time for your audit Log queries and performance impact for business process navigation. If you infrequently run audit Log queries and overall performance is important for you, it may be worth running them without an index at all or create an index only for the time you are doing audit Log analysis. If you plan to extensively run audit log analysis and don't want to impact running business processes, consider copying audit log entries into a new table (again, using a separate bufferpool), then create indexes that best match your queries and run them on the new table. To create a new table and insert the data: CREATE TABLE AUDIT_LOG_COPY LIKE AUDIT_LOG_T INSERT INTO AUDIT_LOG_COPY SELECT * FROM AUDIT_LOG_T usually won't work in a real life scenario since large amounts of audit log table data will exceed the log file size of the database. Thus, you should use the faster DB2 export/load statements to copy the table, for example: CREATE TABLE AUDIT_LOG_COPY LIKE AUDIT_LOG_T EXPORT TO C:\TMP\AUDIT_LOG_T OF IXF SELECT * FROM AUDIT_LOG_T LOAD FROM C:\TMP\AUDIT_LOG_T OF IXF INSERT INTO AUDIT_LOG_COPY 5.4 Deleting Audit Log Data Audit log data can increase rapidly in a production environment and you might reach a condition where even a simple DELETE FROM AUDIT_LOG_T fails because the amount of data exceeds the current log file settings. Process Choreographer ships with a powerful utility to delete entries in the audit log: BPEAuditLogDelete {-processtime processtime| -time time|-all [slice]} Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 14 -dbs database_system -dbn database_name [-u userId] [-p password] [-host host] [-port port] This utility uses slicing to delete entries in the audit log table - that is, only the specified amount of entries are deleted in a series of single transactions. Apart from this feature, the utility can also delete 'old' audit log entries, written before a given time. Check the description in the Info Center for more details and how to run BPEAuditLogDelete. 6. JDBC Program to Query Audit Log Writing a small JDBC program to query the audit log has the following advantages: • • • • Flexibility You can pass command line arguments to a program, thus allow multiple variations of the query to be run with a single program. Performance If your JDBC program uses prepared statements and parameter markers, queries are processed faster if they are found in the database statement cache. Capabilities If it is hard to write a single ('closed') SQL query for the desired task, it might be easier to run multiple queries and post-process the results in a program. Usability A program can present retrieved data in a user friendly form (for example using graphics in a GUI), making it easier to read and use. Here is a snippet of Java code that you might want to use as a starting point - it takes the name of a process template as command line parameter and returns start dates of all corresponding instances. The code assumes again a DB2 Universal Database - make sure you have the CLI-based JDBC Driver (db2java.zip) in your classpath when compiling and running this sample code You can also use the Universal JDBC Driver (db2jcc.jar) that ships with DB2 Universal Database V8 - make sure you replace references to COM.ibm.db2.jdbc.DB2DataSource with com.ibm.db2.jcc.DB2SimpleDataSource. import java.sql.*; import java.util.Calendar; import java.util.TimeZone; public class AuditLogInfo { public static void main(String argv[]) { if( argv.length != 1 ) Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 15 { System.err.println("usage: " + AuditLogInfo.class.getName () + " <process template name>"); return; } System.out.println("Start times of process instances for: " + argv[0] ); COM.ibm.db2.jdbc.DB2DataSource db2ds = new COM.ibm.db2.jdbc.DB2DataSource(); // Adapt "BPEDB" if you've chosen a different name // during configuration db2ds.setDatabaseName("BPEDB"); } } Calendar cal = Calendar.getInstance( TimeZone.getTimeZone("UTC") ); try { Connection con = db2ds.getConnection(); PreparedStatement pstmt = con.prepareStatement( "SELECT EVENT_TIME FROM AUDIT_LOG_T WHERE " + "PROCESS_TEMPL_NAME = ? AND " + "AUDIT_EVENT=21000"); pstmt.setString(1, argv[0] ); ResultSet resultSet = pstmt.executeQuery(); while( resultSet.next() ) { System.out.println( resultSet.getTimestamp(1, cal) ); } resultSet.close(); pstmt.close(); con.close(); } catch( SQLException exc ) { exc.printStackTrace(); } To keep the code simple, it lacks proper error and resource handling (such as closing resources in case of an exception etc.), but these few lines of code demonstrate how easy it is to start with a small Java application. Note that a Calendar object with a UTC timezone is defined to be used when retrieving the timestamp from the JDBC ResultSet - this does the appropriate timezone conversion from UTC timestamps stored in database to the timezone defined in the local environment. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 16 6.1 Accessing Audit Log Inside WebSphere Application Server Enterprise Beyond using plain SQL statements or a J2SE based JDBC program, you can exploit the power of WebSphere Application Enterprise to build an enterprise applications that uses Servlets, EJBs and WebSphere resources to connect to the database table. Because Process Choreographer runs in WebSphere Application Server Enterprise, there is already a data source pointing to the database. You simply have to add a resource reference to this data source for your enterprise application and can use a JNDI lookup to retrieve it. You can use CMP Enterprise Java Beans to wrapper the Audit Log table and allow users to access information via the dynamic EJB query language. WebSphere Application Server offers performance benefits, such as statement caching and allows you to access audit log information from any remote location using a Web browser. 7. Query Process Choreographer's Views In addition to the audit log table, Process Choreographer defines a couple of views that have current runtime information about processes, activities, events etc. The whitepaper about work items and the query API() (see [1] "WebSphere Application Server Enterprise Process Choreographer Work items and the query() API call") discusses these views in detail and demonstrates how the query() API can be used to retrieve data. Running queries on the views is a good choice if business processes are not defined to write audit log information. However, all business process instance data will usually be deleted once the process is completed. Thus, queries on the views are usually not feasible to gather statistical or historical data. Furthermore, queries on the views affect performance of running business processes and may cause locking issues (if a query requires a table scan on one of the underlying tables, a large amount of read-locks might be acquired that can cause lock wait or even deadlocks). 8. Summary Process Choreographer's audit log data is a powerful data stock - together with the capabilities provided by the SQL query language, it is easy to gather information about relevant data and interesting coherences of business processes. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 17 References [1] Frank Neumann: WebSphere Application Server Enterprise Process Choreographer Work items and the query() API call http://www.ibm.com/developerworks/websphere/library/techarticles/wasid/WPC_Queries /WPCQueries.html Trademark attributions and copyrights © IBM Corporation 1994-2004. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM Web site pages may contain other proprietary notices and copyright information which should be observed. The following terms are trademarks of the IBM Corporation in the United States, other countries, or both: IBM, WebSphere. Special attributions Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Oracle, Oracle 8i and Oracle 9i are trademarks of Oracle Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. Process Choreographer - Enhanced Audit Log Data Analysis and Query Page 18