* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download TMF Introduction - Hewlett Packard Enterprise
Survey
Document related concepts
Microsoft Access wikipedia , lookup
Global serializability wikipedia , lookup
Oracle Database wikipedia , lookup
Tandem Computers wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Functional Database Model wikipedia , lookup
Relational model wikipedia , lookup
Database model wikipedia , lookup
Clusterpoint wikipedia , lookup
Commitment ordering wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Versant Object Database wikipedia , lookup
Transcript
HP NonStop TMF Introduction Abstract This manual introduces the concepts of transaction processing and the features of the HP NonStop™ Transaction Management Facility (TMF). This manual is directed to business professionals, application designers and programmers, and system managers. Product Version TMF H01 Supported Release Version Updates (RVUs) This publication supports H06.05 and all subsequent H-series release version updates (RVUs) until otherwise indicated by its replacement publication. Part Number Published 540134-002 February 2006 Document History Part Number Product Version Published 540134-001 TMF H01 July 2005 540134-002 TMF H01 February 2006 HP NonStop TMF Introduction Index Examples What’s New in This Manual vii Manual Information vii New and Changed Information Figures Tables vii About This Manual ix Who Should Read This Manual? ix How This Manual is Organized x TMF Documentation x Other Documentation xi Notation Conventions xii 1. TMF Overview What Characterizes the TMF Environment? 1-3 Distributed Data 1-3 Database Distribution 1-3 Database Consistency 1-5 Transaction Concurrency 1-6 What Does TMF Do? 1-8 Transaction Protection 1-8 Database Recovery 1-10 Why Use TMF? 1-14 Point-of-Sale Application Scenario 1-14 Financial Application Scenario 1-14 Insurance Application Scenario 1-15 Travel Agency Application Scenario 1-15 Manufacturing Application Scenario 1-15 How Does TMF Integrate With OLTP Software? 1-16 Applications 1-17 Application Languages 1-17 Application Control Subsystem 1-17 Transaction Management Facility 1-17 Hewlett-Packard Company—540134-002 i 1. TMF Overview (continued) Contents 1. TMF Overview (continued) Database Management Software 1-17 NonStop OS, File System, and Disk Process 1-18 Who Uses TMF? 1-18 Application Designers and Programmers 1-18 Database Administrators and System Managers 1-18 System Operators 1-19 What Are the Benefits of TMF? 1-19 2. Transaction Coordination Coordinating Local Transactions 2-1 Coordinating Distributed Transactions 2-2 Resource Managers and Transaction Gateways 2-3 How Distributed Transaction Coordination Works 2-4 Transaction Identification 2-6 Transaction Definition by the Application 2-6 Physical Transaction Identification 2-9 Locking Mechanism 2-10 Two-Phase Commit Protocol 2-11 Summary of Transaction Coordination 2-12 3. Transaction Protection and Database Recovery Audit Trails 3-2 Master Audit Trail 3-3 Auxiliary Audit Trail 3-3 Audit-Trail Files and Rollover 3-3 Audit Volumes 3-5 Active-Audit Volume 3-5 Overflow-Audit Volume 3-5 Restore-Audit Volume 3-5 Audit Dumps 3-6 Automatic Audit Dumping 3-6 Transaction Backout 3-8 Initiating Transaction Backout 3-8 Backout for Distributed Transactions Volume Recovery 3-10 Online Dumps 3-12 File Recovery 3-13 3-9 HP NonStop TMF Introduction —540134-002 ii 3. Transaction Protection and Database Recovery (continued) Contents 3. Transaction Protection and Database Recovery (continued) Summary of Transaction Protection and Database Recovery TMF Transaction Protection 3-16 Database Recovery 3-16 3-16 4. Application Performance Performance Comparison 4-1 Performance Versus Transaction Size 4-2 Writes to Database Files 4-3 Writes to Audit-Trail Files 4-5 Performance Versus Transaction Rate 4-5 5. TMF Application Programming Overview of Application Programming 5-1 TMF Functions in Application Programming 5-1 The Requester/Server Model 5-2 Accessing and Locking Data 5-2 Controlling Transaction Isolation 5-3 Accessing SQL Data 5-5 Accessing Enscribe Data 5-5 Managing Deadlock 5-7 Controlling Transactions 5-8 Transaction Control for Single-Threaded Design 5-8 Transaction Control for Multithreaded Design 5-9 Designing Applications for the Pathway Environment 5-9 Assigning Roles to Application Modules 5-9 Developing Requesters Using SCREEN COBOL 5-11 Developing Servers for Database Access 5-12 Summary of Transaction Design Tasks for Applications in the Pathway Environment 5-16 6. TMF System Management TMFCOM Command Interpreter 6-1 Subsystem Programmatic Interface (SPI) 6-2 TMF Process Communication 6-3 Event Management 6-4 Tasks for Managing TMF 6-5 Configuring TMF 6-5 Designating Audited Files 6-5 Specifying Compression of Selected Audit Records 6-5 HP NonStop TMF Introduction —540134-002 iii 6. TMF System Management (continued) Contents 6. TMF System Management (continued) Managing Audit and Online Dumps Monitoring TMF 6-6 Managing the TMF Catalog 6-6 6-6 Index Examples Example 2-1. Transaction Control Using Embedded SQL 2-8 Figures Figure 1-1. Figure 1-2. Figure 1-3. Figure 1-4. Figure 1-5. Figure 1-6. Figure 1-7. Figure 1-8. Figure 1-9. Figure 2-1. Figure 2-2. Figure 2-3. Figure 2-4. Figure 2-5. Figure 2-6. Figure 2-7. Figure 3-1. Figure 3-2. Figure 3-3. Figure 3-4. Figure 3-5. Figure 3-6. Figure 3-7. Figure 4-1. Figure 4-2. Figure 4-3. Figure 4-4. Figure 5-1. Figure 5-2. The Role of TMF in Transaction Processing 1-1 TMF Protecting a Distributed Database 1-4 Database Consistency in a Banking Transaction 1-5 Lack of Concurrency Control 1-6 Concurrency Control 1-7 Serialized Transactions 1-9 The Audit Trail and Audited Database Files 1-11 Distributed Audit Information 1-12 The Role of TMF in Transaction Processing 1-16 Local Transaction 2-2 A Transaction Gateway 2-4 Commit Coordinator and Subordinate Transaction Managers 2-5 Transaction Control in a Pathway Environment 2-7 Relationship Between TCP and Server Processes 2-9 Two-Phase Commit 2-11 Stages of a Transaction 2-12 Audit-Trail Records 3-2 Audit-Trail Rollover 3-4 Automatic Audit-Trail Dump 3-7 Network Backout 3-9 Recovering a Database With Volume Recovery 3-11 Online and Audit Dumps for File Recovery 3-12 Restoring Files With File Recovery 3-14 Performance Comparison With and Without TMF Protection 4-1 Transaction Input/Output Example 4-2 Unbuffered and Buffered Input/Output 4-4 Throughput Comparison Depending on Transaction Rate 4-6 Deadlock Example 5-7 Modules of a Typical OLTP Pathway Application 5-10 HP NonStop TMF Introduction —540134-002 iv Figures (continued) Contents Figures (continued) Figure 5-3. Figure 5-4. Figure 5-5. Figure 5-6. Figure 6-1. Figure 6-2. Passing the Current Transaction to a Server 5-11 Mixed Database Access by an HP COBOL Server 5-13 A Long Transaction 5-14 A Context-Saving Transaction 5-15 Managing a Central and Remote Site 6-2 TMF Process Communication 6-4 Tables Table 2-1. Sample Transaction Control Statements 2-8 HP NonStop TMF Introduction —540134-002 v Contents HP NonStop TMF Introduction —540134-002 vi What’s New in This Manual Manual Information HP NonStop TMF Introduction Abstract This manual introduces the concepts of transaction processing and the features of the HP NonStop™ Transaction Management Facility (TMF). This manual is directed to business professionals, application designers and programmers, and system managers. Product Version TMF H01 Supported Release Version Updates (RVUs) This publication supports H06.04 and all subsequent H-series release version updates (RVUs) until otherwise indicated by its replacement publication. Part Number Published 540134-002 November 2005 Document History Part Number Product Version Published 540134-001 TMF H01 July 2005 540134-002 TMF H01 November 2005 New and Changed Information This edition of this manual has been updated to include the following changes: • • Section 1, TMF Overview, updates the discussions Application Control Subsystem on page 1-17 and NonStop OS, File System, and Disk Process on page 1-18 to include current product names. Section 2, Transaction Coordination, updates the discussions Coordinating Distributed Transactions on page 2-2 and Resource Managers and Transaction Gateways on page 2-3 to include current product names. HP NonStop TMF Introduction —540134-002 vii New and Changed Information What’s New in This Manual • • • Section 3, Transaction Protection and Database Recovery, revises the following discussions: ° Restore-Audit Volume on page 3-5 mentions a new option for retaining audittrail files restored from tape on restore-audit volumes indefinitely to enhance performance. ° File Recovery on page 3-13 discusses recovery of files to different volumes, subvolumes, and file IDs than the ones on which they were dumped. Section 4, Application Performance, corrects technical errors in the discussion Writes to Database Files on page 4-3. Section 5, TMF Application Programming, revises the following discussions: ° ° • Managing Deadlock on page 5-7 corrects technical terminology. Assigning Roles to Application Modules on page 5-9 and Developing Requesters Using SCREEN COBOL on page 5-11 include current product names. Section 6, TMF System Management, revises the following discussions: ° ° Configuring TMF on page 6-5 updates capabilities for audit-trails. Designating Audited Files on page 6-5 includes current product names and slight operational revisions. HP NonStop TMF Introduction —540134-002 viii About This Manual This manual introduces the basic concepts of transaction processing and the features of the HP NonStop™ Transaction Management Facility (TMF). TMF is a software product that provides transaction protection, database consistency, and database recovery on HP NonStop servers. Use this manual to learn how TMF works to protect business data in local and distributed databases. To learn more about this manual, read the following topics: Topic Page Who Should Read This Manual? ix How This Manual is Organized x TMF Documentation x Other Documentation xi Who Should Read This Manual? This manual is intended for business professionals, application designers, application programmers, and system managers who need to know: • • • How using TMF can benefit a business How TMF software maintains high OLTP application performance How to use TMF functionality to manage transactions and database recovery This manual provides the foundation for understanding all other TMF manuals. Although this manual requires a basic understanding of the NonStop server architecture and the HP NonStop operating system, readers need not have a previous transaction management background. HP NonStop TMF Introduction —540134-002 ix How This Manual is Organized About This Manual How This Manual is Organized This manual begins with the basic concepts of transaction processing and specific details of TMF implementation. Next, the manual covers application programming and system management. Depending upon your responsibilities, different sections may be of most interest to you: Section Business Professionals Application Designers & Programmers System Managers Section 1, TMF Overview X X X Section 2, Transaction Coordination X X X Section 3, Transaction Protection and Database Recovery X X X Section 4, Application Performance X X X Section 5, TMF Application Programming X Section 6, TMF System Management X TMF Documentation Readers of this manual might also refer to the following documentation describing TMF. Audience Title All readers TMF Introduction (This manual) TMF Glossary System managers and operators TMF Planning and Configuration Guide TMF Operations and Recovery Guide TMF Reference Manual Application programmers TMF Application Programmer’s Guide TMF Management Programming Manual Documentation Description • • TMF Introduction Read this manual first. It provides a general overview of TMF concepts and capabilities for business professionals, application designers and programmers, and system managers and administrators. TMF Glossary Refer to this manual to look up technical terms used in the TMF documentation set. HP NonStop TMF Introduction —540134-002 x Other Documentation About This Manual • • • • • TMF Planning and Configuration Guide Read this guide for information about how to plan, configure, and manage a TMF environment. This guide also describes how to move your applications between an earlier TMF product version and the current one. TMF Operations and Recovery Guide Read this guide for information about how to perform and monitor standard TMF operations, obtain online and audit dumps, and respond to a variety of TMF exception conditions. TMF Reference Manual Refer to this manual for information about how to use the TMFCOM command interface to the TMF software. This manual includes syntax, cautionary considerations, and command examples for TMFCOM. TMF Application Programmer’s Guide Read this guide for information about how to design requester and server modules to run in the TMF programming environment. Also read it for information about a set of procedures that are helpful in examining the content of TMF audit trails. TMF Management Programming Manual Read this manual for information about how to use the Subsystem Programmatic Interface (SPI) to communicate programmatically with the TMF software. Also read it for information about how to programmatically interpret Event Management Service (EMS) event messages generated by TMF. Other Documentation Several other HP products and interfaces support limited TMF operations. For information about how they relate to TMF, see the introductions, user guides, and reference manuals covering these products: • • • • • • • • • • • NonStop Operating System Procedure Calls NonStop Operating System TACL Command Interface NonStop SQL Relational Database Management System NonStop Transaction Services (TS/MP) NonStop TUXEDO Distributed Systems Management/Software Configuration Manager (DSM/SCM) Event Management Service (EMS) File Utility Program (FUP) Measure Subsystem Pathway/iTS Surveyor Subsystem HP NonStop TMF Introduction —540134-002 xi Notation Conventions About This Manual • Various programming languages, including HP COBOL for NonStop Systems, Pathway SCREEN COBOL, FORTRAN, TAL, C, C++, and SQL (NonStop SQL implementation). Notation Conventions Hypertext Links Blue underline is used to indicate a hypertext link within text. By clicking a passage of text with a blue underline, you are taken to the location described. For example: This requirement is described under Backup DAM Volumes and Physical Disk Drives on page 3-2. Change Bar Notation Change bars are used to indicate substantive differences between this edition of the manual and the preceding edition. Change bars are vertical rules placed in the right margin of changed portions of text, figures, tables, examples, and so on. Change bars highlight new or revised information. For example: The message types specified in the REPORT clause are different in the COBOL environment and the Common Run-Time Environment (CRE). The CRE has many new message types and some new message type codes for old message types. In the CRE, the message type SYSTEM includes all messages except LOGICAL-CLOSE and LOGICAL-OPEN. HP NonStop TMF Introduction —540134-002 xii 1 TMF Overview TMF furnishes transaction protection, database consistency, and database recovery critical in high-volume transaction processing. It sustains high performance for online transaction processing (OLTP) applications, as well as online query processing (OLQP) activities, batch processing applications, and decision-support systems. This section introduces TMF, discussing these topics: Topic Page What Characterizes the TMF Environment? 1-3 What Does TMF Do? 1-8 Why Use TMF? 1-14 How Does TMF Integrate With OLTP Software? 1-16 Who Uses TMF? 1-18 What Are the Benefits of TMF? 1-19 To support OLTP applications, TMF can monitor thousands of complex transactions sent by hundreds of users to a common database. The database can be distributed among many disks on a system (node), or across many nodes in a network. The database can be a SQL relational database, an Enscribe database, or a combination of both. The role of TMF in transaction processing is illustrated in Figure 1-1. TMF also supports heterogeneous transaction processing through the use of gateways. In this kind of processing, TMF on an HP NonStop™ server communicates with a foreign transaction manager on another platform to cooperate in transactionbased activities. Figure 1-1. The Role of TMF in Transaction Processing \SYS OLTP Software Transaction Entry TMF Database Log of Transaction Records VST001.vsd HP NonStop TMF Introduction —540134-002 1 -1 TMF Overview For OLTP applications, TMF provides: • • All the complex operations for concurrent transactions and database consistency, making these operations transparent to both users and application programmers. Database recovery through three mechanisms: • • • Transaction backout, which recovers the database from application or transaction failure. Volume recovery, which recovers the database in the most typical recovery situations: a disk volume or system failure. File recovery, which reconstructs database files when the copy on the data volume is not usable. TMF performs these operations while maintaining high performance for applications. In addition, TMF simplifies the tasks of users who develop and manage OLTP applications. HP NonStop TMF Introduction —540134-002 1 -2 What Characterizes the TMF Environment? TMF Overview What Characterizes the TMF Environment? TMF works in the OLTP environment to protect transactions for business operations and maintain the consistency of the database. In general, a transaction is a multistep operation that changes the database. For example, a transaction for an airline reservation could include the operations of adding the reservation to the airline passenger list, issuing a ticket, and adding the ticket price to accounts receivable. OLTP applications, in particular, are critical applications for businesses. These applications require timely information that reflects the current state of the business. The transactions and database must be protected because many users are accessing the same database. TMF manages two major processing problems: • • Database consistency Transaction concurrency TMF manages these problems regardless of the distribution of the data. Distributed Data Often OLTP applications must access data that resides in different computer systems at customer sites, in different types of databases, or on different platforms. To access data on other platforms, TMF can interoperate with foreign transaction managers and resource managers through transaction gateways, which translate from one protocol to another. Database Distribution Frequently, the databases in OLTP applications are distributed so that data resides where it is most frequently used. This distribution contrasts with undistributed data that resides at a central site. A distributed database is illustrated in Figure 1-2. Both manufacturing and sales databases reside in Seoul and Chicago, and only sales databases reside in San Francisco and London, where there are only sales offices. TMF is active at all sites, to protect the transactions and the database. Distribution refers either to the location of data or to the location of processes that access the data. A distributed database resides on more than one node of a network; nodes (computer systems) can be dispersed geographically. Distributed transactions involve transaction manager activity on more than one node. Transactions processed on two or more nodes by TMF are called homogeneous distributed transactions. Transactions processed under the control of multiple transaction managers (TMF on a NonStop server and one or more foreign transaction managers on one or more other platforms) are called heterogeneous distributed transactions. HP NonStop TMF Introduction —540134-002 1 -3 Database Distribution TMF Overview TMF features that maintain consistency for distributed databases are transparent to application programmers. TMF equally protects transactions that affect only a local database and transactions that affect a database distributed on many nodes. Figure 1-2. TMF Protecting a Distributed Database \SAN FRANCISCO \CHICAGO TMF TMF SALES MANUF SALES \LONDON \SEOUL TMF TMF MANUF SALES SALES VST002.vsd HP NonStop TMF Introduction —540134-002 1 -4 Database Consistency TMF Overview Database Consistency When you design a database, you establish criteria for the relationships among its data; for example, an account balance must equal credits to the balance minus debits to the balance. When the database satisfies these criteria, the database is considered to be consistent. The changes a transaction makes to a database should transform the database from one consistent state to a new consistent state. The changes, however, can make the database inconsistent at points during the transformation. For example, consider transaction T1 that transfers $500 from a savings account to a checking account, as shown in Figure 1-3. The database is inconsistent after the debit on savings (Operation 1) but before the credit on checking: at that moment, the database violates the criteria for balancing accounts because the $500 debited is not represented. After the credit (Operation 2), the database is consistent again. Figure 1-3. Database Consistency in a Banking Transaction BEGIN T1 Transfer $500 from SAVINGS to CHECKING Operation 1 Debit SAVINGS $500 Operation 2 Credit CHECKING $500 END T1 SAVINGS CHECKING Balance Balance State of Database $4500 $50 Consistent $4000 $50 Inconsistent $4000 $550 Consistent VST003.vsd TMF provides database consistency by ensuring either that all the operations in the transaction are applied to the database as a unit or that none of the operations affect the database. An application delimits the operations of a transaction with logical begin and end transaction statements, thus defining the TMF transaction. With the transaction defined, TMF treats all operations in the transaction as a unit. HP NonStop TMF Introduction —540134-002 1 -5 Transaction Concurrency TMF Overview Transaction Concurrency To process the transaction correctly, an application program must have consistent input from the database, regardless of any concurrent changes being made to the database. To illustrate the problems introduced when there is no protection for concurrency, assume two concurrent transactions each check the balance of an account and then deposit funds to the account (see Figure 1-4). Transaction T1 queries the balance, which is $50, and then deposits $500 to the account. Another concurrent transaction, T2, queries the account balance, determines the balance is $50, and deposits $100 to the balance. These concurrent transactions demonstrate the problem when the $500 transferred by T1 is lost. Figure 1-4. Lack of Concurrency Control CHECKING Balance BEGIN T1 Get current balance ($50) Deposit $500 NEWBAL=$50+$500 Writes NEWBAL of $550 $50 $550 $150 END T1 BEGIN T2 Get current balance ($50) • • • Deposit $100 NEWBAL=$50 +$100 Writes NEWBAL of $150 END T2 ERROR VST004.vsd Both transactions T1 and T2 retrieve the balance information of $50. Then, because of a lack of concurrency control, T1 is overwritten by T2. This is just one of the problems in managing concurrent transactions. HP NonStop TMF Introduction —540134-002 1 -6 Transaction Concurrency TMF Overview TMF provides concurrency control by managing locks on data accessed and updated in a TMF transaction. The locks isolate transactions from one another so that each transaction appears to be operating alone on the database. This allows database access by many users. Figure 1-5 illustrates concurrency control. Figure 1-5. Concurrency Control CHECKING Balance BEGIN T1 Get current balance ($50) $50 Deposit $500 T1 locks the account NEWBAL=$50+$500 Writes NEWBAL of $550 BEGIN T2 Get current balance ($50) • • • $550 Deposit $100 Account is locked; T2 must wait • • • END T1 T1 releases the lock Deposit $100 T2 locks the account $650 NEWBAL=$550 +$100 Writes NEWBAL of $650 END T2 T2 releases the lock VST050.vsd HP NonStop TMF Introduction —540134-002 1 -7 What Does TMF Do? TMF Overview What Does TMF Do? TMF protects transactions and performs database recovery: • • • It ensures database consistency by protecting transactions from many potential hazards, including program failures, system component failures, and communication failures. It protects transactions from one another, thus providing database consistency despite concurrent transactions. It performs database recovery by using transaction-audit information. Transaction Protection Because transactions usually consist of a series of operations, processing more than one transaction at a time can threaten database consistency and concurrency, making transaction management complex; TMF manages this complexity. How TMF manages transactions is best described by summarizing the properties of transactions: • • • Each transaction is a single unit. Transactions can be serialized. Transactions are permanent. Transactions that have these properties are fully protected. By ensuring these properties, TMF provides the maximum protection for transactions and ensures database consistency. A TMF Transaction Is a Unit TMF protects the transaction as a single unit, making sure that either all or none of the changes in a transaction are applied to the database. For example, consider the previously described money transfer transaction (Figure 1-3), which consists of two operations on the database. If a power failure occurred just before the credit (Operation 2) and interrupted the system, TMF would ensure that when the system restarted, Operation 1 (the debit) would not be applied to the database. TMF protects the database by backing out any partial transactions applied to the database. Application programs must designate the beginning and ending of a transaction. TMF manages all the operations to commit or back out the transaction, as follows: • • When a transaction ends and all the work is applied to the database, the transaction is committed. When a transaction ends but none of the work is to be applied, the transaction is logically rolled back. In TMF terminology, this is an aborted transaction. TMF HP NonStop TMF Introduction —540134-002 1 -8 Transaction Protection TMF Overview performs an abort operation by backing out all changes the transaction made to the database. TMF Transactions Can Be Serialized TMF protects the transaction from other concurrent transactions so that each transaction appears to operate alone against the database. It provides this protection by locking rows or records that might be inserted, updated, or deleted. The locks are held until the transaction is committed. The locks prevent other transactions from accessing data affected by uncommitted updates. The transaction concurrency example in Figure 1-6 illustrates two transactions. Locks provide concurrency control, and the final balance receives credits from both transactions, T1 and T2. Figure 1-6. Serialized Transactions CHECKING BEGIN T2 BEGIN T1 Balance $50 Reads balance of $50 (with lock) NEWBAL=$50+$500 Writes NEWBAL of $550 Balance $550 END T1 Release Reads balance request (with lock) • • • • • Access balance of $550 NEWBAL = $550 + $100 Balance $650 Writes NEWBAL of $650 Release END T2 Legend Denotes record lock for transaction 1 (T1) Denotes record lock for transaction 2 (T2) VST005.vsd HP NonStop TMF Introduction —540134-002 1 -9 Database Recovery TMF Overview TMF Transactions Are Permanent Once a transaction is committed to the database, all the changes made by the transaction remain in the database, despite any subsequent failure. TMF recovery features ensure transaction permanence. Database Recovery TMF maintains the consistency of the database if a transaction fails to complete successfully, even if any of the following failures or problems occur: • • • • • • A TMF operator aborts a transaction that an application had previously initiated. Program logic determines that an initiated transaction cannot complete and, therefore, aborts the transaction. Some single-component failure interrupts a transaction, causing it to be aborted: for example, the processor that is processing the transaction fails. Both disks of a mirrored pair fail while transactions are changing data on the disks. The primary disk process controlling a disk fails while transactions are changing data on the disk. A system failure occurs because: • • Multiple system components (processors or disks) fail and cause the entire system to become unavailable. An extended power failure or human error causes a total system failure. TMF provides database recovery for failures ranging from a single-component failure to a total system failure, using the following features: • • • • • Audit trails Transaction backout Volume recovery Online dumps and audit dumps File recovery HP NonStop TMF Introduction —540134-002 1- 10 Database Recovery TMF Overview Log of Updates in Audit Trails An audit trail, sometimes called a “transaction log,” is a series of files in which TMF records information about a transaction’s changes to a database. An audit trail contains audit records. Audit records contain information about each transaction, such as the final outcome of transaction commit or abort operations, as well as before-images and after-images of all data records changed by each transaction. • • Before-images are copied portions of the data records before the transaction changed them; they are used to back out a transaction. After-images are copied portions of the changed records; they are used in volume recovery and file recovery. The files or tables protected by TMF are called audited files. Only audited files have change records logged to audit trails. Files that are not protected by TMF are nonaudited files and do not have changes logged. Site personnel can choose which files are to be audited on a file-by-file basis, depending on application requirements. The relationship of an audit trail to audited database files is shown in Figure 1-7. Figure 1-7. The Audit Trail and Audited Database Files $AUDIT R1 R2 R3 Audit Trail (Before-images and After-images) and Control Records $DB Server R1 R2 R3 R4 Audited Files Non-Audited File UPDATES UPDATES VST040.vsd HP NonStop TMF Introduction —540134-002 1- 11 Database Recovery TMF Overview The audit-trail records contain both logical and physical audit information. The records include before-images and after-images of audited data, and records of certain transaction status information. Audit trails also include records about operations that affect the physical structure of data files, such as block splits. If the database is distributed over two or more nodes of a network, TMF maintains separate audit trails on each node. Audit information for a distributed database is shown at each node in Figure 1-8. Figure 1-8. Distributed Audit Information \SAN FRANCISCO TMF \CHICAGO AUDIT SALES TMF MANUF AUDIT SALES \LONDON \SEOUL TMF MANUF TMF AUDIT AUDIT SALES SALES VST006.vsd When a transaction is distributed over two or more systems, the before-images and after-images are written to an audit trail on the node where the changed file resides. For files partitioned across nodes, the before-images and after-images are written to the audit trail on the node where the changed partition resides. Transaction status records are written to the audit trails on all nodes that perform any work for a transaction. HP NonStop TMF Introduction —540134-002 1- 12 Database Recovery TMF Overview Transaction Backout Transaction backout reapplies before-images to database records to undo the effects of an aborted transaction. If the transaction affects data distributed over a network, backout occurs independently at each network node. If the transaction is distributed, backout is managed independently by each participating transaction manager or resource manager. The result of backing out a transaction is the same as if the transaction’s changes had never occurred. Volume Recovery After most volume or system failures in a NonStop system, volume recovery provides automatic and relatively quick (usually only a few minutes) database recovery. Volume recovery ensures that the effects of all committed transactions are applied to the database files, and that the effects of uncommitted transactions are backed out. Online Dumps and Audit Dumps Online dumps are copies of database files that are dumped, or copied, to a secondary storage medium while the application is running and the database is online. Similarly, audit dumps are copies of audit-trail files that are dumped while the application is running. The dump destination can be either local or remote. Online dumps with audit dumps support file recovery, the ability to recover database files. The online dumps preserve the data as of a particular point in time; the audit dumps, along with current audit-trail files, record the changes made to the data since that time. To recover the database, TMF restores the dumps to disk and applies changes from the audit records to the database. File Recovery Explicitly initiated by the operator, file recovery recovers database files after a disk media failure or incorrect program update. It uses online and audit dumps to restore consistency to damaged files. During file recovery, TMF first restores the database files affected by the failure, using online dumps. Then it searches the audit trail for the audit records of all transactions that changed the database files since the last online dump of these database files, redoes all changes, and backs out the changes of uncommitted transactions. In this way, file recovery ensures that the effects of committed transactions are written to the files and that the effects of aborted transactions are backed out of the files. HP NonStop TMF Introduction —540134-002 1- 13 Why Use TMF? TMF Overview Why Use TMF? OLTP applications are characterized by concurrent operations on shared data that is often distributed in a network of systems. They require database consistency, almost continuous data availability, and very high performance: all provided by TMF. Each of the following scenarios presents an OLTP problem that can occur, and discusses the TMF solution to that problem. Point-of-Sale Application Scenario Problem: Cancelling a Transaction A clerk using an OLTP point-of-sale application finishes recording 11 pieces of clothing purchased by customer Smith and submits the charge card number for authorization of the credit purchase. The reply is negative: “Do not accept the charge.” Assume the customer has no alternate method of payment. Solution: Transaction Backout The clerk has to cancel the transaction. Because this is an OLTP application, all the purchases were recorded as they were entered; now they must be undone. Using transaction backout, TMF undoes all the individual sales in the transaction. In this point-of-sale application, TMF ensures all operations in the transaction are undone, so the database remains consistent. The clerk simply needs to cancel the transaction. Financial Application Scenario Problem: Node Failure A broker enters a transaction to sell 200 shares of stock just as the node in which the server executing the transaction on the database fails. Solution: Transaction Backout TMF detects the failure and backs out the transaction. The broker’s program determines that the transaction has failed and automatically restarts it. This second attempt is successful. In this stock transaction, TMF protects the transaction by ensuring that it completes as a unit. TMF also keeps the database consistent despite the failure of a major component, the node. The broker notices none of these problems because the submitted transaction completes quickly. HP NonStop TMF Introduction —540134-002 1- 14 Insurance Application Scenario TMF Overview Insurance Application Scenario Problem: Communication Failure An insurance agent at a remote site submits information on the sale of a new policy. A fierce storm interrupts all communication lines so that the transaction cannot reach headquarters to record information about the sale. Solution: Transaction Backout TMF detects that the headquarters system is not available and backs out all the operations in the transaction to maintain database consistency. The insurance agent does not have to worry about what part of the new policy was recorded: the agent just has to wait until the required systems are available to resubmit the transaction about the new policy. Travel Agency Application Scenario Problem: Stopping the System The air conditioning fails in the computer room of a travel agency, and the room temperature gets so hot that the computers have to be shut down to avoid physical damage. The screens of all the agents using the system stop responding to input. All the transactions in progress are interrupted. Solution: Volume Recovery Computer room personnel fix the air conditioning and cold load the system. Then the database must be recovered. Recovery happens automatically when TMF is started. The volume recovery process backs out any changes made by incomplete transactions, which were all transactions in progress at the time of the shutdown. Volume recovery puts the database online in a consistent state. The agents with transactions in progress just before the shutdown must verify whether their transactions completed, and resubmit any that were incomplete. The agents do not have to be concerned about partially completed transactions because TMF ensures either that the whole transaction is applied or that none of it is. Manufacturing Application Scenario Problem: Disk Volume Failure In a manufacturing application, the head crashes on a nonmirrored volume that stores a portion of the local inventory database. The volume has to be replaced and recovered. Solution: File Recovery Applications that require inventory data run with no problems as long as the applications do not require data from the damaged volume. Any transactions in progress that changed data on that volume are backed out by TMF to prevent partial HP NonStop TMF Introduction —540134-002 1- 15 How Does TMF Integrate With OLTP Software? TMF Overview transactions, which can cause the database to be inconsistent. TMF maintains the consistency of the company’s database. In this manufacturing application, TMF and the database management system allow the work of the company to continue, except for work requiring data from the damaged volume. After replacing the damaged disk, site personnel use the file recovery facility to restore the database to a consistent state that represents the database just before the crash. The inventory data residing on the volume is then ready for more transactions. In this manufacturing application, TMF recovers the damaged volume. How Does TMF Integrate With OLTP Software? TMF must be integrated with other HP products to protect transactions and database consistency. The relationship of TMF to other products (in a Pathway transaction processing environment) is illustrated in Figure 1-9. Descriptions of these components follow. Figure 1-9. The Role of TMF in Transaction Processing Applications (optionally generated by Pathmaker) Application Languages Applic. Control Subsystem Embedded SQL Application C C++ HP COBOL NonStop TUXEDO Application Enscribe Application SCREEN COBOL TAL Java Pathway/iTS CORBA Application Embedded SQL FORTRAN NonStop TS/MP Transaction Mgt. Subsystem NonStop TMF File System NonStop SQL Enscribe Operating System NonStop OS VST107.vsd HP NonStop TMF Introduction —540134-002 1- 16 Applications TMF Overview Applications Several different types of applications can use TMF for transaction management, including the following: • • HP NonStop SQL provides SQL preprocessors and a SQL compiler. With these tools, programmers can create their own programmatic interfaces to a SQL database by compiling and executing applications that use embedded SQL statements to query the database. HP NonStop TUXEDO serves as an open transaction processing monitor for HP NonStop servers running the HP NonStop Open System Services (OSS) environment. Pathmaker is an application development tool that helps programmers define screens and transactions for applications. After programmers design the screens, the data items, and the transactions, Pathmaker can generate requesters, servers, and a configuration file. Pathmaker can generate applications for both SQL databases and Enscribe databases. Application Languages Application languages are used to code programs that access audited database files. SCREEN COBOL is the language designed for displaying and receiving data in screen programs, which are referred to as “requesters.” Other languages— C, C++, HP COBOL, FORTRAN, and TAL—can be used to code servers. SQL supports SQL statements embedded in host programs in such languages as C, C++, HP COBOL, and TAL to access a SQL database. Application Control Subsystem The HP NonStop Pathway/iTS product provides a simplified, terminal-oriented interface for transaction design and control. Pathway/iTS allows programs that are written to run at a single terminal to be used at hundreds of terminals. Most transactions are started by users entering them through terminals, so Pathway/iTS provides for communication with TMF about transactions in progress. The TS/MP software provides the underlying server support for Pathway/iTS. Transaction Management Facility TMF maintains consistency for the databases that are changed by the transactions, and ensures that transactions access only consistent data. Database Management Software SQL is a distributed relational database management system. SQL requires TMF to protect the data dictionary and uses it to protect user data in audited tables. HP NonStop TMF Introduction —540134-002 1- 17 NonStop OS, File System, and Disk Process TMF Overview Enscribe is a record manager for accessing and changing Enscribe database files. TMF protects user data in audited Enscribe files. NonStop OS, File System, and Disk Process The NonStop OS, file system, and disk process provide several features to maintain data integrity, including continuous availability, structural integrity of files, protection of storage media, and data-sharing protection. These features help to ensure that changes are made successfully to a database, regardless of any single-component failures. The NonStop OS provides a reliable environment so that NonStop TS/MP and Pathway/iTS, TMF, SQL, and Enscribe operations can proceed, regardless of singlecomponent failures. Who Uses TMF? Because TMF is integrated with HP OLTP software and has many interfaces, several kinds of users are affected by it: application designers and programmers, database administrators and system managers, and system operators. Each of these groups is responsible in its own way for providing the most effective use of TMF. Application Designers and Programmers TMF can be used by applications written in a number of languages implemented for NonStop servers. Although TMF makes the task of application programming easier, programmers must be aware of TMF requirements, and design their applications to: • • Include the appropriate set of operations to define the beginning and end of each transaction. Use TMF’s ability to lock data that will be changed by coding applications for the locking techniques required for the database. Database Administrators and System Managers Database administrators and system managers must ensure the adequacy of system facilities required to support TMF. Their responsibilities include: • • • • Determining and specifying which files in a database should be audited. Configuring characteristics of audit trails such as size, location, volumes for audittrail overflow, and volumes for restoring audit-trail files. Configuring audited data volumes, and specifying the audit trail to which TMF sends audit records that result from audited data volume activity. Determining how often to make audit dumps and how many generations of dumps to retain. HP NonStop TMF Introduction —540134-002 1- 18 System Operators TMF Overview • Determining how TMF is to be controlled and monitored: • • Through TMFCOM, the command interface for TMF. Through TMFSERVE, which provides programmatic access to TMF through the Subsystem Programmatic Interface (SPI). System Operators System operators are responsible for the following tasks: • • • • • • • Monitoring TMF status. Starting and stopping TMF. Reading event messages to determine the status of recovery and other operations. Generating routine dumps, such as online dumps, audit dumps, and online snapshots. Mounting and unmounting secondary storage media for dumps. Starting the file recovery process after a media failure or an accidental purge. Mounting secondary storage media as requested by the file recovery process. What Are the Benefits of TMF? Using TMF provides these benefits: • Increased programmer productivity for application development and maintenance Programmers can focus on the business tasks to be accomplished by transactions, rather than writing code to protect transactions and recover databases. • Database consistency Transactions are applied consistently to the database, despite potential accidents such as program failures, system failures, and media failures. • Transaction and database protection for distributed data TMF maintains consistency transparently across all nodes of a network-distributed database, regardless of where the transaction changing the database originates or how the database is distributed geographically. • Very high database availability Because TMF provides volume recovery to a database after a system failure, a business loses only a few minutes to recovery downtime for most recoveries. HP NonStop TMF Introduction —540134-002 1- 19 What Are the Benefits of TMF? TMF Overview • Increased high performance for transaction processing Because TMF is integrated with the file system and disk process, using it increases performance for high-volume OLTP applications. TMF provides database protection and increased throughput. • Database recovery The volume recovery and file recovery facilities together provide complete protection for database files. • An application-readable log of transactions Application programs can use TMF audit-reading routines to read the transaction records written to an audit trail. HP NonStop TMF Introduction —540134-002 1- 20 2 Transaction Coordination TMF coordinates transactions to ensure database consistency and provide maximum concurrent access to shared data. This coordination takes place regardless of: • • The location of the data, whether local or remote. The involvement of other software (such as transaction gateways and resource managers) used to support distributed transactions. TMF coordinates transactions through transaction identification, a locking mechanism, and the two-phase commit protocol. To explain transaction coordination, this section discusses the following topics: Topic Page Coordinating Local Transactions 2-1 Coordinating Distributed Transactions 2-2 Transaction Identification 2-6 Locking Mechanism 2-10 Two-Phase Commit Protocol 2-11 Summary of Transaction Coordination 2-12 Coordinating Local Transactions Local transactions are transactions that involve requesters, servers, and files that all reside on the same system (node). Local transactions, in other words, are processed by TMF on a single node. Figure 2-1 illustrates a sample node (\DALLAS) with two audited volumes ($DATA1 and $DATA2), and one active-audit volume ($AUDIT). A transaction could access data on both audited volumes; TMF coordinates the transaction’s activity at the node. HP NonStop TMF Introduction —540134-002 2 -1 Coordinating Distributed Transactions Transaction Coordination Figure 2-1. Local Transaction \DALLAS $DATA1 User Application $DATA2 TMF $AUDIT VST043.vsd Coordinating Distributed Transactions Distributed transactions are transactions that access data located at multiple nodes of a network, or that access data by means of requesters or servers at multiple nodes of a network. There are two kinds of distributed transactions: • • Homogeneous transactions are distributed transactions that are processed by the same type of transaction manager operating on the same platform. For example, part of the transaction may be processed by TMF at NonStop Server (Node) A and another part may be processed by TMF at NonStop Server (Node) B. Heterogeneous transactions are distributed transactions that are processed by different types of transaction managers operating on multiple platforms. For example, one part of a transaction may be processed by TMF on a NonStop server, and another part by TUXEDO or BEA WebLogic Server running on another platform. As another example, WebLogic Server can start a transaction on a NonStop server and then subcontract parts of it to WebLogic Server running on one or more foreign servers. Alternatively, WebLogic Server on another type of system can begin a transaction and then subcontract parts of it to the NonStop server. TMF is an open transaction manager; its commit protocol is open and public, and it supports coordination with resource managers and remote transaction managers through the use of transaction gateways. These elements work together to make possible any kind of distributed transaction processing, whether homogeneous or heterogeneous. They enable HP customers to integrate data located on many different types of databases and platforms to conduct their business operations. HP NonStop TMF Introduction —540134-002 2 -2 Resource Managers and Transaction Gateways Transaction Coordination Resource Managers and Transaction Gateways In general, a resource manager is a software entity that manages a certain part of a computer system’s shared resources. Many other software entities can request access to these resources from time to time, using services that the resource manager provides. This general concept of a resource manager is treated extensively in the literature of the Open Group, a vendor and technology-neutral industrial consortium dedicated to enterprise interoperability. Note. For more information about heterogeneous transaction processing and resource managers in general, see the Open Group Transaction Processing Publications, which can be ordered through many bookstores and from the following Web location: http://www.opengroup.org/pubs/catalog/tp.htm When the shared resources are persistent, such as a database that contains information on disk, the resources of a transaction manager are used to maintain the consistency of that data across updates. With respect to distributed transaction processing, a resource manager is a collection of routines that enables cooperation between different transaction management systems running on different platforms. A resource manager manages transaction objects and provides services to applications or other resource managers. For example, a resource manager may be a database system, such as NonStop SQL, or a transaction queue manager. Resource managers are run by a process known as a transaction gateway. The gateway provides communication between the local and remote transaction managers. This capability, in turn, enables communication between different processing environments—for example, between TMF running on a NonStop server and a foreign transaction manager running on another platform. The gateway executes the resource manager routines to support this communication. (In fact, the gateway itself is regarded as a communication resource manager (CRM) in Open Group terminology.) The gateway handles all communications between the local and remote nodes regarding transactions. In particular, the local transaction manager communicates with a remote node through the gateway. The local transaction manager (TMF on a NonStop system) treats the gateway as a surrogate resource manager; it does not directly communicate with any remote resource manager. In the remote node, the gateway notifies the remote transaction manager that it has a subordinate relationship to the originating transaction manager. Through the gateway, the local and remote transaction managers cooperatively participate in an agreement protocol to determine the outcome of individual transactions. HP NonStop TMF Introduction —540134-002 2 -3 How Distributed Transaction Coordination Works Transaction Coordination Transaction gateways translate between one protocol and another; they translate imported transactions and transaction IDs to the local protocol, and translate exported transactions and transaction IDs to the remote protocol. The transaction manager using a gateway does not need to know the characteristics of the object it is communicating with at the other end of the gateway; it only needs to know that the object follows the protocol supported by the gateway. You can think of the object at the other end of the gateway as a “black box,” or unknown object as shown in Figure 2-2; TMF makes no assumptions about what the object is. Figure 2-2. A Transaction Gateway Transaction Gateway TMF Transactions Remote TM or RM “Black Box” VST049.vsd How Distributed Transaction Coordination Works Although multiple transaction managers are involved in distributed transactions, one transaction manager must be responsible for determining whether to commit or abort the transaction for all participants. This transaction manager is called the commit coordinator; it coordinates with the other participants to commit or abort the transaction. The commit coordinator is the transaction manager that initiates the transaction. The commit coordinator acts as the parent or superior transaction manager (as shown in Figure 2-3) and records the other child or subordinate systems involved in the transaction. Parent nodes send transactions; child nodes receive transactions. By recording these nodes, the commit coordinator can make sure the transaction is eventually committed or backed out on all participating nodes. HP NonStop TMF Introduction —540134-002 2 -4 How Distributed Transaction Coordination Works Transaction Coordination Figure 2-3. Commit Coordinator and Subordinate Transaction Managers Commit Coordinator Subordinate TM Subordinate TM VST053.vsd The transaction management process (TMP) on each participating node records the parent nodes and child nodes for each transaction. A transaction branch refers to that part of a distributed transaction being processed by one resource manager or transaction manager. Coordination of Child Transactions For child, or outgoing, transactions, TMF treats the gateway as a surrogate resource manager; the gateway manages the remote resource. TMF does not directly communicate with the remote transaction manager or resource manager. The local transaction manager views the gateway as a resource manager; the remote system views the gateway as a transaction manager. Coordination of Parent Transactions For parent, or incoming transactions, TMF acts as a subordinate transaction manager; it commits or aborts the transaction branch as directed by the superior transaction manager. In this situation, TMF is the local commit coordinator. HP NonStop TMF Introduction —540134-002 2 -5 Transaction Identification Transaction Coordination Transaction Identification As discussed in Section 1, TMF Overview, an application commonly makes a series of changes to a database and transforms the database from one state to another. During the transformation, some of the intermediate states of the database fail to satisfy one or more application consistency rules. To solve this problem, a TMF transaction is defined as a group of changes to a database that transform the database from one consistent state to another (despite the fact that individual changes usually make the database inconsistent). This group of changes, then, is logically defined in an application by the transaction’s collective beginning and end. When an application initiates a transaction, TMF associates that transaction with a unique identifier. As TMF manipulates the transaction, it associates that unique identifier with each of the individual changes that make up the transaction. Transaction Definition by the Application Programmers define TMF transactions by designating the operations for the logical work of the transaction. The application specifies the logical work of the transaction; TMF makes all the complex processing involved in transaction control transparent to the programmer. The transaction control statements and the response of TMF are described next. In the Pathway environment, transaction control typically occurs in a requester program, as illustrated in Figure 2-4. The requester, which is a SCREEN COBOL program, uses these transaction control statements: BEGIN-TRANSACTION, END-TRANSACTION, and ABORTTRANSACTION. The server program accesses the database on behalf of the requester program. Servers typically include abort-transaction statements for error handling if they need to undo a transaction. Servers are coded in C, C++, HP COBOL, FORTRAN, or TAL; they can also use SQL statements embedded within most of those languages. HP NonStop TMF Introduction —540134-002 2 -6 Transaction Definition by the Application Transaction Coordination Figure 2-4. Transaction Control in a Pathway Environment File System Requester TMF Log of Transaction Records Server Database TCP SCREEN COBOL Program Transaction Entry ... ... BEGIN-TRANSACTION. ... DECLARE CURSOR... ... READ MSG-IN... OPEN cursor... SEND MESSAGE MSG-OUT... REPLY CODE... ... END-TRANSACTION. ... ... FETCH... ON-ERROR-PROCEDURE. ABORT-TRANSACTION. ... UPDATE... CLOSE cursor... ... ... WRITE REPLYMSG... ... SQL- HP COBOL Program VST008.vsd Begin Transaction The application begins a new TMF transaction, indicating that all subsequent database access is part of the transaction. TMF generates a transaction identifier and sets up all the internal work to begin monitoring a transaction, and then associates the application’s subsequent access of audited database files with the transaction. End Transaction The application ends (commits) a TMF transaction, indicating that all the database changes since the begin-work designation are permanent. TMF makes permanent all the work performed for the duration of the transaction on audited files. HP NonStop TMF Introduction —540134-002 2 -7 Transaction Definition by the Application Transaction Coordination Abort Transaction The application aborts (cancels) a TMF transaction, indicating that any database changes made since the begin-work designation are to be undone. TMF undoes any of the work performed by the transaction on audited files, and ensures that no part of the work is applied to the database. Transaction Control Statements in Programs To designate the transaction control operations, application programmers may include transaction control statements in their programs. They can put these statements in any appropriate module of the application. The syntax to specify these transaction control operations is different for different programming languages and application programming interfaces (APIs), as shown in Table 2-1. Table 2-1. Sample Transaction Control Statements Language or API Begin Transaction End Transaction Abort Transaction SQL BEGIN WORK COMMIT WORK ROLLBACK WORK TMF API BEGINTRANSACTION ENDTRANSACTION ABORTTRANSACTION Guardian System Procedure Calls (TAL) BEGINTRANSACTION ENDTRANSACTION ABORTTRANSACTION SCREEN COBOL BEGIN-TRANSACTION END-TRANSACTION ABORT-TRANSACTION Transaction control statements in an embedded SQL application are illustrated in Example 2-1. Example 2-1. Transaction Control Using Embedded SQL EXEC SQL BEGIN WORK; ... EXEC SQL UPDATE ...; ... if (!strcmp(SQLSTATE, SQLSTATE_OK)) EXEC SQL COMMIT WORK; else EXEC SQL ROLLBACK WORK; HP NonStop TMF Introduction —540134-002 2 -8 Physical Transaction Identification Transaction Coordination Physical Transaction Identification Each TMF transaction is distinguished from other transactions by a transaction identifier or transid. A transaction is a uniquely identified entity known to the system and exists separately from the application that defines the sequence of operations in the transaction. A transaction can span processes; therefore, the system associates a current transaction identifier with each process. For example, after a SCREEN COBOL program has initiated a new transaction identifier by issuing BEGIN-TRANSACTION, the terminal control process (TCP) and server processes cooperate to accomplish the changes required by the transaction, as illustrated in Figure 2-5. TCPs and servers communicate through interprocess messages. To ensure that each action is identified with a specific transaction, the transaction identifier is attached to each message sent by any of the processes. When a process reads the message, the message’s transaction identifier becomes the current transaction identifier for that process. Figure 2-5. Relationship Between TCP and Server Processes Server Class A Begin Trans #1 Transid1 TCP Begin Trans #2 All work by the selected server from Server Class A is identified with Transid1 . Transid1 Update Disk Server Class B Transid2 Database Transid2 All work by the selected server from Server Class B is identified with Transid2 . VST009.vsd A TCP supports the overall processing flow of each terminal by sending messages to the appropriate server processes through the system file $RECEIVE. Since a single transaction can result in a multistep update requiring several servers, each server is associated with the transaction identifier of the SCREEN COBOL program that issued BEGIN-TRANSACTION. All operations on the database are identified with the transaction, and all locks obtained for database access are owned by the transaction, allowing several servers to do work concurrently for the same transaction. HP NonStop TMF Introduction —540134-002 2 -9 Locking Mechanism Transaction Coordination Each server process tries to satisfy the request by performing an appropriate operation on the database. After satisfying the request, the server replies with a message indicating the disposition of the request. For NonStop OS requesters outside the Pathway environment, the requester sends a message to the server via a WRITEREAD system procedure. WRITEREAD includes the current transaction identifier with the message, and it becomes the current transaction of the server as well. The server performs the database access and replies to the requester. In addition to managing the begin and end or abort transaction operations, TMF manages all access to audited data in transactions through the locking mechanism. Locking Mechanism TMF provides a locking mechanism that isolates transactions from one another. Applications protected by TMF make requests for locks on data they read and update. TMF and the file system together manage the locks. Locks have these characteristics: • • • • • Locks are owned by the transaction, not by the processes that initiate database access. Locks are held on all rows (records) inserted by a transaction. Locks are held on all rows changed or deleted by a transaction. Locks can be held on rows that are read-only, provided a program requests locks. Locks remain in effect until a transaction is fully committed or backed out. Because of the TMF locking mechanism, a transaction can be backed out without affecting other concurrent transactions. HP NonStop TMF Introduction —540134-002 2- 10 Two-Phase Commit Protocol Transaction Coordination Two-Phase Commit Protocol TMF uses a two-phase commit protocol, which ensures that the whole transaction commits as a unit. In the first phase, TMF asks each participant in the transaction to prepare to commit. In the second phase, if all participants reply that they are prepared, TMF asks each participant to commit the change. In this way, transactions are not committed unless there is a reasonable expectation that the commit will succeed for all participants. Figure 2-6 illustrates the two-phase commit protocol. Assuming no failures, the transaction is committed successfully. Figure 2-6. Two-Phase Commit Phase 1: Prepare Phase 2: Commit Application 1 Application Commit 6 TMF 2 Prepare Resource Manager TMF Prepare 3 Committed 4 Commit Prepared Commit 5 Resource Manager Resource Manager Committed Resource Manager Legend 1 Application asks to commit the transaction. 2 TMF asks resource managers to prepare to commit. 3 Resource managers reply that they are prepared. 4 TMF asks resource managers to commit. 5 Resource managers commit the transaction and reply to TMF. 6 TMF replies to the application that the transaction is committed. VST045.vsd HP NonStop TMF Introduction —540134-002 2- 11 Summary of Transaction Coordination Transaction Coordination A two-phase commit is analogous to the message interaction in the following example: • A manager wants a staff meeting and asks the administrative assistant to arrange one when everyone can attend. The administrative assistant determines that Thursday at 10:00 looks possible but must call to confirm with all attendees: ° Phase 1—The administrative assistant calls you. After checking your calendar, you reply that you can attend. At that time, you have committed to attend the meeting, but you do not know for sure if it will really happen. You certainly should not, however, make any other plans for the time. ° Phase 2—Later, the administrative assistant calls backs and says that the time is firm. You finally know that the meeting is really going to be held. When a commit-work request is executed in an application protected by TMF, TMF commits the transaction’s changes in two phases to ensure that the changes are permanent and recoverable in the event of a total system failure. If the transaction is distributed, TMF ensures that all changes can be committed at all systems involved in the transaction. If any part of the transaction cannot be committed, the entire transaction is backed out. . Summary of Transaction Coordination The three stages in the life of a successful transaction are shown in Figure 2-7: Figure 2-7. Stages of a Transaction Application TMF Stage 1 BEGIN-TRANSACTION • Starts transaction • Establishes transid Stage 2 Modify Database • Verifies and holds locks • Builds audit records END-TRANSACTION • Verifies transaction is complete • Releases locks File System Audit Trail Database Stage 3 VST012.vsd HP NonStop TMF Introduction —540134-002 2- 12 Transaction Coordination Summary of Transaction Coordination Stage 1 The application issues a request to begin a transaction, thereby creating a new TMF transaction. The transaction is originated and identified as an active recoverable unit. Transaction identifiers are transmitted among processes that do the work of the transaction. Stage 2 The transaction modifies the database by reading, changing, deleting, and inserting rows (records). TMF maintains locks on all data that the transaction inserts, deletes, or changes. Concurrently, TMF builds an audit trail of all database modifications. Stage 3 The application issues the request to commit the transaction, indicating that changes should be permanent. TMF ensures that all changes to the database are recorded in the audit trail so they can be used for any subsequent recovery operations. As the operations for the two-phase commit protocol complete, all locks on the affected data are also released. HP NonStop TMF Introduction —540134-002 2- 13 Transaction Coordination Summary of Transaction Coordination HP NonStop TMF Introduction —540134-002 2- 14 3 Transaction Protection and Database Recovery TMF protects the database through transaction backout, file recovery, and volume recovery, which together ensure that no partially completed transaction affects the database. Should an error or failure occur, TMF uses audit trails and audit dumps to recover the database. This section covers the TMF recovery features, and includes the following topics: Topic Page Audit Trails 3-2 Audit Volumes 3-5 Audit Dumps 3-6 Transaction Backout 3-8 Volume Recovery 3-10 Online Dumps 3-12 File Recovery 3-13 Summary of Transaction Protection and Database Recovery 3-16 HP NonStop TMF Introduction —540134-002 3 -1 Audit Trails Transaction Protection and Database Recovery Audit Trails An audit trail is a series of files that TMF uses to record sequential information about a transaction’s changes to an audited file. Only TMF can rename, purge, or alter audittrail files. The disk volumes that contain audit trails are called active-audit volumes. HP requires that you place audit trail files and audited database files on separate disk volumes so that the database files and audit-trail files cannot be damaged by the same failure. The volumes where audited database files reside are called data volumes. Figure 3-1 illustrates the contents of audit records and associated database files for a transaction that deposits $550 to the Lee checking account, and $100 to the Lee savings account. In this figure, the checking and savings database files reside on audited volumes on disks $DB1 and $DB2; the audit-trail files reside on a separate disk, $AUDIT. Figure 3-1. Audit-Trail Records AA001001 AA001002 CHECKING $DB1 ADAMS LEE WOOD $450 $800 $250 $AUDIT SAVINGS $DB2 BROWN LEE WHITE $400 $200 $600 TX1 (CHECKING) BEFORE LEE $250 AFTER LEE $800 TX1 (SAVINGS) BEFORE LEE $100 AFTER LEE $200 TX1 COMMIT • • • VST013.vsd HP NonStop TMF Introduction —540134-002 3 -2 Transaction Protection and Database Recovery Master Audit Trail Master Audit Trail The master audit trail (MAT) contains TMF control information, such as commit or abort records for each transaction, and information describing the logical ordering of audit information in all audit trails in the system. The master audit trail can also store audit information generated by a set of audited volumes. There must be exactly one master audit trail in a TMF system. The master audit trail is named MAT. Auxiliary Audit Trail Auxiliary audit trails receive the audit information generated by a specified set of audited volumes; they do not contain TMF control information. Although most environments require only a master audit trail, each TMF system can have up to 15 auxiliary audit trails. Auxiliary audit trails can be used to increase overall auditing capacity and throughput, or to logically separate operations related to sets of audited volumes. Auxiliary audit trails are named AUXnn (nn is a two-digit value ranging from 01 to 15). Audit-Trail Files and Rollover TMF preallocates audit-trail files to ensure that disk space is available for these files as it is needed. Audit-trail files are named in sequence on each audit volume, enabling TMF to keep track of which audit-trail files are most recent and which are oldest. TMF requires at least two available audit-trail files whenever audit records need to be written. One is the audit-trail file to which TMF is currently writing; the other file is held in reserve to be used when the current audit-trail file becomes full. This requirement is so important that there is a special TMF mechanism for managing the audit-trail files: audit-trail rollover ensures audit-trail availability, allowing uninterrupted execution of all applications. Rollover involves renaming: just before the current audit-trail file becomes full, TMF renames an audit-trail file to become the new highest-numbered file in the sequence, and resets the new file’s end-of-file pointer to 0. Then, when the current audit-trail file is filled, TMF can proceed to the next file and continue writing audit information uninterrupted. Renaming an existing file takes considerably less time than creating a completely new file. Figure 3-2 illustrates audit-trail rollover. The current audit-trail file is AA000006; the rollover mechanism has already renamed an old file to AA000007 and reset its end-offile pointer to 0. When AA000006 is full, TMF begins writing audit information to AA000007 and rollover occurs: AA000005 is then renamed to AA000008, and its endof-file pointer is reset to 0. HP NonStop TMF Introduction —540134-002 3 -3 Audit-Trail Files and Rollover Transaction Protection and Database Recovery Figure 3-2. Audit-Trail Rollover Before audit trail rollover Current Audit Trail File AA000006 $AUDIT __________ __________ __________ ___EOF___ Next Audit Trail File AA000007 EOF Old Audit Trail File AA000005 __________ __________ __________ __________ __________ ___EOF___ Audit TMF After audit trail rollover Old Audit Trail File $AUDIT Current Audit Trail File Next Audit Trail File AA000006 AA000007 AA000008 __________ __________ __________ __________ __________ ___EOF___ __________ ___EOF___ EOF Audit TMF VST014.vsd HP NonStop TMF Introduction —540134-002 3 -4 Transaction Protection and Database Recovery Audit Volumes Audit Volumes Audit volumes are the disk volumes that contain audit-trail files. There are three types of audit volumes: active-audit volumes, overflow-audit volumes, and restore-audit volumes. Active-Audit Volume The disk volumes that contain active-audit-trail files are called active-audit volumes. HP requires that you place audit-trail files and audited database files on separate disk volumes so that the database files and audit-trail files cannot be damaged by the same failure. Overflow-Audit Volume The system manager can configure overflow-audit volumes to be used when the active audit trail is nearing full capacity and no files are available for a rollover. The oldest audit-trail files are copied to the overflow volumes: this allows audit-trail files to be reused. Typically, overflow volumes are used if TMF is temporarily generating audit records at a higher rate than anticipated when the audit trail was configured, or if secondary storage media is not available for audit dumps. Restore-Audit Volume Restore-audit volumes contain audit-trail files restored from an audit dump as part of a TMF recovery procedure. If you configure automatic audit dumping, you must configure restore-audit volumes. Restore-audit volumes are used during file recovery. During recovery of database files, TMF applies changes from audit-trail files to the database. If TMF requires an audit-trail file that is no longer available on the system, but is archived on tape, TMF restores that file from an audit dump on the tape to a restore-audit volume on disk. TMF then reads the audit-trail file and applies changes to the database. When the recovery process completes, TMF purges this file and all other unneeded files from the restore-audit volume. (System managers can, however, override this standard restore-and-purge mechanism by requesting TMF to retain audit-trail files restored from tape on the restore-audit volume indefinitely, to enhance performance.) For audit-trail files dumped to disk, TMF opens the disk-dump copy directly without involving a restore-audit volume. More information about this recovery operation appears under File Recovery on page 3-13. Since restore-audit volumes are used only during recovery procedures, you can also use restore-audit volumes as overflow volumes. HP NonStop TMF Introduction —540134-002 3 -5 Transaction Protection and Database Recovery Audit Dumps Audit Dumps Eventually, as TMF writes more and more audit records, the amount of space required for audit records in the audit trail could exceed the disk space available for them. As transactions are committed, the need decreases for volume recovery to use audit records pertaining to the oldest committed transactions; file recovery, however, typically needs these records. To free disk space for current audit records, and to preserve the availability of older audit records, TMF copies audit-trail files to audit dumps. These dumps can be written to secondary storage media at either the local site or a remote site. For a dump to tape, TMF prompts the operator to mount a scratch tape when an audit-trail file is ready for dumping. For extra security, TMF can write duplicate, simultaneous dump tapes: the business can then keep one or more sets of tapes off site for protection if a fire or other disaster damages an installation. To preserve an audit trail at an unattended site, a good technique is to perform audit dumps on a remote disk. The audit trail could be dumped to disk at a remote, attended site, and then be backed up to tape. A site should store audit dumps for possible future use. A site that does not dump and preserve audit trails will not be able to perform file recovery. Automatic Audit Dumping Those who are responsible for configuring a TMF system may configure audit trails to be dumped automatically or may dump audit trails as needed. HP recommends that you configure automatic audit dumping to ensure the highest levels of protection. Automatic audit-dumping for an audit trail is illustrated in Figure 3-3. When audit-trail file AA000005 becomes full, TMF begins writing to the next audit-trail file, AA000006. AA000005 is dumped to secondary storage media. Once the file has been dumped, the original file on the audit volume is marked as dumped and eligible for renaming when needed for new audit information. HP NonStop TMF Introduction —540134-002 3 -6 Automatic Audit Dumping Transaction Protection and Database Recovery Figure 3-3. Automatic Audit-Trail Dump TMF Current Audit Trail File Audit AA000006 __________ __________ __________ ___EOF___ Next Audit Trail File Audit Dump Environment Dumped - Old Audit Trail File AA000007 AA000005 EOF __________ __________ __________ __________ __________ ___EOF___ Audit Dump Process Audit Dumps Old Audit Trail Files AA000001 $AUDIT __ __ __ __ __ __ AA000002 __ __ __ __ __ __ AA000003 __ __ __ __ __ __ AA000004 __ __ __ __ __ __ AA000005 __________ __________ __________ __________ __________ ___EOF___ Secondary Storage Media VST115.vsd Audit dumps are recorded in the TMF catalog and are automatically managed by TMF; the transaction management process (TMP) purges dump files as they are no longer needed for recovery operations. HP NonStop TMF Introduction —540134-002 3 -7 Transaction Protection and Database Recovery Transaction Backout Transaction Backout The TMF backout process uses the before-images in the audit trails to undo the effects of an aborted transaction. TMF can back out transactions without affecting database consistency because the record locks and two-phase commit feature prevent transaction interaction, maintaining locks on data until backout completes. During transaction backout, the backout process writes the before-images (saved in the audit trails) back to the files that were affected by a failed transaction. After the effects of a transaction are undone, the locks for the transaction are released. Initiating Transaction Backout A transaction is aborted when an event prevents the transaction from being committed. Possible causes for transaction backout include the following: • • • • • • • • • Under the Pathway environment, a SCREEN COBOL program encounters a RESTART-TRANSACTION or ABORT-TRANSACTION statement. A SCREEN COBOL program is suspended or aborted because of errors or specific Pathway environment commands. The CPU of a Pathway environment server that has work in progress fails. The transaction is aborted, then the terminal control process (TCP) restarts it automatically. The primary TCP that started a transaction is disabled. The transaction’s changes are aborted, and the backup TCP restarts the transaction. If there is no backup TCP, the transaction is aborted but not restarted. The CPU of the primary disk process controlling the disk where an affected data file resides fails before the transaction is committed. The TMF operator aborts the transaction. The application aborts the transaction. The application terminates because of errors. Communication is lost between the participants of a distributed transaction before the transaction is committed. Because the effect of backout is the same as if the transaction had never started, an application program can recover by simply restarting the transaction from the beginning, without having to undo the effects of the transaction. HP NonStop TMF Introduction —540134-002 3 -8 Backout for Distributed Transactions Transaction Protection and Database Recovery Backout for Distributed Transactions For a distributed transaction, loss of communication between participating computer systems (nodes) can result in transaction backout. If a transaction cannot be committed on all participating systems, it is backed out on all participating systems. The transaction is backed out if a communication failure occurs at any point either before the commit or during the phase 1 (prepare) stage of the commit. For example, assume that communication between destination and source nodes is lost before the transaction begins Phase 1 of committing its changes. Each node affected by the loss of communications aborts the transaction; therefore, the transaction’s changes are backed out and its locks are released, as illustrated in Figure 3-4. Figure 3-4. Network Backout Tran-SF1000 at \SANFRAN AUDIT Tran-SF1000 at \LONDON Begin transaction. ... I/O to data at \LONDON ... End transaction. Participate in Tran-SF1000 AUDIT 01 Send prepare message 01 Reply message fails Database 01 Detects failure Backs out the transaction Backs out the transaction Database Transaction operations that are transparent to the application. VST116.vsd Another example is failure during Phase 1 of the commit. Phase 1 is the prepare stage: the commit for the transaction is issued on the local node, and all participating transaction managers and resource managers are asked to prepare to commit. Phase 1 may fail because either the communication path with a participating node transaction manager or resource manager is broken or a participating node replies that it cannot commit the transaction. When Phase 1 fails, the transaction is aborted and backed out on all nodes participating in the transaction. HP NonStop TMF Introduction —540134-002 3 -9 Transaction Protection and Database Recovery Volume Recovery Volume Recovery Volume recovery provides relatively quick database recovery after a volume, disk, or node failure occurs in a NonStop system. The precise recovery time varies depending on the number of files to be recovered, the system resources (particularly processors) available, and transaction lengths. Nevertheless, it usually takes only a few minutes to recover a database. Volume recovery is initiated automatically whenever an audited volume must be recovered. For example, when TMF is started after a failure or an audited volume comes up after a system failure, the volume recovery process runs. Volume recovery uses audit-trail records to return the database to a consistent state. With files protected by TMF, database updates are first written to the audit trail, then to cache for the database volumes, and finally to disk. If a failure causes data to be lost, TMF recovers the volume by applying before-images and after-images from the audit trail to the database. Volume recovery performs the following operations: • • Backs out transactions that had not committed all their changes at the time of the failure. To back out transactions, TMF applies before-images from the audit trail to the appropriate database. Applies committed transactions that were in cache at the time of the failure and not yet applied to the database. To apply committed transactions, TMF applies afterimages from the audit trail to the appropriate database. The volume recovery operation is illustrated in Figure 3-5. Volume recovery applies the after-images for transactions 3 and 4 to the database and backs out the partial effects of transaction 2, which was not committed. HP NonStop TMF Introduction —540134-002 3- 10 Volume Recovery Transaction Protection and Database Recovery Figure 3-5. Recovering a Database With Volume Recovery Database Updates Failure interrrupts the writing of Transaction 2 database changes. Cache Trx 2 Trx 3 Trx 4 $DB Trx 1 Before-images and After-images Trx 2 Volume Recovery Process $DB Trx 1 Trx 3 Trx 4 $AUDIT $AUDIT Before-images and After-images Trx 1 Begin Transaction Update Database Commit Transaction Trx 2 Begin Transaction Update Database Trx 3 Begin Transaction Update Database Commit Transaction Trx 4 Begin Transaction Update Database Commit Transaction ... VST017.vsd HP NonStop TMF Introduction —540134-002 3- 11 Online Dumps Transaction Protection and Database Recovery Online Dumps Online dumps preserve copies of audited database files in case of media failure or other damage to a database such as an accidental purge, or in some cases, incorrect application updates. Each online dump of a file provides an image of the file that can be used by the file recovery process to reconstruct the file. To support file recovery effectively, online dumps must be used with audit dumps provided through automatic audit dumping, as shown in Figure 3-6. The online dump provides an image of the database at a point in time; the audit dumps document all the changes made to the database since that time. Figure 3-6. Online and Audit Dumps for File Recovery Database Audit Online Dump Audit Dump File Recovery VST061.vsd HP recommends that you perform online dumps for critical production databases. Online dumps are essential for most file recovery operations and ensure that the database can be recovered to a consistent state. Through online dumps, audited database files can be copied to tape or disk without stopping active applications and without compromising the consistency of the database. Online dumps can be made while transactions are being processed by the applications. HP NonStop TMF Introduction —540134-002 3- 12 Transaction Protection and Database Recovery File Recovery Online dumps of database files can be initiated manually through the TMFCOM interactive command interface to TMF. Alternatively, they can be started programmatically through the Subsystem Programmatic Interface (SPI). Using SPI, these subsystem management programs are ideal for controlling online dumps at unattended sites. Sites can also manage dumps by using HP Tandem Advanced Command Language (TACL) programs. The online dump process can write duplicate, simultaneous dumps: the business can then keep one or more sets of storage media off site for protection if a fire or other disaster damages an installation. A site should store online dumps for possible future use. A site that does not dump and preserve audited files will not be able to perform file recovery. Online dumps are recorded in the TMF catalog and are managed automatically by TMF; the TMP purges dump files as they are no longer needed for recovery operations. File Recovery If a system or media failure jeopardizes the consistency of one or more audited files, those files cannot be opened again until they have been recovered. Although volume recovery can recover a database in most situations, file recovery is needed to restore consistency to the database after a critical failure has occurred. A critical failure can take place when: • • • • • A disk has become corrupted. A system failure has occurred from which volume recovery cannot recover. An audited file or volume is affected by a failure of a nonmirrored volume or both disks of a mirrored volume. An application incorrectly updates a database. If online dumps of the affected files exist from before the application problem, file recovery can recover the files to a state prior to the incorrect updates. A file is accidentally purged. File recovery uses online dumps of database files and audit dumps to restore damaged files, as shown in Figure 3-7. HP NonStop TMF Introduction —540134-002 3- 13 File Recovery Transaction Protection and Database Recovery Figure 3-7. Restoring Files With File Recovery File Recovery Environment Audit Dumps Online Dumps $RESTORE $DATA Restore Audit Volume Database Volume Catalog $AUDIT Before-Images and After-Images After-Image File Recovery Process Database Restored Copies of Files VST118.vsd HP NonStop TMF Introduction —540134-002 3- 14 File Recovery Transaction Protection and Database Recovery File recovery works by first restoring the data to the state of the last online dump of data files. Then, TMF applies audit information, using both audit dumps and the current audit-trail files. By applying audit information, file recovery can recover files to a time very close to when the failure occurred. This is a typical time-line for file recovery: Online Dump of Data Files Automatic Audit Dump #1 Automatic Audit Dump #2 Failure Current Audit Trail Files Time T1 T2 T3 T4 VST057.vsd Using this time-line, file recovery would involve these steps: 1. A failure has occurred, and a system operator initiates file recovery. 2. The file recovery process first restores the audited files from online dumps to the state they were in when the dump was made: to the time of T1. 3. TMF restores the audit-trail files from audit dumps on tape made after the online dump. TMF restores the audit dumps to restore-audit volumes and applies changes from these audit-trail files to the database. Applying changes from Audit Dump #1 restores the data to the time of T2; applying the changes from Audit Dump #2 restores the data to the time of T3. 4. The file recovery process reads the current audit trail and applies the changes recorded there, including removing the effects of any incomplete transactions. This restores the data to the time of T4, when the failure occurred. The file recovery process generates event messages while it runs, allowing you to monitor the process. When file recovery is complete, the recovered files can be opened. File recovery requires a longer history of audit dumps than volume recovery. It needs audit dumps for all audit-trail files written since the last online dump of a database file. File recovery allows files to be recovered optionally to different volumes, subvolumes, and file IDs than the ones on which they were dumped. The new (target) files, however, must be located on the same node (system) and the same instance of TMF as the old (source) files. File recovery—with appropriate online dumps of database files, audit-trail dumps, and sufficient disk space for its operations—completely protects the database files. HP NonStop TMF Introduction —540134-002 3- 15 Transaction Protection and Database Recovery Summary of Transaction Protection and Database Recovery Summary of Transaction Protection and Database Recovery Database files in a modern transaction-processing environment must always be available: when they are not, businesses can lose revenue. Downtime can be avoided by using hardware such as fault-tolerant disks, achieved by methods such as disk mirroring, and fault-tolerant disk access, achieved through shared disks. When a disk for a mirrored pair fails, the system uses the good disk volume for the application while the damaged volume is being recovered. Mirrored volumes preclude many recovery problems caused by a single disk failure. Other sorts of downtime can be prevented using software. TMF protects applications from downtime in the event of process, processor, or system communication failures. It offers two levels of protection for applications, depending on the recovery requirements of the application and the amount of system resources that TMF can use; these levels of protection are TMF transaction protection and database recovery. TMF Transaction Protection Audited database files automatically receive TMF transaction protection, including transaction management, database consistency, and database recovery. The resources and requirements for TMF transaction protection are: • • Application code that includes transaction control statements to begin, end (commit), or abort (back out) transactions, and appropriate data-locking techniques for the transactions. Sufficient disk space for audit trails. Database Recovery Database recovery includes TMF transaction protection and full protection of database files through file recovery. TMF provides the facilities for file recovery, but using them to protect the database or portions of the database is optional. Business requirements should determine the kind of protection the database needs. Site personnel then must ensure that there are sufficient system resources to support TMF requirements. In addition to the resources needed for transaction protection, database recovery requires the following resources: • • Procedures and storage media drives for making: ° ° Audit dumps Online dumps of any file to be protected by file recovery A fireproof vault or safe place off-site for storing the audit dumps and online dumps HP NonStop TMF Introduction —540134-002 3- 16 4 Application Performance Despite the overhead of auditing and committing records, TMF sustains top performance for high-volume systems. The performance benefits of using TMF increase as transaction size and rates increase. This section examines these benefits in the following topics: Topic Page Performance Comparison 4-1 Performance Versus Transaction Size 4-2 Performance Versus Transaction Rate 4-5 Performance Comparison A performance comparison of transactions protected and not protected by TMF appears in Figure 4-1. Figure 4-1. Performance Comparison With and Without TMF Protection 1 Cost per Transaction 2 0 1 2 3 4 5 6 7 8 9 10 Number of updates per transaction Legend 1 Without TMF protection 2 With TMF protection VST019.vsd HP NonStop TMF Introduction —540134-002 4 -1 Performance Versus Transaction Size Application Performance Without TMF protection, the relative cost for input/output (I/O) operations per transaction increases in direct proportion to the number of I/O operations in the transaction. With TMF protection, the relative cost increases only slightly as the number of I/O operations per transaction increases. Performance Versus Transaction Size The number of records being accessed in a transaction determines the transaction size. The smallest transaction reads and updates one record. The largest transaction could read and update all records in several files. A typical transaction reads and updates one or several records in a number of files. For example, filling out the bill of materials for rocket engines represents large transactions, updating records for many parts in multiple files, charging the work to various accounts, and scheduling. As shown in Figure 4-2, the inventory resides in tables (files) distributed across many volumes; the scheduling and accounting data reside on remote systems. Figure 4-2. Transaction Input/Output Example \MANUF Inventory Inventory Audit Trail Inventory Bills of Materials Transaction \ACT Audit Trail \SCHED Accounts Audit Trail Schedule VST020.vsd Another example is a bank buying an interest-bearing bond. This purchase is a smaller transaction that could involve charging principal and accrued interest to separate accounts, and then crediting the total cost to the brokerage firm from which the bond was purchased. HP NonStop TMF Introduction —540134-002 4 -2 Writes to Database Files Application Performance I/O for these transactions on a database protected by TMF includes writes to the database and writes to audit-trail files. Both types of writes include features that enhance performance for database files under TMF protection. Writes to Database Files Whereas the I/O for reading data is the same whether or not an application is using TMF, the I/O for writing the updated records is different: • • Writes for files under TMF protection can be buffered and held collectively in cache. Writes for nonaudited files, however, should not be buffered and held collectively in cache because these writes are not completed until they actually arrive on electronic media. Buffering writes is similar to keeping a list of instructions that you have to give someone, and then giving them all in one telephone conversation instead of making a separate call for each instruction. Many buffered writes are stored in cache, and then written to disk in one physical access. Because more time is used by the physical access than by any of the other operations involved in writing records to disk, using buffered writes greatly improves performance. For example, an application can end a transaction while the transaction is held in cache; the application does not have to wait for disk access. With files under TMF protection, database updates are first written to the audit trail, then to cache for the database volumes, and finally to disk. If a failure causes data to be lost in cache, TMF protects the database with the before-images and after-images in the audit trail. I/O on files without TMF protection should be unbuffered so that the updates are written immediately to disk. Data records written only to cache can be lost if a double processor failure occurs. Unbuffered and buffered writes are illustrated in Figure 4-3. HP NonStop TMF Introduction —540134-002 4 -3 Writes to Database Files Application Performance Figure 4-3. Unbuffered and Buffered Input/Output Unbuffered I/O One Transaction One Transaction One Transaction One Transaction U U U U R W 1 R 2 W R W R W 3 Time 4 5 3 Time 4 5 Buffered I/O Many Transactions U, U, U, U R W 1 2 Legend R Read data from disk and put into cache U Update data in cache W Write data to disk VST021.vsd HP NonStop TMF Introduction —540134-002 4 -4 Writes to Audit-Trail Files Application Performance Writes to Audit-Trail Files TMF protects transactions and the integrity of the audit trail while optimizing the performance of writing to audit-trail files. TMF provides audit-writing procedures that minimize physical access time. It does this by writing audit records for multiple transactions at the same time. This is called group commit. In addition to commit records, all accumulated audit records are written to disk by the audit-trail disk process, regardless of the transaction to which they apply. Group commits minimize the impact on performance caused by the additional input/output required for auditing files under TMF protection. Sequential writes are much more efficient than random writes. For this reason, TMF makes sequential writes to the audit trail rather than the random writes typically used for accessing data volumes. In an application protected by TMF, even though transactions might access many volumes on a node, the writes go only to the active-audit volume. The single write to the audit trail ensures that the database changes to files are permanent, even if the database is distributed on many volumes of a node. Performance Versus Transaction Rate TMF increases application performance as the transaction rate increases, because of the effects of buffered input/output for data volumes and group commits to the audit trail. Disk writes often occur at specific intervals, based on system events that cause physical writes from cache to disk. As the number of transactions per interval increases, throughput increases. For example, compare throughput of records to disk with throughput of people on a subway. Trains leave the station every 15 minutes. During a time of heavy use, many more people are transported than during a time when only a few people want a ride. The trains are full during heavy use and nearly empty during light use. A comparison of throughput depending on transaction rate is shown in Figure 4-4. For the comparison, assume a running application where all the transactions and their input/output requirements are similar. The input/output for the comparison represents writes for TMF transactions on audited files with buffered writes. At certain intervals, cache for a database disk is written and includes updates for that disk from several transactions. With the slow transaction rate, updates for only a few transactions are written. With the higher rate, updates for thousands of transactions are written. HP NonStop TMF Introduction —540134-002 4 -5 Performance Versus Transaction Rate Application Performance Figure 4-4. Throughput Comparison Depending on Transaction Rate Low Transaction Rate (Few per cycle) High Transaction Rate (Thousands per cycle) Transactions Transactions Legend Time for an I/O cycle — read, update, write Represents I/O for a transaction, assuming all are similar VST022.vsd With a high transaction rate, more records typically are written with every physical access. Because TMF can sustain a high transaction rate, the benefits of TMF protection increase as the transaction rate increases. HP NonStop TMF Introduction —540134-002 4 -6 5 TMF Application Programming The TMF application programming interface (API) is accessible through commands in the following languages: C, C++, HP COBOL, FORTRAN, SQL, SCREEN COBOL, and TAL. TMF frees application programmers from the tasks of managing concurrency and consistency in case of failure. However, those who use TMF must properly define transactions as they code their application programs. This section addresses application programming concepts, and includes the following topics: Topic Page Overview of Application Programming 5-1 Accessing and Locking Data 5-2 Controlling Transactions 5-8 Designing Applications for the Pathway Environment 5-9 Overview of Application Programming As a TMF application program runs, TMF provides underlying functions that greatly simplify the programmatic details of transaction processing. The application programs themselves are typically based on the requester/server model. TMF Functions in Application Programming The TMF interface in application programs provides two major functions: • Accessing and locking data All data records that are to be modified must first be locked. The I/O and locking interface to TMF depends on the programming language used. TMF manages locks when several applications need concurrent locks on the same data. • Controlling transactions Applications must be able to create, commit, and abort transactions. The particular transaction control statements also depend on the programming language. HP NonStop TMF Introduction —540134-002 5 -1 The Requester/Server Model TMF Application Programming The Requester/Server Model In the Pathway application environment, applications consist of two types of programs: requesters and servers. This design allows application logic to reside near the resources it manages. For example, presentation logic is located near workstations or terminal devices; database logic resides in server programs near the database. The requester sends a request to the server, which performs the requested action and then replies to the requester. The requester and server may reside on the same processor or on different processors. Requesters and servers communicate by using the file system or message system that is part of the NonStop OS. For a sample application based on the requester/server model, see Designing Applications for the Pathway Environment on page 5-9. Even though transactions are typically controlled by the requester, most servers need transaction control to back out (abort) a transaction if errors occur. Servers must always reply to requesters to indicate whether they were able to perform the requested work. The reply satisfies TMF requirements for committing or aborting transactions. If a server does not reply to a requester with the status of the transaction, the requester is left with an outstanding request to the server and cannot complete the transaction. Note. In distributed applications, the client/server model is often used. This model extends the requester/server model. In general, but not always, in this model the client process resides on a workstation and the server process resides on a second workstation, minicomputer, or mainframe system. Communication takes the form of request and reply pairs, which are initiated by the client and serviced by the server. Client/server computing is often used to connect different types of workstations or personal computers to a host computer system by means of communication protocols. Accessing and Locking Data TMF automatically manages database access and locking of audited files. It generally operates in the same way, regardless of the API or programming language used to access the database. The syntax the applications use to access a database, however, varies, depending on the programming language and the type of database. The issues involved in accessing and locking data include: • • • • Controlling transaction isolation Accessing SQL data Accessing Enscribe data Managing deadlock HP NonStop TMF Introduction —540134-002 5 -2 Controlling Transaction Isolation TMF Application Programming Controlling Transaction Isolation Applications can achieve different levels of isolation from concurrent programs by using various locking techniques for transactions. With the lowest level of isolation, a transaction accesses data for read operations only, despite any locks by concurrent transactions. With the highest level of isolation, a transaction acquires locks on data for its exclusive use only, prohibiting any access to the data by other transactions. As transaction isolation increases, concurrency decreases: application designers, therefore, need to balance application requirements for concurrency against requirements for isolation. Most databases, including NonStop SQL and Enscribe databases, allow for various levels of isolation; NonStop SQL access, however, provides greater concurrency with a higher level of isolation than does Enscribe access. The following discussion describes levels of isolation—from the lowest to the highest— as choices in locking techniques: • • • Disregarding locks on data (reading uncommitted data) Locking for Read and Update Locking for repeatable access Note. This discussion on locking is general: the different programming languages each provide their own options for more varied control. These are discussed in the individual programming language manuals. Disregarding Locks on Data (Reading Uncommitted Data) This isolation level applies only to reading data and allows a transaction to read data even if it is locked by another transaction. This allows the transaction to gather information without waiting for other transactions to complete. However, the transaction may read data that is in transition or inaccurate, because of uncommitted transactions. For example, an application could retrieve data about a sale in progress that is subsequently backed out (aborted). Transactions that disregard locks have no isolation from other transactions; therefore, they have the highest level of concurrency. Embedded SQL applications disregard locks for data by specifying READ UNCOMMITTED. Applications using Enscribe databases disregard locks for data by specifying a read with no lock. Locking Data for Read and Update When locking for read and update, an application acquires locks on the accessed data. The locks protect the application from accessing data affected by uncommitted updates of other transactions. Typically, the data that an application reads has a shared lock, but the data that the application updates has an exclusive lock. A shared lock allows any number of HP NonStop TMF Introduction —540134-002 5 -3 Controlling Transaction Isolation TMF Application Programming processes to read the same data, but prevents any process from writing to the locked data or reading it without intent to rewrite. An exclusive lock allows only the transaction that holds the lock to access the data. Using shared locks improves concurrency; using exclusive locks improves accuracy. Locks on updated, inserted, or deleted data are retained until the transaction either commits or is backed out. For read-only access, a lock is held on the current position (row or record being read) and is released when the next row or record becomes the current position. Locks on data read are typically released by the application before the transaction commits or aborts. This locking provides average transaction isolation and average concurrency. To lock SQL data for read and update, the application specifies STABLE ACCESS. To lock Enscribe data for read and update, the application specifies both shared locks and read statements with locks. Locking for Repeatable Access When locking for repeatable access, the transaction has exclusive use of data and complete isolation from concurrent transactions. This type of access provides maximum insulation from the effects of other transactions. With repeatable access enabled, a transaction can run as though it were the only active transaction in the system. The transaction will not read uncommitted changes made by other transactions. Furthermore, other transactions cannot alter any uncommitted changes made by this transaction. If your transaction has updated a set of detail items, subtotals, and totals so that they are consistent with one another, no other transaction can disturb that consistency. Other transactions cannot update, delete, or insert any data within the entire range of rows or records accessed by your transaction. This guarantees your transaction the opportunity to reread previously read rows or records and see exactly the same data values each time. For audited tables in SQL, the transaction acquires shared locks on all accessed data and exclusive locks on any inserted or updated data. All locks are held until the end of the transaction. This isolation level prevents other transactions from inserting or modifying rows in the range of rows examined by the transaction. Repeatable access provides a high level of isolation; this access has a lower level of concurrency than stable access, but a much higher level of concurrency than file locks. To lock SQL data for repeatable access, the application specifies SERIALIZABLE or REPEATABLE READ. For Enscribe data, the application must specify file locks to gain exclusive use of entire files. The application-specified file locks should be released after all transaction operations are completed. File locks give an application nearly complete isolation, but they can greatly impact concurrency. HP NonStop TMF Introduction —540134-002 5 -4 Accessing SQL Data TMF Application Programming Accessing SQL Data The NonStop SQL interface to TMF is implicit through the SQL data manipulation statements SELECT, FETCH, DELETE, INSERT, and UPDATE. The application selects the level of isolation. The locks to provide the selected isolation are subsequently managed by TMF and the file system. Applications can access a NonStop SQL database by using SQL statements embedded in host language programs, including C, C++, HP COBOL, and TAL. The following example shows a cursor that accesses data from the PARTS table and declares the intention of updating the data. (A cursor is a mechanism through which a program retrieves a set of rows defined by the SELECT statement.) EXEC SQL DECLARE list_by_partdesc_cursor CURSOR FOR SELECT partnum, partdesc, price, qty_available FROM sales.parts WHERE (partdesc = :partdesc OF parts) SERIALIZABLE ACCESS FOR UPDATE OF PARTDESC, PRICE, QTY_AVAILABLE END-EXEC. After defining the set of rows, the program can retrieve them one-by-one and perform update or delete operations on them; the program processes the cursor like a sequential file. The row in which the cursor is currently located is locked for exclusive access because the FOR UPDATE clause was specified. As the program accesses rows through the cursor, the rows remain locked until the transaction commits. If this cursor were defined without the FOR UPDATE clause, the stable access option would cause only the row in which the cursor is currently located to be locked with a shared lock. The lock would be released when the next row becomes the current position. If this cursor were defined without the FOR UPDATE clause but with repeatable access, all rows accessed through the cursor would remain locked until the transaction commits. Accessing Enscribe Data Access to an Enscribe database is achieved either through programming language statements or, for some languages, through calls to the system procedures for the NonStop OS. When applications access audited Enscribe files, TMF automatically manages their transactions and data locking. Standard language input/output provides access to Enscribe data for the HP COBOL and FORTRAN languages. TACL has a built-in function, #REQUESTER, that provides for reading and writing to Enscribe files. System procedure calls provide access to Enscribe data for these languages: C, C++, and TAL. HP NonStop TMF Introduction —540134-002 5 -5 Accessing Enscribe Data TMF Application Programming The syntax for accessing Enscribe data and specifying isolation level of the application varies depending on the programming language. Typically, servers issue lock requests for all the data they update and for some of the data they read. The servers are responsible for issuing unlock requests when they finish their tasks, but they also need to provide for potential errors. Typically, servers include requests to unlock data as follows: • • For successful database access, the server issues an unlock file statement and then replies to the requester. For unsuccessful database access, the server omits unlock file statements (therefore retaining the locks) and then replies to the requester about the failure. As examples of accessing Enscribe data, consider the following uses of an HP COBOL application. Using Locks With READ and REWRITE Statements in an HP COBOL Program The READ statement specifies locking for the record being read: READ EMP-MASTER WITH LOCK KEY IS EMP-NO-KEY OF EMP-REC. IF EMP-FIELDS OF EMP-REC = SAVED-EMP-FIELDS OF EUPDATE-MSG PERFORM 970-REWRITE-EMP 970-REWRITE-EMP. REWRITE EMP-REC WITH UNLOCK. READ WITH LOCK ensures that no other user can read the record; it acquires a lock on the record for subsequent updating. For REWRITE, TMF requires that the transaction has previously locked the record. UNLOCK on the REWRITE statement indicates that the program has completed accessing the record. The locks held by the transaction on the rewritten record are held until the transaction is either committed or backed out. Using Repeatable Reads in an HP COBOL Program The program locks records it reads and uses in the transaction even though the records are not changed. By locking the records, the program ensures that the transaction’s reads are repeatable: READ EMP-MASTER WITH LOCK KEY IS EMP-NO-KEY OF EMP-REC. IF EMP-FIELDS OF EMP-REC = SAVED-EMP-FIELDS OF EUPDATE-MSG . . ELSE UNLOCKRECORD EMP-MASTER Until it issued UNLOCKRECORD, the program had repeatable reads. HP NonStop TMF Introduction —540134-002 5 -6 Managing Deadlock TMF Application Programming After the UNLOCKRECORD request, the locks are released immediately, but only for the unchanged records; locks on updated records are held until the transaction is either committed or backed out. Using File Locks for Repeatable Access If an application issues LOCKFILE in a transaction, updates records while it holds the lock granted by LOCKFILE, and then issues UNLOCKFILE before issuing ENDTRANSACTION, the file lock is held until the transaction commits. Managing Deadlock Transaction deadlock occurs when two transactions cannot proceed because they are each waiting for the other to release a lock. Planning applications requires considering how locks affect concurrency and how to avoid deadlock. If deadlock occurs, the application needs to detect it and take appropriate action. In Figure 5-1, deadlock results from this sequence of row (record) locking operations: 1. Transaction T1 accesses and locks row A. 2. Transaction T2 accesses and locks row B. 3. Transaction T1 attempts to access and lock row B and has to wait. 4. Transaction T2 attempts to access and lock row A and has to wait. Figure 5-1. Deadlock Example CHECKING T1 Debits Balance A $100,00 T2 BAL A BAL B Credit Balance B Debits Balance B $500,00 Credit Balance A Deadlock VST028.vsd In this example, neither transaction can proceed. This situation can be avoided by setting a time limit on how long a statement should wait before being granted a lock request for access: if a request exceeds the time limit, the statement times out. HP NonStop TMF Introduction —540134-002 5 -7 Controlling Transactions TMF Application Programming Because timeout is detectable (the system issues a timeout error message), the program can try the statement again, or abort the transaction and start the whole transaction again. Timeout can be used to detect deadlock in either an SQL database or an Enscribe database. For applications using SQL tables, there is a default time limit that applies if the application does not set one. For applications using Enscribe files, the application must set a time limit; there is no default limit. Controlling Transactions The application programming interfaces to TMF include transaction control statements that mark the beginning, committing, or backing out of a transaction. The recommended application design is for the requester, rather than the server, to handle transaction control; however, servers are responsible for backing out transactions for some error conditions. In general, applications should use single-threaded requesters and servers. A singlethreaded process is one that can deal with only one TMF transaction during any specific period of time; in other words, the requester or server module never has more than one transaction outstanding at a time. Some applications, however, use multi-threaded requesters and servers. A multithreaded process is one that can deal with more than one TMF transaction concurrently; it can, for instance, initiate multiple transactions and then switch processing from one to another, or it can accept multiple work requests and then alternately switch between one request and another. Transaction Control for Single-Threaded Design The syntax for transaction control varies, depending on the programming language and API. See Table 2-1, Sample Transaction Control Statements, on page 2-8 for transaction control statements for different programming languages and APIs. The system procedure calls provide the most comprehensive interface to TMF for transaction control: • • • BEGINTRANSACTION causes a new transaction identifier to be created. The new transaction identifier becomes the current transaction identifier for the process that called BEGINTRANSACTION. ENDTRANSACTION causes the changes associated with the current transaction identifier to be committed. ABORTTRANSACTION causes the changes associated with the current transaction identifier to be backed out. The program can restart the transaction at the BEGINTRANSACTION call under a new transaction identifier. HP NonStop TMF Introduction —540134-002 5 -8 TMF Application Programming Transaction Control for Multithreaded Design C, C++, HP COBOL, and FORTRAN use these calls. HP COBOL programmers use the ENTER statement. FORTRAN programmers use calls to the TAL procedures. • • SQL statements are BEGIN WORK, COMMIT WORK, and ROLLBACK WORK. TACL built-in functions are #BEGINTRANSACTION, #ENDTRANSACTION, and #ABORTTRANSACTION. Transaction Control for Multithreaded Design Multithreaded applications use the same transaction control statements as described for single-threaded design, along with calls to the following TAL procedures: • • • RESUMETRANSACTION, used by programmers coding multithreaded requester processes. To resume a particular transaction, the application calls the procedure with the transaction identifier that was returned from the initial BEGINTRANSACTION call. The specified transaction identifier becomes the current transaction identifier for the process calling RESUMETRANSACTION. GETTRANSID, used by programmers to obtain the current transaction identifier of the calling process. ACTIVATERECEIVETRANSID, used by programmers coding $RECEIVE-queuing servers; these servers queue messages from several requesters concurrently before replying. For dealing with heterogeneous transactions, additional control statements are available. These are described in the TMF Application Programmer’s Guide. Designing Applications for the Pathway Environment Although TMF protects databases in environments other than the Pathway environment, programmers most commonly design their applications for the Pathway environment. The following discussion provides a survey of Pathway application design. Assigning Roles to Application Modules An OLTP application defines TMF transactions and accesses audited tables or files in the database. The modules of a typical OLTP application and their roles in the transaction environment are illustrated in Figure 5-2. HP NonStop TMF Introduction —540134-002 5 -9 Assigning Roles to Application Modules TMF Application Programming The roles of the modules are: • • Requester—the module that begins and ends a transaction and specifies the kind of work to be performed in the transaction. Typically, the requester is a SCREEN COBOL program run by a terminal control process (TCP) in the Pathway environment. Server—the module that performs database access on behalf of the requester. The server can be a program written in a number of application languages. SCREEN COBOL requesters are written to run at a single terminal. With Pathway/iTS however, copies of individual requesters can be run simultaneously on thousands of terminals. Figure 5-2. Modules of a Typical OLTP Pathway Application Requester Server TCP SCREEN COBOL Program Database Transaction Entry ... BEGIN-TRANSACTION. ... SEND MESSAGE MSG-OUT... REPLY CODE... ... END-TRANSACTION. ... ON-ERROR-PROCEDURE. ABORT-TRANSACTION. ... Application Language Program VST025.vsd The server programs typically perform database input/output within a single active transaction. They are often written in HP COBOL. To access a NonStop SQL database, the program uses embedded SQL statements; to access an Enscribe database, the program uses HP COBOL input/output statements. Servers that access a NonStop SQL database can be written in languages that allow embedded SQL statements, including C, HP COBOL, or TAL. Server programs that access an Enscribe database can be written in C, C++, FORTRAN, HP COBOL, or TAL. Through the facilities of NonStop TS/MP, multiple copies of a single server program run concurrently. Pathway/iTS simplifies the designing and coding of requester programs by providing multithreaded functionality. TMF simplifies designing applications, as well HP NonStop TMF Introduction —540134-002 5- 10 Developing Requesters Using SCREEN COBOL TMF Application Programming as coding requesters and servers, by providing concurrency control, protecting transactions, and managing database recovery. Developing Requesters Using SCREEN COBOL The SCREEN COBOL statements BEGIN-TRANSACTION, END-TRANSACTION, ABORT-TRANSACTION, and RESTART-TRANSACTION enable Pathway/iTS applications to use TMF. BEGIN-TRANSACTION identifies the beginning of a sequence of operations treated as a single transaction. When this statement executes, the terminal enters transaction mode, TMF starts a new transaction, and the new transaction identifier (transid) is assigned to the terminal by the TCP. After beginning a TMF transaction, the requester always passes the transaction identifier as the current transid to the server. The transid is associated with the server. The server’s access to the database is performed under the transaction. Passing the current transaction to the server is shown in Figure 5-3. Figure 5-3. Passing the Current Transaction to a Server Assign Transid Server Requester Access Identified by Transid TCP SCREEN COBOL Program Transaction Entry Database Receive Transid ... BEGIN-TRANSACTION. ... SEND MESSAGE MSG-OUT... REPLY CODE... ... END-TRANSACTION. ... ON-ERROR-PROCEDURE. ABORT-TRANSACTION. .. . ... READ MSG-IN OPEN cursor... ... FETCH... ... UPDATE... CLOSE cursor... ... WRITE REPLY-MSG .. . SQL-COBOL85 Program VST026.vsd The server receives messages from the requesters in the system file $RECEIVE, provided for interprocess communication. The server then performs the requested operations, prepares a response to the requester, and sends a reply to the requester. HP NonStop TMF Introduction —540134-002 5- 11 Developing Servers for Database Access TMF Application Programming If a transaction is aborted for any reason, all of its changes to audited database files are backed out by TMF. In most cases, the requester program restarts the transaction at the BEGIN-TRANSACTION statement. The new transaction is assigned a new transaction identifier by the TCP. ABORT-TRANSACTION stops the processing of a transaction. TMF backs out all database changes made by the transaction and does not restart the transaction. END-TRANSACTION identifies the end of a transaction and indicates that the transaction’s database changes should be committed. After END-TRANSACTION is completed, the transaction cannot undo any of its changes and the terminal is switched out of transaction mode. RESTART-TRANSACTION causes TMF to back out a transaction and restart it at the BEGIN-TRANSACTION statement with a new transaction identifier. The new transaction identifier is passed to the servers for subsequent database access. Developing Servers for Database Access Applications using a mixed SQL and Enscribe database also have TMF protection for transaction and database recovery. Mixed database access can be either through a single server or multiple servers. TMF monitors all database access on a transaction basis by associating all access with the transaction identifier passed from the requester to the server. An HP COBOL server accessing a mixed database is shown in Figure 5-4. HP COBOL allows embedding SQL statements in the program. The program accesses the SQL database by using the SQL statements. The program accesses the Enscribe database by using HP COBOL statements. Transaction Design Transaction design involves asking the following primary questions: • • What is the work to be accomplished within the application (for example, order entry or account balancing)? How can the work be divided into a number of transactions that can be recovered, if necessary, by TMF? Factors that influence the answers to these questions are: • • • Concurrency—How long should locks on the data be held for a transaction? Performance—How much extra disk input/output, server activity, and TCP memory requirements are involved in the choice of one design over another? Consistency—How large is the unit of recovery for each transaction, and is it adequate for the application? HP NonStop TMF Introduction —540134-002 5- 12 Developing Servers for Database Access TMF Application Programming Figure 5-4. Mixed Database Access by an HP COBOL Server NonStop SQL Database HP COBOL Server Requester Enscribe Database Procedure Division. ... 200-Access-SQL-DB. *With SQL statements. EXEC SQL FETCH... ... 400-Access-ENSCRIBE-DB. *With HP COBOL statements. READ ... ... VST027.vsd If a transaction requires only one screen display (allowing the terminal operator to enter all the required data at once), the transaction design is straightforward and the three preceding questions do not apply. If, however, the transaction requires two or more screen displays to accumulate its data, one of the following two possible designs can be considered: • • A single, long TMF transaction A context-saving TMF transaction Long Transactions A long transaction is one in which a requester accepts data and sends it to servers several times in a single TMF transaction. This design uses the TCP’s ability to provide transaction restart if a TMF transaction is interrupted; however, it precludes a requester issuing a CHECKPOINT statement to ensure recovery of accepted operator input data (checkpoints are not allowed within a TMF transaction). HP NonStop TMF Introduction —540134-002 5- 13 Developing Servers for Database Access TMF Application Programming A typical sequence of SCREEN COBOL actions for a long transaction is shown in Figure 5-5. Figure 5-5. A Long Transaction Requester Display 1 Operator Action Display 2 Operator Action Display 3 Operator Action ACCEPT data • • • BEGIN-TRANSACTION • • • SEND to server • • • ACCEPT data • • • SEND to server • • • ACCEPT data • • • SEND to server • • • END-TRANSACTION Transaction State Server Server Server Database VST029.vsd Advantages of long transactions are: • • Context is minimized because the requester does not collect all input at once. Failure at any point during the TMF transaction causes all updates to audited data to be backed out. HP NonStop TMF Introduction —540134-002 5- 14 Developing Servers for Database Access TMF Application Programming Disadvantages of long transactions are: • • Resources could be locked for an indeterminate amount of time. Locking resources for seconds or minutes might be acceptable; locking resources for hours is not acceptable. An application should ensure that an operator cannot leave to perform some other task before finishing a transaction so that locked resources can be released. Transaction restart requires the operator to reenter input. Context-Saving Transactions A context-saving transaction is one in which a requester accepts and collects all operator input before starting the TMF transaction and sending a database update request to a server. A typical sequence of SCREEN COBOL actions for context-saving transactions is shown in Figure 5-6. Figure 5-6. A Context-Saving Transaction Requester Display 1 Operator Action Display 2 Operator Action Display 3 Operator Action ACCEPT data Save data in TCP's terminal context area. • • • ACCEPT data Save data in TCP's terminal context area. • • • ACCEPT data BEGIN-TRANSACTION Transaction State SEND to server SEND to server Server Database END-TRANSACTION VST030.vsd HP NonStop TMF Introduction —540134-002 5- 15 Summary of Transaction Design Tasks for Applications in the Pathway Environment TMF Application Programming Advantages of context-saving transactions are: • • • Locks are held for a short time. Failure at any point during the TMF transaction causes all updates to audited data to be backed out. Transaction recovery does not require operator action. To avoid having to reenter any operator input for transaction recovery, the requester can use the SCREEN COBOL statement CHECKPOINT to establish a restart point during the contextgathering portion of the transaction. Disadvantages of context-saving transactions are: • • The collected input uses more context than the context used for long transactions. Overuse of context area could affect performance. There can be additional database access if the data-gathering requires reading the database (without locking the data) so that an update request would require rereading the data to verify consistency with the initial read. Summary of Transaction Design Tasks for Applications in the Pathway Environment Designing applications to use TMF in the Pathway environment typically includes these tasks: • • • • Determining which database files are to be audited. Determining the appropriate transaction design strategy for the application and where to put the BEGIN-TRANSACTION and END-TRANSACTION statements. Writing modularized servers so that any server updating audited files is not also updating nonaudited files. Using appropriate locking techniques in the servers to provide the desired level of concurrency and application isolation. HP NonStop TMF Introduction —540134-002 5- 16 6 TMF System Management Users typically perform various TMF system management activities such as configuring TMF itself and monitoring and managing its activities on a routine basis. These operations help ensure that applications continue running with full transaction protection. For performing these operations at either local or remote sites, two convenient interfaces are available: • • TMFCOM, the interactive command interpreter used by system managers and operators. Subsystem Programmatic Interface (SPI), used by application programmers. The TMF interfaces are discussed in this section, along with other related topics: Topic Page TMFCOM Command Interpreter 6-1 Subsystem Programmatic Interface (SPI) 6-2 TMF Process Communication 6-3 Event Management 6-4 Tasks for Managing TMF 6-5 TMFCOM Command Interpreter You can manage and operate TMF by using TMFCOM, the TMF interactive command interface. Through TMFCOM, you enter commands and receive responses at your terminal. Most TMFCOM commands can be included in EDIT files, which operators can submit as command (OBEY) files to perform routine operations. These commands can also be included in TACL macros. TMFCOM provides commands for all TMF operations typically required by system managers and system operators. Complete information about TMFCOM commands appears in TMF Reference Manual. HP NonStop TMF Introduction —540134-002 6 -1 Subsystem Programmatic Interface (SPI) TMF System Management Subsystem Programmatic Interface (SPI) By using the Subsystem Programmatic Interface (SPI), you can construct system management programs that monitor and control the TMF environment for you. Most of the operations available in TMFCOM are also available through SPI. System management programs allow you to transfer as many of the routine management tasks to programs as possible. This approach frees system managers and operators for the work that directly requires their personal judgment and creativity. Information obtained through SPI can be used directly by an application to control its actions, monitor events, and produce reports. As illustrated in Figure 6-1, both TMFCOM and system management programs are useful for controlling TMF at either a central or remote site. The SPI commands for TMF are described in TMF Management Programming Manual. Figure 6-1. Managing a Central and Remote Site \CENTER \AWAY TMFCOM System Management Programs TMFSERVE TMFSERVE TMF TMF Operator System Management Programs TMFCOM Operator Legend Messages in SPI format VST024.vsd HP NonStop TMF Introduction —540134-002 6 -2 TMF Process Communication TMF System Management TMF Process Communication Both TMFCOM and application programs using SPI send their requests to the TMFSERVE process, transmitting them in SPI message format (see Figure 6-1). TMFSERVE is a TMF server process whose main job is to receive and process these requests. TMFSERVE either handles the requests itself, or passes them to other TMF processes through a programmatic messaging system. Basically, each TMFCOM process or user application program interacts with TMFSERVE in a common client-server relationship. TMF process communication is illustrated in Figure 6-2, which also shows some of the major TMF processes. The most significant TMF processes and their functions are: TMP (Transaction Management Process) Coordinates the TMF configuration and distributed transactions, and is in charge of all operations that require a single point of control; it writes transaction status records to audit trails and manages audit trails, data volumes, and other TMF processes. Audit dump Dumps audit trails to disk or tape. Audit restore Restores audit dumps from disk or tape. Backout Backs out transactions as a result of a request to abort a transaction, or as a result of certain failures (for example, processor failures). Catalog Manages the TMF catalog, which contains data about TMF dumps and the media on which they are stored. Copy audit overflow Writes audit-trail files to overflow-audit volumes. File recovery Recovers a set of files as requested by an operator after a media failure. File recovery list Supports the file recovery process. Online dump Dumps audited database files to disk or tape. Online restore Restores audited database files from disk or tape. TMFMON Performs highest-priority, performance critical operations such as managing memory for the TMF Library data structures and notifying the TMP about changes in disk volume status. TMFMON2 Provides a processing environment for TMF Library operations. Volume recovery Performs automatic recovery at startup or any other time data volumes are started for TMF processing. HP NonStop TMF Introduction —540134-002 6 -3 Event Management TMF System Management Figure 6-2. TMF Process Communication TMFCOM System Management Program Operator TMFSERVE TMF Volume Recovery File Recovery TMP Backout Catalog Audit Dump Online Dump TMFMON Legend Messages in SPI format VST023.vsd Event Management TMF generates Event Management Service (EMS) messages. These messages convey information about an event or significant occurrence in the system or application. Some of these messages require operator action, and others simply convey information. They appear on the terminal or workstation screen and are also recorded in the EMS log file. HP NonStop TMF Introduction —540134-002 6 -4 Tasks for Managing TMF TMF System Management Tasks for Managing TMF Managing TMF typically involves: • • • • • • Configuring TMF Designating audited files for TMF protection Specifying compression of selected audit records Managing audit dumps and online dumps Monitoring TMF Managing the TMF catalog Configuring TMF Through TMFCOM, you can create, alter, and determine the status of audit trails, data volumes, and other objects that TMF requires or uses in its operations. For example, you use the ADD AUDITTRAIL command to configure an audit trail and to specify whether audit dumping will be performed for the audit trail. You configure at least one audit trail before starting TMF for the first time, and can add more anytime thereafter while TMF is running. You configure audited data volumes and the TMF catalog, and can alter most configuration attributes, while TMF is running. Designating Audited Files If TMF is to protect a database file (an SQL table or an Enscribe file), the file must first be designated as audited. SQL/MP tables are designated as audited by the AUDIT attribute, used in the standard SQL statements CREATE and ALTER. By default, these tables are audited. The SQL/MP data dictionary is always audited. SQL/MX objects are always audited. In both SQL/MP and SQL/MX, you use the CREATE statement when you create the table or the ALTER statement after you create it. Enscribe files are designated as audited by the File Utility Program (FUP) commands CREATE and ALTER. By default, Enscribe files are not audited. Specifying Compression of Selected Audit Records The disk process can reduce the size of some audit records by including only the changed fields in each before-image or after-image. In a personnel file, for example, a transaction might change only the salary field without affecting the name, address, and other data. With compression, only the salary and some control information are written to the audit trail; the unchanged fields consume no audit trail space. HP NonStop TMF Introduction —540134-002 6 -5 Managing Audit and Online Dumps TMF System Management When audit compression is used, programs cannot use audit-reading routines as easily. These routines have more difficulty reading the audit trails because of the decompressing work involved. Audit compression is controlled on a file-by-file basis for database files with the AUDITCOMPRESS file attribute. Managing Audit and Online Dumps When an audit trail is configured to have an audit dump process, audit dumps are created automatically, as needed. Online dumps can be initiated without operator intervention by running system management programs that use SPI, or by using TACL programs. Monitoring TMF To maintain TMF, you should monitor it each day. You can use TMFCOM to view the status of the different components of TMF. Managing the TMF Catalog The TMF catalog is a set of files residing on the same mirrored disk volume as the TMF control files. The catalog contains two types of control information: a dump directory and a media directory. This information is required for file recovery. The dump directory includes a history of each file that was dumped, including the time and date when it was copied. This directory includes: • • • • The names of the dumped files, and the names of the tape drives used to dump the files, if tape is used A timestamp and unique serial number for each dump The copy number of each dump (because multiple copies can be made for each generation, the copy number identifies each distinct copy of a particular version) The sequence number of the audit-trail file in use at the time of each online dump The media directory contains all known tape-reel identifiers and their current status: scratch, released, bad, or assigned to a particular dump’s serial number. As an online or audit dump progresses, the TMF catalog process writes the necessary information to the catalog in a fail-safe manner. The file recovery process uses this cataloged information to determine which tapes or disk files are required for recovery operations. More information about TMF system management appears in TMF Planning and Configuration Guide and TMF Operations and Recovery Guide. HP NonStop TMF Introduction —540134-002 6 -6 Index A C Aborting transactions 2-8 ABORT-TRANSACTION control statement 5-12 Accessing data 5-2/5-8 After-images 1-11 Application programming languages 1-17 transaction isolation 5-3 Applications assigning roles to modules 5-9 languages used for writing servers 2-6 Audit dumps 1-13, 3-6/3-7, 3-13, 6-6 Audit records 1-11, 3-2, 6-5 Audit trail rollover 3-3 Audit trails 1-11, 1-12, 3-2/3-5, 4-5 Audit volumes active 3-2, 3-5 overflow 3-5 restore 3-5 types of 3-5 Audited files designation 6-5 dumps 3-12 relationship to audit trails 1-11 Auxiliary audit trails 3-3 Cache 3-10, 4-3 Child Transactions 2-5 Command interface for TMF (TMFCOM) 6-1 Commit protocol, two-phase 2-11 Committing transactions 2-7, 2-8 Compression of audit records 6-5 Concurrency control 1-7 versus transaction isolation 5-3 Consistency, database 1-5 Context-saving transactions 5-15 Control statements 2-6/2-8 Controlling transactions 5-8/5-9 B Before-images 1-11, 3-8 Beginning transactions 2-7, 2-8 BEGIN-TRANSACTION control statement 5-11 Buffered writes to disk 4-3 D Data access 5-2/5-8 Data locks 1-9, 2-10, 5-2/5-8 Data volumes 3-2, 3-5 Database consistency 1-5 distribution 1-3 recovery 1-10/1-13, 3-16 Deadlock, managing 5-7 Designating audited files 6-5 Developing servers for database access 5-12 Distributed audit trails 1-12 Distributed transactions 1-3, 2-4 backout of 3-9 coordination of 2-2 Dumps audit 1-13, 3-6, 6-6 online 1-13, 3-12, 6-6 HP NonStop TMF Introduction —540134-002 Index -1 E Index E N EMS messages 6-4 END-TRANSACTION control statement 5-12 Enscribe accessing data 5-5 and TMF 1-18 NonStop OS and TMF 1-18 NonStop SQL accessing data 5-5 and TMF 1-17 F OLTP software and TMF 1-16 Online dumps 1-13, 3-12, 3-13, 6-6 Overflow audit volumes 3-5 File recovery 1-13, 1-15, 3-13/3-15 Financial application scenario 1-14 O P G Insurance application scenario 1-15 Interfaces for managing and operating TMF 6-1 Parent Transactions 2-5 Pathmaker and TMF 1-17 Pathway and TMF 1-17 Pathway transaction design 5-12/5-16 Performance versus transaction rate 4-5/4-6 versus transaction size 4-2/4-5 Point-of-sale application scenario 1-14 Problem/Solution scenarios 1-14/1-16 Procedure calls 5-8 Process communication 6-3 Programmatic interface for TMF (TMFSERVE) 6-2 L R Locking data 5-2/5-8 Locking mechanism 2-10 Locks see Data locks Long transactions 5-13 Reading locked data 5-3 Reading uncommitted data 5-3 Recovery database 1-10 file 1-13, 1-15, 3-13 volume 1-13, 1-15, 3-10 Repeatable access 5-3, 5-4 Requesters 2-6 developing by using SCREEN COBOL 5-11 multithreaded 5-9 Gateways, transaction 2-3/2-4 Group commit 4-5 H Heterogeneous transactions 2-2 Homogeneous transactions 2-2 I M Managing TMF 6-2, 6-5/6-6 Manufacturing application scenario 1-15 Master audit trail (MAT) 3-3 Mixed database access 5-12 Multithreaded requesters and servers 5-9 HP NonStop TMF Introduction —540134-002 Index -2 S Index Requesters (continued) role in application 5-10 single-threaded 5-8 Resource managers 2-3/2-4 RESTART-TRANSACTION control statement 5-12 Restore audit volumes 3-5 Roles of application modules 5-9 Rollover, audit trail 3-3 S Scenarios financial application 1-14 insurance application 1-15 manufacturing application 1-15 point-of-sale application 1-14 problem/solution 1-14/1-16 travel agency application 1-15 SCREEN COBOL 5-11 Serializable access 5-3, 5-4 Servers 2-6 multithreaded 5-9 role in application 5-10 single-threaded 5-8 Single-threaded requesters and servers 5-8 T Tasks for managing TMF 6-5/6-6 Terminal control process (TCP) 2-9 TMF benefits 1-19 catalog 3-13, 6-6 characteristics 1-3/1-7 definition 1-1 environment 1-3/1-7 integrating with OLTP software 1-16 interfaces for managing and operating 6-1 management tasks 6-5/6-6 TMF (continued) managing 6-2 overview 1-1/1-2 performance benefits 4-1/4-6 personnel responsibilities 1-18 process communication 6-3 purpose 1-1 response to transaction control statements 2-6 TMFCOM 6-1 TMFSERVE 6-2 Transaction backout causes 3-8 distributed transactions 3-9 examples 1-14, 1-15 Transaction control 2-6/2-8, 5-8/5-9 Transaction gateways 1-1, 2-3/2-4 Transaction isolation 5-3/5-4 Transaction management process (TMP) 2-5, 3-13 Transaction protection 3-16 Transactions aborted 3-8 child 2-5 concurrency 1-6 context-saving 5-15 control for multithreaded requesters and servers 5-9 for single-threaded requesters and servers 5-8 control statements 5-11/5-12 coordination, summary 2-12 design 5-12/5-16 distributed 1-3, 2-4 coordinating 2-2 transaction backout 3-9 heterogeneous 2-2 homogeneous 2-2 identification 2-6/2-10 HP NonStop TMF Introduction —540134-002 Index -3 V Index Transactions (continued) identifiers 2-9, 5-11 local 2-1 long 5-13 parent 2-5 properties of 1-8/1-10 protection 1-8, 3-16 rate versus performance 4-5 serialized 1-9 size versus performance 4-2 stages of 2-12 Travel agency application scenario 1-15 Two-phase commit protocol 2-11 V Volume recovery 1-13, 1-15, 3-10 W Writes to disk, buffered 4-3 to the audit trail, sequential 4-5 Special Characters $RECEIVE system file 2-9, 5-11 HP NonStop TMF Introduction —540134-002 Index -4