Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Front cover CICS Transaction Gateway for z/OS Version 6.1 Install and configure the Gateway on z/OS Connect from WebSphere on Windows and z/OS Enable global transaction support and SSL security Nigel Williams Colin Alcock Steven Day Tommy Joergensen Rob Jones Phil Wakelin ibm.com/redbooks International Technical Support Organization CICS Transaction Gateway for z/OS Version 6.1 May 2006 SG24-7161-00 Note: Before using this information and the product it supports, read the information in “Notices” on page xi. First Edition (May 2006) This edition applies to CICS Transaction Gateway for z/OS V6.1. © Copyright International Business Machines Corporation 2006. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Part 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1. CICS Transaction Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 What is CICS Transaction Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.1 CICS TG products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.2 CICS TG application programming interfaces. . . . . . . . . . . . . . . . . . . 8 1.2 CICS TG for z/OS modes of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.1 Remote mode of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.2 Local mode of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3 JCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.1 Overview of JCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.3.2 CICS resource adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.4 Using the JCA with different CICS TG topologies . . . . . . . . . . . . . . . . . . . 15 1.4.1 WebSphere Application Server and CICS TG on distributed . . . . . . 16 1.4.2 WebSphere Application Server on distributed, CICS TG on z/OS . . 18 1.4.3 WebSphere Application Server and CICS TG on zSeries . . . . . . . . 19 1.4.4 Mixed version support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Chapter 2. Installing and configuring the CICS TG . . . . . . . . . . . . . . . . . . 25 2.1 Preparing for the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2 Installing CICS TG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2.1 Running the ctginstall script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.2.2 Basic security considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.3 Configuring CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.3.1 Defining an HFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.3.2 Setting permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.3.3 Creating a Gateway daemon configuration file . . . . . . . . . . . . . . . . . 36 2.3.4 Creating an STDENV file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.3.5 Creating the started task JCL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 © Copyright IBM Corp. 2006. All rights reserved. iii 2.4 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.4.1 Creating a CONNECTION definition . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.4.2 Creating a SESSIONS definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.4.3 Installing the definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.4.4 Configuring CICS for RRMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.4.5 Compiling and installing the sample programs . . . . . . . . . . . . . . . . . 46 2.4.6 Editing and assembling DFHCNV . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.4.7 Opening interregion communication . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.5 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.6 Operating CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.6.1 Starting the Gateway daemon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.6.2 Stopping the Gateway daemon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.6.3 Automatic Restart Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.7 Configuring for multiple Gateway daemons . . . . . . . . . . . . . . . . . . . . . . . 55 2.8 Migrating to CICS TG V6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.8.1 Migrating from ctgenvvar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.8.2 Other migration and configuration considerations. . . . . . . . . . . . . . . 58 2.9 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.9.1 Common problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.9.2 Tips and utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.9.3 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Part 2. Connection Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Chapter 3. Gateway daemon connections . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.2 Configuring WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . 80 3.2.1 Installing the resource adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.2.2 Creating connection factories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.2.3 Deploying the sample J2EE application . . . . . . . . . . . . . . . . . . . . . . 89 3.3 Configuring the Gateway daemon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.3.1 TCP/IP connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.3.2 Gateway threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.4 Configuring EXCI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3.4.1 EXCI pipe limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.4.2 Our EXCI settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.4.3 EXCI user-replaceable module: DFHXCURM. . . . . . . . . . . . . . . . . 100 3.4.4 Transactional EXCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.5 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.5.1 CICS CONNECTION and SESSION values . . . . . . . . . . . . . . . . . . 101 3.5.2 Adding a private mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 iv CICS Transaction Gateway for z/OS Version 6.1 3.5.3 Transaction timeouts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.6 Configuring for high scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.6.1 TCP/IP load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.6.2 Dynamic program link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.7 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.7.1 Single Gateway daemon connectivity tests. . . . . . . . . . . . . . . . . . . 107 3.7.2 Cloned Gateway daemon connectivity tests . . . . . . . . . . . . . . . . . . 114 3.8 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.8.1 Common connection problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.8.2 Tips and utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3.8.3 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Chapter 4. Connecting from WebSphere Application Server for z/OS . . 121 4.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 4.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.2 WebSphere Application Server configuration . . . . . . . . . . . . . . . . . . . . . 124 4.2.1 J2EE server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 4.2.2 Defining workload profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.2.3 Installing the CICS ECI resource adapter . . . . . . . . . . . . . . . . . . . . 126 4.2.4 Creating a connection factory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.2.5 Configuring the J2EE server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.2.6 Deploying our sample J2EE application . . . . . . . . . . . . . . . . . . . . . 134 4.3 Configuring EXCI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.1 EXCI pipe limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.2 CTG_PIPE_REUSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.3 Defining the EXCI library to your J2EE Server . . . . . . . . . . . . . . . . 138 4.4 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.4.1 Adding a private mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 4.4.2 Transaction timeouts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.5 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.6 Configuring for high scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4.6.1 TCP/IP load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4.6.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4.6.3 Dynamic program link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.6.4 Other configuration considerations . . . . . . . . . . . . . . . . . . . . . . . . . 151 4.7 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 4.7.1 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Part 3. Transaction Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Chapter 5. Gateway daemon transactions . . . . . . . . . . . . . . . . . . . . . . . . 157 5.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 5.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Contents v 5.2 Transactions - what are they . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.2.1 CICS transactions, tasks, and syncpoints. . . . . . . . . . . . . . . . . . . . 159 5.2.2 Distributed transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 5.3 Resource Recovery Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.3.1 How RRS works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.3.2 CICS TS and RRS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.4 Transactional support in the CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.4.1 Extended logical units of work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.4.2 Transactional support in the CICS ECI resource adapters . . . . . . . 167 5.5 Transactions in WebSphere Application Server . . . . . . . . . . . . . . . . . . . 171 5.6 Configuring for extended LUWs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.6.1 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 5.6.2 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 5.6.3 Configuring CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 5.6.4 Configuring WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 5.6.5 Testing the extended LUW configuration . . . . . . . . . . . . . . . . . . . . 182 5.7 Configuring for XA transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.7.1 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 5.7.2 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 5.7.3 Configuring CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 5.7.4 Configuring WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 5.7.5 Testing the XA transaction configuration . . . . . . . . . . . . . . . . . . . . 201 5.8 Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 5.8.1 Gateway daemon shutdown and restart . . . . . . . . . . . . . . . . . . . . . 219 5.8.2 Gateway daemon restart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 5.8.3 The ctgasi utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 5.9 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 5.9.1 Common transaction problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 5.9.2 Tips and utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Chapter 6. Gateway daemon transactions with TCP/IP load balancing . 225 6.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.2 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3 Transaction affinity considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3.1 Extended LUWs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3.2 XA transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 6.3.3 XA-enabled CICS TG group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 6.4 Configuring an XA-enabled CICS TG group . . . . . . . . . . . . . . . . . . . . . . 230 6.4.1 CICS TG master process configuration . . . . . . . . . . . . . . . . . . . . . 230 6.4.2 Gateway daemon configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 6.5 Testing the CICS TG group configuration . . . . . . . . . . . . . . . . . . . . . . . . 234 6.5.1 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 6.5.2 Test scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 vi CICS Transaction Gateway for z/OS Version 6.1 6.6 Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 6.6.1 CICS TG group startup and shutdown . . . . . . . . . . . . . . . . . . . . . . 248 6.6.2 CICS TG group commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 6.7 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 6.7.1 Tracing with ctgmaster trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Chapter 7. Transactions with WebSphere Application Server for z/OS . 253 7.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 7.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 7.2 Resource Recovery Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 7.3 Transactional support with WebSphere on z/OS . . . . . . . . . . . . . . . . . . 255 7.3.1 Transactional support in the CICS ECI resource adapters . . . . . . . 256 7.4 Configuring for global transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.4.1 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.4.2 Configuring CICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.4.3 Configuring WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 7.4.4 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 7.5 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 7.5.1 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 Part 4. Security Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Chapter 8. Gateway daemon security . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 8.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 8.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 8.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 8.2 Introduction to CICS and CICS TG security . . . . . . . . . . . . . . . . . . . . . . 283 8.3 Configuring CICS and CICS TG security . . . . . . . . . . . . . . . . . . . . . . . . 286 8.4 Configuring JCA security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 8.4.1 J2C authentication data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 8.4.2 Setting user ID and password on the ECIConnectionSpec. . . . . . . 296 8.5 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 8.5.1 Testing security with Java program EciB1 . . . . . . . . . . . . . . . . . . . 296 8.5.2 Testing security with J2EE application CTGTesterCCI . . . . . . . . . . 299 8.6 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 8.6.1 Common security problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 8.6.2 Tips and utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 Chapter 9. Enabling SSL with the Gateway daemon . . . . . . . . . . . . . . . . 309 9.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 9.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 9.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 9.2 Secure Sockets Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 9.2.1 JSSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 Contents vii 9.2.2 Cipher suites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313 9.2.3 Hardware cryptography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 9.2.4 The hwkeytool command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 9.2.5 Java configuration for using hardware cryptography . . . . . . . . . . . 315 9.3 Configuring server authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 9.3.1 Creating a keystore on z/OS using keytool . . . . . . . . . . . . . . . . . . . 316 9.3.2 Creating a key ring and certificates on z/OS using RACF . . . . . . . 319 9.3.3 Configuring CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 9.3.4 Testing server authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 9.4 Configuring client authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 9.4.1 Creating the client certificate on a Windows client machine . . . . . . 329 9.4.2 Extracting a client personal certificate. . . . . . . . . . . . . . . . . . . . . . . 330 9.4.3 Configuring CICS TG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 9.4.4 Testing client authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 9.5 Migration considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 9.5.1 SystemSSL to JSSE migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 9.6 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 9.6.1 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Chapter 10. Security with WebSphere Application Server for z/OS . . . . 339 10.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 10.1.1 Software checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 10.1.2 Definitions checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 10.2 Configuring CICS and CICS TG security . . . . . . . . . . . . . . . . . . . . . . . 342 10.3 Configuring JCA security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.3.1 Thread identity support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.4 Testing the configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 10.4.1 Testing thread identity support . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 10.5 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Part 5. Appendixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 Appendix A. Sample applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 The CTGTesterCCI application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 Application overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 The CTGTesterCCIXA application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 Application overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 Appendix B. DFHCNV and CICS data conversion . . . . . . . . . . . . . . . . . . 379 Data conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 DFHCNV and the mirror program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 Code page aware Java programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 Code page aware Java programs without DFHCNV. . . . . . . . . . . . . . . . . 382 viii CICS Transaction Gateway for z/OS Version 6.1 Appendix C. EXCI Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 EXCI trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Appendix D. Additional material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Locating the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Using the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 System requirements for downloading the Web material . . . . . . . . . . . . . 394 How to use the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 Contents ix x CICS Transaction Gateway for z/OS Version 6.1 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. © Copyright IBM Corp. 2006. All rights reserved. xi Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: Redbooks (logo) ibm.com® z/OS® z/VM® zSeries® AIX® CICS® CICSPlex® DB2® ™ HiperSockets™ Iterations® IBM® IMS™ Language Environment® MVS™ OS/390® POWER™ Rational® Redbooks™ RACF® SupportPac™ Tivoli® TXSeries® VisualAge® WebSphere® The following terms are trademarks of other companies: Enterprise JavaBeans, EJB, Java, Java Naming and Directory Interface, JavaBeans, JavaServer, JDBC, JDK, JSP, JVM, J2EE, Solaris, Sun, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. xii CICS Transaction Gateway for z/OS Version 6.1 Preface The CICS® Transaction Gateway (CICS TG) provides the IBM® implementation of the J2EE™ Connector Architecture (JCA) for access to CICS from WebSphere® Application Server. CICS TG V6 currently supports the following platforms: z/OS®, Linux® on zSeries®, AIX® , Linux on Intel®, Microsoft® Windows®, HP-UX, and Sun™ Solaris™ operating environments.The qualities of service of the CICS TG are highest when deploying on the z/OS platform. This configuration provides significantly improved performance and better management of connections, security, and transactions. In this IBM Redbook, we introduce the new facilities of the CICS TG for z/OS V6.0 and V6.1, which provide significant improvements in the areas of transactional integration, systems management, performance, security and ease of use. Included are a set of configuration scenarios for a variety of different z/OS deployment topologies for integrating J2EE applications on WebSphere Application Server with CICS TS for z/OS. The topologies illustrate how to use the new CICS ECI XA resource adapter to provide two-phase commit transactional integration between WebSphere Application Server and CICS TS. Also included are details on how to: Dynamically administer the Gateway daemon from the z/OS console Use the new JES logging capabilities Configure remote connections from WebSphere Application Server running on a Windows platform Configure local connections from WebSphere Application Server for z/OS Enable secure interoperability between WebSphere Application Server and CICS Exploit a RACF® key ring when configuring JSSE SSL support © Copyright IBM Corp. 2006. All rights reserved. xiii The team that wrote this redbook This International Technical Support Organization (ITSO) redbook was produced by a team of specialists from around the world working at the Product Solutions and Support Centre in IBM Montpellier, France. Figure 1 The team posing in the IBM Montpellier vineyard Colin Alcock is an IT Software Support Specialist based in IBM Farnborough, UK. He has worked in IBM providing customer support for CICS products since 1989 and currently supports the CICS TS, TXSeries® CICS and CICS Transaction Gateway products. Colin has worked on two other ITSO residencies, and previously worked in the CICS TS change team. Steven Day works for the CICS TG Level 3 Service Team at IBM Hursley. He has 20 years experience in mainframe systems programming. Tommy Joergensen is a Senior IT Specialist working for IBM Global Services in IBM Denmark. He has more than 25 years of experience working in CICS technical support, including three years at IBM Hursley. In recent years he has delivered services at large accounts in Denmark for both the CICS and WebSphere products. Tommy is the IBM representative in the CICS working group of the Nordic Share Guide organization. xiv CICS Transaction Gateway for z/OS Version 6.1 Rob Jones is a Software Engineer working for IBM at the Hursley software development laboratory in the United Kingdom. He has ten years of experience in the CICS Transaction Processing area, including development experience with CICS Transaction Server for z/OS and the CICS Transaction Gateway. He holds a degree in Computer Science and Mathematics from Durham University. He specializes in the z/OS platform aspects of the CICS TG product, most recently working on CTGBATCH and ctgmaster components. Phil Wakelin is the CICS Transaction Gateway Technical Planner, responsible for the product's future development. Based at IBM Hursley, he's an IBM Certified Solutions Expert in CICS Web Enablement and the author of many papers, IBM Redbooks™, and CICS SupportPacs. Nigel Williams is a Certified IT Specialist working in the IBM Design Centre for On Demand Business in Montpellier. He specializes in core business transformation, connectors, and service-oriented architectures. He is the author of several papers and IBM Redbooks and he speaks frequently on CICS and WebSphere topics. Previously, Nigel worked at the Hursley software lab as a software developer, in systems test and as customer support for the CICS Early Support Program. He holds a degree in Mathematics and Economics from Surrey University. Thanks to the following people for their contributions to this project: Phil Hanson and Andy Bates of IBM Hursley, and Chris Rayns of the ITSO Poughkeepsie, for their support with this project. Gary Flood and Pete Masters of the IBM Hursley Lab, for explaining the workings of the CICS Transaction Gateway V6.1 XA transaction support. Patrick Kappeler of IBM Montpellier, and Andrew Smithson of IBM Hursley for their help with the SSL test scenarios. Philippe Richard of IBM Montpellier for providing excellent systems support. Phil Wakelin, John Joro, Kevin Kinney and David Seager, who were the authors of the IBM Redbook CICS Transaction Gateway V5 The WebSphere Connector for CICS, SG24-6133. The following people for the significant amount of time that they have spent reviewing and for their detailed review comments: Mike Cox of the Washington System Center Mitch Johnson of IBM zSeries Services Sylvie Lemariey of IBM Montpellier Daniel McGinnes of IBM Hursley Alan Roessle of IBM Montpellier Steve Wall of IBM Montpelier Adrian Warman of IBM Hursley Preface xv Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners, or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. To learn more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html Comments welcome Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an e-mail to: [email protected] Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400 xvi CICS Transaction Gateway for z/OS Version 6.1 Part 1 Part 1 Introduction In this part, we give a broad overview of the CICS Transaction Gateway. We outline the major CICS TG components and describe the different topologies in which you can use the CICS TG. We also describe how we installed and configured the CICS TG on our system and how we tested the installation. © Copyright IBM Corp. 2006. All rights reserved. 1 2 CICS Transaction Gateway for z/OS Version 6.1 1 Chapter 1. CICS Transaction Gateway The CICS Transaction Gateway (CICS TG) is a market-leading connector that provides a high performing, secure, scalable, and tightly integrated method of e-business access to CICS. It enables CICS customers to expand the value of their investment in classic CICS applications and usually requires no changes to existing CICS applications. It is easy to install and has flexible configuration options that require minimal changes to CICS. It provides a range of networking options and provides a choice of Java™ and non-Java client programming interfaces. In this chapter, we introduce the CICS TG components, the application programming interfaces (APIs) that it provides. We also discuss the J2EE Connector Architecture (JCA), the CICS JCA resource adapters, and the topologies in which you can deploy the CICS TG. © Copyright IBM Corp. 2006. All rights reserved. 3 1.1 What is CICS Transaction Gateway The CICS TG is a set of client and server software components that allow a remote client application to invoke services in a CICS region. The client application can be either a Java application or a non-Java application using either C, C++, COBOL or COM interfaces (depending on the platform used). When a Java application is used, then the application can be any type of client (such as an applet, a servlet, or an enterprise bean). However, in the J2EE environment, the application is typically a servlet or enterprise bean that is deployed into a J2EE application server such as WebSphere Application Server. 1.1.1 CICS TG products With CICS TG V6 and later there are now the following two distinct orderable CICS TG products: CICS TG for Multiplatforms CICS TG on z/OS CICS TG for Multiplatforms V6.0 CICS TG for Multiplatforms V6.0 is supported on the following range of operating systems and platforms and is designed to support connectivity to all in-service CICS servers: Linux on zSeries Linux on Intel Linux on POWER™ AIX HP-UX (on PA-RISC) Sun Solaris (on SPARC) Windows XP, Windows 2000, and Windows 2003 This version was announced on 30 November 2004 in a U.S. announcement letter 204-2284, which is available at: http://www-306.ibm.com/fcgi-bin/common/ssi/ssialias?infotype=an&subtype= ca&supplier=897&letternum=ENUS204-284 CICS TG for Multiplatforms is comprised of the following main runtime components: The Gateway daemon, which listens for incoming work and manages the threads and connections necessary to ensure good performance. The Client daemon, which provides the communication to CICS servers and the non-Java APIs. 4 CICS Transaction Gateway for z/OS Version 6.1 A Java class library or JCA resource adapter, which is deployed into the client runtime environment. When used in a JCA environment the resource adapter is deployed into the J2EE application server. A Java client program can connect to a remote Gateway daemon using the TCP or SSL protocols. The Client daemon then provides the transport drivers to connect to the CICS server, as shown in Figure 1-1. Note that because the non-Java APIs are provided by the Client daemon, there is no remote connectivity support for non-Java clients. Note: The CICS Universal Client V6, which provides the Client daemon functions within CICS TG, is available as a separate product on the AIX, Linux, and Windows platforms. CICS Universal Client provides the transport protocols and non-Java programming interfaces for single user access to CICS. Distributed platform CICS TG JNI module Java TCP or SSL Client ctgclient.jar cicseci.rar cicseciXA.rar cicsepi.rar z/OS, OS/390 VSE, TxSeries Gateway daemon Protocol handler ctgjni.dll ctgserver.jar Client daemon APPC TCP62 TCP/IP CICS Server Figure 1-1 Components of CICS TG for Multiplatforms CICS TG for Multiplatforms V6.0 provides the following improvements: Runtime performance: – The runtime performance of the Client daemon has been enhanced by improvements in the internal tracing mechanism. – Support for the Native Posix Thread Library (NPTL) threading model is provided for the RHEL V3 and SLES 9 Linux distributions, providing for improved performance and scalability for multi-threaded applications. Chapter 1. CICS Transaction Gateway 5 Systems management: – A normal shutdown mechanism has been introduced allowing the Gateway daemon to be terminated in either a controlled or immediate manner. – A command-based systems administration tool (ctgadmin) provides improved usability and the added capability of invoking shutdown processing. – Support for a system daemon process and a new daemon process (ctgd) in UNIX® and Linux environments. – APPC network connections to CICS servers are now supported for Linux on zSeries using the facilities of IBM Communications Server. – Management of Gateway daemon log files is now possible, allowing a maximum file size and number of log archives to be specified. Security A new configuration option for SSL connections allows the negotiated SSL cipher suite to be controlled, providing for increased control of security levels when using SSL connections. Installation Installation and migration is simplified using Install Shield Multiplatform (ISMP) on all platforms. Java J2EE Connector Architecture (JCA) 1.5 supports provides for enhancements in the management of connections, security, and transactions in supported J2EE V1.4 application servers. CICS TG for z/OS V6.1 CICS TG for z/OS V6.1 is the latest version of the z/OS product. It is supported on z/OS V1R4 and later and supports connectivity to CICS TS for z/OS V1.3, V2.2, V2.3, and V3.1. The product was announced on 04 October 2005 in a U.S. announcement letter 205-248, which is available at: http://www-306.ibm.com/fcgi-bin/common/ssi/ssialias?infotype=an&subtype= ca&supplier=897&letternum=ENUS205-248 6 CICS Transaction Gateway for z/OS Version 6.1 CICS TG for z/OS uses the external communication interface (EXCI) interface provided by CICS TS to communicate with CICS (Figure 1-2). It does not include the Client daemon and does not provide any support for non-Java based applications because this support is provided through the CICS EXCI interface. z/OS CICS TG Java Java Client Client ctgclient.jar cicseci.rar cicseciXA.rar TCP or SSL JNI module Gateway daemon Protocol handler libctgjni.so ctgserver.jar EXCI CICS TS MRO IRC Figure 1-2 Components of CICS TG for z/OS CICS TG for z/OS V6.1 has many improvements principally in the area of systems management, usability, and performance as follows: XA transaction support enables CICS Transaction Server (CICS TS) for z/OS to participate in a global two-phase commit transaction that is initiated in a distributed J2EE V1.4 application server, such as WebSphere Application Server V6.0. Management of the Gateway daemon from SDSF provides better system administration capabilities, including a new normal shutdown mechanism. Direction of output to JES provides improved management for runtime messages. An option to limit the number of EXCI pipes to one per thread improves availability and scalability. Security Enhancements include the following: – Storage of SSL private keys and certificates within the RACF database. – Improved support for hardware cryptography cards when using SSL. – A new external configuration option for SSL encryption allowing the SSL cipher suite to be specified. Chapter 1. CICS Transaction Gateway 7 Connection enhancements: – A load-balancing feature for XA transactions that enables a group of Gateways to act as one virtual endpoint for incoming TCP/IP requests, helping to maximize scalability and availability. – An automatic TCP/IP subsystem reconnect feature that allows new TCP/IP connections to be made to the Gateway daemon after a restart of the TCP/IP subsystem. This leads to improved availability and less operator intervention. – An IP address binding feature that allows administrators to control which network interfaces can be used by the Gateway daemon, providing improved control of security and ease of network management. – A socket connection timeout feature that provides the option to specify a timeout duration on pending socket connection requests from the J2EE application to a remote Gateway daemon, giving improved recovery in the TCP/IP network. SMP/E packaging and installation, which simplifies the process of installing, migrating, and applying corrective maintenance to the CICS TG. 1.1.2 CICS TG application programming interfaces CICS TG for Multiplatforms provides the following programming interfaces for accessing CICS applications: External Call Interface (ECI) External Presentation Interface (EPI) External Security Interface (ESI) Note: The CICS TG on z/OS supports only the ECI, because of its reliance on the CICS EXCI function that provides the call interface and connectivity between the Gateway daemon and the CICS TS region. External Call Interface The ECI is used for calling COMMAREA-based CICS programs. The COMMAREA is the buffer that is used for passing the data between the client and the CICS server. CICS sees the client request as a distributed program link (DPL) request. The ECI enables a user application to call a CICS program synchronously or asynchronously. It enables the design of new applications to be optimized for client-server operation, with the business logic on the server and the presentation logic on the client. 8 CICS Transaction Gateway for z/OS Version 6.1 An ECI request can be invoked from a Java application using a variety of different interfaces: The ECIRequest class that is provided by the CICS TG base classes. This interface provides a simple procedural type interface to the ECI. It is supported in any Java environment (such as an stand-alone application) and provides similar capabilities to the JCA. However, it does not provide the same qualities of service (such as XA transaction support). The Common Client Interface (CCI) that is provided by the CICS ECI resource adapters (cicseci.rar or cicseciXA.rar). These classes define a standard architecture for connecting the Java 2 Platform Enterprise Edition (J2EE) platform to a heterogeneous EIS such as CICS. Java applications interact with resource adapters using the Common Client Interface (CCI), which is a common framework of classes extended by each resource adapter to allow communication with a specific EIS. CCF: Before the existence of the JCA, IBM recognized a need for a common way to connect to EIS systems and introduced the Common Connector Framework (CCF) through the IBM VisualAge® for Java product. CCF was similar in concept to JCA. However, CCF was not an open specification and did not provide the same support for system contracts and qualities of service (such as transaction support and connection pooling). CICS TG V6.1 no longer provides support for the CCF. However, a SupportPac™ (CE52) is available that provides runtime support for CCF using a CICS TG V6 Java client. It is available at: http://www.ibm.com/software/htp/cics/ctg/support External Presentation Interface The EPI is used for invoking 3270-based transactions. A terminal is installed in CICS, and CICS sees the request as running on a remote terminal controlled by the CICS TG. An EPI request can be invoked from a Java application using one of three different interfaces: The EPIRequest class provided by the CICS TG base classes. This class provides a Java interface to the EPI, and is used for invoking 3270-based transactions. Due to its low-level nature, using it for developing EPI applications requires a strong knowledge of CICS and 3270 data streams. Chapter 1. CICS Transaction Gateway 9 The EPI support classes, which provide high-level constructs for handling 3270 data streams. A wide range of classes is provided including AID, FieldData, Screen, Terminal, Map and MapData. These are used to represent the interface to a CICS 3270 terminal, and the resulting 3270 response. The Common Client Interface (CCI) provided by the CICS EPI resource adapter (cicsepi.rar). EPI is typically used when it is not possible to separate the presentation logic from the business logic of an application, and allows the reformatting of the user interface, without modification of the CICS application. For a more detailed discussion on the benefits of separating business logic from presentation logic refer to Revealed! Architecting e-business Access to CICS, SG24-5466. External Security Interface The ESI is used for verifying and changing the user ID and password information held in the CICS external security manager (ESM), such as RACF. It is based on the CICS Password Expiration Management (PEM) function. ESI calls to CICS can only be made using the ESIRequest class that is provided by the CICS TG base classes. This class provides a Java interface to the ESI, and provides two simple PEM requester functions: Verify Password Allows a client application to verify that a password matches the password for a given user ID that is stored by the CICS ESM. Change Password Allows a client application to change the password that is held by the CICS ESM for a given user ID. There is no other interface available for the ESI, although both the EPI and ECI allow user IDs and passwords to be flowed within the actual requests. In this case the user ID and password is authenticated either within CICS, if using a Multiplatform CICS TG, or within the CICS TG, if using the CICS TG for z/OS. 1.2 CICS TG for z/OS modes of operation This redbook focuses on installing and configuring the CICS TG for z/OS for two different modes of operation: remote and local. 1.2.1 Remote mode of operation The remote mode of operation uses the Gateway daemon as a long running task which listens on specified ports for incoming ECI requests and then forwards 10 CICS Transaction Gateway for z/OS Version 6.1 them to the CICS server via the EXCI protocol. The Gateway deamon runs in its own MVS™ address space and provides connection and thread management as shown in Figure 1-1 on page 5. 1.2.2 Local mode of operation If the CICS TG is to be used on the same machine as WebSphere Application Server, it is more efficient to use the CICS TG classes within WebSphere Application Server to provide the gateway functionality (Figure 1-3). This mode of operation allows WebSphere Application server to manage the connections and threads and reduces the communications overhead. This configuration is known as the local mode of operation. WebSphere Application Server Java Client HTTP JSP Servlet EJB CCI Z/OS or Distributed CICS TG Resource adapter CICS Server EXCI or Client daemon Figure 1-3 CICS TG in local mode with WebSphere Application Server There is further discussion of the local and remote modes of operation in “Using the JCA with different CICS TG topologies” on page 15. 1.3 JCA The J2EE connector architecture (JCA) defines a standard for connecting the Java 2 Platform, Enterprise Edition (J2EE) to heterogeneous Enterprise Information Systems (EIS) such as CICS. The architecture defines a set of scalable, secure, and transactional mechanisms that enable the integration of EISs with application servers and enterprise applications. A proprietary J2EE application server, such as IBM WebSphere Application Server, can be extended to support the resource adapter architecture and is then assured of a seamless Chapter 1. CICS Transaction Gateway 11 connectivity to multiple EISs. Likewise, an EIS vendor provides one standard resource adapter with the capability to plug into any application server that supports the connector architecture. 1.3.1 Overview of JCA Example 1-4 shows the key parts of the JCA: the deployable components (resource adapters), the programming interface they implement (CCI), and the qualities of service that they implement (system contracts). J2EE Server (e.g WebSphere Application Server) Container-Component Contract Application Component (e.g EJB) Common Client Interface (CCI) ConnectionFactory cf =<JNDI lookup> Connection connection =cf.getConnection(); System Contracts Interaction interaction =connection.createInteraction(); interaction.execute(<input and output data>); interaction.close(); connection.close(); ƒ Connection Management ƒ Transaction Management ƒ Security Management Resource Adapter (e.g CICS ECI resource adapter) EIS Specific Interface Enterprise Information System (e.g CICS) Figure 1-4 J2EE Connector Architecture (JCA) Resource adapters A resource adapter is a system-level software driver that a Java application uses to connect to an EIS. A resource adapter is installed into an application server and provides connectivity between the EIS, the application server, and the enterprise application. CCI The Common Client Interface (CCI) defines a standard client API for application components to access multiple resource adapters. This API can be used directly, or enterprise application integration frameworks can be used to generate EIS access code for the developer. The CCI is designed to be an EIS independent API, such that an enterprise application development tool can produce code for any J2EE compliant resource adapter that implements the CCI interface. 12 CICS Transaction Gateway for z/OS Version 6.1 The CCI has the following additional characteristics: It forms a base level API for EIS access on which higher level functionality specific to an EIS can be built. It provides an API that is consistent with other APIs in the J2EE platform, such as JDBC™. It is targeted primarily towards application development tools and enterprise application integration frameworks, rather than Java developers using the CCI API directly. It is an optional part of the JCA specification. System contracts The Connector architecture defines a standard set of system-level contracts between a J2EE server and a resource adapter.The standard contracts are as follows: A connection management contract that lets a J2EE server pool connections to an underlying EIS, and lets application components connect to an EIS. This leads to a scalable application environment that can support a large number of clients requiring access to EIS systems. A transaction management contract between the transaction manager and an EIS that supports transactional access to EIS resource managers. This contract lets a J2EE server use a transaction manager to manage transactions across multiple resource managers. This contract also supports transactions that are managed internal to an EIS resource manager without the necessity of involving an external transaction manager. A security contract that enables secure access to an EIS. This contract provides support for a secure application environment, which reduces security threats to the EIS and protects valuable information resources managed by the EIS. These system contracts are transparent to the application developer, meaning they do not have to implement these services themselves. JCA Version 1.5 In addition to the standard connection, transaction and security system contracts, JCA Version 1.5 provides four additional system contracts: Life cycle management, allowing an application server to control the startup and termination of a resource adapter. This is not supported by the CICS TG or by WebSphere Application Server. Chapter 1. CICS Transaction Gateway 13 Work management, allowing a resource adapter to submit work instances to an application server for execution. This is not supported by the CICS TG or by WebSphere Application Server. Message inflow, allowing a resource adapter to asynchronously deliver messages to message endpoints residing in the application server Transaction inflow, allowing a resource adapter to import a transaction context from the EIS system into an application server. Note: The CICS resource adapters do not exploit the new JCA V1.5 system contracts currently. JCA 1.5 also provides two optimizations to the transaction management and connection management contracts, which are exploited in the JCA 1.5 resource adapters provided by CICS TG V6.0 and V6.1: Lazy connection association This contract provides for potential improved connection pooling in J2EE application servers. It allows the resource adapter to disassociate a cached connection handle from the managed connection once the connection handle is no longer being used by the J2EE component. This allows J2EE applications to use the get-use-cache model for connection handles, and for the underlying connections to still be efficiently pooled by the Connection Pool manager in WebSphere Application Server. Lazy transaction enlistment This contract optimizes transaction enlistment calls so that the resource adapter only participates in a transaction if it is actually invoked, rather than being enlisted whenever a reference exists within the J2EE component. This provides for potential better performance if transactional JCA applications are used in an J2EE application server. 1.3.2 CICS resource adapters The following resource adapters are provided for use with the CICS TG for Multiplatforms. The resource adapter archives (RAR files) are supplied in the <install_path>/deployable directory. 14 cicseci.rar The CICS ECI resource adapter, which provides one-phase transactional support. cicsepi.rar The CICS EPI resource adapter, which is non-transactional. CICS Transaction Gateway for z/OS Version 6.1 The following resource adapters are provided for use with the CICS TG for z/OS. The resource adapter archives (RAR files) are supplied in the <install_path>/deployable directory. cicseci.rar The CICS ECI resource adapter, which provides one-phase transaction support when installed into WebSphere Application Server on a distributed platform, and two-phase transactional support when deployed into WebSphere Application Server for z/OS. cicseciXA.rar The CICS ECI XA resource adapter, which provides two-phase transactional support when deployed into WebSphere Application Server on any supported platform. 1.4 Using the JCA with different CICS TG topologies The qualities of service provided by the JCA vary, depending on the topology in use. The most common topologies in use today are shown in Figure 1-5 and are as follows: Topology 1. WebSphere Application Server and the CICS Transaction Gateway are both deployed on a distributed (non-zSeries) platform. Topology 2. WebSphere Application Server is deployed on a distributed platform and CICS Transaction Gateway is deployed on a z/OS system. Topology 3. Both WebSphere Application Server and CICS Transaction Gateway are deployed on zSeries servers. Chapter 1. CICS Transaction Gateway 15 Topology 1 WebSphere Server and CICS TG Topology 2 zSeries WebSphere Application Server HTML CICS TG z/OS Topology 3 Network CICS WebSphere and CICS TG on zSeries Figure 1-5 JCA deployment topologies 1.4.1 WebSphere Application Server and CICS TG on distributed In this topology both WebSphere Application Server and CICS TG are deployed on one of the distributed platforms such as AIX or Linux (Figure 1-6). Distributed platform z/OS WebSphere Application Server V6 CICS TS HTML JSP Servlet CICS TG V6.0 EJB CCI CICS ECI resource adapter Client daemon SNA or TCP62 or TCP/IP C O M M A R E A COBOL application Figure 1-6 WebSphere Application Server and CICS TG on distributed platform The Gateway daemon is not required in this configuration because the CICS TG is used in local mode to invoke the Client daemon native libraries directly from the J2EE enterprise bean. The Client daemon then flows the ECI request to the CICS server using either TCP/IP, a Systems Network Architecture (SNA) connection or a TCP62 connection. The management of connections, 16 CICS Transaction Gateway for z/OS Version 6.1 transaction and security is controlled within WebSphere Application Server through a combination of both configuration parameters and application deployment descriptors. Note: The TCP62 protocol allows SNA flows to be encapsulated in TCP/IP packets and works in conjunction with the Anynet LU6.2/IP support in VTAM. The specific qualities of service (in terms of the JCA system contracts) that apply to this topology are as follows. Connection pooling Pooling of connections is provided seamlessly by the pool manager component of WebSphere Application Server allowing the reuse of connections to the resource adapter between multiple J2EE components. Note: The TCP/IP or SNA network connections from the Client daemon into the CICS region are managed and re-used by the Client daemon component of the CICS TG and are not subject to the JCA connection pooling system contract. Transaction management The CICS ECI resource adapter (cicseci.rar) only supports the LocalTransaction interface. As a result, the scope of the transaction is limited to the Resource Manager (that is, the associated connection factory and the specified CICS server). Such Resource Manager LocalTransactions only support one-phase commit processing. However, if the J2EE application server supports the JCA option of last-resource commit optimization, an ECI interaction can participate in a global transaction, provided that it is the only one-phase-commit resource in the global transaction. This function is provided by the last-participant support in WebSphere Application Server V6.0 and WebSphere Application Server Enterprise Edition V5.0. Security management Security credentials (user ID and password) propagated through to CICS from WebSphere Application Server can be determined by the application (component-managed) or by the Web or EJB™ container (container-managed). Container-managed sign-on is recommended because it is good practice to separate the business logic of an application from qualities of service, such as security and transactions. In this topology, however, the principal means of enabling container-managed authentication is by specifying the user ID and password in a JCA authentication entry (also Chapter 1. CICS Transaction Gateway 17 known as an alias) and associating the alias with the resource reference when the J2EE application is deployed. 1.4.2 WebSphere Application Server on distributed, CICS TG on z/OS When WebSphere Application Server is deployed on one of the distributed platforms, it is possible to access CICS through a Gateway daemon running on z/OS (Figure 1-7). Distributed z/OS WebSphere Application Server V6 CICS TG V6.1 HTML JSP Servlet Gateway Daemon CICS TG V6.1 EJB CCI CICS ECI or CICS ECI XA resource adapter TCP SSL JNI EXCI COBOL Application CICS TS V3.1 Figure 1-7 WebSphere Application Server on distributed and CICS TG on z/OS The protocol specified in the connection settings of the connection factory is one of the remote protocols (TCP or SSL). The communication from the Gateway daemon on z/OS to the CICS region uses the EXCI. This topology enables integration with z/OS internet protocol (IP) workload-management functions, including Sysplex Distributor and TCP/IP port sharing. Use of these technologies allows an individual Gateway daemon to be removed as a single point of failure and enables incoming work to be efficiently balanced across multiple Gateway daemons in multiple z/OS logical partitions (LPARs). The specific JCA qualities of service that apply to this topology are as follows: Connection pooling The connection pool represents physical network connections between WebSphere Application Server and the Gateway daemon on z/OS. In such a configuration, it is essential to have an efficient connection-pooling 18 CICS Transaction Gateway for z/OS Version 6.1 mechanism because otherwise, a significant proportion of the time from making the connection to receiving the result from CICS Transaction Server and closing the connection can be in the creation and destruction of the connection itself. The JCA connection-pooling mechanism mitigates this overhead by allowing connections to be pooled by the WebSphere Application Server pool manager. For information about configuring connections with this topology see Chapter 3, “Gateway daemon connections” on page 77. Transaction management In this topology, two-phase commit global transactions are now supported, using the CICS ECI XA resource adapter. Also, one-phase commit transactions using the CICS ECI resource adapter are still supported. Note that in either case, if the enterprise bean is deployed with a transactional deployment descriptor (for example, a value of REQUIRED), the resulting ECI request to the CICS region uses transactional EXCI. In this case, both the Gateway daemon and the CICS region need to be configured to register with Resource Recovery Services (RRS). For information about configuring transactions with this topology, see Chapter 5, “Gateway daemon transactions” on page 157 and Chapter 6, “Gateway daemon transactions with TCP/IP load balancing” on page 225. Security management In this configuration, the Gateway daemon is the entry point to the zSeries system in which your CICS system is running, so it is normal for the Gateway daemon to perform an authentication check for incoming ECI requests from clients. However, after the user has authenticated to WebSphere Application Server, a password might not be available to send to the Gateway daemon. In this case, therefore, you need to devise a way to establish a trust relationship between WebSphere Application Server and the Gateway daemon so that WebSphere Application Server can be trusted to flow only the user ID on the request through to CICS Transaction Gateway. Solutions such as SSL client authentication and virtual private networks (VPNs) can be used to establish such a trust relationship. For information about configuring security with this topology, see Chapter 8, “Gateway daemon security” on page 281 and Chapter 9, “Enabling SSL with the Gateway daemon” on page 309. 1.4.3 WebSphere Application Server and CICS TG on zSeries In a zSeries topology, WebSphere Application Server can be deployed on either a z/OS system or on a Linux operating system. The qualities of service differ between these two topologies and are therefore discussed separately. Chapter 1. CICS Transaction Gateway 19 WebSphere Application Server and CICS TG on z/OS In this topology (Figure 1-8), only the CICS ECI resource adapters are supported. The most common z/OS configuration uses the local mode of operation. This results in a direct cross-memory EXCI connection between WebSphere Application Server and CICS. Figure 1-8 shows an application deployed to WebSphere Application Server for z/OS. z/OS WebSphere Application Server V6 CICS TS V3.1 HTML JSP Servlet CICS TG V6.1 EJB CCI CICS ECI or CICS ECI XA resource adapter EXCI C O M M A R E A COBOL application Figure 1-8 WebSphere Application Server and CICS TG on z/OS CICS TG V5.01 introduced remote connection support for this topology, however, the highest qualities of service can be achieved when a local connection is used. The specific JCA qualities of service that apply to this topology are as follows: Connection pooling The connection pool is a set of connection objects managed by WebSphere Application Server, which is not directly associated with the EXCI pipes used for communication to CICS. For information about configuring connections with this topology, see Chapter 4, “Connecting from WebSphere Application Server for z/OS” on page 121. Transaction management A two-phase commit capability is provided through RRS, an integral part of z/OS. RRS acts as the external transaction coordinator for z/OS in managing the transaction scope between WebSphere Application Server, CICS and other z/OS subsystems, including IMS™, DB2®, and WebSphere MQ. For information about configuring transactions with this topology, see Chapter 7, “Transactions with WebSphere Application Server for z/OS” on page 253. 20 CICS Transaction Gateway for z/OS Version 6.1 Security management Both container-managed and component-managed sign-on are supported. In this topology, the ECI resource adapters flow only the user ID to CICS with the ECI request - it is assumed that the user is already authenticated by WebSphere Application Server. When using container-managed sign-on, a z/OS specific functionality known as thread identity support is provided by WebSphere Application Server for z/OS. For information about configuring security with this topology, see Chapter 10, “Security with WebSphere Application Server for z/OS” on page 339. CICS TG for Linux on zSeries WebSphere Application Server and CICS TG deployed within Linux on zSeries (Figure 1-9) provides a flexible and scalable environment based on the virtualization capabilities of IBM z/VM® and Linux systems. zSeries z/OS Linux on zSeries HTML CICS TS WebSphere Application Server V6 JSP APPC, TCP62 or TCP/IP Servlet CICS TG V6.0 EJB CCI CICS ECI resource adapter Client daemon Comms Server or HiperSockets C O M M A R E A COBOL application Figure 1-9 WebSphere Application Server and CICS TG for Linux on zSeries The JCA qualities of service for this topology are almost identical to those described in 1.4.1, “WebSphere Application Server and CICS TG on distributed” on page 16 because Linux on zSeries (within a JCA and CICS TG scenario) can be treated as a distributed platform. A significant exception to this generalization is that the HiperSockets™ function can be used to provide a highly efficient cross-memory transport for TCP/IP based communication into CICS (using the ECI over TCP/IP function of CICS TS V2.2 and later releases). Alternatively, an APPC connection to CICS TS can be provided by IBM Communications Server for Linux on zSeries. Chapter 1. CICS Transaction Gateway 21 1.4.4 Mixed version support The client-server nature of the CICS TG means that different levels of the Gateway daemon and the Java client (including the resource adapter) can interoperate. This provides a degree of flexibility in deployment, allowing software components to be updated at different times. However, because of the defined internal flow levels and of the JNI interface, you should follow these rules should to ensure interoperability: 1. Remote Java clients connecting to a Gateway daemon, must use a CICS resource adapter of the same (or lower) level CICS TG version as the Gateway daemon. 2. If the CICS TG is used in local mode within WebSphere Application Server, then the Java client and the CICS TG JNI library must be at the same CICS TG level. Full details of the latest supported software combinations are provided on the ibm.com® support pages for the CICS TG available at: http://www.ibm.com/software/htp/cics/ctg/support Here are some examples of supported configurations: Using a JCA 1.5 resource adapter (from CICS TG V6.1 or CICS TG V6.0) deployed in WebSphere Application Server V6 to connect to a CICS TG V6.1 Gateway daemon is supported. Using a JCA 1.0 resource adapter (from CICS TG v5.1) deployed in WebSphere Application Server v5.1 to connect to CICS TG V6.1 Gateway daemon is supported. Using a JCA 1.0 resource adapter (from CICS TG v5.1) deployed in WebSphere Application Server V6 to connect to a CICS TG V5.1 Gateway daemon is supported. 22 CICS Transaction Gateway for z/OS Version 6.1 Here are some examples of non-supported configurations: Using a JCA 1.5 resource adapter (from CICS TG V6.1 or CICS TG V6.0) deployed in WebSphere Application Server v5 is not supported. Using a JCA 1.5 resource adapter (from CICS TG V6.1 or CICS TG V6.0) deployed in WebSphere Application Server V6 to connect to a CICS TG V5.1 daemon is not supported. Using a CICS TG V5.1 JNI library for use in local mode with a JCA 1.5 resource adapter (from CICS TG V6.0) with WebSphere Application Server V6.0 is not supported. Note: Do not mix the levels of CICS resource adapters that are deployed on a WebSphere Application Server node. For example, do not install a resource adapter from CICS TG V6.1 to a node that has a V6.0 ECI or EPI resource adapter installed. The resource adapter version is shown in the ra.xml file in the resource adapter. Chapter 1. CICS Transaction Gateway 23 24 CICS Transaction Gateway for z/OS Version 6.1 2 Chapter 2. Installing and configuring the CICS TG In this chapter, we describe how to install and configure the CICS Transaction Gateway for z/OS V6.1 and how to test the installation with one of the sample Java applications that is supplied with the CICS TG. © Copyright IBM Corp. 2006. All rights reserved. 25 2.1 Preparing for the installation The following sections detail the software levels that we use in our configuration and include instructions on how to install the CICS TG for z/OS. z/OS UNIX System Services CICS TS region CICS TG Java application EciB1 Gateway daemon EXCI EC01 tx1.mop.ibm.com Figure 2-1 Software components: CICS TG for z/OS The CICS TG for z/OS runs in the z/OS UNIX System Services (USS) environment. You need a mix of traditional MVS skills and UNIX skills in order to install and to configure the CICS TG. 2.1.1 Software checklist We use the following software levels: z/OS V1R6 CICS Transaction Gateway for z/OS V6.1 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2 CICS Transaction Server for z/OS V3.1 The minimum levels that are supported with CICS TG V6.1 are: z/OS V1R4 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2 CICS Transaction Server for z/OS V1.3 26 CICS Transaction Gateway for z/OS Version 6.1 2.1.2 Definitions checklist Table 2-1 summarizes the definitions that we use in this scenario. Before you configure the products, we recommend that you acquire definitions for each value that is listed in this table. Table 2-1 Definitions checklist Value Gateway daemon CICS TS hostname tx1.mop.ibm.com - IP address 9.100.193.122 - TCP/IP port 13101 - Job name CITGR1G CITGR1CI APPLID - A6POTGR1 NETNAME CITGR1G - CONNECTION - GR1G 2.2 Installing CICS TG During the SMP/E installation of CICS TG, the HFS data set OMVS.CTG61.HFS that contains the CICS TG product files was created and mounted at the mount point /usr/lpp/cicstg/ctg610 (known as the <install_path>). For more information about the SMP/E installation, see the Program Directory for CICS Transaction Gateway for z/OS, GI10-2587. To install the product, you need to perform the following steps: 1. Run the ctginstall script. 2. Consider basic security setup. Chapter 2. Installing and configuring the CICS TG 27 2.2.1 Running the ctginstall script After the SMP/E installation, you must run the ctginstall script by executing the ctginstall command in OMVS (ctginstall 40 can be used to limit the screen width if required). When you run the script, you must agree to the licensing agreement, as shown in Example 2-1. Tip: When running the ctginstall script, you might see INPUT in the lower right-hand corner of your screen which means that OMVS has put your terminal to sleep. Press PF10 to refresh the screen. Example 2-1 Partial listing of the CICS TG license agreement CITGTJ: /usr/lpp/cicstg/ctg610>ctginstall To install CICS Transaction Gateway you must accept the following license agreement. If the license file does not display correctly, try restarting the installation using the command: '/usr/lpp/cicstg/ctg610/bin/ctgsetup 40'. Enter 1 to view the license, or 2 to quit. 1 >>> Beginning of the License File >>> International Program License Agreement Part 1 - General Terms BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE PROGRAM YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT... _______________________________________________________________________________ Enter 1 to page down, 2 to page up, 3 to accept or 4 to decline. 3 You have accepted the license agreement. Wait while the installation continues. The installation is complete. License files can be viewed using the command: '/usr/lpp/cicstg/ctg610/bin/ctgbrowse' CITGTJ: /usr/lpp/cicstg/ctg610> 28 CICS Transaction Gateway for z/OS Version 6.1 Figure 2-2 shows the resulting directory structure that follows the installation of CICS TG. ct ct ct ct ct ct ct ct ct ct /usr/lpp/cicstg/ctg610 Installation directory bin executables resource ctgcfg resources classes CTG Java class library docs CTG documentation ctgzoscfg non-z/OS config tool deployable J2EE RAR files ga r m ga si gbr owse gc f g gc onvenv ge nv var s amp gl og. hl p gmas t er gmsg. hl p gmsgs ct gs amp. i ni ct gs et up ct gs et up_ I BM- 93 0 ct gs et up_ I BM- 93 5 ct gs et up_ I BM- 93 9 ct gs t a r t l i bc t gj ni . s o l i bc t gj ni _g. s o l i bc t gmvs . s o t e st z os ccf 2 . j a r ci c s j 2e e. j a r conf t ool . j a r connect or . j ar ct gc l i e nt . j ar ct gs ampl es . j a r ct gs er v er . j ar mas k . j a r psk . j ar t gui de. j ar samples java com ibm ctg samples server CICS COBOL source code eci ECI samples j2ee J2EE samples security Security exit samples Figure 2-2 CICS TG z/OS directory structure Note: All the directories also contain an IBM directory that contains files with names in the form CTGnnnnn, where nnnnn are digits. These directories and files are part of the SMP/E installation. You should not alter or remove them. The /usr/lpp/cicstg/ctg610/bin directory contains the following files which are referred to later in this chapter: ctgsamp.ini ctgenvvarsamp ctgstart Sample CICS TG configuration file. Sampled ctgenvvar script. Shell script to start the Gateway daemon. You can browse files using TSO OBROWSE or ISHELL. Chapter 2. Installing and configuring the CICS TG 29 The /usr/lpp/cicstg/classes directory contains the following JAR files: ctgclient.jar Java class library ctgserver.jar Classes for use by Gateway daemon ctgsamples.jar Samples cicsj2ee.jar, ccf2.jar, connector.jar J2EE classes mask.jar, psk.jar, guide.jar Classes used by the configuration tool Tip: The contents of a JAR or zipped file can be listed with the USS command in OMVS: jar -tvf <file> 2.2.2 Basic security considerations You must perform the following basic RACF security tasks before starting the Gateway daemon: 1. Set up a user ID for the Gateway daemon started task. 2. Allow the Gateway daemon user ID read access to program-controlled libraries. 3. Define programs and libraries as program-controlled. These tasks enable basic RACF security permissions. For information about configuring a more secure environment, see Chapter 8, “Gateway daemon security” on page 281. Setting up a user ID for the Gateway daemon started task The user ID under which the Gateway daemon started task runs should: Have an OMVS segment defined. Be in a group that has an OMVS segment. Be defined without a password. Have READ access to the RACF profile that protects the TCPIP.STANDARD.TCPXLBIN data set. 30 CICS Transaction Gateway for z/OS Version 6.1 We ran our Gateway daemon as a started task under user ID CITGR1G. There are two methods of defining a default user ID to a started task: Use the RACF exit - ICHRIN03. Use the STARTED class in RACF. The following example shows the user ID CITGR1G in the CICS group being assigned as the user ID for the CITGR1G task: RDEF STARTED CITGR1G* STDATA(USER(CITGR1G) PRIVILEGED(NO) TRUSTED(NO)) Allow Gateway daemon access to program-controlled MVS libraries If the z/OS system has the BPX.SERVER FACILITY class profile defined within RACF, then the user ID under which the Gateway daemon runs must be permitted to this profile. The following example shows the PERMIT command that we issue for our CITGR1G Gateway daemon user ID: PERMIT BPX.SERVER CLASS(FACILITY) ID(CITGR1G) ACCESS(READ) PERMIT BPX.FILEATTR.PROGCTL CLASS(FACILITY) ID(CITGR1G) ACCESS(READ) If the BPX.SERVER FACILITY class is not defined, the Gateway daemon user ID must be defined with a UID of 0 (that is, be a root user). Defining programs and libraries as program controlled USS program control allows RACF to secure USS executables as though they were MVS programs. If a nonprogram controlled program is loaded into the Gateway daemon, the address space becomes dirty, and program control is lost, which results in security failures in the Gateway daemon. Tip: The ls -E command from an OMVS screen shows the extended attributes of HFS file, including the program control attribute p. After the SMP/E installation of CICS TG, we noted that the HFS files which need to be program controlled had been marked with the extended attribute +p. Chapter 2. Installing and configuring the CICS TG 31 Tip: The ctgstart script is installed with the shared bit s set (Example 2-2) and should be left with this setting. This setting causes the _BPX_SHAREAS environment variable to be used, which allows the required sharing of the address space with CTGBATCH and non-sharing of the address space when using ctgstart from USS. Example 2-2 Results of the ls -E command showing the extended attributes of ctgstart CITGCA: /usr/lpp/cicstg/ctg610/bin>ls -E total 6200 drwxrwxr-x 2 CITGT2G CTGV61 8192 Sep 17 13:05 IBM . -rwxrwxr-x --s- 2 CITGT2G CTGV61 29217 Sep 17 13:05 ctgsetup_IBM-930 -rwxrwxr-x --s- 2 CITGT2G CTGV61 29217 Sep 17 13:05 ctgsetup_IBM-935 -rwxrwxr-x --s- 2 CITGT2G CTGV61 29217 Sep 17 13:05 ctgsetup_IBM-939 -rwxrwxr-x -ps- 2 CITGT2G CTGV61 37522 Sep 17 13:05 ctgstart -rwxrwxr-x -ps- 2 CITGT2G CTGV61 856064 Sep 15 13:13 libctgjni.so -rwxrwxr-x -ps- 2 CITGT2G CTGV61 856064 Sep 15 13:13 libctgjni_g.so -rwxrwxr-x -ps- 2 CITGT2G CTGV61 172032 Sep 17 13:05 libctgmvs.so drwxrwxr-x 12 CITGT2G CTGV61 8192 Sep 17 13:05 resource -rwxrwxr-x --s- 2 CITGT2G CTGV61 8192 Sep 17 13:05 testzos If the files are subsequently modified, they lose their program controlled status and might need to be reset, as follows: extattr +p <install_path>/bin/lib*.so extattr +ps <install_path>/bin/ctgstart The Java SDK must also be program controlled. This option is set by default when the Java SDK is installed. The Gateway daemon loads required files from MVS libraries. Of these libraries, SCTGLOAD, SDFHEXCI, SDFHLINK and SCEERUN2 must be program controlled. To give these libraries PROGRAM CONTROL status, issue the following RACF commands: RALTER RALTER RALTER RALTER PROGRAM PROGRAM PROGRAM PROGRAM * * * * ADDMEM(’CTGV61.SCTGLOAD’//NOPADCHK) ADDMEM(‘CICSTS31.CICS.SDFHEXCI’//NOPADCHK) ADDMEM(’SYS1.CICSTS31.CICS.SDFHLINK’//NOPADCHK) ADDMEM(’CEE.SCEERUN2’//NOPADCHK) Then, make the program control settings active with the following command: SETROPTS WHEN(PROGRAM) REFRESH You can find more information about the RACF commands in the RACF Systems Programmer’s Guide, SC28-1913. 32 CICS Transaction Gateway for z/OS Version 6.1 Advanced Program Function (APF) Authorization If you plan to use the new XA transaction support in CICS TG V6.1, CTGINIT (supplied in the CICS TG SCTGLINK library) must be in the LNKLST and in an APF authorized library. For more details see “AFP Authorization of CTGINIT” on page 197. 2.3 Configuring CICS TG In this section, we describe how to configure CICS TG. To configure CICS TG, you need to: 1. 2. 3. 4. 5. Define an HFS. Set permissions. Create a Gateway daemon configuration file. Create an STDENV file. Create the started task JCL. 2.3.1 Defining an HFS In this section, we show how we created and mounted HFS data sets to hold trace and configuration data. Creating a data set In our text environment, we did not want the configuration and trace files to be written to the same HFS as the CICS TG product executables, which we mounted with read only access. We needed the Gateway daemon to be able to write traces to HFS. Also, we needed to be able to alter the configuration files. So, we created two new HFS data sets: HFS OMVS.CITGTRC.HFS OMVS.CITGCFG.HFS Mount point /cicstg/ctg610/trace /cicstg/ctg610/config An HFS can be allocated using ISPF option 3.2, an IEFBR14 batch job, or through the ISHELL menus. Tip: zSeries File System (zFS) has improved caching functionality compared with HFS. This improved caching can provide significant performance benefits for log and trace files in a zFS compared with equivalent logs on an HFS. This redbook does not discuss zFS. For information about zFS, see z/OS Distributed File Service zSeries File System Implementation z/OS V1R7, SG24-6580. Chapter 2. Installing and configuring the CICS TG 33 Mounting the HFS In OMVS, change to the root directory and use the OMVS mkdir command to create the /cicstg/ctg610/trace and /cicstg/ctg610/config directories (Figure 2-3). Example 2-3 Creating a mount point directory CITGCA: CITGCA: CITGCA: CITGCA: CITGCA: CITGCA: /CITG/CA>cd / />mkdir cicstg />mkdir cicstg/ctg610 />mkdir cicstg/ctg610/trace />mkdir cicstg/ctg610/config /> Then, mount each of the HFS data sets using the ISHELL File_systems menu (see Figure 2-3). Figure 2-3 Using ISHELL to mount OMVS.CITGTRC.HFS 34 CICS Transaction Gateway for z/OS Version 6.1 In our scenario, we wanted our HFS file systems to be available when our z/OS system was idle. So, we added MOUNT statements for /cicstg/ctg610/trace and /cicstg/ctg610/config to our BPXPRMxx member in SYS1.PARMLIB (Example 2-4). Example 2-4 MOUNT statements for the CICS TG config and trace HFSs MOUNT FILESYSTEM('OMVS.CITGTRC.HFS') TYPE(HFS) MODE(RDWR) MOUNTPOINT('/cicstg/ctg610/trace') MOUNT FILESYSTEM('OMVS.CITGCFG.HFS') TYPE(HFS) MODE(RDWR) MOUNTPOINT('/cicstg/ctg610/config') 2.3.2 Setting permissions To be able to read the configuration file, the Gateway daemon user ID needs to have: Execute permission on the directories (to traverse the directories) Read permission on the configuration file. The Gateway daemon also needs write access the traces directory. In our testing, we planned to have a number of Gateway daemons, all running under their own user IDs. In this environment, each daemon reads its configuration file from the configuration directory and writes trace output to the trace directory. To enable this, we connected all of the Gateway daemon user IDs to the CTGV61 group (group id 7100), which was made the owning group of the following directories: /cicstg /cicstg/ctg610 /cicstg/ctg610/config /cicstg/ctg610/trace We use the ISHELL, specifying the relevant directory and using the Directory → Attributes → Edit → Owning Group menu selection (as shown in Figure 2-4). Chapter 2. Installing and configuring the CICS TG 35 Figure 2-4 Changing the GID with ISHELL We then changed the permissions for the /cicstg and /cicstg/ctg610 directories to 750 and gave the configuration file CITGR1G.ini permissions of 660. For the /cicstg/ctg610/config and /cicstg/ctg610/trace directories we gave permissions 770. 2.3.3 Creating a Gateway daemon configuration file CICS TG uses a Gateway daemon configuration file for its configuration parameters. The default name for this configuration file is CTG.INI. A sample configuration file is supplied in the <install_path>/bin/ctgsamp.ini file. In our testing, we decided to used the job name of our Gateway daemon for our configuration file (for example, CITGR1G.ini) so that we could differentiate between configuration files for different Gateway daemons in the same directory. To create our configuration file we: Copied the sample configuration file /usr/lpp/cicstg/ctg610/bin/ctgsamp.ini to /cicstg/ctg610/config/CITGR1G.ini. 36 CICS Transaction Gateway for z/OS Version 6.1 Edited CITGR1G.ini as shown in (Example 2-5). Example 2-5 CITGR1G.ini changes SECTION GATEWAY connectionlogging=on nonames=on [email protected][email protected]=com.ibm. ctg.server.TCPHandler [email protected]=port=13101;\ connecttimeout=2000;\ idletimeout=600000;\ pingfrequency=60000 We set the port parameter to 13101, and we also set connectionlogging to on (as an aid to debugging any connection problems that we might encounter). The connectionlogging parameter logs client connections and disconnections. For more detail about other connection related configuration file settings, see 3.3.2, “Gateway threads” on page 92. 2.3.4 Creating an STDENV file Before the Gateway daemon can be started, you need to set some environment variables in an STDENV file, which can be an MVS sequential file, a PDS member, or an HFS file. A sample STDENV is supplied as a member of SCTGSAMP. Tip: We recommend that you use a PDS or PDSE member for your STDENV. We found that an STDENV allocated as a sequential data set cannot be updated while the Gateway daemon is running. You need to set the following environment variables: CICSCLI CLASSPATH PATH DFHJVPIPE DFHJVSYSTEM STEPLIB Chapter 2. Installing and configuring the CICS TG 37 We copied the supplied STDENV member to CITG.CTG610.STDENV(CITGR1G) and then did the following: Changed the environment variables Removed the comments Ordered the environment variables alphabetically Example 2-6 shows the environment variable settings that we used. Example 2-6 CITG.CTG610.STDENV(CITGR1G) _BPX_SHAREAS=YES CICSCLI=/cicstg/ctg610/config/CITGR1G.ini CLASSPATH=/usr/lpp/cicstg/ctg610/classes/ctgsamples.jar COLUMNS=80 CTG_RRMNAME=CTG.RRM.CITGR1G.IBM.UA DFHJVPIPE=CITGR1G DFHJVSYSTEM_00=A6POTGR1-Primary server PATH=/bin:/usr/lpp/java142/J1.4/bin TMPDIR=/cicstg/ctg610/config TZ=GMT-2 Description of the environment variables The following list provides a description of the main environment variables and the values that we set: _BPX_SHAREAS The Gateway daemon can be started in its own address space or share the address space of the process that started it. If you are using CTGBATCH to start the Gateway daemon (see 2.6.1, “Starting the Gateway daemon” on page 51), ensure that _BPX_SHAREAS=YES is set in the STDENV DD statement. If starting the Gateway daemon from a USS shell prompt (such as OMVS), set _BPX_SHAREAS=NO in the ctgenvvar script to force the use of a clean address space. We specified _BPX_SHAREAS=YES because we started the Gateway daemon using CTGBATCH and used one address space. CICSCLI You can specify a different path to the configuration file by setting the location of the configuration file. This variable is important when running multiple Gateway daemon started tasks. We set CICSCLI to /cicstg/ctg610/config/CITGR1G.ini to allow each Gateway daemon started task to have its own configuration file. 38 CICS Transaction Gateway for z/OS Version 6.1 CLASSPATH Includes a concatenation of fully qualified jar files or HFS directories that contain class files. If running the Gateway daemon with the sample security exits, include <install_path>/classes/ctgsamples.jar in the CLASSPATH. The ctgstart script appends the main product jar files to the classpath. We set CLASSPATH to /usr/lpp/cicstg/ctg610/classes/ctgsamples.jar. CTG_RRMNAME This is the name with which the Gateway daemon registers with Resource Recovery Services (RRS) for transactional EXCI requests to CICS. The CTG_RRMNAME must conform to the naming rules for RRS groups. The name can consist of the following printable characters: – – – – Alphanumeric characters: A-Z and 0-9 National characters: $ (X'5B'), # (X'7B'), @ (X'7C') The period (.) The underscore (_) RRM name restrictions include: – The name cannot start with a blank or contain embedded blanks. – Lowercase characters are converted to uppercase characters. – To avoid naming conflicts, use A through C or G through I as the first character. – The length of CTG_RRMNAME must not exceed 32 characters and must end with the characters IBM.UA. Note that different rules apply if XA transactions are enabled, for more details see 5.6.3, “Configuring CICS TG” on page 175. If you do not specify a CTG_RRMNAME, a unique CTG_RRMNAME is generated by the CICS TG. We set CTG_RRMNAME to CTG.RRM.CITGR1G.IBM.UA. COLUMNS Specifies the output width of the screen for CICS TG messages. We left this at the COLUMNS=80 supplied in the sample STDENV. DFHJVPIPE The value specified here must match the NETNAME in the CICS connection definition to be used for an EXCI connection using a specific pipe. The default connection name in the IBM-supplied group DFH$EXCI is EXCS, which has a NETNAME of BATCHCLI. We created our own connection definition, and used a NETNAME of CITGR1G. Chapter 2. Installing and configuring the CICS TG 39 DFHJVSYSTEM The syntax of this variable is DFHJVSYSTEM_nn=aaaaaaaa literal, where: – nn = 0 to 99 – aaaaaaaa = the APPLID of the CICS region – literal = a description of the CICS region Note the setting of this variable is optional, and only affects the result of the ECIRequest.listSystems() method and does not affect which CICS regions you can connect to. We set DFHJVSYSTEM_00 to A6POTGR1 (our CICS APPLID). PATH The PATH statement includes the binary directory where Java is located. We set PATH to /usr/lpp/java142/J1.4/bin. STEPLIB This variable identifies the libraries containing EXCI options. It also identifies the EXCI load libraries. If a modified EXCI options table DFHXCOPT is to be included, it must appear before the EXCI load library SDFHEXCI in order to override the default DFHXCOPT table. At this stage, we did not specify a value for the STEPLIB variable, because we initially used the default DFHXCOPT options table provided in the SDFHEXCI library. See “TIMEOUT settings in DFHXCOPT” on page 98 for information about how we subsequently changed the EXCI options. TMPDIR TMPDIR is a USS variable defining a temporary directory which is used whilst executing OMVS commands. The default location for this directory is /tmp. We kept the TMPDIR within the CICS TG HFS in directory /cicstg/ctg610/traces. TZ TZ is a USS environment variable which allows the mapping of the system time (stored in GMT) to a local time zone. For full details on TZ see z/OS V1R6 UNIX System Services Command Reference, SA22-7802. We set TZ=GMT-2 so that the Gateway daemon messages were reported with the correct local time. For a full description of all the CICS TG environment variables see CICS Transaction Gateway for z/OS Administration, SC34-6672. 40 CICS Transaction Gateway for z/OS Version 6.1 Order of precedence of CICS TG environment variables Environment variables passed by CTGBATCH to the Gateway daemon can be set in one of the following ways: Using the ctgenvvar script file Within the JCL using a data set referenced by the STDENV DD card The ctgenvvar script is best used when you run the Gateway daemon from the OMVS command line using the ctgstart command (Figure 2-5). The search order for environment variables used by CTGBATCH is as follows: 1. If STDENV DD statement is defined in the startup JCL, environment variables specified in the reference file are used. 2. If the CTGENVVAR environment variable is set, the variables are taken from the referenced file and these override environment variables set by STDENV. 3. If CTGENVVAR is not set, or the ctgenvvar file which the CTGENVVAR variable specifies does not exist, the Gateway daemon looks in the <install_path>/bin directory for a ctgenvvar file. If it finds one, any environment variables which it contains override those already set by the STDENV referenced file. No STDENV specified ? Yes Use the variables specified CTGENVVAR specified and present ? No Yes No Use the variables specified, overiding any previous definition ctgenvvar in <install_path>/bin? Yes Use the variables specified, overiding any previous definition Start Gateway daemon Figure 2-5 Concatenation sequence for CICS TG variables Chapter 2. Installing and configuring the CICS TG 41 For ease of maintenance, we placed our environment variables in an STDENV member. To ensure that these variables were not overridden, we set the CTGENVVAR variable and did not have a ctgenvvar file in our <install_path>/bin directory. 2.3.5 Creating the started task JCL In 2.6.1, “Starting the Gateway daemon” on page 51, we discuss the different ways in which the Gateway daemon can be started. We recommend that you start the Gateway deamon with a started task job. A sample is supplied in member CTGPROC of SCTGSAMP that you can use as a base for the Gateway daemon started task JCL. In our scenario, we copied CTGPROC, removed the comments, and specified values for CTGUSR, CTGHLQ, REGION, and the ctgstart path as shown in Example 2-7. Example 2-7 JCL for Gateway daemon started task //CITGR1G PROC // SET CTGHLQ='CTGV61' // SET CTGUSR='CITG.CTG610.STDENV(CITGR1G)' // SET LEOPTS='/' //CTG EXEC PGM=CTGBATCH,REGION=250M, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/bin/ctgstart -noinput ' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //CTGDBG DD DUMMY //STDENV DD DSN=&CTGUSR.,DISP=SHR //* We recommend that you specify a specific REGION size and do not specify REGION=0M. While REGION=0M requests that the job receives as large a region as it requires, the actual allocation can be limited by the z/OS IEFUSI exit, resulting in a region size that is much smaller than expected. The IEFUSI exit is used in a z/OS system to limit the REGION size that is allowed to jobs and can override the allocation with a much lower system default value. This limitation can result in a Gateway daemon running with an unexpectedly small storage allocation, suffering storage and performance problems. If a specific value is set for REGION and the requested storage is not available, the job will fail, allowing early problem determination. We specified a REGION size of 250M, which we considered reasonable for our testing purposes. 42 CICS Transaction Gateway for z/OS Version 6.1 2.4 Configuring CICS In this section, we deal with the following CICS configuration tasks: 1. 2. 3. 4. 5. 6. 7. Creating a CONNECTION definition. Creating a SESSIONS definition. Installing the definitions. Configuring CICS for RRMS. Compiling and installing the sample program. Editing and assembling DFHCNV. Opening interregion communication. 2.4.1 Creating a CONNECTION definition An EXCI connection for the CICS TG must be defined in CICS. You can copy the supplied EXCS CONNECTION definition from the DFH$EXCI group and update it to reflect the required definitions. You can install either generic or specific EXCI connections in your CICS regions to allow them to communicate with the Gateway daemon. For a specific EXCI connection, the value specified to the Gateway daemon environment variable DFHJVPIPE must be the same as the NETNAME option on the connection definition. The netname itself is an arbitrary value. A generic connection is defined in much the same way, except that the NETNAME option is left blank. Note: We recommend that you use specific EXCI connections because this gives you more control over which Gateway daemons can access which CICS regions and also makes problem determination easier. We copied the EXCS connection to our own CITGR1 group as GR1G and set NETNAME to CITGR1G (Figure 2-6). Chapter 2. Installing and configuring the CICS TG 43 OBJECT CHARACTERISTICS CICS RELEASE = 0640 CEDA View CONnection( GR1G ) CONnection : GR1G Group : CITGR1 DEscription : SAMPLE EXCI SPECIFIC CONNECTION CONNECTION IDENTIFIERS Netname : CITGR1G INDsys : REMOTE ATTRIBUTES REMOTESYSTem : REMOTEName : REMOTESYSNet : CONNECTION PROPERTIES ACcessmethod : IRc Vtam | IRc | INdirect | Xm PRotocol : Exci Appc | Lu61 | Exci Conntype : Specific Generic | Specific SInglesess : No No | Yes DAtastream : User User | 3270 | SCs | STrfield | Lms + RECordformat : U U | Vb SYSID=TGR1 APPLID=A6POTGR1 PF 1 HELP 2 COM 3 END 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL Figure 2-6 GR1G connection definition 2.4.2 Creating a SESSIONS definition The EXCS SESSIONS definition can also be copied from group DFH$EXCI and the connection attribute of the session definition altered to refer to the previously defined CONNECTION. It is also possible to set the RECEIVECOUNT and RECEIVEPFX parameters. It is desirable to do this because: RECEIVECOUNT defines the maximum number of EXCI pipes to be used for the defined connection. RECEIVEPFX allows the definition of a single or double character prefix so that the usage of each session can be easily identified. If a single character RECEIVEPFX is specified the RECEIVECOUNT can be a maximum of 999. If a two character RECEIVEPFX is specified the RECEIVECOUNT can be a maximum of 99. We defined session GR1G, setting the RECEIVEPFX to 1 and the RECEIVECOUNT to 100 (Figure 2-7). See 3.5, “Configuring CICS” on page 100 for more details about the definition of EXCI CONNNECTION and SESSIONS. 44 CICS Transaction Gateway for z/OS Version 6.1 OBJECT CHARACTERISTICS CICS RELEASE = 0640 CEDA View Sessions( GR1G ) Sessions : GR1G Group : CITGR1 DEscription : Sample EXCI Specific sessions definition SESSION IDENTIFIERS Connection : GR1G SESSName : NETnameq : MOdename : SESSION PROPERTIES Protocol : Exci Appc | Lu61 | Exci MAximum : 000 , 000 0-999 RECEIVEPfx : 1 RECEIVECount : 100 1-999 SENDPfx : SENDCount : 1-999 SENDSize : 04096 1-30720 + RECEIVESize : 04096 1-30720 SYSID=TGR1 APPLID=A6POTGR1 PF 1 HELP 2 COM 3 END 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL Figure 2-7 GR1G SESSIONS definition 2.4.3 Installing the definitions We activated the CICS definitions by installing the CITGR1 group using the CEDA INSTALL GROUP(CITGR1G) command. 2.4.4 Configuring CICS for RRMS Because we intend to test CICS TG transactional support in our scenario, we configure CICS to register with Resource Recovery Management Services (RRMS). It is a requirement that RRMS support is functioning and CICS is registered with RRMS if CICS is to handle extended logical units of work, because RRMS is the resource manager for these extended transactions. In our environment, we enabled CICS registration with RRMS by setting the CICS system initialization parameter RRMS=YES. Tip: It is possible to verify that RRMS is open using the CEMT INQUIRE RRMS command. Chapter 2. Installing and configuring the CICS TG 45 2.4.5 Compiling and installing the sample programs The sample program ec01.ccp is provided with the CICS TG in the /usr/lpp/cicstg/ctg610/samples/server directory. For our testing, we copied the program to our CICS source library, compiled the program, and added the load module to our program load library. 2.4.6 Editing and assembling DFHCNV You can use the EciB1 sample application that is provided with the CICS TG to test the Gateway daemon configuration. EciB1 is a Java program that runs in the USS environment, a UNICODE environment. The Java program receives data that is produced by the EC01 COBOL program running in CICS, an EBCDIC environment. The data is passed in a COMMAREA. To ensure that this COMMAREA is converted correctly between UNICODE and EBCDIC, we created a DFHCNV table entry for program EC01 with SRVERCP=037,CLINTCP=850. For further details about using DFHCNV, refer to Appendix B, “DFHCNV and CICS data conversion” on page 379. 2.4.7 Opening interregion communication It is possible to check the status of CICS interregion communication using the CEMT INQURE IRC command (Example 2-8). If IRC is closed, you can issue the CEMT SET IRC OPEN command to open CICS IRC. Example 2-8 CEMT INQUIRE IRC command INQUIRE IRC STATUS: RESULTS - OVERTYPE TO MODIFY Irc Ope 2.5 Testing the configuration This section explains how to verify that your Gateway daemon is functioning correctly using: 46 The CTGTESTR sample JCL CICS TG STDOUT output The ping command The netstat command CICS Transaction Gateway for z/OS Version 6.1 To test our configuration we used the CTGTESTR sample job supplied in the SCTGSAMP data set. This job runs the ctgtest script from the <install_path>/samples directory. The ctgtest script is designed to be run without user input so that it can be executed from a batch job. The script uses the EciB1 sample program. We updated the CICSTESTR job (Example 2-9) with the following: URL and PORT used by our Gateway daemon PATH variable set to the HFS directory of the Java binary directory CLASSPATH variable set to the HFS directory of ctgclient.jar Example 2-9 CTGTESTR job //CITGCAT1 JOB (0),MSGCLASS=X,CLASS=A,NOTIFY=&SYSUID,REGION=250M // SET CTGHLQ='CTGV61' // SET LEOPTS='/' //TEST EXEC PGM=CTGBATCH, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/samples/ctgtest USERID // PASSWORD tx1.mop.ibm.com 13101' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //STDENV DD * PATH=/bin:/usr/lpp/java/J1.4/bin CLASSPATH=/usr/lpp/cicstg/ctg610/classes/ctgclient.jar /* // Chapter 2. Installing and configuring the CICS TG 47 EciB1 flows an ECI request to a connected CICS region through a Gateway daemon (Figure 2-8). tx1.mop.ibm.com z/OS CICS CICS TG CICS Application Gateway daemon JNI EXCI EC01 CITGR1G 13101 ECI request CTGBATCH UNIX System Services ctgstart A6POTGR1 CTGTESTR JCL JVM EciB1 ctgclient.jar ctgserver.jar Figure 2-8 CTGTESTR test scenario The output of this application is the date and time as formatted by the CICS COBOL program EC01. We verified that our configuration was set up correctly as follows: 1. We started the Gateway daemon as a started task from SDSF using the /S CITGR1G command, as shown in Example 2-10. Example 2-10 Results of /S CITGR1G written to the SDSF log S CITGR1G $HASP100 CITGR1G ON STCINRDR IEF695I START CITGR1G WITH JOBNAME CITGR1G IS ASSIGNED TO USER CITGR1G , GROUP CITG $HASP373 CITGR1G STARTED 2. When the Gateway daemon had started, we checked the STDOUT to ensure that the Gateway daemon had started correctly (Example 2-11). Message CTG6524I shows that the TCP protocol handler has started successfully so 48 CICS Transaction Gateway for z/OS Version 6.1 that the Gateway daemon can accept incoming work. Message CTG5612I shows that the Gateway daemon has successfully initialized. Example 2-11 Successful start of Gateway daemon as seen in STDOUT CTG6400I CTG8400I CTG6577I CTG6502I CTG6526I CTG6574I CTG6738I CTG6898I CTG6981I CTG6505I CTG6524I CTG6512I CTG6898I CICS Transaction Gateway is starting Using configuration file /cicstg/ctg610/config/CITGR1G.ini. Java version is 1.4.2 Initial ConnectionManagers = 1, Maximum ConnectionManagers = 100 Initial Workers = 1, Maximum Workers = 100 Connection logging has been disabled XA transaction support is not enabled 250 EXCI pipes are available for use by the CICS Transaction Gateway. Successfully initialized JNI library Successfully created initial ConnectionManager and Worker threads Successfully started handler for the tcp: protocol on port 13101 CICS Transaction Gateway initialization complete 250 EXCI pipes are available for use by the CICS Transaction Gateway. 3. We checked the state of the TCP/IP sockets on our USS TCP/IP stack using the netstat command from OMVS. Example 2-12 shows the Gateway daemon listening on port 13101. Example 2-12 OMVS netstat CITGCA: /CITG/CA>netstat MVS TCP/IP NETSTAT CS V1R6 TCPIP Name: TCPIP1 User Id Conn Local Socket Foreign Socket ------- ---------------------------CITGR1G 00003378 0.0.0.0..13101 0.0.0.0..0 16:45:22 State ----Listen 4. We checked basic IP connectivity from our workstation to z/OS using the ping command from a Windows 2000 command prompt (Example 2-13). Example 2-13 Ping to z/OS verifying hosts or DNS setup C:\>ping tx1.mop.ibm.com Pinging tx1.mop.ibm.com [9.100.193.122] with 32 bytes of data: Reply Reply Reply Reply from from from from 9.100.193.122: 9.100.193.122: 9.100.193.122: 9.100.193.122: bytes=32 bytes=32 bytes=32 bytes=32 time<10ms time<10ms time<10ms time<10ms TTL=63 TTL=63 TTL=63 TTL=63 Ping statistics for 9.100.193.122: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms Chapter 2. Installing and configuring the CICS TG 49 5. We performed our basic test by submitting the CTGTESTR job. The job finished with a completion code of 0. Checking the job output we could see the date and time returned by the CICS program EC01 (Example 2-14). Example 2-14 OMVS CTGTESTR test output ctgtest: Executing EciB1 with userid = USERID and parameters: tx1.mop.ibm.com 13101 CICS Transaction Gateway Basic ECI Sample 1 Usage: java com.ibm.ctg.samples.eci.EciB1 [Gateway URL] [Gateway Port Number] [SSL Keyring] [SSL Password] To enable client tracing, run the sample with the following Java option: -Dgateway.T.trace=on The address of the Gateway has been set to tx1.mop.ibm.com Port:13101 CICS Servers Defined: . 1. A6POTGR1 -Primary server Choose Server to connect to, or q to quit: Program EC01 returned with data:.Hex: 31392f30392f30352031363a30303a30330 .ASCII text: 19/09/05 16:00:03. Note that the completion code indicates whether the CTGTESTR job successfully completed or not, rather than the result of the ECI call itself. The completion code is still 0 even if the CICS region is not active, however, the job output shows an ECI_ERR_NO_CICS error (Example 2-15). Example 2-15 Results of CTGTEST when CICS is not available ctgtest: Executing EciB1 with userid = USERID and parameters: tx1.mop.ibm.com 13101 CICS Transaction Gateway Basic ECI Sample 1 Usage: java com.ibm.ctg.samples.eci.EciB1 [Gateway URL] [Gateway Port Number] [SSL Keyring] [SSL Password] To enable client tracing, run the sample with the following Java option: -Dgateway.T.trace=on The address of the Gateway has been set to tx1.mop.ibm.com Port:13101 CICS Servers Defined: 1. A6POTGR1 -Primary server Choose Server to connect to, or q to quit: ECI returned: ECI_ERR_NO_CICS Abend code was null 50 CICS Transaction Gateway for z/OS Version 6.1 2.6 Operating CICS TG In this section, we discuss the operation of CICS TG, including how to do the following: 1. Starting the Gateway daemon. 2. Stopping the Gateway daemon. 3. Automatic Restart Management (ARM). 2.6.1 Starting the Gateway daemon You can start the Gateway daemon either from a USS command line or by a JCL batch job. We do not discuss the use of the ctgstart script from USS because the use of a batch job has many advantages, including more flexibility in the routing of Gateway daemon messages and the option to use ARM. There are two alternatives for starting the Gateway daemon from JCL: BPXBATCH CTGBATCH BPXBATCH Prior to Version 6 of CICS TG, starting the Gateway daemon was dependent on the z/OS supplied utility BPXBATCH. Also, in previous releases, all log messages were written to the USS stdout and stderr destinations, and BPXBATCH can only map the USS stdout and stderr destinations to HFS files. Starting the Gateway daemon using BPXBATCH is still supported with CICS TG V6.1 and is described in CICS Transaction Gateway for z/OS Administration, SC34-6672. However, the recommended way to start the Gateway daemon from JCL is to use CTGBATCH. CTGBATCH CTGBATCH is a Language Environment® batch mode program which is supplied with CICS TG for z/OS V6 and later. It is used to start the Gateway daemon via JCL started task and to pass the required environment variables to the USS ctgstart script. CTGBATCH has advantages over BPXBATCH in that it can route Gateway daemon messages to the following destinations: The JES log An MVS sequential data set An HFS file Chapter 2. Installing and configuring the CICS TG 51 If you start the Gateway daemon from JCL using CTGBATCH (or BPXBATCH), it can register with the z/OS Automatic Restart Manager component (see 2.6.3, “Automatic Restart Manager” on page 53 for more information). You can start CTGBATCH from a JCL batch job or as a started task procedure. We recommend that you start the Gateway daemon via CTGBATCH from a started task procedure. We started the Gateway daemon using a procedure called CITGR1G in SYS1.PROCLIB (see Example 2-7 on page 42). 2.6.2 Stopping the Gateway daemon In CICS TG V6, you can stop the Gateway daemon in a controlled manner, rather than using the SDSF CANCEL command. Normal shutdown A normal shutdown quiesces the Gateway daemon so that all the work that is in progress completes but no new work is started. After all the work in progress has completed, the Gateway daemon then shuts down. A normal shutdown is requested with the following SDSF command: /F JOB_NAME,APPL=SHUT|SHUTDOWN Tip: The SDSF stop command /P JOB_NAME that specifies the Gateway daemon job name has exactly the same effect as /F JOB_NAME,APPL=SHUT (as shown in Example 2-16). Example 2-16 Normal shutdown using /P CITGR1G P CITGR1G BPXM023I (CITGR1G) 856 CTG8239I Response received from CICS Transaction Gateway CTG8235I Normal shutdown was requested Immediate shutdown An immediate shutdown terminates all work in progress and breaks any active connections. The Gateway daemon does not accept any new connections and shuts down immediately. An immediate shutdown is requested with the following SDSF command: /F JOB_NAME,APPL=SHUT|SHUTDOWN,IMM|IMMEDIATE 52 CICS Transaction Gateway for z/OS Version 6.1 Example 2-17 Immediate shutdown using /F CITGR1G,APPL=SHUT,IMM F CITGR1G,APPL=SHUT,IMM BPXM023I (CITGR1G) 864 CTG8239I Response received from CICS Transaction Gateway CTG8234I Immediate shutdown was requested $HASP395 CITGR1G ENDED Note: If you have requested a normal shutdown and the Gateway daemon is taking too long to shutdown, it is possible to request an immediate shutdown subsequently. For shutdown considerations when using extended logical units of work or XA transactions, see 5.8, “Operations” on page 219. 2.6.3 Automatic Restart Manager The CICS TG is enabled to use Automatic Restart Manager (ARM), which is a z/OS facility that allows the automated restart of an MVS subsystem. MVS automatic restart management is a sysplex-wide integrated automatic restart mechanism that can restart: An MVS subsystem in place if it abends (or if a monitor program notifies ARM of a stall condition). A failed MVS image. The CTGARM utility is supplied in the CTGARM member of SCTGAUTH and can be used to register and deregister the CICS TG with ARM. Tip: To use CTGARM the SCTGAUTH load library must be defined as APF-authorized. Example 2-18 shows how we removed the comments from the CTGARM-related code in the supplied CTGPROC and wrapped it around our existing JCL that we created originally in Example 2-7 on page 42. We then started the Gateway daemon with the /S CITGR1G,CTGID=CITGR1G command. We specified ARM_TYPE as SYLVL2, which is the default. Example 2-18 CICS TG started task JCL with ARM registration //CITGR1G PROC // SET CTGHLQ='CTGV61' // SET CTGUSR='CITG.CTG610.STDENV(CITGR1G)' // SET LEOPTS='/' //* Chapter 2. Installing and configuring the CICS TG 53 //* Optional step: register with ARM //* //REG EXEC PGM=CTGARM, // PARM='REGISTER CTG&CTGID. SYSLVL2' //STEPLIB DD DSN=&CTGHLQ..SCTGAUTH,DISP=SHR //SYSPRINT DD SYSOUT=* //* //*Main step: run CTG //* Only runs if successfully registered with ARM //* // IF RC <5 THEN //CTG EXEC PGM=CTGBATCH,REGION=250M, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/bin/ctgstart -noinput ' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR // DD DSN=CITG.CITGR1G.USERLOAD,DISP=SHR // DD DSN=CICSTS31.CICS.SDFHEXCI,DISP=SHR //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //CTGDBG DD DUMMY //STDENV DD DSN=&CTGUSR.,DISP=SHR //* //* Optional step: deregister with ARM //* Only runs if CTG terminated cleanly //* //DEREG EXEC PGM=CTGARM, // PARM='DEREGISTER' //STEPLIB DD DSN=&CTGHLQ..SCTGAUTH,DISP=SHR //SYSPRINT DD SYSOUT=* // ENDIF // //PEND //* The syntax of the CTGARM registration is ’R[egister] ARM_ID [ARM_TYPE]’. The restart order is governed by the ARM_TYPE where a job registered with SYSLVL0 will be restarted before one registered with SYSLVL1, and so on. For more details on Automatic Restart Management, see z/OSV1R2.0 MVS Setting up a Sysplex, SA22–7625. The startup job expects the CTGID to be passed on the SDSF start command. If the CTGID is not passed on the start command the ARM registration fails and a the following message is written to the SYSPRINT DD destination: CTG6022E Registration failed - ARM_ID is invalid Return code 8 is returned to the CTGBATCH job, and the Gateway daemon does not start (there is a conditional statement requiring a return code less than five). 54 CICS Transaction Gateway for z/OS Version 6.1 The ARM registration results in a number of ARM related CTG messages which are written to the SYSPRINT location (Figure 2-19). Similar messages are also written when the Gateway daemon is shut down. Example 2-19 ARM registration messages sent to SYSPRINT CTG6000I CTGARM - CTG Automatic Restart Manager Batch Utility CTG6002I CTG6003I CTG6004I CTG6005I Parameters processed: Command: Register ARM_ID: CTGCITGR1G ARM_TYPE: SYSLVL2 CTG6034I Registering with ARM CTG6006I Results are: CTG6007I Return code: 00000000 CTG6008I Reason code: 00000000 CTG6031I This is first time register with ARM CTG6035I Telling ARM we are ready CTG6006I Results are: CTG6007I Return code: 00000000 CTG6008I Reason code: 00000000 CTG6037I Successfully registered with ARM CTG6001I End of CTGARM - CTG Automatic Restart Manager Batch Utility Note: The USS ctgarm utility that is supplied in <install_path>/bin is provided for migration purposes only. 2.7 Configuring for multiple Gateway daemons After we had tested that our single Gateway daemon could communicate with a CICS TS region successfully, we decided to set up a second Gateway daemon (see Table 2-2). You might run multiple Gateway daemons if: You need test and production Gateway daemon regions. You are upgrading to a new version. You require a second Gateway Daemon for a different e-business application. Chapter 2. Installing and configuring the CICS TG 55 In 3.7.2, “Cloned Gateway daemon connectivity tests” on page 114, we demonstrate a similar scenario in which we clone Gateway daemons. You would use a cloned configuration if: You need more than 250 simultaneous pipes to CICS regions. You need to workload manage across multiple cloned Gateway daemons. Table 2-2 Definitions checklist with second Gateway daemon and CICS regions Value Gateway daemon Second Gateway daemon CICS TS Region 1 CICS TS Region 2 hostname tx1.mop.ibm.com tx1.mop.ibm.com - - IP address 9.100.193.122 9.100.193.122 - - TCP/IP port 13101 11101 - - Job name CITGR1G CITGS1G CITGR1CI CITGS1CI APPLID - - A6POTGR1 A6POTGS1 NETNAME CITGR1G CITGS1G - - CONNECTION - - GR1G GS1R Consider the following areas when setting up a second Gateway daemon: Set RACF permissions on the Gateway daemon started task user ID. Create started task JCL. Create new directories in USS. Create a new configuration file. – Change the Gateway daemon TCP/IP port Create a new STDENV member. – – – – – Set the name and location of the configuration file Set DFHJVPIPE name Set the DFHJVSYSTEM_00 name Set the CLASSPATH Set the RRMNAME Create CICS definitions. – Connection – Sessions 56 CICS Transaction Gateway for z/OS Version 6.1 We list each step in this process. However, because we assume that you have performed these tasks previously to set up the first Gateway daemon, we do not go into each step in detail. To set up a second Gateway daemon: 1. Set RACF permissions on the Gateway daemon started task user ID. We set up a new started task user ID, CITGS1G and set the required permissions. 2. Create started task JCL in the Proclib. We copied our CITGR1G started task job as CITGS1G and changed it to reference a new STDENV member CITG.CTG610.STDENV(CITGS1G). 3. Create new directories in USS. We used the same cicstg/ctg610/config directory for the Gateway configuration file and /cicstg/ctg610/trace directory for trace files for both Gateway daemons. You might want to use a separate HFS for the configuration files and traces of each Gateway daemon, in which case you would need to create the directories and set the required permissions (as shown in 2.3.2, “Setting permissions” on page 35). 4. Create a new configuration file. We copied /cicstg/ctg610/config/CITGR1G.ini to /cicstg/ctg610/config/CITGS1G.ini and changed the port from 13101 to 11101. 5. Create a new STDENV member. We copied CITG.CTG610.STDENV(CITGR1G) to CITG.CTG610.STDENV(CITGS1G) and made the following changes: a. We set CICSCLI to /cicstg/ctg610/config/CITGS1G.ini. b. We changed DFHJVPIPE to CITGS1G to match the NETNAME which we specified on our CONNECTION definition in CICS region CITGS1CI. c. We changed CTG_RRMNAME to CTG.RRM.CITGS1G.IBM.UA. 6. Create CICS definitions. In our CITGS1CI CICS region, we defined the following resources: d. CONNECTION We created a GS1G CONNECTION definition and set NETNAME to CITGS1G. e. SESSIONS We created a GS1G SESSION definition and set it the CONNECTION it referenced to GS1G and the RECEIVEPFX to 2. Chapter 2. Installing and configuring the CICS TG 57 2.8 Migrating to CICS TG V6.1 Direct migration to CICS TG V6.1 from earlier versions of CICS TG for z/OS is not supported. It is necessary to install CICS TG for z/OS V6.1 and then migrate your settings to the new installation. In this section, we discuss: 1. Migrating from ctgenvvar. 2. Other migration and configuration considerations. 2.8.1 Migrating from ctgenvvar The sample ctgconvenv script is supplied in the <install_path>/bin directory. You can use this script to migrate ctgenvvar settings from a pre-CICS TG V6 system. The script converts a ctgenvvar script into the STDENV style settings which can be used by CTGBATCH. You can invoke the ctgconvenv script from: A USS command line with the following command: ctgconenv [wrap width] [source file name] A batch job. A sample job to invoke ctgconvenv is supplied in SCTGSAMP(CTGCONV). Note that ctgconvenv is not tolerant of environment variables that it does not recognize. If it finds a variable that it cannot identify, it stops parsing the ctgenvvar. Tip: If the ctgenvvar script that is specified in the ctgconvenv command does not have the execute permissions set for the user ID running the ctgconvenv, you will receive the following message: CTG6124W Source file filename not found or not executable. 2.8.2 Other migration and configuration considerations In this section, we list other migration and configuration considerations. Default Configuration file name The default configuration file name has changed from CTG.INI in CICS TG V5 to ctg.ini. If both exist in the <install_path>/bin directory the ctg.ini file will be used. You should specify the configuration file in the CICSCLI environment variable to ensure that the Gateway daemon starts with the required settings. 58 CICS Transaction Gateway for z/OS Version 6.1 Configuration tool There are two variations of the CICS TG configuration tool that is shipped with CICS TG V6.1: The ctgcfg tool is supplied in the <install_path>/bin directory and runs on z/OS and is accessed via X-windows. The ctgfzoscfg tool is supplied in the <install_path>/ctgzoscfg directory and can be transferred to a distributed platform using FTP. Both tools generate a configuration file and a ctgenvvar script. The advantage of using these tools is that they perform validation checking on the values specified. We did not use a configuration tool. Features removed In CICS TG V6 the following features are removed: Support for HTTP and HTTPS protocols. Disabling the validation of units of work. Allowing Java Client applications to obtain generic replies. The Handler wakeup timeout. The resource adapter specific for WebSphere Application Server for z/OS (cicsecirrs.rar) is no longer supplied because the CICS ECI resource adapters used on all platforms are now identical. SystemSSL SystemSSL is not supported with CICS TG V6. For details on migration from SystemSSL to JSSE (Java Secure Sockets Extension), see 9.5.1, “SystemSSL to JSSE migration” on page 335. WebSphere Application Server The CICS ECI resource adapters that are supplied with the CICS TG V6.1 for z/OS are supported with the following J2EE application servers: WebSphere Application Server V6.0.1 for z/OS WebSphere Application Server V6.0. Only the 32-bit application servers on the following platforms are supported: – – – – – – – AIX Windows Linux on Intel Linux on POWER Linux on zSeries HP-UX PA-RISC Solaris SPARC Chapter 2. Installing and configuring the CICS TG 59 In remote mode, earlier versions of the CICS ECI resource adapters can be used to connect to a CICS TG V6.1 daemon. To do this you must use a version of the resource adapter that is supported with the specific WebSphere Application Server in question. When installing the CICS TG V6.1 resource adapters, it is not possible to connect to Gateway daemons of a lower version or release level. If it is necessary to interoperate with Gateway daemons of a lower version or release, then you must maintain separate application servers, each with the appropriate level of resource adapter. In local mode, the resource adapters and CICS TG must be at the same level. For detailed information about connecting from WebSphere Application Server, see Part 2, “Connection Management” on page 75. TCP/IP workload management If you are currently using a TCP/IP workload management capability, such as Sysplex Distributor, to workload manage TCP/IP requests across a set of Gateway daemons, there are additional migration considerations if you intend to use the new XA transaction support available with CICS TG V6.1. For more information, see Chapter 6, “Gateway daemon transactions with TCP/IP load balancing” on page 225. 2.9 Problem determination In this section, we present information that we learned while installing the CICS TG and general information about problem determination and tracing. 2.9.1 Common problems In this section, we list some common problems that you might experience when installing and testing the CICS TG. ECI_ERR_NO_CICS This error is caused by a failure in an attempt to connect to a CICS region. Check STDOUT for the following message: CCL6822E Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = xx, subreason field-2 = 0, Cics_Rc = -3. 60 CICS Transaction Gateway for z/OS Version 6.1 Subreason field-1 identifies the cause of the problem: Subreason field-1 = 92 = X’005C’ means IRERRNLG “System not logged on” IRC is not open. Check using the CICS command CEMT INQ IRC. Check that the CICS region user ID has READ access to the RACF FACILITY class profile DFHAPPL.<CICS_applid>. Subreason field-1 = 104 = X’0068’ means IRERRNSS “Secondary system not in primary LCB” This error occurs because the value specified for the CICS TG DFHJVPIPE environment variable does not correspond with the NETNAME value of any installed CICS CONNECTION definition. Check that DFHJVPIPE has been set correctly and that a CONNECTION definition with a NETNAME value the same as DFHJVPIPE has been installed on the CICS region. Abend S806 The abend S806 indicates that there has been a failure to load a module. The Gateway daemon abends with abend code S806 because it is unable to find a module which it requires. You should: Check the syslog for message: CSV003I REQUESTED MODULE DFHXCPRX NOT FOUND Ensure that the STEPLIB is set correctly in the STDENV file, using a statement such as: EXCI_LOADLIB="CICSTS31.CICS.SDFHEXCI" Check that the profile of the user ID which runs the Gateway daemon does not set the STEPLIB to point to a different library that can contain conflicting modules. Note: In CICS TG V6, JNI code is loaded during initialization. Thus, this error occurs at Gateway daemon startup, rather than on the first ECI call (as was the case with CICS TG V5). Chapter 2. Installing and configuring the CICS TG 61 CTG6114E The CICS TG STDERR message shown in Example 2-20 indicates that the CICS TG has not been correctly installed. Example 2-20 Starting Gateway daemon when ctginstall has not been run CTG6114E CICS Transaction Gateway has not been fully installed. Run the command '/usr/lpp/cicstg/ctg610/bin/ctgsetup' to complete the installation. 09/17/05 11:09:35:231 [0] CTG0827W CTGBATCH Child completed with a non-zero return code 256 You should run the ctginstall script that is supplied in /usr/lpp/cicstg/ctg610/ (which executes the ctgsetup script in the <install_path>/bin directory) to complete installation of the CICS TG. 2.9.2 Tips and utilities This section includes useful commands and utilities for debugging problems with your configuration. z/OS TCP/IP commands We found the following TCP/IP commands useful when analyzing network problems with our CICS TG for z/OS: HOMETEST Issue this as a TSO command. If there is a valid TCP/IP address or host name, this command tells you what it is, along with other configuration information. NETSTAT Issue this command to obtain information about the TCP/IP sockets that are in use (TSO and other platforms). The USS version of this command can be invoked using the onetstat command. NSLOOKUP <IP address> Issue this as a TSO command. A valid TCP/IP address tells you the host name and vice versa. PING Use this to determine if an address is active and to resolve host names to IP addresses (Example 2-21). 62 CICS Transaction Gateway for z/OS Version 6.1 Example 2-21 Pinging the z/OS host tso ping wtsc66oe.itso.ibm.com CS V1R6: Pinging host TX1.MOP.IBM.COM (9.100.193.122) Ping #1 response took 0.000 seconds. Java version To display the level of the Java Development Kit (JDK™) installed on your system, use the java -fullversion command as follows: >java -fullversion >java full version "J2RE 1.4.2 IBM z/OS Persistent Reusable VM build cm142-20050623" To determine where the JDK is installed on your system, issue the following command to search for the location of the Java executable: >type java >java is /usr/lpp/java142/J1.4/bin/java If this reports that java is not found, then you need to add the JDK directory to your PATH environment variable. This is best added in the USS /etc/profile so that all users can use Java. BPXPRM parameters The SYS1.PARMCLIB member BPXPRMxx contains parameters which are used when USS is initialized. The parameters that are currently in effect can be displayed by the command /D OMVS,OPTIONS (Example 2-22). Example 2-22 SYS1.PARMLIB(BPXPRMxx) displayed by /D OMVS,OPTIONS D OMVS,OPTIONS BPXO043I 14.25.36 DISPLAY OMVS 343 OMVS 000D ACTIVE OMVS=(00,V8) CURRENT UNIX CONFIGURATION SETTINGS: MAXPROCSYS = 900 MAXPROCUSER MAXFILEPROC = 10000 MAXFILESIZE MAXCPUTIME = 2147483647 MAXUIDS MAXPTYS = 800 MAXMMAPAREA = 40960 MAXASSIZE MAXTHREADS = 15000 MAXTHREADTASKS MAXCORESIZE = 2147483647 MAXSHAREPAGES IPCMSGQBYTES = 2147483647 IPCMSGQMNUM IPCMSGNIDS = 500 IPCSEMNIDS IPCSEMNOPS = 25 IPCSEMNSEMS IPCSHMMPAGES = 524287 IPCSHMNIDS IPCSHMNSEGS = 500 IPCSHMSPAGES SUPERUSER = BPXROOT FORKCOPY = 200 = NOLIMIT = 200 = 2147483647 = 5000 = 131072 = 10000 = 500 = 1000 = 500 = 262144 = COW Chapter 2. Installing and configuring the CICS TG 63 STEPLIBLIST = USERIDALIASTABLE= /etc/ualiastable PRIORITYPG VALUES: NONE PRIORITYGOAL VALUES: NONE MAXQUEUEDSIGS = 1000 SHRLIBRGNSIZE SHRLIBMAXPAGES = 4096 VERSION SYSCALL COUNTS = NO TTYGROUP SYSPLEX = NO BRLM SERVER LIMMSG = NONE AUTOCVT RESOLVER PROC = RESOLV1 AUTHPGMLIST = NONE SWA = BELOW = = = = = 67108864 / TTY N/A OFF You can also make dynamic changes by using the SETOMVS command as follows: SETOMVS MAXPROCUSER=5000 Note: To make these parameter changes permanent, you must edit the BPXPRMxx member of your SYS1.PARMLIB library. The changes take effect after the next IPL. BPXPRMxx also contains the mount commands for your HFS structure. To display which HFS directories are mounted (available), you can use the /D OMVS, F command as shown in Example 2-23. Example 2-23 Partial listing of HFSs mounted to USS D OMVS,F BPXO045I 14.31.40 DISPLAY OMVS 353 OMVS 000D ACTIVE OMVS=(00,V8) TYPENAME DEVICE ----------STATUS----------AUTOMNT 23 ACTIVE NAME=*AMD/u PATH=/u HFS 124 ACTIVE NAME=OMVS.CITGCFG.HFS PATH=/cicstg/ctg610/config HFS 122 ACTIVE NAME=OMVS.CITGTRC.HFS PATH=/cicstg/ctg610/trace HFS 110 ACTIVE NAME=OMVS.CTG6127.HFS PATH=/usr/lpp/cicstg/ctg610 MODE RDWR RDWR RDWR RDWR For more information about these settings, refer to z/OS V1R6 UNIX System Services Planning, GA22-7800. 64 CICS Transaction Gateway for z/OS Version 6.1 CTGDBG Including a dummy DD statement in the Gateway daemon started task JCL results in extra messages being sent to STDOUT and STDERR (Example 2-24). Example 2-24 Started task JCL specifying //CTGDBG DD DUMMY //CITGR1G PROC // SET CTGHLQ='CTGV61' // SET CTGUSR='CITG.CTG610.STDENV(CITGR1G)' // SET LEOPTS='/' //CTG EXEC PGM=CTGBATCH,REGION=250M, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/bin/ctgstart -noinput ' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR // DD DSN=CITG.CITGR1G.USERLOAD,DISP=SHR //* DD DSN=CICSTS31.CICS.SDFHEXCI,DISP=SHR //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //CTGDBG DD DUMMY //STDENV DD DSN=&CTGUSR.,DISP=SHR //* We found it useful to have the details of the environment, especially the environment variables, which are routed to STDOUT at startup (Example 2-25) Example 2-25 STDOUT with //CTGDBG DD DUMMY specified CTG0826I CTGBATCH Parsed STDENV entry _BPX_SHAREAS=YES CTG0826I CTGBATCH Parsed STDENV entry CICSCLI=/cicstg/ctg610/config/CITGR1G.ini CTG0826I CTGBATCH Parsed STDENV entry CLASSPATH=/usr/lpp/cicstg/ctg610/classes/ctgsamples.jar CTG0826I CTGBATCH Parsed STDENV entry COLUMNS=80 CTG0826I CTGBATCH Parsed STDENV entry CTG_RRMNAME=CTG.RRM.CITGR1G.IBM.UA CTG0826I CTGBATCH Parsed STDENV entry DFHJVPIPE=CITGR1G CTG0826I CTGBATCH Parsed STDENV entry DFHJVSYSTEM_00=A6POTGR1-Primary server CTG0826I CTGBATCH Parsed STDENV entry PATH=/bin:/usr/lpp/java142/J1.4/bin CTG0826I CTGBATCH Parsed STDENV entry TMPDIR=/cicstg/ctg610/config CTG0826I CTGBATCH Parsed STDENV entry TZ=GMT-2 CTG0813I CTGBATCH RLIMIT_AS reports current=258M, system max=2047M CTG0815I CTGBATCH CWD=/CITG/R1 CTG0817I CTGBATCH PID=83886125 CTG0818I CTGBATCH LOCALE=C CTG0819I CTGBATCH POSIX=1 USS Version=5 CTG0811I CTGBATCH Runtime env PATH=/bin:/usr/lpp/java142/J1.4/bin CTG0811I CTGBATCH Runtime env CICSCLI=/cicstg/ctg610/config/CITGR1G.ini CTG0811I CTGBATCH Runtime env TMPDIR=/cicstg/ctg610/config CTG0811I CTGBATCH Runtime env TZ=GMT-2 CTG0811I CTGBATCH Runtime env _BPX_SHAREAS=YES CTG0820I CTGBATCH Userid=CITGR1G UID=13 GID=7100. CTG0821I CTGBATCH Initial dir=/CITG/R1 Initial program=/bin/sh. Chapter 2. Installing and configuring the CICS TG 65 CTG0825I CTGBATCH Spawned child PID=67109054 CTG6400I CICS Transaction Gateway is starting CTG8400I Using configuration file /cicstg/ctg610/config/CITGR1G.ini. CTG6577I Java version is 1.4.2 CTG6502I Initial ConnectionManagers = 1, Maximum ConnectionManagers = 100 CTG6526I Initial Workers = 1, Maximum Workers = 100 CTG6738I XA transaction support is not enabled CTG6898I 250 EXCI pipes are available for use by the CICS Transaction Gateway. CTG6981I Successfully initialized JNI library CTG6505I Successfully created initial ConnectionManager and Worker threads CTG6524I Successfully started handler for the tcp: protocol on port 13101 CTG6512I CICS Transaction Gateway initialization complete LEOPTS The LEOPTS parameter on CTGBATCH allows Language Environment override options to be passed to LE. If // SET LEOPTS='/' in Example 2-18 on page 53 is replaced with the following, then the LE runtime options and storage report is routed to STDERR: // SET LEOPTS='RPTOPTS(ON),RPTSTG(ON)/' Tip: The PARM string is limited to 100 bytes. If you find that you need to code very long property strings, you can use the CTGSTART_OPTS environment variable in STDENV. The df . command Issuing the command df . on a USS command line gives information about the mount table entry for the current HFS. Example 2-26 Use of the df . command to show filesystem mount information CITGCA: /cicstg/ctg610/config>df . Mounted on Filesystem Avail/Total Files /cicstg/ctg610/config (OMVS.CITGCFG.HFS) 1208/1440 Available CITGCA: /cicstg/ctg610/config> Status 4294967289 Changing the code page for CICS TG messages We installed the CICS TG using the IBM-1047 code page. It is possible to convert the installation to another code page by running the following command: <install_path>/bin/ctgmsgs <language code> <code set> 66 CICS Transaction Gateway for z/OS Version 6.1 The user ID which runs this command needs write permission to the <install_path>/bin directory. Converting the installation to another code page will not affect messages used by the CTGBATCH program. To display the supported language and code set combinations, issue the command (Example 2-27): <install_path>/bin/ctgmsgs -? Example 2-27 Output of ctgmsgs -? This utility is used to change the language of user messages. ------------------------------------------------------------Language --------------------en US English ja Japanese zh Simplified Chinese Locale -------------En_US Ja_JP Ja_JP.IBM-930 Zh_CN Code Set ---------IBM-1047 IBM-939 IBM-930 IBM-935 The syntax of this command is: ctgmsgs <Language> <Code Set> Error with bad permissions on the directory We tried to start the Gateway daemon when we had not set the correct permissions to allow it to traverse the /cicstg/ctg610/config directory. The result was that the Gateway daemon failed to start and messages shown in Example 2-28 were written to the Gateway daemon job log. Example 2-28 Starting Gateway daemon with incorrect permissions IEF695I START CITGR1G WITH JOBNAME CITGR1G IS ASSIGNED TO USER CITGR1G , GROUP CITG $HASP373 CITGR1G STARTED $HASP100 BPXAS ON STCINRDR $HASP373 BPXAS STARTED ICH408I USER(CITGR1G ) GROUP(CITG ) NAME(CTG GATEWAY DAEMON U) 324 /cicstg/ctg610/config/CITGR1G.ini CL(DIRSRCH ) FID(01D6C5F0F0F0F100010400002DFB0000) INSUFFICIENT AUTHORITY TO OPEN ACCESS INTENT(--X) ACCESS ALLOWED(OTHER ---) EFFECTIVE UID(0000000013) EFFECTIVE GID(0000007100) $HASP395 CITGR1G ENDED We needed to set execute permission on the directories for our group using the ISHELL command, specifying the relevant directory, and using menu selection Directory → Attributes → Edit → Owning Group as shown in Figure 2-4 on page 36. Chapter 2. Installing and configuring the CICS TG 67 Listing libraries in the LINKLIST We found it helpful to check the libraries in the LINKLIST using the MVS command /D PROG,LNKLST, which gave the results in the log as shown in Example 2-29. Example 2-29 Displaying libraries in the LINKLIST D PROG,LNKLST CSV470I 16.36.01 LNKLST DISPLAY 141 LNKLST SET LNKLSTLA LNKAUTH=LNKLST ENTRY APF VOLUME DSNAME 56 A CICS31 SYS1.CICSTS31.CICS.SDFHLINK 57 A CICS31 CICSTS31.CICS.SDFHEXCI 2.9.3 Tracing CICS TG on z/OS provides three types of trace: Gateway trace JNI trace EXCI trace The Gateway daemon and JNI traces can be started statically (where the trace parameters are specified at startup) or dynamically (where the trace parameters are specified during the normal operation of the Gateway daemon). The EXCI trace can be formatted from a dump of the Gateway daemon address space, or routed to the MVS Generalized Trace Facility (GTF). Gateway trace The Gateway trace captures Gateway daemon requests and responses. Static trace (Gateway daemon) You can specify the following parameters on the ctgstart command when starting the Gateway daemon: trace=on (enables trace) tfile=<pathname> (specifies the destination of the Gateway trace) The Gateway daemon must be stopped and restarted in order to change these settings. Setting the trace options statically is useful on occasions when you need to trace Gateway daemon initialization. 68 CICS Transaction Gateway for z/OS Version 6.1 Dynamic trace (Gateway daemon) With CICS TG V6, it is also possible to administer Gateway tracing dynamically from the SDSF console using the following commands: /F <job_name>,APPL=TRACE,TLEVEL=0|1|2|3|4 0 No trace information is output. 1 Exception tracing. Only exceptions are traced. 2 Trace exceptions, and entry and exit of methods. 3 Trace exceptions, some internals, and entry and exit of methods. 4 Full debug tracing. /F <job_name>,APPL=TRACE,TFILESIZE= filesize Specifies the maximum size, in kilobytes, of the Gateway trace output file, for example 50000. /F <job_name>,APPL=TRACE,DUMPOFSET Specifies the offset from which displays of any data blocks start, for example 512. If the offset is greater than the total length of data to be displayed, an offset of 0 is used. You cannot use this together with the fulldatadump option. /F <job_name>,APPL=TRACE,TRUNCATIONSIZE= Specifies the byte at which to stop the hex dump, for example 2000. It defines the endpoint, not the number of bytes to display. So if on a dump of size 40 you set the dumpoffset to 11, and the truncationsize to 25, you will see 15 bytes (from 11 to 25). You cannot use this together with the fulldatadump option. /F <job_name>,APPL=TRACE,FULLDATADUMP Sets the dumpoffset to 0 and ignores any value specified in truncationsize. The current trace settings can be displayed by issuing the following command (Example 2-30). /D <job_name>,APPL=TRACE Chapter 2. Installing and configuring the CICS TG 69 Example 2-30 Results of the command /D CITGR1G,APPL=TRACE BPXM023I (CITGR1G) CTG8239I Response received from CICS Transaction Gateway Gateway Daemon trace settings: tlevel=0 truncationsize=80 dumpoffset=0 tfile=/cicstg/ctg610/trace/CITGR1G.gwytrc tfilesize=0 JNI trace settings: jnilevel=0 jnifile=/cicstg/ctg610/trace/CITGR1G.jnitrc To demonstrate the Gateway trace we issued the command /F CITGR1G,APPL=TRACE,TLEVEL=2 (Example 2-31) and used the supplied sample EciB1 to send a request to the gateway daemon. Example 2-31 Activating Gateway trace with /D CITGR1G,APPL=TRACE, TLEVEL=2 BPXM023I (CITGR1G) CTG8239I Response received from CICS Transaction Gateway Gateway Daemon trace settings successfully changed: tlevel=2 The Gateway trace is written to the destination specified in the tfile configuration parameter, in our case /cicstg/ctg610/trace/CITGR1G.gwytrc. Example 2-32 shows sample Gateway trace output of an ECI request for CICS APPLID A6POTGR1 and an error response Rc=-3 (ECI_ERR_NO_CICS). Example 2-32 Edited version of our Gateway trace 12:38:33:038 Thread-2: .MBeanServerImpl:<- [setAttributes] = [tracelevel=2] 12:38:36:335 ConnectionManager-9: .GatewayRequest:-> [readPaddedString] (java.io.DataInputStream@48feeec0, 3) 12:38:36:335 ConnectionManager-9: .GatewayRequest:<- [readPaddedString] = ECI 12:38:36:335 ConnectionManager-9: S-C: -> [ClientMessages: getMessage()] (2, null, ECI, 500010, 1, 0, 96) 12:38:36:335 ConnectionManager-9: S-C: -> [ClientMessages: getMsg()] (null, 2, 0) 12:38:36:335 ConnectionManager-9: S-C: <- [ClientMessages: validateResourceBundle()] = com.ibm.ctg.client.ResourceWrapper@6966eec1 12:38:36:336 ConnectionManager-9: S-C: <- [ClientMessages: getMsg()] = CTG6602I GatewayRequest type = ECI, flow version = 500010, flow type = 1, Gateway return code = 0, length of data following the header = 96 12:38:36:336 ConnectionManager-9: S-C: CTG6602I GatewayRequest type = ECI, flow version = 500010, flow type = 1, Gateway return code = 0, length of data following the header = 96 . 70 CICS Transaction Gateway for z/OS Version 6.1 12:38:36:393 Worker-9: S-C: CTG6695I Before JNI CcicsECI(1) : Server=A6POTGR1, Program=EC01, Commarea.Length=18, ExtendMode=0, LUW=0. 12:38:36:451 Worker-9: S-C: CTG6698I After JNI CcicsECI(1) : Rc=-3, LUW=0, Null stripped commmarea len=0 . 12:38:36:452 Worker-9: .ServerECIRequest:<- [getCicsRcString()] = ECI_ERR_NO_CICS 12:38:36:453 Worker-9: S-C: CTG6721I ECIRequest: Call_Type = ECI_SYNC, Cics_Rc = ECI_ERR_NO_CICS, Abend_Code = , Luw_Token = 0, Message_Qualifier = 0. . 12:38:36:455 Worker-9: S-C: CTG6515I Finished work request. [Worker-9] JNI trace The JNI trace captures information about ECI requests and the EXCI flows which the Gateway daemon initiates. Static trace (JNI) The following environment variables can be specified in STDENV to enable JNI tracing at startup. This is useful if you need to investigate JNI problems which occur during Gateway daemon startup. CTG_JNI_TRACE=<pathname> CTG_JNI_TRACE_ON=YES Dynamic trace (JNI) The JNI trace can also be started and stopped dynamically using the SDSF MODIFY command: /F <job_name>,APPL=TRACE,JNILEVEL=0|1 0 No trace information is output. 1 On. Tips: The Gateway and JNI trace destinations cannot be set dynamically. If you do not define specific destinations for the Gateway and JNI traces the output is written to STDERR. If both traces are written to STDERR, trace entries are interleaved making it easier to diagnose problems. More than one modify command can be issued at one time, for example, the command /F CITGR1G,APPL=TRACE,JNILEVEL=1,TLEVEL=2 sets both JNI and Gateway tracing on. Chapter 2. Installing and configuring the CICS TG 71 Example 2-33 shows sample JNI trace output of an ECI request for CICS APPLID A6POTGR1 and an EXCI response 8, EXCI reason 203 (NO_CICS). Example 2-33 Edited version of our JNI trace CTG6865I ECI Server List Entry Parameters. Number of Systems = 20. CTG6866I ECI Server List Exit Parameters. Cics_RC = 0. CTG6800I ECI parameters on entry. Call_Type = 1, Extend_Mode = 0, Luw_Token = 0, Commarea_Length = 18, Cics_Rc = 0, Flags = 0, outbound Commarea length = 18. CTG6801I Performed parameter conversion. parameter name = jstrServer, parameter value = A6POTGR1. CTG6801I Performed parameter conversion. parameter name = jstrProgram, parameter value = EC01 . CTG6802I Input commarea information. commarea length = 18, commarea address = fac5588. CTG8602I JNI Method zos_SetContext entry CTG8604I JNI Method zos_SetContext exit (rc = 0). CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6870I EXCI Open pipe gave a Retryable Response. Allocate, open will be retried a further 5 times. CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6822E EXCI function error. Function Call = 3, Response = 8, Reason = 203, Subreason field-1 = 0x5c, subreason field-2 = 0x00, Cics_Rc = -3. CTG6805I ECI parameters on exit. Call_Type = 1, Extend_Mode = 0, Luw_Token = 0, Commarea_Length = 18, Cics_Rc = -3, AV = 0. CTG8602I JNI Method zos_ResetContext entry CTG8632I No XID present on ECI Request. CTG8604I JNI Method zos_ResetContext exit (rc = 0). We see that the Gateway daemon attempted to call program EC01 but received an error. Because this is a retryable error (Response code = 8) the request is retried a further five times before the response Rc=-3 is passed back to the application. Note: Details of the EXCI response and reason codes can be found in the CICS External Interfaces Guide. Subreason codes are also documented in hex in the DFHIRSDS member of CICSTS31.CICS.SDFHMAC. 72 CICS Transaction Gateway for z/OS Version 6.1 EXCI trace The CICS TG writes trace entries to the EXCI trace. The trace entries are in the CICS trace EXCI format, so the trace entries in a dump can be printed using standard z/OS utilities. For detailed information about using EXCI trace, see Appendix C, “EXCI Trace” on page 385. Chapter 2. Installing and configuring the CICS TG 73 74 CICS Transaction Gateway for z/OS Version 6.1 Part 2 Part 2 Connection Management In this part, we provide detailed information about how we connected WebSphere Application Server to CICS using the CICS TG. We cover two different topologies: Running WebSphere Application Server on Windows Running WebSphere Application Server on z/OS We discuss the configuration of connections within WebSphere Application Server, the Gateway daemon, and CICS. © Copyright IBM Corp. 2006. All rights reserved. 75 76 CICS Transaction Gateway for z/OS Version 6.1 3 Chapter 3. Gateway daemon connections This chapter provides details on how we configured our test environment to allow our J2EE application running on WebSphere Application Server on Windows to connect through the CICS Transaction Gateway on z/OS to an application running in CICS Transaction Server on z/OS. First, we document how we prepared the system and the settings that we used. Then, we provide details on how we set up and modified the connections between each of the following tiers in our test environment: Configuring WebSphere Application Server Configuring the Gateway daemon Configuring EXCI Configuring CICS Finally, in 3.7, “Testing the configuration” on page 106, we explain how we tested the environment. © Copyright IBM Corp. 2006. All rights reserved. 77 3.1 Preparation In our testing, we deployed a sample J2EE application on WebSphere Application Server for Windows and connected to CICS using the CICS ECI resource adapter provided by CICS TG for z/OS. This chapter concentrates on how to configure the connections between each tier (Figure 3-1). It does not provide details on how to install all of the software components, and it assumes a working knowledge of CICS or WebSphere Application Server where appropriate. Windows z/OS WebSphere Application Server V6 CICS TG V6.1 HTML JSP Gateway Daemon Servlet CICS TG V6.1 EJB CCI CICS ECI resource adapter TCP JNI EXCI COBOL Application CICS TS V3.1 Figure 3-1 Software components: Connection management scenario Full details about how to install the CICS TG for z/OS and run the IVP are provided in Chapter 2, “Installing and configuring the CICS TG” on page 25. 78 CICS Transaction Gateway for z/OS Version 6.1 3.1.1 Software checklist Table 3-1 shows the levels of software that we used for this configuration. Table 3-1 Software checklist Windows z/OS Internet Explorer V6.0 z/OS V1R6 Windows 2000 SP4 CICS Transaction Gateway for z/OS V6.1 IBM WebSphere Application Server Network Deployment V6.0.2 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. CICS Transaction Server for z/OS V3.1 The CICS ECI resource adapters that are supplied with the CICS TG V6.1 for z/OS are supported with WebSphere Application Server V6.0 (32-bit) or later versions. You can use earlier versions of the CICS ECI resource adapter to connect to a CICS TG V6.1 daemon. You must use a version of the resource adapter that is supported with the specific version of WebSphere Application Server that you are using. Note: Although we installed WebSphere Application Server Network Deployment, we did not use any of the Network Deployment clustering functions and only configured a base application server environment. 3.1.2 Definitions checklist Table 3-2 lists the definitions that we used to configure the scenario. These values are the same that we used in Chapter 2, “Installing and configuring the CICS TG” on page 25. Table 3-2 Definitions checklist Value WebSphere Gateway daemon CICS TS hostname cam21-pc11 tx1.mop.ibm.com - IP address (dhcp) 9.100.193.122 - TCP/IP port - 13101 - Job name - CITGR1G CITGR1CI APPLID - - A6POTGR1 NETNAME - CITGR1G - CONNECTION - - GR1G Chapter 3. Gateway daemon connections 79 3.2 Configuring WebSphere Application Server In this section, we discuss the configuration of our WebSphere Application Server on Windows 2000. We detail the following steps: Installing the resource adapter Creating connection factories Deploying the sample J2EE application 3.2.1 Installing the resource adapter The CICS TG V6.1 now provides two JCA resource adapters, cicseci.rar and cicseciXA.rar. In our scenario, we used the cicseci.rar, which provides the same level of local transaction support as provided previously in CICS TG V6.0 and earlier versions. The cicseciXA.rar provides additional XA transactional support and is covered in Chapter 5, “Gateway daemon transactions” on page 157. Note: Because both resource adapters provide the same CCI application interface and connection management capability, our sample J2EE application CTGTesterCCI can be used with either resource adapter. Transferring cicseci.rar to the workstation In order to install the resource adapter into WebSphere, we first needed to transfer the cicseci.rar file from z/OS to our Windows workstation. We used the Windows ftp command line utility to connect to the z/OS FTP daemon and download the file from the HFS directory /usr/lpp/ctg/ctg610/deployable to our local c:\temp directory (Example 3-1). Example 3-1 FTP of cicseci.rar to workstation C:\temp>ftp tx1.mop.ibm.com Connected to tx1.mop.ibm.com. 220-FTPD11 IBM FTP CS V1R6 at TX1.MOP.IBM.COM, 16:27:14 on 2005-09-16. 220 Connection will close if idle for more than 5 minutes. User (tx1.mop.ibm.com:(none)): citgt2g 331 Send password please. Password: 230 CITGT2G is logged on. Working directory is "CITGT2G.". ftp> cd /usr/lpp/cicstg/ctg610/deployable 250 HFS directory /usr/lpp/cicstg/ctg610/deployable is the current working directory. ftp> bin 200 Representation type is Image ftp> get cicseci.rar 200 Port request OK. 125 Sending data set /usr/lpp/cicstg/ctg610/deployable/cicseci.rar 250 Transfer completed successfully. ftp: 750023 bytes received in 0.16Seconds 4777.22Kbytes/sec. ftp> 80 CICS Transaction Gateway for z/OS Version 6.1 To install the resource adapter into WebSphere, do the following: 1. Start the Application Server and log into the Administrative Console using the following URL: http://cam21-pc11:9060/ibm/console/ 2. In the Administrative Console Welcome panel, click Resources and then Resource adapters. The panel in Figure 3-2 displays. Figure 3-2 Administrative Console - installing RAR 1 3. Click Install RAR. In the Install RAR file panel, click Local Path and then Browse to locate the resource adapter cicseci.rar on the Windows workstation (Figure 3-3). Figure 3-3 Administrative Console - installing RAR 2 Chapter 3. Gateway daemon connections 81 4. Click Next and then OK when the Resource Adapters panel displays. The ECIResourceAdapter is now added to the list of installed adapters (Figure 3-4). Figure 3-4 Administrative Console - installing RAR 3 5. Add the changes to the master repository by clicking Save and then clicking Save again. Note: When installing the CICS TG V6.1 resource adapters, it is possible to connect to Gateway daemons of a lower version or release level. If it is necessary to interoperate with Gateway daemons of a lower version or release, then you must maintain separate application servers, each with the appropriate level of resource adapter. Conversely, application servers with resource adapters from a lower CICS TG version or release can connect to Gateway daemons of a higher version or release but will not be able to use all of the features available at the version or release. 3.2.2 Creating connection factories Each installed resource adapter should have one or more connection factories associated with it. The connection factory is used to define the connection to the target CICS system. As such, it must contain the following information: The CICS region that will be the target of requests via this connection factory. The connection details of the Gateway daemon that will be used. In addition, the connection factory can contain other optional information, such as: The security credentials that will be used for communication. The mirror transaction name that should be used. 82 CICS Transaction Gateway for z/OS Version 6.1 Tip: At least one connection factory is required to use a resource adapter. However, it is possible to create multiple connection factories each with different options (such as differing CICS region names) to allow different applications to use connections with different properties or to different CICS regions. Creating a connection factory This section explains how to create and configure a connection factory for the CICS ECI resource adapter. This connection factory is used in the next sections to test the connectivity to the CICS region. To create a connection factory: 1. From the Resource Adapters panel (Figure 3-4 on page 82), click ECIResourceAdapter and the Configuration panel (Figure 3-5) displays. This panel lists the General Properties and Additional Properties of the resource adapter. There is usually no need to change the general properties of the resource adapter. Figure 3-5 Administrative Console - Creating a connection factory 1 Chapter 3. Gateway daemon connections 83 2. Click J2C connection factories and then click New, which displays the General Properties panel of the connection factory to be created (Figure 3-6). Figure 3-6 Administrative Console - Creating a connection factory 2 There are many properties displayed in this window. However, the only important property at this stage is the Name field. We entered the value CITGR1CI-tx1-13101, which is a concatenation of the name of the target CICS region, the TCP/IP host name and the port the Gateway daemon is using. 3. Enter a description of this connection factory and click Apply. 84 CICS Transaction Gateway for z/OS Version 6.1 4. After applying these settings, the Additional Properties links are enabled. Click Custom Properties, which takes you to the CICS ECI resource adapter custom properties panel (Figure 3-7). Figure 3-7 Administrative Console - Creating a connection factory 2 5. The custom properties are the parameters that are used by the CICS ECI resource adapter to communicate with a Gateway daemon and a target CICS region. Set the values as shown in Figure 3-7 and then save the configuration. In our scenario, we entered the following settings: – ConnectionURL This is the URL that defines the TCP/IP host name and protocol that are used to communicate with the Gateway daemon. We used the value tcp://tx1.mop.ibm.com. – PortNumber This is the TCP/IP port number on which the Gateway is listening. It defaults to 2006. We used the value 13101. – ServerName This is the APPLID of the CICS region to which requests are sent. We used the value A6POTGR1. If it is not set, then the value from the DFHJVSYSTEM_00 variable is used by the Gateway daemon. Chapter 3. Gateway daemon connections 85 – TPNName This is the CICS TPN name and is used as the name of the attached mirror transaction that requests run under in CICS. It overrides TranName and defaults to CSMI when using the CICS TG on z/OS. We used the value ECIT, which we then had to define as a private mirror transaction in our target CICS region (see 3.5.2, “Adding a private mirror” on page 101). Tip: In some cases, for example if you plan to use a large number of different CICS private mirror names, it might be necessary for the application itself to set the TPNName or TranName dynamically. This can be done using the ECIInteractionSpec methods setTPNName() or setTranName(). – SocketConnectTimeout This is the timeout in milliseconds that a socket connection request will wait for when an attempt is made to connect to a remote Gateway daemon. We used the value 10. We accepted the defaults for the following parameters: UserName This is the user ID to be flowed to the Gateway and onto CICS. Password This is the password to be flowed to the Gateway. TranName This is the name CICS uses for the EIBTRNID of the mirror transaction. It does not affect the name of the attached mirror transaction and is overridden by TPNName. ClientSecurity This is the name of the CICS TG client security exit. If specified the exit must be added to the system classpath. ServerSecurity This is the name of the CICS TG server security exit. If specified the exit must be added to the system classpath. KeyRingClass This is the SSL keyring name for the JSSE keystore. KeyRingPassword This is the password used to access the SSL keyring. 86 CICS Transaction Gateway for z/OS Version 6.1 TraceLevel This is the JCA trace level, and can take one of the following values: 0 1 2 3 Disable all tracing Output exception trace stacks Output method entry and exit stack traces Output debug trace entries Connection pool settings After you have defined the connection factory, click Connection pool properties on the resource adapter connection factory panel and enter values to tune the connection pool (Figure 3-6). Figure 3-8 Administrative Console - connection pool settings Chapter 3. Gateway daemon connections 87 In our scenario, we entered the following settings: Connection timeout Specifies how long, in seconds, a getconnection() request by the J2EE application can wait if there are no connections available in the free pool and no new connections can be created. If a timeout occurs, a ConnectionWaitTimeoutException is thrown. We set this to 10 seconds in order to queue requests but to timeout requests early if no connection becomes available. By default, the request waits for 180 seconds. Maximum connections This is the maximum number of connections that can be created within the connection pool. We set this to 55 in order to demonstrate how to limit the number of simultaneous connections between WebSphere and a Gateway daemon. We also set the maximum number of Gateway connection manager threads to 55 (see 3.3.2, “Gateway threads” on page 92). Minimum connections This is the minimum number of connections that will be maintained in the connection pool after the pool manager maintenance thread has run. We set this to 10, the same as the initial number of Gateway connection manager threads. Reap time This is the interval at which the pool manager maintenance thread will run. We let this default to three minutes. Unused timeout This is the amount of time that connections will be allowed to remain idle before being eligible for closure by the pool manager maintenance thread. We set this to 600 so that connections would be closed down after 10 minutes of inactivity. Aged timeout This is the amount of time that connections will be allowed to age before being eligible for closure by the pool manager maintenance thread. We set this to zero (0) because we did not want old connections to be closed. PurgePolicy This specifies how to purge connections when a fatal connection error is detected by the pool manager. We set this to Failing Connection Only, which means a connection is purged only if it is used and found to be bad. The alternative is EntirePool, and in this case the connection pool is flushed if a fatal connection error occurs. 88 CICS Transaction Gateway for z/OS Version 6.1 Tip: We noted that either terminating the Gateway or the CICS region would lead to a fatal connection error. After you set these values, click OK and then save the settings to the master repository. 3.2.3 Deploying the sample J2EE application In this section, we describe how to deploy the test application CTGTesterCCI. For further details about our sample, refer to Appendix A, “Sample applications” on page 361. To deploy the sample application, do the following: 1. In the WebSphere Application Server Administrative Console Welcome panel, click Applications → Install New Application. The panel in Figure 3-9 displays. Figure 3-9 Administrative Console - installing the EAR 2. Enter the location of the EAR file (CTGTesterCCI.ear) and click Next twice and then click Continue when prompted with the Application Security Warnings page. You are then prompted with the details of the eight step process for installing new applications. Chapter 3. Gateway daemon connections 89 3. Click Next through each step until you reach Step 5 Map Resource References to Resources, where you perform the following actions: a. Select the JNDI name eis/CITGR1CI-tx1-13101 for the existing Connection Factory that you just created. b. Scroll down the page and select the EJB module CTGTesterCCIEJB in the J2EE application. c. Scroll back to the top of the page and click Apply. Figure 3-10 shows the final result. In the bottom panel, the resource reference ECI in the EJB CTGTesterEJBCCI is now mapped to the JNDI name eis/CITGR1CI-tx1-13101, which represents our connection factory. The resource reference ECI in the J2EE application now uses the connection factory CITGR1CI-tx1-13101 and the properties that are contained within it. Figure 3-10 Administrative Console - mapping resource references 90 CICS Transaction Gateway for z/OS Version 6.1 4. Click Next three times and then click Finish to exit the process. Confirm that the deployment was successful and save your changes to the WebSphere repository. 3.3 Configuring the Gateway daemon In order to configure the CICS TG environment, you might need to make changes to both the Gateway daemon settings and the TCP/IP environment. 3.3.1 TCP/IP connections When configuring the Gateway daemon, consider the following items regarding the usage of TCP/IP. Choosing a port The Gateway daemon requires use of a TCP/IP port. This defaults to 2006, but we used 13101 for our CITGR1G Gateway. The Gateway fails to start if the protocol handler cannot bind to the port at startup. So, it is important that you coordinate which jobs can access which ports. You can achieve this coordination through the use of the TCP/IP PORT reservation statement, which limits the applications that can use a specific port. We added the following value to our TCP/IP profile data set OMVS1.TCPIP.TCPPARMS(PROFILE) to ensure that only the job named CITGR1G could bind to port 13101: 13101 TCP CITGR1G ; Production Gateway daemon We then varied our changes to TCP/IP online using the MVS TCP/IP command: /V TCPIP,,O,OMVS1.TCPIP.TCPPARMS(PROFILE) Tip: You can use the USS netstat -o command to display the port reservation list for restricted ports. IP address binding By default the CICS TG protocol handlers bind to all available IP addresses (this is referred to as INADDR_ANY) and to all available TCP/IP stacks. To allow you to control access to your z/OS system, it is likely you will want to control this behavior if you are using multiple stacks or IP addresses (including VIPA addresses). Chapter 3. Gateway daemon connections 91 On our system, we had two IP addresses (9.100.193.121 and 9.100.193.122) configured. We decided to restrict our Gateway daemon to using only the 9.100.193.122 address. So, we added a bind parameter to our Gateway configuration file CITGR1G.ini (Example 3-2). Tip: You can use the USS netstat -d command to list the configured IP adapters on you system. Example 3-2 IP address binding in Gateway configuration file [email protected]=com.ibm.ctg.server.TCPHandler [email protected]=port=13101;\ connecttimeout=2000;\ idletimeout=600000;\ pingfrequency=60000;\ bind=9.100.193.122; Note: The IP address binding function that is provided by the bind parameter in the Gateway configuration file is new function in CICS TG V6.1. However, if you are running a previous version of the Gateway, you can achieve similar functionality through the use of the BIND control for INADDR_ANY function in TCP/IP. Multiple IP stacks We were only using a single TCP/IP stack. So, we did not need to make any modification to the TCP/IP bind behavior. However, if required, you can set the environment variable _BPXK_SETIBMOPT_TRANSPORT in STDENV to specify the name of an individual TCP/IP stack to which you want the Gateway daemon to bind. For more details on TCP/IP subsystem management refer to Communications Server for z/OS V1R2 TCP/IP Implementation Guide Volume 1: Base and TN3270 Configuration, SG24-5227. 3.3.2 Gateway threads The Gateway daemon receives network requests from remote Java clients, and is itself a sophisticated multi-threaded Java application that runs in the UNIX System Services environment. It can handle multiple requests simultaneously and has a set of properties configured in the Gateway configuration file. Two pools of threads can be configured: the connection manager threads and the worker threads. 92 CICS Transaction Gateway for z/OS Version 6.1 For each connected client (such as a connection in the WebSphere connection pool), one connection manager thread is used in the Gateway daemon, and is held until the client closes the connection or the Gateway times out the connection. In order for an ECI call to be performed via an allocated connection manager thread, a thread must be allocated from the worker thread pool for the duration of the ECI request. This relationship is summarized in Figure 3-11. Java client Gateway daemon Connection Managers JavaGatew ay.open() JavaGateway.flow() Workers Connect ECI ECI request Wait ECI reply JavaGateway.close() Disconnect Figure 3-11 CICS TG threading model The connection manager threads limit the maximum number of connected Java clients, and the worker threads limit the number of concurrent ECI calls that can be issued by these attached clients. The initial and maximum numbers of these connection manager and worker threads are set in the Gateway configuration file using the maxconnect and maxworker parameters. Requests from connection manager threads for worker threads can be queued internally, but will be timed out according to the workertimeout parameter. Chapter 3. Gateway daemon connections 93 Gateway configuration parameters To improve the operational characteristics of our Gateway daemon, we made changes to our Gateway configuration file as shown in Example 3-3. Example 3-3 Modified Gateway configuration file settings closetimeout=10000 initconnect=10 initworker=10 maxconnect=55 maxworker=55 nonames=on workertimeout=0 [email protected]=com.ibm.ctg.server.TCPHandler [email protected]=port=13101;\ connecttimeout=0;\ idletimeout=600000;\ pingfrequency=30000;\ bind=9.100.193.122; Here is a description of the parameters that we changed: closetimeout=10000 This parameter controls the amount of time a connection manager waits for in progress requests to complete if a Java client disconnects before a worker thread has completed its work. We set this to 10,000 ms (10 seconds) to limit any delays in cleanup processing. initconnect=10 This is the initial number of connection manager threads that are allocated when the Gateway daemon starts. Having pre-allocated connection manager threads can improve the response time of the initial ECI requests. We set it to the same value as the minimum number of connections in the connection factory (see 3.2.2, “Creating connection factories” on page 82). initworker=10 This is the initial number of worker threads that are allocated when the Gateway daemon starts. Having pre-allocated worker threads can improve the response time of the initial ECI requests. We set it to the same value as initconnect. maxconnect=55 This is the maximum number of connection manager threads that can be used and corresponds to the maximum number of socket connections that the Gateway daemon accepts. We set this to 55 which is the same as the value that we specified for the maximum number of connections in the connection factory (see 3.2.2, “Creating connection factories” on page 82). 94 CICS Transaction Gateway for z/OS Version 6.1 maxworker=55 This is the maximum number of worker threads that can be used. We set this to 55 which is the same as the upper limit of transactions that can be active and queued in our CICS TRANCLASS (see Figure 3-13 on page 102). nonames=on This parameter means that TCP/IP host names in messages will not be resolved to IP addresses. We set nonames=on because we did not want the Gateway daemon to use Domain Name Services (DNS) to convert IP names to addresses because this can cause a significant performance overhead. workertimeout=0 This parameter sets the amount of time that a connection manager thread can be suspended waiting for a free worker thread. We set this value to 0 as we did not wish to queue requests internally in our Gateway daemon. connecttimeout=0 This parameter sets the amount of time the TCP/IP protocol handler waits for a connection manager thread to become available after a connection has been established. We set this to zero (0) because we did not want to queue request internally in our Gateway daemon. idletimeout=600000 This parameter sets the amount of time the Gateway daemon keeps an unused connection alive. We set this to 60,000 ms (10 minutes), which is the same as the unused timeout setting in the connection factory. pingfrequency=30000 This parameter sets the amount of time between ping requests that are sent by the Gateway daemon to the resource adapter. We set this to 30,000 ms (30 seconds) because we wanted to decrease the amount of time that it took for the Gateway to clean-up after a network failure. Chapter 3. Gateway daemon connections 95 3.4 Configuring EXCI The EXCI is used by the CICS TG for z/OS to send requests to the CICS region specified in the ECI request. To make an ECI call, the CICS TG for z/OS uses the EXCI CALL interface. The EXCI CALL interface provides a low-level API for calling a CICS program using a COMMAREA, and consists of the following six commands which control the usage of the CICS sessions (which equate to EXCI pipes): 1. 2. 3. 4. 5. 6. Initialize_User Allocate_Pipe Open_Pipe DPL_Request Close_Pipe Deallocate_Pipe The Gateway daemon by default does not perform a Deallocate_Pipe after it flows the initial ECI request from any given worker thread (Figure 3-12). Gateway Daemon Worker threads EXCI 1st request • Allocate • Open • DPL request • Close Next request • Open • DPL request • Close Gateway Termination CICS • Deallocate Figure 3-12 Default Gateway daemon EXCI pipe usage An EXCI pipe stays allocated for the lifetime of the thread and is only deallocated if one of the following events occurs: An error is received by the Gateway worker thread on an Open_Pipe call (such as when a CICS connection has been closed). The Gateway daemon terminates. The worker thread allocates another pipe to a different CICS applid, and the CTG_PIPE_REUSE environment variable is set to ONE (see “CTG_PIPE_REUSE value” on page 98 for a description of this variable). 96 CICS Transaction Gateway for z/OS Version 6.1 For all connected Gateway daemons on z/OS the CICS region requires both a CONNECTION and SESSIONS definition. The CONNECTION definition identifies the remote system, and the SESSIONS definition associated with this CONNECTION determines the properties of the sessions. In the following sections, we provide further information about the system limits that apply to EXCI pipes and the values that we used in our configuration. You can find further information about using EXCI pipe in the CICS External Interfaces Guide, SC33-1944. 3.4.1 EXCI pipe limits Every address space which uses EXCI pipes (such as a Gateway daemon) is limited to a maximum number of pipes that it can allocate to all attached CICS regions. The EXCI pipe limit rules are as follows: In CICS TS V1.3, the limit is 100 EXCI pipes in total per calling address space. In CICS TS V2.2 or higher the PTF for APAR PQ92943 allows a MVS system wide LOGONLIM to be set, which controls the pipe limit. The upper limit is 250 and the default is 100. The maximum pipe usage in a CICS region is limited by the RECEIVECOUNT parameter defined in the CICS MRO SESSIONS definition, this can be up to 999 (depending on the number of characters in the RECEIVEPREFIX). Tip: In CICS TG V6 additional diagnostic messages are written to STDERR by the CICS TG if pipe usage reaches 90% or 100% of the EXCI LOGONLIM. 3.4.2 Our EXCI settings In our environment we made changes to the following systems setting to provide for increased throughput and more predictable response times when using the EXCI: EXCI LOGON limit CTG_PIPE_REUSE value TIMEOUT settings in DFHXCOPT EXCI LOGON limit In order to exploit an increased worker thread count in any of our Gateway daemons, we increased the EXCI pipe limit on our z/OS LPAR from the default of 100 to the upper limit of 250. We added the following line to our SYS1.PARMLIB1(DFHSSI00) member. LOGONLIM=250 Chapter 3. Gateway daemon connections 97 We then had to IPL our LPAR in order to activate this change. For more detail about how to configure the LOGONLIM, refer to the CICS Transaction Server for z/OS Installation Guide V3.1, GC34-6326. CTG_PIPE_REUSE value By default if a Gateway is used to connect to multiple CICS regions, then worker threads can allocate EXCI pipes potentially to multiple regions. If this occurs, it can cause unpredictable pipe shortage issues. In CICS TG V6.0, a new environment variable called CTG_PIPE_REUSE was introduced. If this environment variable is set to ONE, then a worker thread only allocates one pipe and deallocates this pipe if it needs to reallocate the pipe to a different CICS region. This provides a more predictable usage model but with a small cost in performance. The exact performance cost depends on the frequency of pipe deallocations. The default setting for CTG_PIPE_REUSE is ALL, which gives the same pipe allocation behavior as in CICS TG V5 (that is, worker threads can allocate multiple pipes to multiple CICS regions). We set the value CTG_PIPE_REUSE=ONE in the STDENV for our Gateway CITGR1G. Because we only had one CICS region in our setup, this setting had no immediate effect. However, setting it is good practice if you anticipate that the Gateway daemon will connect to multiple CICS regions at some time in the future, and your aim is to ensure a predictable operating environment. Attention: Because the EXCI call Deallocate_Pipe has a cost associated to it, setting CTG_PIPE_REUSE=ONE can have a detrimental affect on performance if your Gateway daemon is servicing EXCI requests for many different CICS regions. This cost has been mitigated to some extent in CICS TS V2.3, so we recommend that is the minimum level if using this function. Note, however, that the benefit of its use is that it gives a predictable usage model that allows you to predict accurately absolute pipe usage, irrespective of what connections are made. TIMEOUT settings in DFHXCOPT In our testing, we decided that we required an EXCI timeout setting to ensure that our Gateway would not wait indefinitely for transactions that might have stalled in CICS. This setting can be achieved by setting a TIMEOUT in the EXCI options table DFHXCOPT. This setting is then used by the EXCI to terminate any requests that have been waiting too long for a DPL_Request command to complete. 98 CICS Transaction Gateway for z/OS Version 6.1 We created a DFHXCOPT table (Example 3-4) with a timeout value of 90 seconds (which is equivalent to 9000 hundreths). Example 3-4 EXCI options table, DFHXCOPT *********************************************************************** DFHXCO TYPE=CSECT, * TIMEOUT=9000, 90 seconds * TRACE=OFF, Only Exception trace entries * TRACESZE=16, 16K trace table * DURETRY=30, Retry SDUMPS for 30 seconds * TRAP=OFF, DFHXCTRA - OFF * GTF=OFF, GTF - OFF * MSGCASE=MIXED, Mixed case messages * CICSSVC=0, EXCI will obtain CICS SVC number * CONFDATA=SHOW, Show user commarea data in trace * SURROGCHK=YES Perform surrogate-user check END DFHXCOPT **************************** Bottom of Data **************************** Note: Although the EXCI terminates the TCB running the request (and thus the Gateway worker thread), the task in CICS will not abend until it tries to return data back to the Gateway. Thus, in a CICS stall condition, it is quite possible for the CICS application to abend after the Gateway worker thread has returned the error back to the application. In our testing, we assembled the table and link-edited this into our CICS TG LOADLIB CITG.CITGR1G.USERLOAD, which we added to our STEPLIB DD statement ahead of the CICS SDFHEXCI library, as shown in Example 3-5. We also needed to define the CITG.CITGR1G.USERLOAD library as program controlled. Example 3-5 Updated STEPLIB //CITGR1G PROC // SET CTGHLQ='CTGV61' // SET CTGUSR='CITG.CTG610.STDENV(CITGR1G)' //CTG EXEC PGM=CTGBATCH,REGION=0M, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/bin/ctgstart -noinput ' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR // DD DSN=CITG.CITGR1G.USERLOAD,DISP=SHR Chapter 3. Gateway daemon connections 99 Note: We allowed the value of CICSSVC in the DFHXCOPT table to default to zero (0), which means that EXCI gets the CICS SVC number from z/OS. Specify 0 only when you are sure that at least one CICS region has logged on to DFHIRP during the life of the z/OS IPL. If you use the default and the EXCI requests the SVC from z/OS, the request fails if no CICS region has logged on to DFHIRP. Alternatively, you can specify the same SVC number that is used by the CICS MRO regions that reside in the z/OS image where the Gateway daemon is running. 3.4.3 EXCI user-replaceable module: DFHXCURM The user-replaceable module DFHXCURM is invoked on every EXCI Allocate_Pipe request and also after detection of any EXCI retryable errors. This occurs under one of three circumstances: The target CICS region is not available. There are no pipes available on the target CICS region. IRC has not been active since the last IPL. DFHXCURM can be used to remove the affinity of an EXCI request to a given CICS region by dynamically modifying the APPLID in the EXCI request. It can also be used to provide limited workload balancing of EXCI requests across multiple CICS regions from the CICS TG for z/OS. 3.4.4 Transactional EXCI Transactional EXCI works together with Recovery Resource Services (RRS), a component of z/OS, in allowing multiple EXCI calls to become part of one logical unit of work. This means that the CICS TG for z/OS can be used by other transactional systems (such as WebSphere Application Server) to make transaction requests to CICS, which can be coordinated using a two-phase commit mechanism between the two systems. For full details on enabling transactions with the CICS TG see Part 3, “Transaction Management” on page 155. 3.5 Configuring CICS In this section, we provide details on the modifications that we made to our CICS region to provide a more production-like environment. We cover the following settings. CICS CONNECTION and SESSION values Adding a private mirror Transaction timeouts 100 CICS Transaction Gateway for z/OS Version 6.1 For further details about the basic CICS configuration changes that we made to enable communication from our CICS TG, refer to Chapter 2, “Installing and configuring the CICS TG” on page 25. 3.5.1 CICS CONNECTION and SESSION values The process of creating CICS CONNECTION and SESSIONS definitions is described in Figure 2-6 on page 44 and Figure 2-7 on page 45. However, for the purpose of exploiting our increased pipe limit, we updated the EXCI session count in our SESSIONS definition by setting the value RECEIVECOUNT=999. We then activated this change by closing IRC, discarding the existing connection, installing the new definitions, and re-opening IRC. Note: Under very fast moving workloads, it is possible that closed EXCI pipes are not freed up quickly enough to be re-used by incoming open requests. So, if you set the RECEIVECOUNT to something less than the maximum 999, you should make sure that you set it slightly greater than the maximum potential number of EXCI pipes that will be allocated to your CICS region. 3.5.2 Adding a private mirror Rather than running our ECI request under the default mirror transaction (CSMI) in our testing environment, we decided to define our own private mirror. This decision has the following advantages: It allows individual CICS monitoring and statistics information to be recorded for specific applications. It allows specific security attributes to be set for the transaction. It allows a specific TRANCLASS to be specified, which can control the queuing and scheduling priorities of the CICS application relative to the other transactions executing in the CICS region. To define our private mirror, we performed the following steps: 1. We defined a mirror transaction ECIT, by copying the default mirror transaction definition CSMI to a new RDO group using the following command: CEDA COPY GR(DFHISC) TRAN(CSMI) AS(ECIT) GROUP(ECIPROG) 2. We updated this definition to specify a TRANCLASS of TCLASS1 using the following command: CEDA ALTER TRANS(ECIT) GROUP(ECIPROG) TRANCLASS(TCLASS1). Chapter 3. Gateway daemon connections 101 3. We defined a new TRANCLASS resource definition to control the attributes of the ECIT transaction with a MAXACTIVE value of 50 and a PURGETHRESHOLD of 6 (Figure 3-13). This TRANCLASS limits our CICS region to running 50 simultaneous ECIT transactions and also queues a further five requests before it refuses to queue further transactions. This is significantly lower than the CICS region MAXTASKS value of 100, thus ensuring that the CICS region is not paralyzed with requests for the ECIT transaction. OVERTYPE TO MODIFY CEDA ALter TRANClass( TCLASS1 TRANClass : TCLASS1 Group : ECIPROG Description ==> CLASS LIMITS Maxactive ==> 050 Purgethresh ==> 0000006 CICS RELEASE = 0640 ) 0-999 No | 1-1000000 SYSID=TGR1 APPLID=A6POTGR1 PF 1 HELP 2 COM 3 END 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL Figure 3-13 CICS TRANCLASS definition 4. We then installed the ECIT transaction and the TRANCLASS using the following command and added the ECIPROG group to our CICS startup list TGR1: CEDA INSTALL GROUP(ECIPROG) 3.5.3 Transaction timeouts We did not set any timeout parameters in our CICS environment (such as RTIMOUT or DTIMOUT), because neither of these were applicable to our scenario. However, we recommend that you do consider setting a timeout on the file manager (such as DB2), because it is generally perceived as best practice to time out suspended tasks as close as possible to the file operation likely to cause delays. For further details about the end-to-end timeout settings that we used in our testing, refer to “End-to-end settings” on page 107. 102 CICS Transaction Gateway for z/OS Version 6.1 3.6 Configuring for high scalability After you have successfully configured and tested your CICS TG configuration, you should consider how you can clone the Gateway daemon in order to improve scalability and availability. The two principal areas for consideration are: How to load balance TCP/IP requests across multiple Gateway daemons How to load balance CICS program link requests dynamically across multiple CICS AORs 3.6.1 TCP/IP load balancing The CICS TG on z/OS is designed for use with the z/OS TCP/IP load balancing topologies, due to its usage of multiple socket connections. You can use the following z/OS TCP/IP load balancing technologies: TCP/IP port sharing Sysplex Distributor TCP/IP Port Sharing TCP/IP port sharing is a feature provided by z/OS Communications Server which allows multiple address spaces (such as Gateway daemons) to listen for TCP/IP requests on the same port. It functions by allowing incoming socket open requests to be distributed among the listening address spaces according to the socket backlog and current usage. It is a very simple feature to implement, and provides effective load balancing across multiple Gateway daemons listening on the same TCP/IP stack in the same LPAR. For details about how we enabled TCP/IP port sharing, see 3.7.2, “Cloned Gateway daemon connectivity tests” on page 114. Sysplex Distributor Sysplex Distributor, an integral part of z/OS Communications Manager, offers the ability to load balance incoming socket open requests across different address spaces running on different TCP/IP stacks (usually on different LPARs). The routing decision is based on real-time socket status and MVS Quality of Service (QoS) criteria. This provides the benefit of balancing work across different MVS images, providing enhanced scalability and failover in a z/OS Parallel Sysplex. Chapter 3. Gateway daemon connections 103 The beauty of Sysplex distributor is that the client view of the parallel sysplex is of a single machine; all the complexity of where and how the work is done, is hidden from the client, but with all the benefits that a parallel sysplex environment brings to TCP/IP networking, including: Virtual IP Addressing (VIPA) Virtual IP addressing (VIPA) allows TCP/IP on z/OS to create IP addresses which do not correspond to a physical connection to the network, but to which requests can be routed, over all the physical connections to the network known to that z/OS. This gives the big benefit that if client requests use a Virtual IP address when connecting to that z/OS machine, and one or more of the physical connections to the network is down, the request can be routed over one of the other physical connections. VIPA also allows z/OS applications to be associated with specific IP addresses. Dynamic VIPA Dynamic VIPA allows the members of a parallel sysplex to share a single VIPA address. One member of the parallel sysplex is configured as the owner of the VIPA and another member is defined as the backup. If the member that owns the dynamic VIPA fails, the parallel sysplex automatically starts routing requests to the backup member. z/OS Workload Manager z/OS Workload Manager and TCP/IP Quality of Service policies (specified using Policy Agent) are used to make intelligent decisions about which is the best instance of the Gateway daemon in the parallel sysplex to process a new request. Note: One important aspect of Sysplex distributor is that the instances of the Gateway daemon must be identical clones, and must be able to handle any affinities which can arise during execution of a client request. 104 CICS Transaction Gateway for z/OS Version 6.1 Figure 3-14 shows the use of both TCP/IP port sharing and Sysplex Distributor. Sysplex Distributor TCP/IP Port Sharing TCP/IP z/OS sysplex LPAR -1 Gateway daemon EXCI CICS router region 1 LPAR - 3 DPL TCP/IP Port Sharing COBOL application LPAR -2 Gateway daemon EXCI CICS router region 2 DPL CICS AORs Figure 3-14 High scalability and availability configuration with the Gateway daemon on z/OS 3.6.2 Dynamic program link Figure 3-14 shows the recommended high availability configuration with CICS TG on z/OS, which is to have a given Gateway daemon connecting to a CICS region on the same LPAR. Such a CICS region is referred to as a router region. This region does not run the CICS application itself but instead routes the program link request to a CICS AOR. CICSPLex SM provides a dynamic routing program which supports the dynamic routing of LINK commands. This provides the ability for applications invoked by ECI requests to be routed dynamically across a CICSplex. Tip: We recommend that a Gateway daemon connects to CICS regions on the same LPAR. The performance and transactional characteristics of an XCF EXCI connection are different from an XM EXCI connection: Certain limits apply to XCF group membership and these limits can be exceeded if many XCF EXCI sessions are allocated. If an ECI request is part of an extended logical unit of work or an XA transaction, the EXCI connection from the Gateway daemon must be an XM EXCI connection. Chapter 3. Gateway daemon connections 105 Restriction: If an ECI request is part of an extended logical unit of work or an XA transaction, the EXCI connection from the Gateway daemon must be to a CICS region running on the same LPAR. However, further XCF links to CICS regions across the CICSPlex® are supported, as shown in Figure 3-14. It is possible to use the EXCI user exit DFHXCURM (see 3.4.3, “EXCI user-replaceable module: DFHXCURM” on page 100) to re-route EXCI requests to an alternative (or backup) CICS router region in case of a failure of the target CICS router region. When using techniques such as Sysplex Distributor with the Gateway daemon, it is important to be aware of a potential affinity-related problem which might be created. For example, in Figure 3-14, if a request for CICS router region1 is routed by Sysplex Distributor to LPAR-1 then a cross-memory (XM) EXCI connection is used between the Gateway daemon and the CICS router. However, if the request is instead routed to LPAR-2 then a cross-system coupling facility (XCF) EXCI connection would be used instead (something that we do not recommend). It is possible to use the EXCI user exit DFHXCURM to resolve this affinity problem by making sure that an EXCI XM connection is always used in preference to an EXCI XCF connection. DFHXCURM is able to dynamically change the APPLID of the target CICS region such that all requests which are routed to the Gateway daemons running on LPAR-1 are passed to CICS router 1 and all requests which are routed to the Gateway daemons on LPAR-2 are passed to CICS router 2. 3.7 Testing the configuration After we installed and configured the software components, we tested our configuration in two scenarios using: A single Gateway daemon Two Gateway daemons configured to use TCP/IP port sharing 106 CICS Transaction Gateway for z/OS Version 6.1 3.7.1 Single Gateway daemon connectivity tests In the following section, we detail how we tested our connectivity environment using our sample J2EE application CTGTesterCCI. Figure 3-15 shows the basic outline of our environment. Windows z/OS WebSphere Application Server V6 CITGR1G HTML JSP Gateway daemon Servlet CTGTesterCCI CCI 13101 CICS TG V6.1 CICS ECI resource adapter JNI TCP EXCI CICS Cam21-pc11 ECIPROG CITGR1CI tx1.mop.ibm.com Figure 3-15 CTGTesterCCI remote connection to Gateway daemon End-to-end settings Table 3-3 summarizes the settings that we used in our testing across the three-tiers (WebSphere, the Gateway daemon, and CICS). Table 3-3 End-to-end connection settings Tier Setting Value WebSphere Application Server EJB container thread pool 100 Connection factory: Minimum connections, maximum connections 10, 55 Connection factory: Unused timeout 10 minutes Connection factory: SocketConnectTimeout 10 milliseconds Connection factory Wait timeout 10 seconds Chapter 3. Gateway daemon connections 107 Tier Setting Value Gateway idletimeout 10 minutes minconnect, maxconnect 10, 55 minworker, maxworker 10,55 idletimeout 10 minutes EXCI LOGONLIM 250 SESSION RECEIVECOUNT 999 Mirror TRANCLASS LIMIT (active, queued) 50, 5 MAXTASKS 100 CICS The reasoning we used in setting these values can be summarized by four rules: 1. Queue early, this saves resources as each request consumes system resources (we were queuing at the connection factory level). 2. Timeout at the source of the file I/O, because this is where the delays are likely to originate from. 3. Do not define infinite queues, because they can lead to infinite delays (such as a purgethreshold of NO in the CICS TRANCLASS). 4. Set resources that are potential bottlenecks (for example, the maximum number of EXCI pipes) to the highest values unless they are to be configured to act as points of queuing. How you tune your own system is dependent on a wide number of different factors. However, from our experience, we suggest that you apply the following specific rules to a WebSphere, JCA, CICS TG scenario such as ours: Set the maximum number of connection manager threads (maxconnect) in the Gateway daemon equal to the sum of the maximum number of connections in all the connection factories that connect to the Gateway. Set the initial number of connection manager threads (initconnect) equal to the minimum number of managed connections in the connection pool in all the connection factories that connect to the Gateway. Set the maximum number of worker threads to: – Less than or equal to the maximum number of connection manager threads. – Less than or equal to the EXCI LOGONLIM. 108 CICS Transaction Gateway for z/OS Version 6.1 Set the initial number of worker threads equal to the initial number of connection manager threads. Set the RECEIVECOUNT for the CICS SESSIONS definition: – At least as big as the EXCI LOGONLIM. – Larger than or equal to the sum of all the worker threads in the Gateway daemons that can connect through this connection (if multiple Gateways share a connection). Testing the configuration To test our configuration, we used three different Web browser instances, each of which invoked our test application CTGTesterCCI (Figure 3-16), using the following URL: http://cam21-pc11:9080/CTGTesterCCIWeb/index.jsp Figure 3-16 CTGTesterCCI application Chapter 3. Gateway daemon connections 109 We specified as input a COMMAREA of D (meaning the CICS program ECIPROG would delay for one minute), and 3 iterations. For further details about our sample application refer to Appendix A, “Sample applications” on page 361. This test results in each invocation from the Web browser driving three calls to CICS and lasting approximately three minutes in total. During this period, we monitored the connection state in each tier using the following utilities: WebSphere TCP/IP CICS TG CICS Tivoli® Performance Viewer netstat stdout log CEMT command The next sections discuss each of these tools. Tivoli Performance Viewer Tivoli Performance Viewer is provided with WebSphere Application Server and provides runtime systems monitoring for a wide variety of resources in WebSphere. First, we enabled the PMI function in the Administrative Console as follows: We selected Monitoring and Tuning → Performance Monitoring Infrastructure (PMI) in the Welcome panel, and then selected server1. We then selected Enable Performance Monitoring Infrastructure (PMI) and Extended. Finally, we selected OK to apply these changes and then saved them to the repository and restarted the Application Server. After the restart, in the Welcome panel, we selected Monitoring and Tuning → Performance Viewer → Current Activity → server1. In the left-side panel of the screen, we selected server1 → Performance Modules → JCA Connection Pools → ECIResourceAdapter → eis/CITGR1CI-tx1-13101 to enable collection of the extended performance metrics for our connection factory (Figure 3-17). 110 CICS Transaction Gateway for z/OS Version 6.1 Figure 3-17 Tivoli Performance Viewer - Current Activity We then selected View Table to view the information in tabular form and selected the following statistics to be displayed: CreateCount This is the total number of managed connections that have been created within the connection pool. CloseCount This is the total number of managed connection that have been closed within the connection pool. Poolsize This is the current number of managed connections in the pool. FreePoolSize This is the number of managed connections that are not allocated to a J2EE component. Chapter 3. Gateway daemon connections 111 The resulting table then displayed in our browser (Figure 3-18). Figure 3-18 Tivoli Performance Viewer - connection factory metrics From this information, we can see that three connections were created at 15:49:48, one was recorded as free at 15:50:48, and then three were recorded as free at 15:51:48. After this, all three connections in the pool remain free for use by other applications. Monitoring sockets with netstat At the same time as we monitored our connection factory, we also monitored the socket status. We used the USS command netstat -a I<IP address> to show the status of sockets connected from z/OS to our Windows workstation (Example 3-6). Example 3-6 TCP/IP socket status CITGPW: /SYSTEM/tmp>netstat -a -I9.100.199.239 MVS TCP/IP NETSTAT CS V1R6 TCPIP Name: TCPIP1 15:41:30 User Id Conn Local Socket Foreign Socket State ------- -------------------------------CITGR1G 000052AE 9.100.193.122..13101 9.100.199.239..2047 Establsh CITGR1G 000052B0 9.100.193.122..13101 9.100.199.239..2050 Establsh CITGR1G 000052B3 9.100.193.122..13101 9.100.199.239..2051 Establsh This example shows that three sockets were established from z/OS to WebSphere on our Windows machine. These sockets remained active after the EJB requests had completed due to the connection pooling of WebSphere Application Server. 112 CICS Transaction Gateway for z/OS Version 6.1 Gateway monitoring To monitor the Gateway daemon connection usage we viewed the STDOUT log, which enables us to view the status of connections (Example 3-7). Example 3-7 Gateway STDOUT log 09/21/05 15:49:30:280 [0] CTG6506I Client connected. [ConnectionManager-9] tcp@Socket[addr=9.100.199.239,port=2047,localport=13101] 09/21/05 15:49:32:399 [0] CTG6506I Client connected. [ConnectionManager-8] tcp@Socket[addr=9.100.199.239,port=2050,localport=13101] 09/21/05 15:49:35:877 [0] CTG6506I Client connected. [ConnectionManager-7] tcp@Socket[addr=9.100.199.239,port=2051,localport=13101] These messages show that three connection manager threads were allocated one for each connection and that these connections remained active after the EJB requests had completed. CICS monitoring While our three Web browser sessions were running, we used CEMT to monitor the status of our mirror transaction ECIT in CICS. We used the CEMT INQ EXCI and CEMT I TRANCLASS commands. Figure 3-19 shows that there are three EXCI requests active from the job CITGR1G running on system X1 (our LPAR). I EXCI STATUS: RESULTS Exc(CITGR1G.CITGR1G. - X1 Exc(CITGR1G.CITGR1G. - X1 Exc(CITGR1G.CITGR1G. - X1 ) Tas(0001417) ) Tas(0001418) ) Tas(0001419) RESPONSE: NORMAL PF 1 HELP 3 END SYSID=TGR1 APPLID=A6POTGR1 TIME: 15.49.50 DATE: 09.21.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF 5 VAR Figure 3-19 CEMT INQUIRE EXCI Chapter 3. Gateway daemon connections 113 The CEMT INQ TRANCLASS command (Figure 3-20) shows that three tasks were active in our TRANCLASS TCLASS1. I TRANCLASS(TCLASS1) STATUS: RESULTS - OVERTYPE TO MODIFY Tcl(TCLASS1 ) Max( 050 ) Act(003) Pur( 0000006 ) Que(000000) RESPONSE: NORMAL PF 1 HELP 3 END SYSID=TGR1 APPLID=A6POTGR1 TIME: 15.49.55 DATE: 09.21.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF 5 VAR Figure 3-20 CEMT INQUIRE TRANCLASS 3.7.2 Cloned Gateway daemon connectivity tests In this section, we show how we cloned our Gateway daemon CITGR1G and tested a TCP/IP port sharing configuration. Cloning a Gateway daemon provides for improved failover and increased throughput. Figure 3-21 shows the outline of our configuration. Windows z/OS WebSphere Application Server V6 CITGR1G/CITGR2G HTML JSP Gateway daemon Servlet CTGTesterCCI CCI 13101 CICS TG V6.1 CICS ECI resource adapter JNI TCP EXCI ECIPROG Cam21-pc11 CITGR1CI tx1.mop.ibm.com Figure 3-21 CTGTesterCCI remote connection to cloned Gateway daemons 114 CICS Transaction Gateway for z/OS Version 6.1 To set up our cloned Gateway daemons, we performed the following tasks: Created another Gateway daemon start task CITGR2G. Created a new STDENV input member for CITGR2G, although note that we shared the same Gateway configuration file as used by CITGR1G (thus we inherited all the same Gateway settings). Defined a new DFHJVPIPE name of CITGR2G in the STDENV member for our Gateway daemon CITGR2G. Defined new CICS CONNECTION and SESSSIONS definitions in our CICS region CITGR1CI, specifying the NETNAME of CITGR2G, and a unique session prefix. Created a new user ID CITGR2G (as described in “Setting up a user ID for the Gateway daemon started task” on page 30) and used the STARTED class in RACF to associate the new user ID with the CITGR2G job. Modified the MRO bind security settings for CICS, by giving user ID CITGR2G READ access to the RACF FACILITY class DFHAPPL.A6POTGR1. Modified our TCP/IP PORT statement to allow CITGR2G to bind to and share the port 13101 as follows: 13101 TCP CITGR1G SHAREPORT 13101 TCP CITGR2G SHAREPORT ; Gateway daemon port shared ; Gateway daemon port shared We then restarted TCP/IP to update the port allocations. Monitoring cloned Gateway usage After we had setup our cloned Gateway daemons, we started each one in turn and ensured that they both started and that they had bound to port 13101 on IP address 9.100.193.122. We then used two Web browser instances to send two requests to the CTGTesterCCI application, each of which had a COMMAREA input of D. This caused the two requests to run in parallel. During the test we monitored the STDOUT log of each Gateway daemon to view the status of connections (Example 3-8 and Example 3-9). Example 3-8 CITGR1G Gateway STDOUT log 09/22/05 17:06:53:193 [0] CTG6485I Successfully started handler for the tcp: protocol on port 13101, bound to address /9.100.193.122 09/22/05 17:06:53:601 [0] CTG6512I CICS Transaction Gateway initialization complete 09/22/05 17:07:14:287 [0] CTG6506I Client connected. [ConnectionManager-9] tcp@Socket[addr=9.100.199.239,port=2560,localport=13101] Chapter 3. Gateway daemon connections 115 Example 3-9 CITGR2G Gateway STDOUT log 09/22/05 17:06:55:922 [0] CTG6485I Successfully started handler for the tcp: protocol on port 13101, bound to address /9.100.193.122 09/22/05 17:06:56:311 [0] CTG6512I CICS Transaction Gateway initialization complete 09/22/05 17:07:13:534 [0] CTG6506I Client connected. [ConnectionManager-9] tcp@Socket[addr=9.100.199.239,port=2559,localport=13101] From the STDOUT logs we can see that a socket connection was established from the WebSphere connection factory to each of the cloned Gateway daemons, thus proving that the TCP/IP port sharing component is load balancing the socket connections across our two Gateway daemons. 3.8 Problem determination In this section, we document common connection problems and information that we learned while configuring our connection test scenarios. For general problem determination guidance, see 2.9, “Problem determination” on page 60. 3.8.1 Common connection problems In this section, we list some common connectivity problems that you might experience. ECI_ERR_RESOURCE_SHORTAGE If the Gateway daemon tries to allocate more pipes than the available number of CICS sessions, the EXCI Open_Pipe call fails with a RETRYABLE response, and reason code 202 is logged in the CICS TG STDOUT log. The ECI application receives a return code -16 (ECI_ERR_RESOURCE_SHORTAGE). This situation can occur even if the client application program has allocated fewer pipes than the number of receive sessions defined on the target CICS connection. This is because CICS can be in the process of cleaning up a pipe from a Close_Pipe request. For this reason, we recommend that you specify a larger RECEIVECOUNT value than is theoretically necessary when defining the SESSIONS resource definition to CICS. ECI_ERR_SYSTEM_ERROR If the number of EXCI pipes in use exceeds the CICS EXCI pipe limit, the next EXCI Allocate_Pipe call fails with a SYSTEM_ERROR response and reason code 608. The ECI application receives return code -9 (ECI_ERR_SYSTEM_ERROR). 116 CICS Transaction Gateway for z/OS Version 6.1 Because this means a limitation in the sending address space has been reached, you should consider one of the following options: Increasing the MVS LOGONLIM if using CICS TS V2.2 or higher. Decreasing the worker thread count and cloning your Gateway regions, to provide increased capacity if required. Setting the CTG_PIPE_REUSE=ONE variable, to provide a more predictable pipe usage If you define the DFHJVPIPE environment variable incorrectly as blanks, the EXCI Initialize_User fails with a USER_ERROR response and reason code 403 (INVALID_APPL_NAME). You should specify a non-blank value for DFHJVPIPE in STDENV. 3.8.2 Tips and utilities This section includes additional tips and utilities for problem determination of connection-related problems. For general tips and information about standard utilities, such as the various Gateway daemon traces, see 2.9.2, “Tips and utilities” on page 62. Testing a Connection pool failure To simulate a connection failure we performed an immediate shutdown on our Gateway daemon by cancelling the Gateway. We then resubmitted a request from our test J2EE application and monitored the state of the sockets and the connection pool. After the Gateway was cancelled, we noticed the following results: Our test application returned the following exception: javax.resource.spi.CommException: CTG9630E IOException occurred in communication with CICS The netstat socket status showed all the sockets in a TimeWait state. The CloseCount in Tivoli Performance Viewer went up to 1 and the PoolSize decreased to 2. This was triggered by the IOException driving a fatal connection error on the connection that it tried to use. Figure 3-22 shows the results from our Tivoli Performance Viewer. Chapter 3. Gateway daemon connections 117 Figure 3-22 Tivoli Performance Viewer - connection factory counters 3.8.3 Tracing In this section, we outline the main traces that you can enable within the WebSphere environment in order to diagnose problems with a connection from WebSphere Application Server on a distributed platform to the Gateway daemon running on z/OS. Java client trace You can set CICS TG Java client tracing programmatically or as a system property. CICS TG Java client tracing can be set programmatically by adding the following statements to the Java application: com.ibm.ctg.client.T.setDebugOn(true); com.ibm.ctg.client.T.setTimingOn(true); Both of these traces go to the stderr output stream, so that they appear in the standard error log file of the WebSphere Application Server. Example 3-10 shows the stderr log on our system. Example 3-10 WebSphere stderr log location C:\Program Files\IBM\WebSphere\AppServer/profiles/AppSrv01\logs\server1\SystemErr 118 CICS Transaction Gateway for z/OS Version 6.1 In our CTGTesterCCI application the Trace option on the index.jsp welcome page controls this tracing. Note: Enabling CICS TG Java client tracing in an application enables it for all applications running in the application server, as a consequence of the static nature of the T class. You can also set Java client trace using the WebSphere Administrative Console. The following system properties control CICS TG Java client trace within the application server: Dgateway.T=on Dgateway.T.setTFile=<filename> You can set these properties from the WebSphere Administrative Console as follows: 1. Click Servers → Application Servers → server1. 2. In the Server Infrastructure configuration settings, expand Java and Process Management and click Process Definition → Java Virtual Machine. 3. Enter the Java client trace system properties as Generic JVM™ Arguments. Note: If you do not explicitly set a file name, the trace entries are written to stderr. J2EE resource adapter trace Resource adapter tracing is used to control tracing of the flow of execution through the CCI classes. When using a managed environment, J2EE resource adapter tracing is enabled by setting the TraceLevel parameter in the connection factory. You can set the tracelevel to one of the following values: 0 1 2 3 Disable all tracing Output exception trace stacks Output method entry and exit stack traces Output debug trace entries Chapter 3. Gateway daemon connections 119 120 CICS Transaction Gateway for z/OS Version 6.1 4 Chapter 4. Connecting from WebSphere Application Server for z/OS This chapter provides details about how we configured our test environment to allow our J2EE application running on WebSphere Application Server for z/OS to connect to an application running in CICS Transaction Server on z/OS using the CICS Transaction Gateway for z/OS. © Copyright IBM Corp. 2006. All rights reserved. 121 4.1 Preparation In our testing, we deployed our sample J2EE application on WebSphere Application Server for z/OS and connected to CICS using the CICS ECI resource adapter that is provided by CICS TG for z/OS. This chapter concentrates on how to configure the connections between each tier (as shown in Figure 4-1). It does not provide details about how to install all of the software components, and it assumes a working knowledge of CICS or WebSphere Application Server where appropriate. Note: In this chapter we configure a local connection from WebSphere Application Server to a CICS region running in the same LPAR. The CICS TG also supports remote connections from WebSphere Application Server for z/OS. See Chapter 7, “Transactions with WebSphere Application Server for z/OS” on page 253 for an example of how to use a remote connection from WebSphere on z/OS. z/OS WebSphere Application Server V6 CICS TS V3.1 HTML JSP Servlet CICS TG V6.1 EJB CCI CICS ECI resource adapter EXCI C O M M A R E A COBOL application Figure 4-1 Software components: CICS TG and WebSphere Application Server for z/OS 122 CICS Transaction Gateway for z/OS Version 6.1 4.1.1 Software checklist Table 4-1 shows the levels of software that we used for this configuration. Table 4-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS V1.6 Windows 2000 Service Pack 4 WebSphere Application Server V6.0.1 a CICS Transaction Server V3.1 CICS Transaction Gateway for z/OS V6.1 IBM SDK for z/OS Java 2 Technology Edition,V1.4.2 SR2 a. WebSphere Application Server for z/OS V6.01 is the minimum WebSphere level supported with the CICS TG V6.1 ECI resource adapters. 4.1.2 Definitions checklist Table 4-2 lists the definitions that we used to configure the scenario. Table 4-2 Definitions checklist: WebSphere Application Server for z/OS Value WebSphere Application Server for z/OS hostname tx1.mop.ibm.com TCP/IP port 13880 Job names Daemon region: CITGRDMN Control region: CITGRS1 Servant region: CITGRS1S APPLID CICS TS CITGR1CI A6POTGR1 NETNAME CITGRS1 CONNECTION GRS1 Private mirror ECIW Chapter 4. Connecting from WebSphere Application Server for z/OS 123 4.2 WebSphere Application Server configuration In this section, we outline the configuration of our WebSphere Application Server for z/OS. We discuss the following: J2EE server configuration Defining the workload profile Installing the resource adapter Creating connection factories Deploying our sample J2EE application 4.2.1 J2EE server configuration Our WebSphere Application Server for z/OS configuration is a base Application Server node (Figure 4-2). It consists of: One application server, which in turn consists of: – One control region – One servant region (where the J2EE applications execute) One daemon (the location agent, responsible for resolving IIOP requests) One node (a logical grouping of managed servers) One cell (a logical grouping of one or more nodes) z/OS Cell (CITGR_cell1) Node (CITGR_node1) J2EE Server (CITGR_server1) Daemon (CITGRDMN) Control region (CITGRS1) Servant region (CITGRS1S) Figure 4-2 WebSphere Application Server for z/OS - base configuration For details about setting up such an environment, refer to WebSphere Application Server for z/OS V6.0.1, Installing your application serving environment, GA22-7957. 124 CICS Transaction Gateway for z/OS Version 6.1 Note: The WebSphere Application Server configuration that we used in our tests is a very simple one. For further considerations when connecting to CICS from a Network Deployment configuration, see 4.6, “Configuring for high scalability” on page 149. 4.2.2 Defining workload profile The WebSphere Application Server Workload profile controls the number of threads used in a servant region. The workload profile can have one of four values, ISOLATE, IOBOUND (the default), CPUBOUND, or LONGWAIT. We changed the value to LONGWAIT, which is the recommended value for applications that make frequent calls to other subsystems such as CICS. When LONGWAIT is specified each servant region is able to use a maximum of 40 threads. Note: The number of threads in a J2EE servant region is controlled by the server workload profile, which can have the following values: IOBOUND - 3 * the number of CPUs (max of 3*32=96) CPUBOUND - the number of CPUs on line (max 32) ISOLATE - 1 thread LONGWAIT - 40 threads To change the setting, we use the WebSphere Administrative Console as follows: 1. Click Servers, and then Application servers. 2. Select our server CITGR_server1 and on the presented panel, click Container Services, and then ORB Service. 3. Click z/OS additional settings. 4. In the next panel, change the default IOBOUND to LONGWAIT (Figure 4-3). 5. Click OK, save the configuration, and restart the J2EE server. Chapter 4. Connecting from WebSphere Application Server for z/OS 125 Figure 4-3 Administrative Console - Workload profile 4.2.3 Installing the CICS ECI resource adapter CICS TG V6.1 provides two ECI resource adapters, cicseci.rar and cicseciXA.rar. For this scenario, we used the cicseci.rar adapter. Note: WebSphere Application Server for z/OS provides two-phase commit support for local connections when using either of the CICS ECI resource adapters. However, only the cicseciXA.rar provides two-phase commit support when connecting to CICS using a remote connection via a z/OS Gateway daemon. See Chapter 7, “Transactions with WebSphere Application Server for z/OS” on page 253 for more information about transactional support. To install the resource adapter: 1. We started our Application Server and logged into the Administrative Console using the following URL in our Web browser: http://tx1.mop.ibm.com:13880/ibm/console/ 2. We were presented with the login page and entered CITGRADM (the WebSphere administration user ID) in the User ID field and clicked login. 126 CICS Transaction Gateway for z/OS Version 6.1 3. In the Administrative Console Welcome panel, we clicked Resources and then clicked Resource Adapters. We selected the Node level view, clicked Apply, and clicked Install RAR. 4. On the presented panel (Figure 4-4), we selected Server path, and filled in the path to the cicseci.rar file, which in our environment was /usr/lpp/cicstg/ctg610/deployable/cicseci.rar. We then clicked next. Figure 4-4 Administrative Console - specifying the path of the RAR 5. On the following panel (Figure 4-5), for the Native Path property, we entered the full path to the CICS TG bin directory where the .so files were stored. The directory on our system was /usr/lpp/cicstg/ctg610/bin. Chapter 4. Connecting from WebSphere Application Server for z/OS 127 Figure 4-5 Administrative Console - specifying the native path 6. We accepted the defaults for the other properties on this panel and scrolled to the bottom of the panel and clicked OK. We clicked Save twice to save the modified configuration. 4.2.4 Creating a connection factory After installing the resource adapter, you need to create a connection factory which targets the CICS region. To create a connection factory, we did the following: 1. From the Administrative Console, we clicked Resources, Resource Adapters and then clicked ECIResourceAdapter, which is the resource adapter that we just installed. 2. On the presented panel, we clicked J2C Connection Factories and then we clicked New. 128 CICS Transaction Gateway for z/OS Version 6.1 3. We were then presented with the panel shown in Figure 4-6. We filled in the Name field. The name we gave the connection factory is CITGR1CI-tx1-local. In this name, CITGR1CI is the CICS region to which we connect, tx1 is the z/OS LPAR, and local designates that we are using a local connection. We allowed the security settings to default and clicked OK. Figure 4-6 CICS connection without security or XA 4. We let the rest of the properties default, clicked Apply, and then clicked Save to save the configuration. 5. After these settings were applied and the connection factory was selected, the Additional Properties links were then enabled. We clicked Custom Properties, which took us to the CICS ECI resource adapter properties panel (Figure 4-7). Chapter 4. Connecting from WebSphere Application Server for z/OS 129 6. The custom properties are the parameters that are used by the CICS ECI resource adapter to communicate with the target CICS region. We set the following parameters by selecting each parameter, filling in the value for the parameter, and clicking OK. – TPNName This is the CICS TPN name and is used as the name of the attached mirror transaction that requests will run in CICS. We set this property to ECIW. If the property is not set, the CICS default mirror transaction CSMI is used. – ConnectionURL This is the URL that defines the TCP/IP host name and protocol that are used to communicate with the Gateway daemon. Because we are using a local connection, we set the connection URL to local. Tip: Make sure that you include the colon (:) in local. – PortNumber This is normally the port number that the Gateway daemon listens on. Because we are running in local mode, and there is no Gateway daemon, we set the port number to zero (0). – ServerName This is the APPLID of the CICS region to which requests are sent. We specified the APPLID of our CICS region A6POTGR1. We then saved our configuration. Restriction: For local connections, the CICS region must run on the same MVS image as the WebSphere Application Server. 130 CICS Transaction Gateway for z/OS Version 6.1 Figure 4-7 Administrative Console - custom properties For a full explanation of all the properties, refer to 4.2.4, “Creating a connection factory” on page 128. 7. After we defined the connection factory custom properties, we clicked Connection pool properties on the resource adapter connection factory page and entered values to tune our connection pool. 8. This presented us with the general connection pool properties panel (Figure 4-8) where we entered the following values. Note: For a local connection, the JCA connection pool managed by WebSphere Application Server is a set of local connection objects. The connection objects do not map onto network connections, nor do they map onto EXCI pipes because EXCI pipes are allocated directly by the JVM threads within the J2EE servant region. Chapter 4. Connecting from WebSphere Application Server for z/OS 131 – Connection timeout Specifies how long, in seconds, a getconnection() request by the J2EE application can wait if there are no connections available in the free pool, and no new connections can be created. If a timeout occurs a ConnectionWaitTimeoutException is thrown. We set this to 10 seconds in order to timeout in the event that a connection request cannot be satisfied. Note, however, that such event should not happen in practice because we are not restricting the size of the connection pool. – Maximum connections We set this to a value of zero (0). If maximum connections is set to zero (0), the connection pool is allowed to grow infinitively. In that case the number of threads defined for the WebSphere Application Server servant regions, in our case 40, limits the maximum number of connections in the pool. – Minimum connections We set the minimum number of connections to a value of zero (0). – Reap time This is the interval at which the pool manager maintenance thread will run. With with connection timeout and aged timeout both set to 0, we also set this to zero (0), which stops the pool manager maintenance thread from running without any work to do. – Unused timeout Because we set the reap time to zero (0), we also set the unused timeout to zero (0). – Aged timeout Again, because we set the reap time to zero (0), we also set the aged timeout to zero (0). – PurgePolicy This specifies how connections are being purged when a fatal error is detected. We set this to FailingConnectionOnly. The alternative is EntirePool, which means that all threads are purged. Note: For a detailed description of the custom pool properties, refer to Figure 3-8 on page 87. 132 CICS Transaction Gateway for z/OS Version 6.1 Figure 4-8 Administrative Console - defining connection pool properties 4.2.5 Configuring the J2EE server In our testing, we used a specific EXCI connection when connecting to CICS (see “EXCI connections” on page 139). Therefore, we need to specify the environment variable DFHJVPIPE. Because we are running in local mode, CICS TG environment variables are specified using the WebSphere Administration Console. Chapter 4. Connecting from WebSphere Application Server for z/OS 133 To configure the J2ee server, we logged in to the Administrative Console, clicked Environment → WebSphere Variables → New. On the presented panel (Figure 4-9), we entered DFHJVPIPE in the name field and CITGRS1 in the Value field. Then we clicked OK. Figure 4-9 WebSphere environment variable - DFHJVPIPE 4.2.6 Deploying our sample J2EE application In this section, we describe how we deployed our test application CTGTesterCCI into WebSphere Application Server for z/OS. For details about the application, refer to Appendix A, “Sample applications” on page 361. Deploying CTGTesterCCI To deploy the test application, we did the following: 1. In the Administrative Console Welcome panel, we clicked Applications and Install New Application. We then selected Remote file system and specified the path of the application ear file /CITG/TJ/CTGTesterCCI.ear 134 CICS Transaction Gateway for z/OS Version 6.1 2. We clicked Next (Figure 4-10). Figure 4-10 Administrative Console - deploying J2EE application 3. We clicked Next on the Preparing for the application installation screens and clicked Continue on the Application Security Warnings screen. 4. The Administrative Console now lays out steps in the installation of the application. These steps might differ from one application to another, depending on what the Administrative Console application sees in the deployment descriptors of the EAR file being installed. For example, when the deployment descriptor contains a <resource-ref> tag, the Administrative Console application presents a step to map that resource reference to a resource. 5. We accepted the defaults for all steps, except the Map resource references to resources, as shown in Figure 4-11. 6. We mapped the ECI resource reference to the JNDI name of the Connection Factory that we defined earlier. All of the JNDI names of defined Connection Factories appeared in the pull-down window. We chose the JNDI name that matched the Connection Factory we created previously: eis/CITGR1CI-tx1-local. We selected the EJB module that we wanted to map (CTGTesterCCIEJB) and choose the default authentication method. We then clicked Apply to insert the JNDI name into the EJB module’s JNDI Name field. Chapter 4. Connecting from WebSphere Application Server for z/OS 135 Figure 4-11 Installing CTGTesterCCI - mapping resources 7. We clicked Next three times and then Finish (noted the ADMA5013I: Application CTGTesterCCI installed successfully message) then we clicked Save to master configuration, followed by Save. 8. In the Administrative Console, we clicked Applications and Enterprise Applications, selected the CTGTesterCCI application, and clicked Start. 136 CICS Transaction Gateway for z/OS Version 6.1 4.3 Configuring EXCI When using local connections with WebSphere Application Serer on z/OS, the CICS ECI resource adapters use the EXCI to communicate directly with CICS. The following sections discuss the key points that we considered for our EXCI configuration. 4.3.1 EXCI pipe limits Each WebSphere servant region is limited to a maximum number of pipes that it can allocate to all attached CICS regions. The EXCI pipe limit rules are as follows: In CICS TS V1.3, the limit is 100 EXCI pipes in total per calling address space. In CICS TS V2.2 or higher the PTF for APAR PQ92943 allows a MVS system wide LOGONLIM to be set, which controls the pipe limit. The upper limit is 250 and the default is 100. The maximum pipe usage in a CICS region is limited by the RECEIVECOUNT parameter defined in the CICS MRO SESSIONS definition. This can be a maximum of 999 pipes, depending on the number of characters specified in the RECEIVEPFX. The default EXCI pipe allocation initialization parameter defined in the DFHSSIN routine is LOGONLIM=100. See 4.3.1, “EXCI pipe limits” on page 137 for instructions on how we changed the default setting. 4.3.2 CTG_PIPE_REUSE By default if a J2EE server is connected to multiple CICS regions, a servant region thread can potentially allocate multiple EXCI pipes. If this occurs it can cause unpredictable pipe shortage issues. In CICS TG V6.0 a new environment variable called CTG_PIPE_REUSE was introduced. If this is set to ONE then a thread will only allocate one pipe, and will deallocate the pipe if it needs to reallocate a pipe to a different CICS region. This provides a more predictable usage model, but with a small cost in performance. The exact performance cost depends on the frequency of pipe deallocations. We set the value CTG_PIPE_REUSE=ONE using the WebSphere Administrative Console. We clicked Environment and WebSphere variables and we then added the variable by clicking New (Figure 4-12) and then saved the changes. Chapter 4. Connecting from WebSphere Application Server for z/OS 137 Because we only have one CICS region in our setup, this setting had no immediate effect. However, setting it is good practice if you anticipate that the WebSphere Application Server will connect to multiple CICS regions at some time in the future and your aim is to ensure a predictable operating environment. Figure 4-12 CTG_PIPE_REUSE=ONE setting 4.3.3 Defining the EXCI library to your J2EE Server The J2EE Server that uses the CICS ECI resource adapter needs to load the CICS module DFHXCSTB from the SDFHEXCI library supplied with CICS. This can be enabled in one of the following ways: Place module DFHXCSTB in the link pack area (LPA). Add the SDFHEXCI library to either the system link list concatenation, or to the STEPLIB concatenation of the J2EE Server. We added our SDFHEXCI data set (CICSTS31.CICS.SDFHEXCI) to the MVS system link list. 138 CICS Transaction Gateway for z/OS Version 6.1 4.4 Configuring CICS For the WebSphere Application Server to be able to connect to the CICS TS region, we performed the following tasks: Set CICS system initialization table (SIT) parameters Configured an EXCI connection in CICS Added a private mirror transaction Planned for transaction timeouts SIT parameters Because the CICS TG and WebSphere Application Server use EXCI for communications and MVS resource recovery services (RRS) for transactional support, we enabled this support in CICS using the following SIT parameters: RRMS=YES IRCSTRT=YES ISC=YES We restarted the CICS region to put these parameters into effect. EXCI connections We created the CONNECTION and SESSIONS definitions in CICS. Then, from a CICS screen, we entered the following: CEDA DEFINE CONNECTION(GRS1) GROUP(GRS1) We defined the GRS1 CONNECTION as shown in Figure 4-13. We recommend using specific EXCI pipes, which provides for better monitoring and accounting within CICS. We set Conntype to Specific and Netname to CITGRS1 (the DFHJVPIPE environment variable name that we defined previously using the WebSphere Administrative Console). DEFINE GROUP(GRS1) CONNECTION(GRS1) OVERTYPE TO MODIFY CEDA DEFine CONnection( GRS1 ) CONnection : GRS1 Group : GRS1 DEscription ==> CONNECTION IDENTIFIERS Netname ==> CITGRS1 INDsys ==> CONNECTION PROPERTIES ACcessmethod ==> IRc PRotocol ==> Exci Conntype ==> Specific SInglesess ==> No Vtam | IRc | INdirect | Xm Appc | Lu61 | Exci Generic | Specific No | Yes Figure 4-13 CICS definitions - CEDA Define CONNECTION Chapter 4. Connecting from WebSphere Application Server for z/OS 139 We then defined the SESSIONS: CEDA DEFINE SESSIONS(GRS1) GROUP(GRS1) CONNECTION(GRS1) We defined the GRS1 SESSIONS as shown in Figure 4-13. We set the RECEIVEPFX to 3, which means that the session names for this connection all start with the character 3, and we set the RECEIVECOUNT to 999. DEF SESSIONS(GRS1) GR(GRS1) CONNECTION(GRS1) OVERTYPE TO MODIFY CEDA DEFine Sessions(GRS1 ) Sessions : GRS1 Group : GRS1 DEscription ==> SESSION IDENTIFIERS Connection ==> GRS1 SESSION PROPERTIES Protocol ==> Exci MAximum ==> 000 , 000 RECEIVEPfx ==> 3 RECEIVECount ==> 999 CICS RELEASE = 0640 Appc | Lu61 | Exci 0-999 1-999 SYSID=TGR1 APPLID=A6POTGR1 Figure 4-14 CICS definitions - CEDA Define SESSIONS Next, we installed our definitions using the following command: CEDA INSTALL GROUP(GRS1) 4.4.1 Adding a private mirror Rather than running our ECI request under the default mirror transaction (CSMI) we defined our own private mirror, which has the following advantages: It allows individual CICS monitoring and statistics information to be recorded for the given application. It allows specific security attributes to be set for the transaction. It allows a specific TRANCLASS to be specified, which can control the queuing and scheduling priorities of the CICS application relative to the other transactions executing in the CICS region. 140 CICS Transaction Gateway for z/OS Version 6.1 To define our private mirror we performed the following steps: 1. We defined a mirror transaction ECIW by copying the default mirror transaction definition CSMI to a new RDO group using the following command: CEDA COPY GR(DFHISC) TRAN(CSMI) AS(ECIW) GROUP(ECIPROG) 2. We updated this definition to specify a TRANCLASS of TCLASS2 using the following command: CEDA ALTER TRANS(ECIW) GROUP(ECIPROG) TRANCLASS(TCLASS2) 3. We defined a new TRANCLASS resource definition, in group ECIPROG, to control the attributes of the ECIW transaction with a MAXACTIVE value of 50 and a PURGETHRESHOLD of 5 (Figure 4-15). This TRANCLASS limits our CICS region to running 50 simultaneous ECIW transactions and also queues a further five requests before it refuses to queue further transactions. Note that the TRANCLASS setting is larger than the number of threads defined in the WebSphere Application Server, see 4.2.2, “Defining workload profile” on page 125. However, if we were to connect to our CICS region at a later time from multiple WebSphere servant regions, the TRANCLASS would ensure that the CICS region would not be paralyzed with requests for the ECIW transaction. OVERTYPE TO MODIFY CEDA ALter TRANClass( TCLASS2 TRANClass : TCLASS2 Group : ECIPROG Description ==> CLASS LIMITS Maxactive ==> 050 Purgethresh ==> 0000005 CICS RELEASE = 0640 ) 0-999 No | 1-1000000 SYSID=TGR1 APPLID=A6POTGR1 Figure 4-15 CICS TRANCLASS definition 4. We installed the TRANCLASS using the CEDA INSTALL GROUP(ECIPROG) command and added the ECIPROG group to our CICS startup list TGR1. Chapter 4. Connecting from WebSphere Application Server for z/OS 141 4.4.2 Transaction timeouts We did not set any timeout parameters in our CICS environment (such as RTIMOUT or DTIMOUT), because neither of these were applicable to our scenario. However, we recommend that you consider setting a timeout on the file manager (such as DB2), because it is generally perceived as best practice to time out suspended tasks as close as possible to the file operation that is likely to cause delays. For further details about the end-to-end timeout settings that we used, refer to “End-to-end settings” on page 143. Tip: We found that a suspended task in CICS, for example, caused by someone leaving CEDF running inadvertently, caused a timeout in the WebSphere Application Server which resulted in a servant region failure and an automatic restart. For this reason, we recommend that you attempt to timeout CICS tasks using CICS facilities such as RTIMOUT and DTIMOUT. 4.5 Testing the configuration In this section, we detail how we tested our environment, using our sample J2EE application CTGTesterCCI. Figure 4-16 shows the basic outline of our environment. z/OS WebSphere Application Server V6 CICS TS V3.1 HTML JSP Servlet CTG V6.1 CTGTesterCCI EXCI cicseci.rar CCI GRS1 C O M M A R E A ECIPROG CITGR1CI CITGR_server1 tx1.mop.ibm.com Figure 4-16 CTGTesterCCI connection from WebSphere Application Server for z/OS 142 CICS Transaction Gateway for z/OS Version 6.1 End-to-end settings Table 4-3 summarizes the settings that we used throughout the chapter. Table 4-3 End-to-end settings Tier Setting Value WebSphere Application Server for z/OS Servant region threads (workload profile LONGWAIT) 40 Connection factory: Minimum connections, maximum connections 0,0 Connection factory: Connection timeout 10 Connection factory: Unused timeout 0 Connection factory: Aged timeout 0 EXCI LOGONLIM 250 SESSION RECEIVECOUNT 999 SESSION RECEIVEPFX 3 Mirror TRANCLASSLIMIT(active,queued) 50,5 MAXTASKS 100 CICS To test our configuration, we used a Web browser to invoke the test application CTGTesterCCI. We used the following URL to start the application: http://tx1.mop.ibm.com:13880/CTGTesterCCIWeb/index.jsp Chapter 4. Connecting from WebSphere Application Server for z/OS 143 We were then presented with the screen that is shown in Figure 4-17. We specified a COMMAREA of D as input, which caused the target CICS program to enter a delay of one minute. Then we selected Submit. We repeated these steps on the two other browsers. Figure 4-17 CTGTesterCCI - input screen 144 CICS Transaction Gateway for z/OS Version 6.1 We were presented with the results as shown in Figure 4-18. The information in the COMMAREA is as follows: A6POTGR1 is the applid of the CICS region to which we connected. 01/12/05 13:11:57 is the transaction timestamp CITGR1D (the default CICS user ID) is the user ID the transaction is run under DS is the transaction start code Figure 4-18 CTGTesterCCI - result screen Monitoring During the test period, we monitored the connection state in WebSphere Application Server and CICS, using the following utilities: WebSphere CICS Tivoli Performance Viewer CEMT command Chapter 4. Connecting from WebSphere Application Server for z/OS 145 Tivoli Performance Viewer Tivoli Performance Viewer is provided with WebSphere Application Server and provides runtime systems monitoring for a wide variety of resources in WebSphere. First, we enabled the PMI function in the Administrative Console as follows: 1. We selected Monitoring and Tuning → Performance Monitoring Infrastructure (PMI) in the Welcome panel, and then selected server1. We selected Enable Performance Monitoring Infrastructure (PMI) → Extended. We selected OK to apply these changes and then saved them to the repository and restarted the Application Server. 2. After the restart, in the Welcome panel we selected Monitoring and Tuning → Performance Viewer → Current Activity → CITGR_server1. In the left-side panel of the screen, we selected CITGR_server1 → Performance Modules → JCA Connection Pools → ECIResourceAdapter → eis/CITGR1CI-tx1-local to enable the collection of the extended performance metrics for our connection factory (Figure 4-19). Figure 4-19 Tivoli Performance Viewer - current activity 146 CICS Transaction Gateway for z/OS Version 6.1 3. We clicked View Table to view the information in tabular form and selected the following statistics to display: CreateCount The total number of managed connections that have been created within the connection pool. CloseCount The total number of managed connection that have been closed within the connection pool. Poolsize The current number of managed connections in the pool. Used and unused. FreePoolSize The number of free managed connections currently in the pool. Figure 4-20 Tivoli performance viewer - connection factory metrics From the information, we can see that between 11:10:24 and 11:10:51 there were no connections in the pool. At 11:11:21, we see that two connections were created. In the same interval, we see the pool size is two, with one of the connections not being used, and the FreePoolSize is one. At 11:12:23, an additional connection is created, bringing the PoolSize to a total of three. CICS monitoring While our Web browser sessions were running, we used CEMT to monitor the status of our mirror transaction ECIW in CICS. We used the CEMT INQUIRE EXCI and CEMT INQUIRE TASK commands. Chapter 4. Connecting from WebSphere Application Server for z/OS 147 Figure 4-21 shows that there are three EXCI requests active from the job CITGRS1S (the WebSphere servant region) running on system X1 (our LPAR). I EXCI STATUS: RESULTS Exc(CITGRS1S.CITGRS1S. - X1 Exc(CITGRS1S.CITGRS1S. - X1 Exc(CITGRS1S.CITGRS1S. - X1 RESPONSE: NORMAL PF 1 HELP 3 END ) Tas(0000803) ) Tas(0000806) ) Tas(0000807) 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 13.12.08 DATE: 12.01.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 4-21 CEMT INQUIRE EXCI From the CEMT INQUIRE TASK command (Figure 4-22), we can see the private mirror transaction ECIW running on facilities with the character 3 in the first position of the name, as defined in the CICS SESSIONS RECEIVEPFX (see Figure 4-14 on page 140). I TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000794) Tra(CEMT) Fac(N197) Run Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDFE9EF50170E00B) Tas(0000796) Tra(CEMT) Fac(N196) Sus Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDFE9F042F04DE09) Hty(ZCIOWAIT) Tas(0000803) Tra(ECIW) Fac(31 ) Sus Ter Pri( 001 ) Sta(DS) Use(CITGR1D ) Uow(BDFEACB372242247) Hty(ICWAIT ) Tas(0000806) Tra(ECIW) Fac(310 ) Sus Ter Pri( 001 ) Sta(DS) Use(CITGR1D ) Uow(BDFEACE043EEFE8D) Hty(ICWAIT ) Tas(0000807) Tra(ECIW) Fac(3100) Sus Ter Pri( 001 ) Sta(DS) Use(CITGR1D ) Uow(BDFEACE16CC1EF20) Hty(ICWAIT ) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR Figure 4-22 CEMT INQUIRE TASK 148 SYSID=TGR1 APPLID=A6POTGR1 13.12.03 DATE: 12.01.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF TIME: CICS Transaction Gateway for z/OS Version 6.1 4.6 Configuring for high scalability After you have successfully configured and tested your WebSphere z/OS configuration, you should consider how you can clone the J2EE server in order to improve scalability and availability. The principal areas for consideration are: How to load balance TCP/IP requests across multiple J2EE servers Clustering the J2EE server How to load balance CICS program links dynamically across multiple CICS AORs Other configuration considerations 4.6.1 TCP/IP load balancing WebSphere Application Server for z/OS is designed to work with Sysplex Distributor. Sysplex Distributor is an integral part of z/OS Communications Manager, which offers the ability to load balance incoming socket open requests across different address spaces running on different IP stacks (usually on different LPARs). The routing decision is based on real-time socket status and MVS Quality of Service (QoS) criteria, which provides the benefit of balancing work across different MVS images and enhances scalability and failover in a z/OS Parallel Sysplex. The benefits of Sysplex Distributor described in “Sysplex Distributor” on page 103 apply well to a WebSphere z/OS configuration. 4.6.2 Clustering Clustering is a technique in WebSphere Application Server that allows individual application servers to be combined into a logical whole. When two or more application servers are combined into a cluster, the Deployment Manager administrative interface sees that cluster as an entity into which applications are installed. In other words, the servers inside the cluster are no longer viewed as individual, but rather part of a group (or cluster). An application installed into the cluster is actually installed into all the servers in the cluster. Clusters must stay within a cell, but a cell can span different systems within a parallel sysplex, thus taking advantage of the scalability and availability capabilities offered by the parallel sysplex. Chapter 4. Connecting from WebSphere Application Server for z/OS 149 Figure 4-23 shows the use Sysplex Distributor and a J2EE server cluster. Note that TCP/IP port sharing is not normally required with this configuration. LPAR -1 node 1 Daemon Controller region TCP/IP z/OS sysplex Servant EXCI regions CICS router region 1 LPAR - 3 DPL Sysplex Distributor J2EE server cluster Controller region Servant EXCI regions Daemon LPAR -2 CICS program DPL CICS AORs CICS router region 2 node 2 Figure 4-23 High scalability and availability configuration with WebSphere on z/OS 4.6.3 Dynamic program link Figure 4-23 shows the recommended high availability configuration when using the CICS TG with WebSphere Application Server for z/OS. A given WebSphere servant region connects to a single CICS region. Such a CICS region is referred to as a router region. This region does not run the CICS application itself but instead routes the program link request dynamically to a CICS AOR. It is possible to use the EXCI user exit DFHXCURM (see 3.4.3, “EXCI user-replaceable module: DFHXCURM” on page 100) to re-route EXCI requests to an alternative (or backup) CICS router region in case of a failure of the target CICS router region. Restriction: For local connections, the CICS region must run on the same MVS image as the WebSphere servant region. However, you can define the target CICS program to be remote so that the program link is routed to another CICS region that is running on a different MVS image. CICSPLex SM provides a dynamic routing program which supports the dynamic routing of LINK commands. This program provides the ability for applications invoked by ECI requests to be routed dynamically across a CICSplex. 150 CICS Transaction Gateway for z/OS Version 6.1 4.6.4 Other configuration considerations When creating a configuration similar to that shown in Figure 4-23, there are configuration considerations in addition to those described in 4.2, “WebSphere Application Server configuration” on page 124, including: Installing the resource adapter You must install the CICS ECI resource adapter on each node in the cell to ensure that the implementation code (JAR files and binaries) of the resource adapter is available to the WebSphere servant regions running on each node. Creating connection factories You should create a connection factory for the LPAR specific CICS router region in each of the two WebSphere nodes (node 1 and node 2 in Figure 4-23). Each connection factory is identical except for the ServerName, which determines the APPLID of the CICS region to which requests are sent. On node 1, you need to specify the APPLID of CICS router 1 as the ServerName. On node 2, you need to specify the APPLID of CICS router 2 as the ServerName. The connection factories should be given the same JNDI name, for example, eis/CICS Router. Deploying the J2EE application When you deploy the application, you deploy to the J2EE server cluster rather than to a specific J2EE server. When you map resource references to resources (as shown in Figure 4-11 on page 136), you should map the resource reference to the connection factory with JNDI name eis/CICS Router. 4.7 Problem determination In this section, we document common connection problems and information that we learned while configuring our connection test scenario using WebSphere Application Server for z/OS. For general problem determination guidance see 2.9, “Problem determination” on page 60. CTG9630E You might get the following message for an ECI request after installing the CICS ECI resource adapter: CTG9630E IOException occurred in communication with CICS Check the WebSphere servant region’s job log for other errors. Example 4-1 shows message CTG9630E and CTG6686E. Chapter 4. Connecting from WebSphere Application Server for z/OS 151 Example 4-1 Edited WebSphere servant region job log showing CTG9630E and CTG6686E javax.resource.spi.CommException: CTG9630E IOException occurred in communication with CICS at com.ibm.connector2.cics.ECIManagedConnection.call(ECIManagedConnection.java:1295) at com.ibm.connector2.cics.ECIConnection.call(ECIConnection.java:122) . Caused by: java.io.IOException: CTG6686E Unable to initialize JNI library. [java.lang.UnsatisfiedLinkError Can't find library ctgjni (libctgjni.so) in sun.boot.library.path or java.library.path sun.boot.library.path=/CITG/CellCITGS/AppServer/java/bin This error is caused by the failure to specify correctly the path to the JNI files (/usr/lpp/cicstg/ctg610/bin) in the WebSphere Application Server Resource Adapters → ECIResourceAdapter Native Path field. 4.7.1 Tracing In this section, we outline the main traces that you can enable within the WebSphere environment to diagnose problems with connections from WebSphere Application Server on z/OS. Java client trace CICS TG Java client tracing can be set programmatically. See “Tracing” on page 118 for information about enabling Java client trace. J2EE resource adapter trace Resource adapter tracing is used to control tracing of the flow of execution through the CCI classes. See 3.8.3, “Tracing” on page 118 for information about enabling J2EE resource adapter trace. JNI trace In order to enable JNI trace, specify the CTG_JNI_TRACE and CTG_JNI_TRACE_ON environment variables using the WebSphere Administrative Console. To specify these variables, do the following: 1. Log in to the Administrative Console and click Environment → WebSphere Variables. 2. Click New. 3. Specify the trace environment variable in the name field and the setting in the Value field. Figure 4-24 shows how we enabled JNI trace for WebSphere Application Server for z/OS. 152 CICS Transaction Gateway for z/OS Version 6.1 Figure 4-24 Administrative Console - enabling CICS TG JNI trace 4. We need to connect user ID CITGRS1S to the CTGV61 group (group ID 7100) that we created in 2.3.2, “Setting permissions” on page 35 so that the WebSphere servant region can create the JNI log file. Example 4-2 shows the initial JNI trace entries that are written as the result of the execution of the CTGTesterCCI application. Example 4-2 JNI trace CTG6813I CTG6880I CTG6895I CTG6894I CTG6893I Running under WebSphere z/OS. Specific pipe used. NETNAME=CITGRS1 . IEANTRT successful. Returned token value 0xfa. GetMaxLogon returned 0, maxlogon was set to 250. EXCI pipe reuse set to ONE per TCB. If CTG_JNI_TRACE_ON=YES is defined as a WebSphere environment variable but CTG_JNI_TRACE is not set, then JNI messages are written to STDERR. Because STDERR for our servant region was defined as SYSOUT=*, this combination of settings means that JNI messages were written to SDSF in our scenario. Note: We found that it was necessary to restart the WebSphere application server in order to pick up changes to environment variables. Chapter 4. Connecting from WebSphere Application Server for z/OS 153 154 CICS Transaction Gateway for z/OS Version 6.1 Part 3 Part 3 Transaction Management In this part, we describe the transactional capabilities of the CICS Transaction Gateway for z/OS. We provide detailed information about how we configured transactions between WebSphere Application Server and CICS. We cover two different topologies: Running WebSphere Application Server on Windows Running WebSphere Application Server on z/OS We discuss the configuration of transactions within WebSphere Application Server, the Gateway daemon, and CICS. © Copyright IBM Corp. 2006. All rights reserved. 155 156 CICS Transaction Gateway for z/OS Version 6.1 5 Chapter 5. Gateway daemon transactions In this chapter, we describe the transactional support that is provided by the Gateway daemon on z/OS. We start with an overview of basic transactional concepts and standards such as two-phase commit and global transactions. Then, we describe the new transactional support that is provided with the CICS TG for z/OS, including support for global transactions (also referred to as XA transactions). We document how we prepared the system and the values that we used in our testing environment. Then, we give details about how we set up and modified the different tiers in our test environment to support transactions. We also explain how we tested the environment. For information about how to enable the Gateway daemon to support XA transactions in a TCP/IP workload management configuration, see Chapter 6, “Gateway daemon transactions with TCP/IP load balancing” on page 225. © Copyright IBM Corp. 2006. All rights reserved. 157 5.1 Preparation In our testing, we used WebSphere Application Server for Windows. For information about transactional support with WebSphere Application Server on z/OS, see Chapter 7, “Transactions with WebSphere Application Server for z/OS” on page 253. First, we tested extended LUWs with our sample CTGTesterCCI application using the cicseci.rar. Then we tested the new XA transaction support using our second sample application CTGTesterCCIXA with the cicseciXA.rar (Figure 5-1). Windows z/OS WebSphere Application Server JSP Global transaction Servlet servlet EJB JDBC Transaction Manager CICS ECI XA JCA resource adapter Gateway daemon TCP CICS A EXCI Resource Manager Datasource RRS DB2 Resource Manager Figure 5-1 Software components: Gateway daemon transaction scenarios Figure 5-1 shows a global transaction spanning updates to multiple resource managers, a CICS region and a DB2 database on Windows. The global transaction is coordinated by the transaction manager, WebSphere Application Server. RRS is used to coordinate the resource updates on z/OS. Note: In our global transaction configuration, we document the setup steps for transactions, which update recoverable resources in two CICS regions. However, we do not document the setup or testing of DB2. 158 CICS Transaction Gateway for z/OS Version 6.1 5.1.1 Software checklist Table 5-1 shows the levels of software that we used for this configuration. Table 5-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS 1.6 Windows 2000 SP4 CICS Transaction Gateway for z/OS V6.1 IBM WebSphere Application Server Network Deployment V6.0.2 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. CICS Transaction Server for z/OS V3.1 5.2 Transactions - what are they In this section, we provide a brief overview of the following fundamental transactional concepts and standards: CICS transactions, tasks, and syncpoints Distributed transactions, including the two-phase commit process 5.2.1 CICS transactions, tasks, and syncpoints We refer to a CICS transaction as the work that is initiated in a CICS region and that runs as a CICS task under a four character transaction ID (tranid). At task initiation CICS implicitly starts a unit-of-work (UOW) for all CICS transactions. This is usually the initial boundary of the transactional work to be undertaken. All updates to recoverable resources or requests to other transactional systems are now part of this unit-of-work, until either a synchronization point (syncpoint) is reached within the CICS program or the CICS transaction finishes and the task terminates (Figure 5-2). Transaction initiation Load initial program Start of task EXEC CICS SYNCPOINT End of task Unit-of-work 2 Unit-of-work 1 Figure 5-2 CICS synchronization points Chapter 5. Gateway daemon transactions 159 In certain circumstances, such as though an inter-system distributed program link (DPL) request is used, the CICS transaction that is linked to can be co-coordinated by a remote CICS system. In this instance, the mirror task remains suspended until the end of the transaction, which is referred to as a long-running mirror task (Figure 5-3). 2nd ECI request 1st ECI request CICS region Commit Commit response Start of mirror task termination of mirror task Mirror transaction Mirror task issues SYNCPOINT Mirror task suspends extended unit-of-work Figure 5-3 Link with long-running mirror task Additionally, the converse situation is also possible, where the invoked CICS transaction runs in a separate transactional context to that of the invoking application. This situation is referred to as running with sync-on-return, which refers to the fact that the mirror transaction in CICS issues a syncpoint on returning control to the calling application (Figure 5-4). The use of a sync-on-return type link also allows the called CICS program to issue EXEC CICS syncpoint commands, because it is not sub-ordinate to another transaction manager. LINK with SYNCONRETURN CICS region Mirror transaction Return to mirror transaction Link to user program unit-of-work Figure 5-4 Link with SYNCONRETURN 160 termination of mirror task Start of mirror task CICS Transaction Gateway for z/OS Version 6.1 EXEC CICS SYNCPOINT 5.2.2 Distributed transactions Within a distributed transactional system each distributed system is either referred to as a resource manager or a transaction manager. The transaction manager controls the outcome of the transaction and is responsible for the recovery of it’s resources; it has to implement a logging mechanism in order to be able to coordinate multiple resource managers. The resource managers control access to recoverable resources, and as such have to implement the necessary network flows and logging procedures to provide transactional coordination. In order to enable distributed systems to send and receive transactional requests the partners must send and receive the requests using a common mechanism. This mechanism is known as the two-phase commit process. Two-phase commit This is an architected set of flows that transaction managers use to ensure all resource managers in a transaction can be reliably coordinated, irrespective of any failure. It is implemented by all transactional protocols and the fundamental concepts are essentially the same. Figure 5-5 summarizes the flows according to the XA specification. Other protocols, such as CICS or LU6.2, might use different terminology and variants on the flows. (For further details about CICS syncpoint flows, refer to the CICS Intercommunication Guide, SC34-6448.) Stage 2 - Commit Stage 1 - Prepare Global Transaction 1 Pr e r pa Resource Manager e r pa Pr ep a re 3 Co ed e Pr P re 2 pa re Transaction Manager Global Transaction d m om Transaction Manager Resource Manager m Resource Manager it Co m C Co 4 mm it mm it t e it t ed Resource Manager d Figure 5-5 Two-phase commit In the first phase (or Stage 1), the transaction manager asks all the resource managers to prepare to commit recoverable resources. Each resource manager can vote either positively (prepared) or negatively (failed to prepare). If a Chapter 5. Gateway daemon transactions 161 resource manager replies positively, it is then obliged to follow the eventual outcome of the transaction as determined at the next stage. The resource manager is now described as in-doubt, because it has delegated eventual transaction control to the transaction manager. If a resource manager replies negatively, the transaction manager assumes a backout is needed and the transaction will be backed out by all resource managers. In Stage 2, providing all the resource managers voted positively, the transaction manager replies to each resource manager with a commit flow. Upon receipt of the commit flow, the resource manager finalizes updates to recoverable resources, and releases any locks held on the resources. The resource manager then responds with a final committed flow, which indicates to the transaction manager that it is no longer in-doubt. Although the two-phase commit process is usually a prerequisite to distributed transactional support, there are certain instances where a single-phase commit process can be sufficient. This is referred to as last resource optimization, and is implemented by a variety of transaction managers. It essentially allows the commit decision to be delegated to the one-phase commit resource, allowing the one-phase commit to participate in a global transaction with any number of two-phase commit capable resources (Figure 5-6). Stage 1 - Prepare Stage 2 - Commit Global Transaction Resource Manager (one phase) Resource Manager (one phase) Co mm 3 it Global Transaction 1 Pr Transaction Manager a ep Resource Manager re P re 4 m it Co m Transaction Manager 2 pa Resource Manager Co re Resource Manager 5 mm it Resource Manager Figure 5-6 Last resource optimization At transaction commit, the transaction manager first prepares the two-phase commit resource managers and, if this is successful, the one-phase commit-resource is then called to commit. The two-phase commit resources are 162 CICS Transaction Gateway for z/OS Version 6.1 then committed or rolled back depending on the response of the one-phase commit resource, effectively delegating transaction coordination to the one-phase commit resource. Unlike for a two-phase commit resource, there is no recovery from a communication failure with a one-phase commit resource. Such a communication failure during commit of the one-phase commit resource introduces the risk of a mixed outcome to the transaction. The two-phase commit resources are by default rolled back, but the outcome of the one-phase commit resource is unknown; it could have committed or rolled back. Applications must therefore be configured to accept the additional risk of such heuristic outcomes. Last resource optimization is implemented within WebSphere Application Server V6 as last participant support, and within CICS and APPC flows as last agent optimization. However, applications that are deployed to exploit last participant support are subject to an increased risk of a mixed outcome in a global transaction if the one-phase commit resource fails during the commit processing. Note: You specify whether an application accepts the possibility of a mixed outcome occurring in a global transaction that contains a one-phase resource, when you deploy the application in WebSphere Application Server. 5.3 Resource Recovery Services Resource Recovery Services (RRS), a component of z/OS, provides a global syncpoint manager that any resource manager on z/OS can exploit. It enables transactions to update recoverable resources managed by many resource managers. 5.3.1 How RRS works RRS is an exit manager and does not itself perform tasks such as commit changes to a DB2 database, or backout changes made by a CICS program, but rather it provides a framework for resource managers such as DB2, CICS or IMS to be coordinated by an independent Transaction Manager, such as WebSphere Application Server (see Figure 5-1 on page 158). Technically, by RRS we really mean Recoverable Resource Management Services (RRMS). RRMS consists of three services: Registration services Context Services Resource Recovery Services (RRS) Chapter 5. Gateway daemon transactions 163 These three services are provided by the RRS MVS address space. Registration services The first thing a resource manager must do in order to start using RRS is to identify itself to RRS and supply RRS with its exit routines, for example, prepare, commit and backout routines. To do so, a resource manager uses RRMS registration services. Context services Context services provide an exit manager allowing resource managers to track the changes made by a given unit of work. A context represents the environment that the application is running in and its work request. A work manager (the gateway daemon, for example) can create a context and have an application run under it. A work manager can create more than one context, but at any point in time there can only be one active context for a single task. If the work manager needs to change the context, it has to explicitly switch to a different context. RRS A resource manager registers various exit routines with RRS, which are driven at key points in the lifetime of a transaction, after the resource manager has indicated an interest in that particular transaction. An application can request RRS to commit a Unit of Recovery (UR), for which RRS will then invoke the commit exits of all resource managers involved in the UR. RRS provides a set of ISPF operator panels for displaying and updating RRS information such as units of recovery (URs). Figure 5-7 shows the RRS primary options panel. RRS Select an option and press ENTER: 1 2 3 4 5 6 Browse an RRS log stream Display/Update RRS related Resource Manager information Display/Update RRS Unit of Recovery information Display/Update RRS related Work Manager information Display/Update RRS UR selection criteria profiles Display RRS-related system information Figure 5-7 ISPF - RRS primary options panel We show examples of how to use the RRS ISPF panels in 5.6.5, “Testing the extended LUW configuration” on page 182 and 5.7.5, “Testing the XA transaction configuration” on page 201. 164 CICS Transaction Gateway for z/OS Version 6.1 For general information about RRMS and RRS, refer to Systems Programmer's Guide to Resource Recovery Services (RRS), SG24-6980. Note: Because different parts of a transaction are processed by different tiers of the infrastructure, the transaction is identified in different ways. See “Identifying the transaction” on page 221 for a list of the different transaction identifiers that are used by the different software components that we used in our tests. 5.3.2 CICS TS and RRS During initialization, CICS TS registers exits with RRS as a resource manager and provides RRS with details of CICS exit routines to be driven for the range of transaction life cycle events (prepare, commit, backout, and so forth). Transactional EXCI The Transactional EXCI support that is provided with CICS TS uses RRS to allow multiple EXCI calls to be part of one extended unit-of-work. An EXCI client program can send several DPL requests to the same CICS region (or to different CICS regions within the same MVS image), as well as make calls to other resource managers and have all requests syncpointed by RRS. Note: Transactional EXCI is a pre-requisite for CICS TG transactional support, for both extended LUWs and XA transactions. A Transactional EXCI client program must first establish the transaction context which will tie the DPL requests together. Each DPL request includes a token representing this context, allowing CICS to execute the target program of the DPL request under the correct context. After completing the last DPL request of the sequence, the application calls RRS to drive either commit or rollback of all updates made under the transaction context. Restriction: Transactional EXCI that is provided by CICS TS is restricted to operating within a single MVS image (or LPAR). A Transactional EXCI client cannot link to CICS regions on other MVS images or LPARs within the same sysplex. ABENDBKOUT option in DFHXCOPT By default, CICS does not tell RRS to rollback updates to recoverable resources following an unhandled abend in a Transactional EXCI program (the Chapter 5. Gateway daemon transactions 165 long-running mirror task). This means that a transaction abend, unlike a CICS failure or a failure of the EXCI client, does not trigger an automatic rollback of the RRS UR. The EXCI options table DFHXCOPT has been amended recently to include the new option ABENDBKOUT={NO|YES}, which you can use to control transactional behavior following a CICS transaction abend. This option specifies whether a CICS transaction abend should trigger an automatic rollback of the RRS UR and backout all updates made by the long-running mirror. Tip: Specify ABENDBKOUT=YES in DFHXCOPT if you want to back out updates made by a long-running mirror following a transaction abend. This option is introduced as APAR PK17426 in CICS TS V3.1 and APAR PK17427 in CICS TS V2.2 and V2.3. 5.4 Transactional support in the CICS TG This section outlines the transactional support that is provided by the CICS TG for z/OS V6.1. CICS TG for z/OS operates as a long-running transactional EXCI client. In versions of CICS TG for z/OS prior to CICS TG V6.1, the Gateway daemon has a passive work manager role in relation to RRS using only the registration and context services from RRMS. This limits the Gateway daemon to providing only one-phase commit support. CICS TG 6.1 for z/OS implements the two-phase commit protocol, additionally using Resource Recovery Services (RRS) and taking on a more active role as a syncpoint coordinator. This implementation allows ECI requests through the Gateway daemon on z/OS to fully participate in a global transaction (or XA transaction). Inclusion of support for XA transactions is configurable and is disabled by default in order to ease migration from earlier releases. 5.4.1 Extended logical units of work The ECI and Transactional EXCI communications protocols used by the CICS TG extend the CICS DPL concept (see 5.2.1, “CICS transactions, tasks, and syncpoints” on page 159) by allowing the invoking application to initiate and co-ordinate the CICS unit-of-work. 166 CICS Transaction Gateway for z/OS Version 6.1 Extended LUW support is provided with the base CICS TG ECIRequest class. ECI request calls can be: Nonextended The called CICS program, not the client Java application, controls whether recoverable resources are committed or backed out. Each program link call corresponds to one CICS task and one or more CICS UOWs. Extended The client Java application controls whether recoverable resources are committed or rolled back. Multiple calls are possible, allowing a LUW to be extended across successive ECI requests to the same CICS region as shown in Figure 5-3 on page 160. An extended LUW causes the CICS mirror task to be long running so that it does not terminate on return from the initial call. Multiple program link calls correspond to one CICS task and one CICS UOW. Important: The single MVS image restriction that is imposed by transactional EXCI means that extended ECI requests that are received by a specific Gateway daemon can only link to CICS regions running on the same MVS image as the Gateway daemon. 5.4.2 Transactional support in the CICS ECI resource adapters The transactional capabilities of the CICS ECI resource adapters is built upon a combination of the ability of the CICS TG to extend the unit-of-work and the transaction management system contract which exists between the J2EE server and the resource adapter. JCA and transactions The JCA specifies the system contract for transaction management (as well as other system contracts such as connection management and security management) that exists between the J2EE server (in our case WebSphere Application Server) and the enterprise information system (in our case CICS). Chapter 5. Gateway daemon transactions 167 A resource adapter is required to implement one of the following transaction management contracts, as defined in the resource adapter's deployment descriptor (ra.xml): XATransaction A resource adapter that can participate fully in a two-phase commit process and which can influence the outcome of the global transaction. LocalTransaction A resource adapter that can participate in transactions that are local to the resource manager, but cannot participate in two-phase commit transactions (other than as an only agent or a last participant). NoTransaction A resource adapter with no transactional properties, this can participate in a transactional context but is not influenced by, and has no effect upon, the outcome of the transaction. The CICS TG for z/OS V6.1 provides two CICS ECI resource adapters: cicseci.rar The CICS ECI resource adapter has the same transactional behavior as the cicseci.rar that is provided with CICS TG V6. It supports the LocalTransaction interface. cicseciXA.rar The new CICS ECI XA resource adapter provided with CICS TG V6.1 provides full support for global transactions by implementing XAResource interface. CICS ECI resource adapter (cicseci.rar) The WebSphere transaction manager enables a single one-phase capable resource manager such as the CICS ECI resource adapter to be used within a global transaction in the absence of any other transactional resource managers. When you use the CICS ECI resource adapter, therefore, you can commit multiple JCA requests to the same CICS region as one unit-of-work if the only recoverable resources updated within the transaction are accessed via a single CICS ECI connection factory (see 3.2.2, “Creating connection factories” on page 82 for full details on creating CICS ECI connection factories). You can commit multiple JCA requests in this way by setting the EJB transaction attribute to specify the requests as being part of the same transaction (see “EJB transaction attribute” on page 172). To commit a JCA request to a CICS region as part of a global transaction with other two-phase commit resource managers, it is necessary to use the Last Participant Support function of WebSphere Application Server V6. Using the 168 CICS Transaction Gateway for z/OS Version 6.1 Last Participant Support function is only possible if the global transaction involves a single one-phase capable resource, and involves JCA requests to a single CICS ECI connection factory. When using the CICS ECI resource adapter with WebSphere Application Server on a distributed platform it is not possible to commit multiple JCA requests to different CICS ECI connection factories as part of the same global transaction with other two-phase commit resource managers. Note: When installed into WebSphere Application Server for z/OS, cicseci.rar supports global transactions (see 7.3.1, “Transactional support in the CICS ECI resource adapters” on page 256). CICS ECI XA resource adapter (cicseciXA.rar) The WebSphere Application Server transaction manager can provide coordination, within a transaction, for any number of two-phase capable resource managers (for example, the CICS ECI XA resource adapter and a JDBC resource). The XA transaction support that is provided with CICS TG for z/OS V6.1 enables CICS TS programs to participate fully (with other two-phase commit capable resource managers) in a global transaction that is initiated in a distributed J2EE V1.4 application server, such as WebSphere Application Server V6. Thus, your application can commit JCA requests to multiple CICS regions that are accessed via different connection factories as part of a global transaction with other two-phase commit resource managers such as DB2 and IMS. Important: The single MVS image restriction that is imposed by transactional EXCI means that XA transaction requests that are received by a specific Gateway daemon can only link to CICS regions running on the same MVS image as the Gateway daemon. However, a J2EE application can link to CICS regions on separate MVS images (or sysplexes) within a single global transaction by using multiple connection factories that define connections to multiple Gateway daemons running on different MVS images. Which resource adapter to use Both the ECI and ECI XA resource adapters can be deployed into the same WebSphere Application Server, provided that they originate from the same release of the CICS TG. Thus, you can install both resource adapters, define connection factories for each one, and then map application resource references Chapter 5. Gateway daemon transactions 169 to these connection factories based on whether XA transaction support is required. Important: Running mixed levels of CICS TG resource adapters in the same application server is unsupported and can lead to unpredictable results. For example, you should not install the CICS TG V6.1 ECI XA resource adapter into a WebSphere Application Server along side either of the CICS TG V5.1 or CICS TG V6.0 ECI resource adapters. The following list gives guidelines on which resource adapter to use. If your J2EE application does not use transactions — for example it is an EJB deployed with the transaction attribute NotSupported or Never (see Example 5-1 on page 172) — then you can use either of the CICS ECI resource adapters. If your application needs to commit multiple JCA requests to the same CICS region as one unit-of-work, you should use the CICS ECI resource adapter (cicseci.rar) in order to avoid the overhead of unnecessary XA flows between the WebSphere Application Server and CICS TG. See 5.6, “Configuring for extended LUWs” on page 174 for information about how to configure your environment for extended LUW support. If your application needs to commit JCA requests to multiple CICS regions, accessed via different connection factories, as part of a global transaction with other two-phase commit resource managers, you must use the CICS ECI XA resource adapter (cicseciXA.rar). Important: You should be aware that due to the processing overhead associated with XA transactions, network, and CPU utilization is significantly higher when you use the XA transactional support of the CICS ECI XA resource adapter. See 5.7, “Configuring for XA transactions” on page 193 for information about how to configure your environment for XA transaction support. If your application requires to commit JCA requests to a single CICS region as part of a global transaction with other two-phase commit resource managers: – You should use the CICS ECI XA resource adapter (cicseciXA.rar) if data integrity your primary concern, because last participant support gives an increased risk of a mixed outcome in a global transaction if CICS fails during the commit processing. 170 CICS Transaction Gateway for z/OS Version 6.1 – You could use the last participant support function of WebSphere Application Server V6 and the CICS ECI resource adapter (cicseci.rar) if performance is your primary concern. This avoids the overhead of XA flows between the WebSphere Application Server and CICS TG. 5.5 Transactions in WebSphere Application Server The way that WebSphere applications use transactions depends on the type of application component, as follows: In the EJB container: – A session bean can either use container-managed transactions (where the bean delegates management of transactions to the container) or bean-managed transactions (where the bean manages transactions itself). – Entity beans use container-managed transactions. In the Web container: The Web container has limited transactional support, as its principal function is for servlet and JSP™ components which are primarily concerned with presentation logic. Web components (for example, servlets) can only use bean-managed transactions. Note: We do not discuss Web container transactions in our transaction test scenarios, because it is best practice to use EJBs if you have a need for transactional integration between WebSphere and CICS. Transactions in the EJB Container The EJB container in WebSphere Application Server is suited ideally to the deployment of transactional components and provides support for both container managed transactions and bean managed transactions. Session beans and message-driven beans can employ either type. Entity beans are restricted to container-managed transactions only. Beans using bean-managed transactions are responsible for transaction demarcation and must begin and end a transaction explicitly. Container-managed transactions are the preferred mechanism because this delegates transactional control to the Application Server, which allows the application developer to concentrate on developing the business logic while allowing the transactional properties of the application to be decided upon deployment. The key to transactional control with container-managed transactions is the EJB transaction attribute. Chapter 5. Gateway daemon transactions 171 Note: We do not cover bean-managed transactions in our transaction test scenarios because we recommend that you leave transaction management to the EJB container. EJB transaction attribute The transaction attribute is set in the assembly descriptor section of the EJB deployment descriptor (that is, the file ejb-jar.xml). This attribute is used by the EJB container to control under which circumstances a global transaction is started when a bean method is invoked. The transaction attribute appears in the <container-transaction> section and is specified with the <trans-attribute> tag. For example, the following XML specifies that the execute() method on the CTGTesterCCI bean has the transaction attribute of NotSupported. Example 5-1 EJB Assembly descriptor transaction attribute <assembly-descriptor> <container-transaction> <method> <ejb-name>CTGTesterCCI</ejb-name> <method-intf>Remote</method-intf> <method-name>execute</method-name> </method> <trans-attribute>NotSupported</trans-attribute> </container-transaction> </assembly-descriptor> Table 5-2 lists the possible values for the transaction attribute. Table 5-2 EJB transaction attribute meaning 172 Transaction Attribute Meaning NotSupported Bean method cannot execute within context of an global transaction Required Bean method must execute within context of an global transaction. RequiresNew Bean method must execute within context of a new global transaction. Supports Bean method can execute within the context of a global transaction, depending on whether the caller supplies a transaction context, or not. CICS Transaction Gateway for z/OS Version 6.1 Transaction Attribute Meaning Mandatory Bean method must execute within the context of the client's global transaction. If the caller does not supply a context (in other words there is no global transaction alive), then the execute fails with an exception. Never Bean method transaction. must not be invoked within the context of a global Note: The default EJB transaction attribute is Required. Impact of EJB transaction attribute Table 5-3 shows the resulting type of JCA request behavior, and resulting type of CICS mirror task, for each of the different transaction attributes and depending on which CICS ECI resource adapter is used. Table 5-3 Impact of EJB attribute Transaction Attribute Resulting JCA request cicseci.rar NotSupported CICS mirror task cicseciXA.rar Non-extended LUW SYNCONRETURN Required Extended LUW XA transaction Long running RequiresNew Extended LUW XA transaction Long running Supports Non-extended or extended LUW Non-extended or XA transaction SYNCONRETURN or long running Mandatory Extended LUW or Exception thrown XA transaction or Exception thrown Long running Never Non-extended LUW SYNCONRETURN Chapter 5. Gateway daemon transactions 173 5.6 Configuring for extended LUWs In this section, we show how we configured our setup to support extended LUWs using the CICS ECI resource adapter (cicseci.rar). We needed to consider the following configuration and setup tasks: Configuring CICS Checking that our CICS TS region was enabled to support Transactional EXCI. CICS TG for z/OS Checking that our Gateway daemon was configured correctly to register with RRS. WebSphere Application Server – Updating the transaction attribute for our sample application CTGTesterCCI – Re-deploying the application and mapping the resource reference to a CICS ECI connection factory – Configuring WebSphere transaction logging We do not discuss installation of the CICS TG ECI resource adapter here or give information about defining connection factories, because these topics are covered in detail in 3.2, “Configuring WebSphere Application Server” on page 80. 5.6.1 Definitions checklist Table 5-4 lists the definitions that we used to configure this test scenario. Table 5-4 Definitions checklist Value Gateway daemon CICS TS hostname tx1.mop.ibm.com - IP address 9.100.193.122 - TCP/IP port 13101 - Job name CITGR1G CITGR1CI APPLID NETNAME A6POTGR1 CITGR1G CONNECTION RRM Name 174 GR1G CTG.RRM.CITGR1G.IBM.UA CICS Transaction Gateway for z/OS Version 6.1 5.6.2 Configuring CICS First, we checked that the RRS address space was actively running on the LPAR. We then verified that our CICS region was registered correctly with RRS (the CICS system initialization parameter RRMS=YES must be specified). During CICS initialization, we see the CICS log messages shown in Example 5-2, which indicate that CICS has registered with RRS successfully. Example 5-2 CICS TS RRS initialization log messages DFHRX0100I A6POTGR1 RX domain initialization has started. DFHRX0104I A6POTGR1 The Resource Recovery Services (RRS) exit manager ATR.EXITMGR.IBM is now available. DFHRX0101I A6POTGR1 RX domain initialization has ended. From a CICS terminal, we check that RRS is available for use by issuing the CEMT INQUIRE RRMS command. The response should indicate a status of open, as shown in Figure 5-8. I RRMS STATUS: RESULTS Rrm Ope SYSID=RGT1 APPLID=A6POTGR1 RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR TIME: 17.00.02 DATE: 09.30.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-8 CEMT INQUIRE RRMS 5.6.3 Configuring CICS TG In this test scenario, we use the CITGR1G Gateway daemon as configured in 3.3, “Configuring the Gateway daemon” on page 91, a single Gateway daemon with xasupport=off specified. Note: Running the CICS TG V6.1 with xasupport=off in the Gateway daemon configuration file provides the same transactional capabilities and behavior as CICS TG V6.0. RRM Name considerations The environment variable CTG_RRMNAME is used to specify the name that the Gateway daemon uses when registering with RRS. The rules that apply when Chapter 5. Gateway daemon transactions 175 specifying this name are described in 2.3.4, “Creating an STDENV file” on page 37. When the Gateway daemon is running with the default setting of xasupport=off, it uses only a subset of the RRMS functionality, which does not require it to be authorized. In this configuration, the Gateway daemon does not run as an authorized application and so must use the .UA (unauthorized) suffix when identifying itself via the Registration Services. For this test scenario, we use the same CTG_RRMNAME setting that we defined when we first customized the Gateway daemon (CTG.RRM.CITGR1G.IBM.UA). Important: The value used for CTG_RRMNAME must be a Sysplex-unique value of no more than 32 characters. If the CTG_RRMNAME environment variable is found to be undefined during initialization, the Gateway daemon generates a Sysplex-unique name to use for RRS registration. 5.6.4 Configuring WebSphere In 3.2.3, “Deploying the sample J2EE application” on page 89, we deployed the CTGTesterCCI application into WebSphere Application Server to use container-managed transactions but with the transaction attribute of NotSupported. This deployment gives us the behavior of a non-extended mode ECI request and a SYNCONRETURN DPL in CICS. CICS is acting as both resource manager and transaction manager. We now change from using CICS TS as the transaction manager to using WebSphere Application Server as the transaction manager by updating the transaction attribute on the EJB from NotSupported to Required. The combination of the Required transaction attribute in the EJB deployment descriptor and the use of the ECI resource adapter results in a local transaction in WebSphere Application Server, an extended mode ECI request, and a long-running mirror transaction in CICS. The assembly descriptor can be updated using Rational® Application Developer or the WebSphere Application Server Toolkit. Updating the EJB deployment descriptor Within the application EAR file, the EJB archive CTGTesterCCIEJB.jar contains file META-INF/ejb-jar.xml. This deployment descriptor file contains a section called the assembly descriptor, which defines the transactional attributes for methods provided by the EJB. 176 CICS Transaction Gateway for z/OS Version 6.1 We changed the transaction attribute for the CTGTesterCCI EJB using our development tool Rational Application Developer (Figure 5-9). Figure 5-9 RAD - Updating EJB trans-attribute for CTGTesterCCI After having used RAD to modify the transaction attribute, we exported the application to create an updated EAR file CTGTesterCCI.ear. Example 5-3 is an extract from the CTGTesterCCI EJB deployment descriptor source, showing again the remote EJB method execute updated to use a trans-attribute of Required. Example 5-3 Updated trans-attribute from CTGTesterCCI <assembly-descriptor> <container-transaction> <method> <ejb-name>CTGTesterCCI</ejb-name> <method-intf>Remote</method-intf> <method-name>execute</method-name> </method> <trans-attribute>Required</trans-attribute> </container-transaction> </assembly-descriptor> Chapter 5. Gateway daemon transactions 177 Dealing with CICS transaction abends CICS cannot tell RRS to roll back updates to recoverable resources following an abend of the long-running mirror task. This means that, by default, if the first JCA request of an extended LUW succeeds but a subsequent request abends (perhaps because of a file I/O failure), then the updates made by the first JCA request will be committed together with any updates made by the second request, even if some work did not complete successfully. There are two ways of avoiding such an inconsistency resulting from a CICS transaction abend: The J2EE application can abandon the current local transaction by calling setRollbackOnly() on the current context. In the event of a transaction abend, a CICSTxnAbendException is thrown by the CICS ECI resource adapter and the decision to rollback or commit resources updated by the local transaction can be taken by the J2EE application. The option ABENDBKOUT=YES can be set in the DFHXCOPT table (see “ABENDBKOUT option in DFHXCOPT” on page 165). In the event of a transaction abend, the local transaction is rolled back automatically at the end of the EJB method call. Note: At the time of writing this book, we were unable to test the ABENDBKOUT option. We, therefore, chose to include transaction abend rollback logic in our sample application. We modified the CTGTesterCCI application to issue a setRollbackOnly() on the EJB session context in the catch block for the ECIInteraction.execute() call (see Example A-1 on page 368). This ensures that the CICS abend is propagated onto WebSphere Application Server as the transaction manager and the local transaction (if one exists) is rolled back. Re-deploying the application Using the WebSphere Application Server Administrative Console, we navigated to the Enterprise Applications page, selected CTGTesterCCI and then clicked Update. The next panel requested that we specify the type of update to perform (Figure 5-10). For simplicity, we updated CTGTesterCCI as a complete replacement by selecting Full application and clicked Browse to select the new EAR file. 178 CICS Transaction Gateway for z/OS Version 6.1 Figure 5-10 Updating the Enterprise Application The subsequent steps for completing the deployment of the EAR file are common to those that are described in 3.2.3, “Deploying the sample J2EE application” on page 89. On the Map resource references to resources panel, we were careful to make sure that the Reference Binding ECI was bound to the JNDI name for our CICS ECI connection factory. After completing the sequence of deployment panels, WebSphere Application Server stopped the CTGTesterCCI application, installed the new version, and then restarted the application. Verifying the installed Application transaction attribute To verify that we had updated the application to be transactional, we looked at the newly installed EJB Deployment Descriptor. Using the WebSphere Administrative Console, we navigated to Enterprise Applications → CTGTesterCCI → EJB Modules → CTGTesterCCIEJB.jar → Deployment Descriptor and found the trans-attribute for the CTGTesterCCI EJB Remote execute method had been updated correctly from NotSupported to Required (Figure 5-11). Chapter 5. Gateway daemon transactions 179 Figure 5-11 Transaction attribute of Required Configuring WebSphere transaction logging WebSphere Application Server stores transaction information in a transaction log. The default location for the log is in the WebSphere install directory <install_root>/tranlog/cell_name/node_name/server_name. The WebSphere transaction log settings can be configured using the WebSphere Administrative Console by following the links Application servers → server1 → Container Service → Transaction Service. 180 CICS Transaction Gateway for z/OS Version 6.1 In the configuration panel, you can set the transaction log directory using the following format: directory_name;file_size Where: directory_name is the name of the transaction log directory. file_size is the size specified in bytes. The nK or nM suffix can be used to indicate kilobytes or megabytes. If you do not specify a file size value, the default value of 1M is used. We did not specify a transaction log directory. By default, WebSphere Application Server used the directory as shown in Example 5-4. Example 5-4 WebSphere transaction log location C:\Program Files\IBM\WebSphere\AppServer/profiles/AppSrv01\tranlog\Cam21-Pc11No de01Cell\Cam21-Pc11Node01\server1\transaction In a high transaction load, excessive transaction logging can slow down performance of the application server due to its dependency on the operating system and the underlying storage systems. To achieve better performance, you can designate a new directory for the log files on a separate, physically larger storage system. Note: You can disable transaction logging by defining a log size of zero (0). Specify “;0” as the transaction log name. In the properties page for the Transaction Service, you can optionally change other transaction-related configuration properties. We changed the value for Total transaction lifetime timeout (default 120 seconds). This value sets the number of seconds a transaction can remain inactive before it is ended by the Transaction Service. A value of zero (0) indicates that there is no timeout limit. Because some of our test cases sometimes involve requests that suspend in CICS, we increased this value to 1200 seconds. Chapter 5. Gateway daemon transactions 181 5.6.5 Testing the extended LUW configuration In this section, we detail how we tested an extended LUW using our sample J2EE application CTGTesterCCI. Figure 5-12 shows the basic outline of our environment. We used this configuration for two scenarios: Single Extended LUW with commit Extended LUW workload with commit and rollback The Single Extended LUW test repeats the scenario that is described in 3.7.1, “Single Gateway daemon connectivity tests” on page 107 but highlights the differences, in transactional terms, that are attributed to the modified trans-attribute in the EJB assembly descriptor. The Extended LUW workload builds upon the previous scenario, demonstrating a sequence of requests that lead to both committed and backed-out updates of a recoverable resource. For this test, we show how the status of WebSphere transactions can be monitored using the Tivoli Performance Viewer. Windows z/OS WebSphere Application Server V6 CITGR1G HTML JSP Gateway daemon Servlet CICS TG V6.1 CTGTesterCCI Transaction Required CCI ECI_Extend CICS ECI resource adapter Cam21-pc11 13101 JNI EXCI CICS Registration and Context Services RRS Long Running Mirror ECIT CITGR1CI tx1.mop.ibm.com Figure 5-12 Extended LUW configuration 182 CICS Transaction Gateway for z/OS Version 6.1 To test our configuration we used our test application CTGTesterCCI via its Web browser interface (Figure 5-13). Figure 5-13 CTGTesterCCI multiple iterations in extended LUW During these tests, we specified DW or AW as the COMMAREA input to the ECIPROG program, meaning: DW Write to a recoverable temporary storage queue RECIPROG and then delay for one minute. AW Write to a recoverable temporary storage queue RECIPROG and then abend with abend code ECIP. We used multiple iterations allowing us to demonstrate that multiple invocations of ECIPROG can be made within a local transaction. For further details about our sample application, refer to Appendix A, “Sample applications” on page 361. Chapter 5. Gateway daemon transactions 183 During the course of these scenarios, we monitored the transaction using each of the following : CICS CICS TG RRS ISPF panels WebSphere CEMT command JNI trace output RRS related Work Manager information Tivoli Performance Viewer Single Extended LUW This test scenario used the CTGTesterCCI to perform two iterations, each of which called the same CICS program ECIPROG specifying a COMMAREA of DW. The Web browser appears to be busy for two minutes before receiving the results page, showing the COMMREA returned by the second iteration. At the end of the test, the recoverable TS Queue RECIPROG contained two new entries. CICS CEMT command While the first ECI request was in-flight, we issued the CEMT INQUIRE TASK command. Figure 5-14 shows the ECIT private mirror task running under Task 604, suspended in ICWAIT state. ECIPROG has issued an EXEC CICS DELAY call for 60 seconds. INQUIRE TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000539) Tra(CEMT) Fac(N117) Sus Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDB459523DDD7F0A) Hty(ZCIOWAIT) Tas(0000596) Tra(CEMT) Fac(N185) Sus Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDB5D7858FFC578B) Hty(ZCIOWAIT) Tas(0000604) Tra(ECIT) Fac(11 ) Sus Ter Pri( 001 ) Sta(D ) Use(CITGR1D ) Uow(BDB5DAA6F51C928D) Hty(ICWAIT ) Tas(0000605) Tra(CEMT) Fac(N186) Run Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDB5DABD1425AC4D) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 15.08.25 DATE: 10.04.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-14 CEMT INQUIRE TASK showing the mirror an UOWID 184 CICS Transaction Gateway for z/OS Version 6.1 We then issued the CEMT INQUIRE EXCI command. Figure 5-15 shows CICS Task 604 again and the associated RRS UR identifier BDB5DAA67E5540000000042E01020000. INQUIRE EXCI STATUS: RESULTS Exc(CITGR1G.CITGR1G. - X1 ) Tas(0000604) Uri(BDB5DAA67E5540000000042E01020000) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 15.08.10 DATE: 10.04.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-15 CEMT INQUIRE EXCI showing a UR identifier We see later that the UR identifier shown on CEMT INQUIRE EXCI panel ties up with the active and archived data accessible by the RRS panels is ISPF. The INQUIRE EXCI command provides the information to correlate an active long-running mirror task with the RRS UR identifier associated with the wider transaction. The Exc field highlights the job name and connection name with which the UR identifier is associated. RRS ISPF panels RRS provides a set of ISPF operator panels for displaying and updating RRS information such as Units of Recovery (URs). We selected option 3 from the RRS primary menu panel and the RRS Unit of Recovery Selection panel was displayed. We pressed enter to see a list of URs and we then selected to view the UR with UR identifier BDB5DAA67E5540000000042E01020000 (Figure 5-16). Chapter 5. Gateway daemon transactions 185 RRS Unit of Recovery Details Row 1 to 1 of 1 Commands r-Remove Interest v-View URI Details UR identifier : BDB5DAA67E5540000000042E01020000 Create time : 2005/10/04 13:08:01.680410 Comments : UR state : InFlight UR type : Prot System : X1 Logging Group : X1 SURID : N/A Work Manager Name : CTG.RRM.CITGR1G.IBM.UA Display Work IDs Display IDs formatted Luwid . : Present Eid . . : Not Present Xid . . : Present Expressions of Interest: S RM Name Type Role DFHRXDM.A6POTGR1.IBM Prot Participant ******************************* Bottom of data ******************************** Command ===> F1=Help F2=Split F3=Exit F7=Up F8=Down Scroll ===> PAGE F9=Swap F12=Cancel Figure 5-16 RRS UR identifier details Figure 5-16 shows: That the UR is in-flight That the RRMS work manager is the Gateway daemon with RM name CTG.RRM.CITGR1G.IBM.UA That the CICS region with RM name DFHRXDM.A6POTGR1.IBM is a participant in the UR For most processing, URs are fleeting and you will normally have difficulty in displaying them. However, RRS provides an archive log for all URs so that you can investigate URs after a transaction has completed (see “RRS Archive logs” on page 187). CICS TG JNI trace By default, details of specific transactional requests are not externalized by the CICS TG. It is essentially a passive component between WebSphere Application Server and CICS. Therefore, to uncover evidence of transactions in the Gateway daemon, we turned on JNI trace. Example 5-5 shows the JNI trace for our extended LUW test scenario. The trace has been edited and annotated for clarity purposes and only selected trace points are shown (although the correct sequence has been preserved). 186 CICS Transaction Gateway for z/OS Version 6.1 Example 5-5 JNI trace of an Extended LUW 1st ECI request - EXTENDED MODE DPL with LUW-token=0 ---------------------------------------------------15:08:01.669 CTG6800I ECI parameters on entry. Call_Type = 12, Extend_Mode = 1, Luw_Token = 0, Commarea_Length = 38, Cics_Rc = 0, Flags = 0, outbound Commarea length = 2. 15:08:01.680 CTG6886I About to issue EXCI DPL call. Pipe token =0xf6c8140 , eci system name =A6POTGR1, program name =ECIPROG . 15:09:01.758 CTG6804I Returned commarea information. Commarea length = 38, commarea address = f909dc0, inbound commarea data length =38. 15:09:01.759 CTG6805I ECI parameters on exit. Call_Type = 12, Extend_Mode = 1, Luw_Token = 16777217, Commarea_Length = 38, Cics_Rc = 0, AV = 0. 2nd ECI request - EXTENDED MODE DPL with LU-token=16777217 ---------------------------------------------------------15:09:01.764 CTG6800I ECI parameters on entry. Call_Type = 12, Extend_Mode = 1, Luw_Token = 16777217, Commarea_Length = 38, Cics_Rc = 0, Flags = 0, outbound Commarea length = 2. 15:09:01.766 CTG6886I About to issue EXCI DPL call. Pipe token =0xf6c8140 , eci system name =A6POTGR1, program name =ECIPROG . 15:10:01.789 CTG6804I Returned commarea information. Commarea length = 38, commarea address = f907b50, inbound commarea data length =38. 15:10:01.790 CTG6805I ECI parameters on exit. Call_Type = 12, Extend_Mode = 1, Luw_Token = 16777217, Commarea_Length = 38, Cics_Rc = 0, AV = 0. 3rd ECI request - EXTENDED MODE COMMIT with LU-token=16777217 ------------------------------------------------------------15:10:01.795 CTG6800I ECI parameters on entry. Call_Type = 1, Extend_Mode = 2, Luw_Token = 16777217, Commarea_Length = 0, Cics_Rc = 0, Flags = 0, outbound Commarea length = 0. 15:10:01.796 CTG6804I Returned commarea information. Commarea length = 0, commarea address = 0, inbound commarea data length =0. 15:10:01.803 CTG6805I ECI parameters on exit. Call_Type = 1, Extend_Mode = 2, Luw_Token = 0, Commarea_Length = 0, Cics_Rc = 0, AV = 0. The sequence of trace entries is as follows: 1. The first call to program ECIPROG initiates a new transaction. We see Extend_Mode=1 (ECI_EXTENDED) and LUW_Token=0. When this first call completes successfully, the COMMAREA data has been received and an Luw_Token 16777217 that represents the Extended LUW is assigned. 2. The second call to ECIPROG continues within the same context: Extend_Mode=1 and the same Luw_Token. The call ends normally. 3. The third call is for Extend_Mode=2 (ECI_COMMIT) with the same Luw_Token. This call requests the long-running mirror task in CICS to perform a syncpoint, committing all updates to recoverable resources performed during the extended LUW. RRS Archive logs The RRS archive log shows information relating to all RRS URs, including completed URs. We selected option 1 from the RRS primary menu panel and the Chapter 5. Gateway daemon transactions 187 Browse an RRS log stream menu panel was displayed. We then chose to view the summary information of the RRS archive log. Attention: If the write activity to the RRS ARCHIVE logs is very high, this might impact the performance throughput of ALL RRS transactions if this log stream is defined and actively in use by RRS. This log stream is optional and only needed for any type of post transaction history type of investigation. We use the RRS archive log in order to illustrate the behavior of transactions. However, in a production system we recommend that you disable RRS archive logging. We searched the RRS Archive log for CTG.RRM.CITGR1G.IBM.UA (the value of CTG_RRMNAME for our Gateway daemon). After we located the UR entries that were associated with our Gateway, we were able to check that the transaction was indeed Committed (Figure 5-17). The URID corresponds to the UR that we observed during the time when the transaction was in-flight. Menu Utilities Compilers Help ------------------------------------------------------------------------------BROWSE CITGRJ.ATR.REPORT Line 00062429 Col 001 080 X1 2005/10/04 15:10:01.802427 BLOCKID=00000000001ADC68 URID=BDB5DAA67E5540000000042E01020000 JOBNAME=CITGR1G USERID=CITGR1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGR1G.IBM.UA SYNCPOINT=Commit RETURN CODE=00000000 START=2005/10/04 13:10:01.796392 COMPLETE=2005/10/04 13:10:01.801917 EXITFLAGS=00800000 LUWID=PSSCX9.A6POTGR1 B5DAA6F51481 0001 TID= GTID= FORMATID=003654931682 (decimal) D9D9D4E2 (hexadecimal) GTRID= F0F0F2F0F6F4F1F1F6C9C2D4F5F1F0F0F0F0F0F0F0F6F5F5F2F9BDB5DAA6F51A1E0D BQUAL= D9D9D4E24BBDB5DAA6F51A236DF0F0F2F0F6F4F1F1F6C9C2D4F5F1F0F0F0F0F0F0F0F6F5F5F2F9 Command ===> F1=Help F2=Split F10=Left F11=Right F3=Exit F12=Cancel F5=Rfind F7=Up Scroll ===> PAGE F8=Down F9=Swap Figure 5-17 RRS Archive summary log entry for committed transaction Extended LUW workload This test scenario used 15 Web browsers, each running the CTGTesterCCI application and specifying a single interaction per request. The COMMAREA data sent used for the request was either DW (write to recoverable TS Queue 188 CICS Transaction Gateway for z/OS Version 6.1 and then delay) or AW (write to recoverable TS Queue and then abend). The workload consisted of the following: Five requests with DW Five requests with AW Five requests with DW Each of the first five requests started a separate local transaction, the first of which spanned only one minute but the fifth of which spanned five minutes because it needed to wait for the lock on the TS Queue. Requests 6, 7, 8, 9, and 10 each made their TS Queue updates but then abended with abend code ECIP. Each failed request caused an ResourceException to be thrown by the ECIInteraction.execute() call from the session EJB. Upon catching this exception, the application issues setRollbackOnly() on the transaction context. At the end of the execute() method, the WebSphere Application Server EJB container finds that a rollback has been requested and drives a rollback of the transaction. Requests 11 through 15 then followed the same pattern as requests1 through 5 and made their updates to the TS Queue over the last five minutes of the test. At the end of the test, the recoverable TS Queue RECIPROG contained ten new entries. We used this test scenario to demonstrate the monitoring of WebSphere transactions using Tivoli Performance Viewer and to show a JNI trace of an extended LUW abnormal termination. Tivoli Performance Viewer - Transaction Manager module Configuring Tivoli Performance Viewer is covered in “Tivoli Performance Viewer” on page 110. For the purpose of transaction monitoring, we modified the settings using the WebSphere Administrative Console as follows: 1. We selected Monitoring and Tuning → Performance Monitoring Infrastructure (PMI) in the Welcome panel and then selected server1. In the Currently monitored statistic set panel, we switched from the Extended set to Basic. Because we are only using local transactions in these scenarios, there is no need to distinguish between local and global transactions. 2. We then selected OK to apply these changes, saved them to the repository, and restarted the application server. 3. After the restart, in the Welcome panel we selected Monitoring and Tuning → Performance Viewer → Current Activity → server1. In the left-side panel of the screen, we selected server1 → Performance Modules. We disabled the Connection Pool monitoring that we enabled previously by Chapter 5. Gateway daemon transactions 189 deselecting JCA Connection Pools → ECIResourceAdapter → eis/CITGR1CI-tx1-13101, and we select Transaction Manager (as shown in Figure 5-18). Figure 5-18 Tivoli Performance Viewer - Current Transaction Activity Figure 5-19 shows a snapshot of the Transaction Manager statistics that are provided by the Basic statistics set. The snapshot was taken after requests 1 through 13 have completed but while requests 14 and 15 are still in-flight. The (concurrent) active transactions peaked at 15 after all 15 requests have been submitted and gradually fall as each transaction waiting for write access to the TS Q is dispatched and the following one minute delay expires. After the first five requests complete, the graph shows that five transaction rollback events appear to occur at an instant. The requests submitted with COMMAREA data AW did not contain the one minute delay, and so all five transaction rollbacks occurred within one monitoring interval. After this point, the number of active transactions was down to five. Each of the remaining transactions completes, bumping the transaction commit count up by one at the end of each one minute delay. At the end of the monitoring period snapshot, the graph ends with the last two transactions still in-flight. 190 CICS Transaction Gateway for z/OS Version 6.1 Figure 5-19 The Tivoli Performance Viewer Transaction Manager during the Extended LUW scenario CICS CEBR command We used the CICS-supplied transaction CEBR to monitor the state of the recoverable TS Queue RECIPROG. The data in each entry reflects the APPLID, date and time, CICS user ID and task start code written at the start of each invocation (before the delay). Chapter 5. Gateway daemon transactions 191 Figure 5-20 shows that the first five requests started at approximately one second intervals. There is then a gap in the time stamps, during which five requests result in task abends and the TS Queue updates are backed out. Finally, before the last five requests end normally and the TS Queue updates are made. CEBR TSQ RECIPROG SYSID ENTER COMMAND ===> ************************** 00001 A6POTGR1 05/10/05 11:54:49 00002 A6POTGR1 05/10/05 11:54:50 00003 A6POTGR1 05/10/05 11:54:51 00004 A6POTGR1 05/10/05 11:54:51 00005 A6POTGR1 05/10/05 11:54:52 00006 A6POTGR1 05/10/05 11:55:16 00007 A6POTGR1 05/10/05 11:55:17 00008 A6POTGR1 05/10/05 11:55:18 00009 A6POTGR1 05/10/05 11:55:19 00010 A6POTGR1 05/10/05 11:55:19 ************************* PF1 : PF4 : PF7 : PF10: HELP VIEW TOP SCROLL BACK HALF SCROLL BACK FULL PF2 : PF5 : PF8 : PF11: TGR1 REC 1 OF 10 COL 1 OF 38 TOP OF QUEUE ******************************* CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D CITGR1D D BOTTOM OF QUEUE ***************************** SWITCH HEX/CHAR VIEW BOTTOM SCROLL FORWARD HALF SCROLL FORWARD FULL PF3 : PF6 : PF9 : PF12: TERMINATE BROWSE REPEAT LAST FIND UNDEFINED UNDEFINED Figure 5-20 CEBR transaction showing committed TS Queue updates CICS TG JNI trace Example 5-6 shows the JNI trace for one extended LUW which ends abnormally during the workload test scenario. The trace has been edited and annotated for clarity purposes, and only selected trace points are shown (although the correct sequence has been preserved). Example 5-6 JNI trace an Extended LUW abnormal termination ECI request - EXTENDED MODE DPL with LUW-token=0 -----------------------------------------------11:55:05.362 CTG6800I ECI parameters on entry. Call_Type = 12, Extend_Mode = 1, Luw_Token = 0, Commarea_Length = 38, Cics_Rc = 0, Flags = 0, outbound Commarea length = 2. 11:55:05.365 CTG6886I About to issue EXCI DPL call. Pipe token =0x17f98180 , eci system name =A6POTGR1, program name =ECIPROG . Error response - Cics_Rc=-7 means transaction abended ----------------------------------------------------11:59:50.055 CTG6887I EXCI DPL call returned. 11:59:50.056 CTG6822E EXCI function error. Function Call = 6, Response = 12, Reason = 422, Subreason field-1 = 0x00, subreason field-2 = 0x00, Cics_Rc = -7. 11:59:50.056 CTG6823E EXCI DPL_REQUEST specific error. RESP value = 0x00, RESP2 value = 0x00, Abend Code = ECIP, Cics_Rc = -7. 192 CICS Transaction Gateway for z/OS Version 6.1 11:59:50.056 CTG6805I ECI parameters on exit. Commarea_Length = 38, Cics_Rc = -7, AV = 0. Call_Type = 12, Extend_Mode = 1, Luw_Token = 16842753, EXTEND_MODE 4 means ECI_BACKOUT (WebSphere Application Server has requested the Gateway daemon to rollback) ------------------------------11:59:50.068 CTG6800I ECI parameters on entry. Call_Type = 1, Extend_Mode = 4, Luw_Token = 16842753, Commarea_Length = 0, Cics_Rc = 0, Flags = 0, outbound Commarea length = 0. 11:59:50.069 CTG6804I Returned commarea information. Commarea length = 0, commarea address = 0, inbound commarea data length =0. 11:59:50.072 CTG6805I ECI parameters on exit. Call_Type = 1, Extend_Mode = 4, Luw_Token = 0, Commarea_Length = 0, Cics_Rc = 0, AV = 0. The sequence of trace entries is as follows: 1. The first trace entry shows the initiation of an Extended LUW (Extend_Mode=1, Luw_Token=0) for the invocation of ECIPROG. However, the supplied COMMAREA data in this case was AW, instructing the target program to issue an abend (code ECIP) after writing to the TS Queue. 2. The second trace entry shows that the abended task resulted in Cics_Rc=-7 (ECI_ERR_TRANSACTION_ABEND). It is this error that percolates back to the Session EJB as a ResourceException, thrown by the ECIInteraction.execute() method. 3. After the failure of the ECI request, the CTGTesterCCI application catches the Resource exception and calls the setRollbackOnly() method on the current context. Therefore, when the container managed transaction reaches syncpoint, WebSphere Application Server rolls back the local transaction. We see the Extend_Mode=4 (ECI_BACKOUT) request in the JNI trace (together with the Luw_Token 16842753 which was returned on the initial ECI request). The backout call to CICS then succeeds with Cics_Rc=0. Note that we would also see evidence of this extended LUW failure in the RRS archive log. 5.7 Configuring for XA transactions In this section, we show how we configured our setup to support XA transactions using the CICS ECI XA resource adapter (cicseciXA.rar). We needed to consider the following configuration and setup tasks: Configuring CICS Checking that our CICS TS region was enabled to support Transactional EXCI. Chapter 5. Gateway daemon transactions 193 CICS TG for z/OS – Enabling CTGRRMS services – Configuring the Gateway daemons to register with RRS – Configuring the Gateway daemons with XA support enabled WebSphere Application Server – – – – – Updating the J2EE application logic Installing the CICS TG XA resource adapter Defining connection factories based on the XA resource adapter Configuring WebSphere transaction logging Deploying our sample application CTGTesterCCIXA 5.7.1 Definitions checklist Table 5-5 lists the definitions that we used to configure the scenario. Table 5-5 Definitions checklist Value Gateway daemon 1 Gateway daemon 2 CICS region 1 hostname tx1.mop.ibm.com tx1b.mop.ibm.com - IP address 9.100.193.122 9.100.193.121 - TCP/IP port 13101 12101 - Job name CITGR1G CITGT1G CITGR1CI CITGT1CI A6POTGR1 A6POTGT1 APPLID NETNAME CITGR1G CITGT1G CONNECTION RRM Name 194 GR1G CTG.RRM.CITGR1G.IBM .UA CTG.RRM.CITGT1G.IBM .UA CICS Transaction Gateway for z/OS Version 6.1 CICS region 2 GT1G 5.7.2 Configuring CICS No additional CICS setup is required for XA transaction support over and above the setup documented for Extended LUW support in 5.6.2, “Configuring CICS” on page 175. 5.7.3 Configuring CICS TG We needed to consider the following CICS TG configuration steps in order to enable XA transaction support: Enabling CTGRRMS services Naming considerations for RRM Enabling XA support in the Gateway daemon Enabling CTGRRMS services To provide support for XA transactions, the Gateway daemon makes use of RRMS facilities which require the calling address space to be registered as an authorized resource manager. A new address space called CTGRRMS acts as an agent for the Gateway daemon, providing access to the RRS capabilities required to support XA transactions. CTGRRMS is a non-terminating address space, which will usually remain active until the next IPL of the MVS image or LPAR. CTGRRMS does not execute code itself beyond it’s own initialization, but does provide services used by any Gateway daemon in the same LPAR which supports XA transactions. During initialization, a Gateway daemon with XA transaction support enabled checks the availability of the CTGRRMS address space. If CTGRRMS is not available, then the Gateway daemon requests the CTGRRMS Address Space Initiator (ctgasi) to start up CTGRRMS. Gateway daemon initialization fails if the CTGRRMS address space is unavailable and cannot be started. Example 5-7 shows the messages which are written to syslog for a successful initialization of CTGRRMS. Example 5-7 Syslog - CTGRRMS initialization messages CTG6241I Initializing CTGRRMS Services IEF170I 1 CTGRRMS CTG6241I Initializing CTGRRMS Services CTG6242I CTGRRMS Services open for business IEF170I 1 CTGRRMS CTG6242I CTGRRMS Services open for business The ctgasi utility also has a command interface which can be used in the event that the CTGRRMS address space needs to be shut down for maintenance Chapter 5. Gateway daemon transactions 195 purposes, see 5.8, “Operations” on page 219 for more information about using ctgasi commands. Usage or initiation of the CTGRRMS address space is restricted to users with UPDATE authority to the RACF FACILITY class profile CTG.RRMS.SERVICE. We gave the user IDs that are used for the Gateway daemon started tasks access to this profile as follows: REDEFINE CTG.RRMS.SERVICE CLASS(FACILITY) UACC(NONE) PERMIT CTG.RRMS.SERVICE CLASS(FACILITY) ID(CITGR1G) ACCESS(UPDATE) PERMIT CTG.RRMS.SERVICE CLASS(FACILITY) ID(CITGT1G) ACCESS(UPDATE) In order to initiate the CTGRRMS address space, the hlq.SCTGLINK PDS member CTGINIT must be placed into the Linked List Area (LLA). We added CTGV61.SCTGLINK to the LLA and then refreshed the LLA using the MVS command: F LLA,REFRESH Naming considerations for RRM The environment variable CTG_RRMNAME is used to specify the name that the Gateway daemon will use when registering with RRS. The rules that apply when specifying this name are described in 2.3.4, “Creating an STDENV file” on page 37. The same guidelines for defining the CTG_RRMNAME environment variable apply to a Gateway daemon with XA support enabled, except that the requirement for the .UA (unauthorized) suffix does not apply. The name must be Sysplex-unique and a maximum of 32 characters. We decided to keep the same CTG_RRMNAME names for our Gateway daemons so that the names used for RRS registration were CTG.RRM.CITGR1G.IBM.UA and CTG.RRM.CITGT1G.IBM.UA. Tip: We were tempted to change the setting of CTG_RRMNAME to not have the .UA suffix. However, in Chapter 6, “Gateway daemon transactions with TCP/IP load balancing” on page 225, we show how these Gateway daemons can be used in a CICS TG group and the CTG_RRMNAME of Gateway daemons in a CICS TG group must be suffixed with .UA. We, therefore, decided to keep the same name throughout the sequence of tests. Enabling XA support in the Gateway daemon CICS TG V6.1 has a single configuration parameter (xasupport=on) which determines whether XA transactions are supported and, thus, whether the additional XA transaction infrastructure, configuration, and setup is required. 196 CICS Transaction Gateway for z/OS Version 6.1 To ease migration from earlier versions of CICS TG you might find it convenient to run the Gateway daemon with the default setting of xasupport=off, until such time as XA transaction support is required. Here, we explore the configuration steps required when xasupport=on is specified. We edited the Gateway daemon configuration files CITGR1G.ini and CITGT1G.ini to specify xasupport=on. The CTG_XA_MAX_TRAN environment variable specifies the maximum number of concurrent XA transactions that the Gateway daemon can support. We used the default setting of 1000. Note: If the Gateway daemon is part of a TCP/IP load balancing group, CTG_XA_MAX_TRAN is specified by the CICS TG master process responsible for the group (see Chapter 6, “Gateway daemon transactions with TCP/IP load balancing” on page 225). After starting the Gateway daemon we noted the following message during initialization: CTG6737I XA transaction support is enabled AFP Authorization of CTGINIT Some MVS system functions, such as functions provided by RRS, are sensitive and access to these functions is restricted to only authorized programs, operating in a privileged mode know as supervisor state. to avoid compromising the security and integrity of the system. MVS provides a mechanism called the Authorized Program Facility (APF) to restrict the access to these sensitive system functions or programs to avoid integrity exposures. This allows you to specify which libraries contain those special functions or programs and prevent unauthorized programs from using them. Those libraries are then called APF libraries. By default the libraries in the LNKLST concatenation are authorized (unless LNKAUTH=APFTAB is specified in the IEASYSxx parameter list). However, if the system accesses the libraries in the LNKLST concatenation via JOBLIB or STEPLIB DD statements, the system does not consider those libraries authorized unless you specified the library name in the APF list by using either the IEAAPFxx or the PROGxx parmlib member. If a JCL DD statement concatenates an authorized library in any order with an unauthorized library, the entire set of concatenated libraries becomes unauthorized. In order for a Gateway daemon to use XA support, CTGINIT (supplied in the SCTGLINK library) must be in the LNKLST and in an APF authorized library. Chapter 5. Gateway daemon transactions 197 Tip: You can use the D PROG,APF,ALL console command to display the APF authorized library list. For more details about LNKIST and authorized libraries, refer to ABCs of z/OS System Programming Volume 2, SG24-6982. 5.7.4 Configuring WebSphere To test our XA transaction configuration we wrote a new application CTGTesterCCIXA which is an extension of the CTGTesterCCI application. CTGTesterCCIXA contains two ECI resource references and can be used to call programs running in two CICS regions via different connection factories. For further details about CTGTesterCCIXA refer to Appendix A, “Sample applications” on page 361. Dealing with CICS transaction abends CICS cannot tell RRS to rollback updates to recoverable resources following an unhandled abend of the long-running mirror task. Thus, by default, if the first JCA request of an XA transaction succeeds but the second JCA request fails (because of a CICS transaction abend), the updates made by the first request are committed but the updates made by the second request cannot be committed. There are two ways of avoiding such an inconsistency resulting from a CICS transaction abend: The J2EE application can abandon the current global transaction by calling setRollbackOnly() on the current context. In the event of a transaction abend, a CICSTxnAbendException is thrown by the CICS ECI XA resource adapter and the decision to rollback or to commit resources that are updated by the global transaction can be taken by the J2EE application. The option ABENDBKOUT=YES can be set in the DFHXCOPT table (see “ABENDBKOUT option in DFHXCOPT” on page 165). In the event of a transaction abend, the global transaction is rolled back automatically at the end of the EJB method call. Note: At the time of writing this book , we were unable to test the ABENDBKOUT option. We, therefore, chose to include transaction abend rollback logic in the sample application CTGTesterCCIXA. 198 CICS Transaction Gateway for z/OS Version 6.1 We coded the CTGTesterCCIXA application to issue a setRollbackOnly() on the EJB session context in the catch block for the ECIInteraction.execute() call. This ensures that the CICS abend is propagated onto WebSphere Application Server as the transaction manager and that the global transaction (if one exists) is rolled back. We used the same code as described in Example A-1 on page 368. Installing the cicseciXA resource adapter The process of installing the CICS ECI XA resource adapter (cicseciXA.rar) is identical to that described for the ECI resource adapter (see 3.2.1, “Installing the resource adapter” on page 80). Both resource adapters can be installed at the same time. Figure 5-21 shows the two resource adapters installed into our WebSphere Application Server for Windows. Figure 5-21 Administrative Console - CICS ECI resource adapters Chapter 5. Gateway daemon transactions 199 Creating connection factories The process of creating connection factories using the CICS ECI XA resource adapter is the same as described in 3.2.2, “Creating connection factories” on page 82. We created two connection factories CITGR1CI-tx1-13101-XA and CITGR1CI-tx1b-12101-XA based upon the CICS ECI XA resource adapter . Figure 5-22 shows our connection factories in the WebSphere Administrative Console. Figure 5-22 Administrative Console - CICS ECI XA connection factories The two connection factories represent two connections to two different CICS regions which are accessed via two different Gateway daemons. The Gateway daemons each bind to different IP addresses. Tip: Before using the XA transaction support, you should ensure that your WebSphere Application Server level contains the fix for APAR PQ99940. This APAR fixes a connection pooling performance problem following a connection failure, for example, as a result of a CICS region or Gateway daemon failure. The fix is contained in V6.0.2 of WebSphere Application Server for multi-platforms. Configuring WebSphere transaction logging We used the same WebSphere transaction logging configuration that we describe in “Configuring WebSphere transaction logging” on page 180. Deploying our sample application To deploy the CTGTesterCCIXA application we followed the same procedure as is described in 3.2.3, “Deploying the sample J2EE application” on page 89. This time we needed to map two resource references using the WebSphere Administrative Console, as shown in Figure 5-23. 200 CICS Transaction Gateway for z/OS Version 6.1 Figure 5-23 Administrative Console - mapping XA resource references After CTGTesterCCIXA was successfully deployed, we started the application. 5.7.5 Testing the XA transaction configuration In this section, we detail how we tested the XA transaction configuration using our sample J2EE application CTGTesterCCIXA. Figure 5-24 shows the basic outline of our environment. We used this configuration for two scenarios: Normal termination of an XA transaction An update and commit of recoverable resources in two CICS regions within one XA transaction, via two Gateway daemons (see “XA transaction normal termination” on page 204). Abnormal termination of an XA transaction A CICS subsystem failure during an in-flight XA transaction (see “XA transaction abnormal termination” on page 214). Chapter 5. Gateway daemon transactions 201 tx1.mop.ibm.com CITGR1G Gateway daemon ECI Request within XA txn Windows 13101 HTML z/OS CICS EXCI JNI WebSphere Application Server V6 CITGR1CI (applid A6POTGR1) Long Running Mirror ECIT JSP/Servlet CICS TG V6.1 CCI Transaction Required Cam21-pc11 RRS CTGRRMS CTGTesterCCIXA CICS ECI XA Resource Adapter ECI Request within XA txn Gateway daemon 12101 CICS EXCI JNI CITGT1G tx1b.mop.ibm.com Long Running Mirror ECIT CITGT1CI (applid A6POTGT1) Figure 5-24 XA transaction scenario Note: On our system the Gateway daemons bind to different TCP/IP addresses but are actually running on the same z/OS image. In practice, they could be running on two completely different systems. To test our configuration, we used our test application CTGTesterCCIXA via its Web browser interface (Figure 5-25). 202 CICS Transaction Gateway for z/OS Version 6.1 Figure 5-25 CTGTesterCCIXA Start Web browser interface During these tests we specified DW as the input to the COMMAREA, which means that the ECIPROG program writes to a recoverable TS Queue and then delays for one minute. Chapter 5. Gateway daemon transactions 203 The iteration count was set to 1 for transaction leg 1 and 5 for transaction leg 2. Thus, clicking Submit results in a total of six ECI calls: 1 request to CICS region CITGR1CI (applid A6POTGR1) and 5 requests to CICS region CITGT1CI (applid A6POTGT1). This iteration count was done to give us six minutes during which to monitor the transaction state across the various components of the system. Note: This scenario is for testing purposes only. Normal transactions would not be expected to delay in this way. During the course of these scenarios, we monitored the transaction using each of the following : CICS CICS TG RRS ISPF panels CEMT command JNI trace output RRS related Work Manager information We discuss each of these tools in the next section: XA transaction normal termination This test demonstrates successful update and commit of resources on two CICS regions accessed via different connection factories within the scope of a single global transaction. The Web browser appeared to be busy for six minutes before displaying the results page that contained the COMMAREAs returned from the two legs of the global transaction. At the end of the test, the recoverable TS Queue RECIPROG on CICS region CITGR1CI (applid A6POTGR1) contained one new entry, and the recoverable TS Queue RECIPROG on CICS region CITGR1CI (applid A6POTGT1) contained five new entries. CICS CEMT command While the second ECI request was in-flight on A6POTGT1, we issued the CEMT INQUIRE TASK command on both A6POTGR1 and A6POTGT1. Figure 5-26 shows the ECIT mirror task running on A6POTGR1 as Task 182 suspended in RRMSEXIT state, which means that CICS is waiting for a commit or backout flow from RRS. Program ECIPROG on A6POTGR1 had completed at this stage but was waiting for syncpoint of the global transaction from the transaction manager (WebSphere Application Server) to be communicated via the Gateway daemon CITGR1G and RRS. Also at this point in time, the mirror transaction on A6POTGT1 was suspended in ICWAIT due to the active EXEC CICS DELAY in ECIPROG. 204 CICS Transaction Gateway for z/OS Version 6.1 INQUIRE TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000181) Tra(CEMT) Fac(N197) Run Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDBD86A2A138664D) Tas(0000182) Tra(ECIT) Sus Tas Pri( 001 ) Sta(TO) Use(CITGR1D ) Uow(BDBD87EC2BB8A361) Hty(RRMSEXIT) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 17.41.40 DATE: 10.10.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-26 CEMT INQUIRE TASK on A6POTGR1 with the mirror in RRMSEXIT state On A6POTGT1, we also issued the CEMT INQUIRE EXCI command. Figure 5-27 shows the EXCI information for the long-running mirror task ECIT (task 67) including the associated RRS UR identifier. INQUIRE EXCI STATUS: RESULTS Exc(CITGT1G.CITGT1G. - X1 ) Tas(0000067) Uri(BDBD88267E554A5C0000004801020000) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGT1 APPLID=A6POTGT1 TIME: 17.42.09 DATE: 10.10.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-27 CEMT INQUIRE EXCI on A6POTGT1 showing a UR identifier Later, we show that the UR identifier shown on CEMT INQUIRE EXCI panel matched the active and archived data accessible by the RRS ISPF panels (see Figure 5-28 on page 209). The INQUIRE EXCI command provides the information to correlate an active long-running mirror task in a CICS region, with the RRS UR id associated with a global transaction. The Exc field highlights the job name and connection name with which the UR identifier is associated. CICS TG JNI trace By default, details of specific transactional requests are not externalized by the CICS TG. It is essentially a passive component between WebSphere Application Server and CICS. Therefore, in order to uncover evidence of transactions in the Gateway daemon, we turned on JNI trace. Chapter 5. Gateway daemon transactions 205 In the previous Extended LUW scenario, we concentrated on the individual ECI requests and the use of LUW tokens. In this case, we highlight the start, prepare, and commit flows that are associated with XA transactions. We also highlight the role of the X/Open identifier (XID). Note: An X/Open identifier (XID) acts as the identifier for a distributed unit of work. An XID consists of a Global transaction identifier (GTRID) and a Branch qualifier (BQUAL). We found that the Gateway daemon trace reveals the presence of XID information for an ECI request, but does not trace out the actual value. JNI trace writes out the full formatted XID information that we could use to correlate with information available via the RRS ISPF panels. Example 5-8 shows the JNI trace for the ECI requests flowed to A6POTGT1 through the Gateway daemon CITGT1G. The trace has been edited and annotated for clarity purposes, and only selected trace points are shown (although the correct sequence has been preserved). Example 5-8 Annotated summary of JNI trace from CITGT1G for leg 2 of the global transaction Initialization of XA support ---------------------------17:39:03.076 CTG8660I Two phase commit support enabled Start of an XA transaction -------------------------17:41:30.747 CTG8602I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXAStart entry 17:41:30.748 CTG9205I Before call to Begin_Context 17:41:30.749 CTG9206I After successful call to Begin_Context 17:41:30.761 CTG9206I After successful call to Express_UR_Interest UR ID : BDBD88267E554A5C0000004801020000 UR Context : BDBD88262CAE980D0200000002A3E450 UR Interest : 7E34C1000000001E0055000155555555 XID.formatID : 1463898948 XID.gtrid : 00-0F 00000106 D5D837F0 00000001 00000039 10-1F 056726FA 5740F82D ADB83B28 0239F181 20-23 BC7F5B5C XID.bqual : 00-0F 00000106 D5D837F0 00000001 00000039 10-1F 056726FA 5740F82D ADB83B28 0239F181 20-2F BC7F5B5C 00000001 00000000 00000000 30-35 00000000 0002 17:41:30.765 CTG8604I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXAStart exit (rc = 0). Gateway process sequence of five ECI Requests as part of XA transaction ---------------------------------------------------------------------- 206 CICS Transaction Gateway for z/OS Version 6.1 17:41:30.835 CTG6800I ECI parameters on entry. Call_Type = 12, Extend_Mode = 0, Luw_Token = 0, Commarea_Length = 38, Cics_Rc = 0, Flags = 0, outbound Commarea length = 2. 17:41:30.837 CTG8635I XID present on ECI Request. 17:41:30.862 CTG6886I About to issue EXCI DPL call. Pipe token=0xf6c6510 , eci system name=A6POTGT1, program name=ECIPROG . 17:42:31.122 CTG6887I EXCI DPL call returned. 17:42:31.123 CTG6805I ECI parameters on exit. Call_Type = 12, Extend_Mode = 0, Luw_Token = 0, Commarea_Length = 38, Cics_Rc = 0, AV = 0. *Above DPL sequence repeated five times* Gateway receives prepare from WebSphere Application Server ---------------------------------------------------------17:46:32.576 CTG8602I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXAPrepare entry 17:46:32.587 CTG8604I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXAPrepare exit (rc = 0). Gateway receives Commit from WebSphere Application Server --------------------------------------------------------17:46:32.794 CTG8602I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXACommit entry 17:46:32.806 CTG8604I JNI Method Java_com_ibm_ctg_server_ServerXARequest_CcicsXACommit exit (rc = 0). The sequence of trace entries is as follows: 1. The first trace entry that we highlight shows that the Gateway daemon has initialized with XA support enabled. 2. The second set of trace entries show the start of the XA transaction and the Global transaction identifier (GTRID) and Branch qualifier (BQUAL). In the next section we explain how we obtained and subsequently used the GTRID to monitor the in-flight transaction using the RRS ISPF panels. 3. The third set of entries show the ECI requests. Note: Extend_Mode=0 and Luw_token=0 for these ECI requests, but it is the presence of an Xid which indicate that the requests are part of a global transaction. 4. The fourth set of entries show the prepare flow of the two-phase commit process. 5. The fifth set of entries show the commit flow of the two-phase commit process. RRS panels from ISPF The RRS panels in ISPF can be used to monitor progress of in-flight transactions down to a very low level of detail. We navigated through the menus, drilling down from the Gateway RRMNAME to the status of individual RM exit routines for a particular transaction. Chapter 5. Gateway daemon transactions 207 We show here, for illustrative purposes, a worked example of how the RRS panels can be navigated in order to uncover information about an in-flight UOW. For most processing, URs are fleeting and you will normally have difficulty in displaying them. However, RRS provides archive logs for all URs so that you can investigate URs after a transaction has completed (for example, see “RRS Unit of Recovery log” on page 212). 1. We selected option 2 (Display/Update RRS related Resource Manager information) from the RRS primary menu panel. 2. On the RRS Resource Manager selection panel, we entered the pattern CTG.RRM.CITG?1G.IBM.UA. The question mark (?) is used here as a wildcard. Tip: In order to specify selections in this way on the RRS ISPF panels, it is important to have a structured naming convention for CTG_RRMNAME. 3. The next panel (RRS Resource Manager List) showed both CITGR1G and CITGT1G resource managers in Run state. Because the first leg of our transaction runs on CITGR1G, we entered u-View URs against that Resource Manager entry (during the first minute of the test, there is no UR active in CITGT1G). 4. The RRS Unit of Recovery List panel showed a single in-flight UR associated with our Resource Manager CITGR1G. We selected v-View Details, which gave us the RRS Unit of Recovery Details panel (Figure 5-28). Figure 5-28 shows: – That the UR BDBD87EC7E5546E80000007C01020000 is in-flight – That the RRMS work manager is the Gateway daemon with RM name CTG.RRM.CITGR1G.IBM.UA – That the Gateway daemon is a syncpoint coordinator or SDSRM (Server Distributed syncpoint Resource Manager) – That the CICS region with RM name DFHRXDM.A6POTGR1.IBM is a participant in the UR 208 CICS Transaction Gateway for z/OS Version 6.1 RRS Unit of Recovery Details Row 1 to 2 of 2 Commands r-Remove Interest v-View URI Details UR identifier : BDBD87EC7E5546E80000007C01020000 Create time : 2005/10/10 15:40:29.803693 Comments : UR state : InFlight UR type : Prot System : X1 Logging Group : X1 SURID : N/A Work Manager Name : CTG.RRM.CITGR1G.IBM.UA / Display Work IDs / Display IDs formatted Luwid . : Present Eid . . : Not Present Xid . . : Present Expressions of Interest: S RM Name Type Role CTG.RRM.CITGR1G.IBM.UA Prot SDSRM DFHRXDM.A6POTGR1.IBM Prot Participant ******************************* Bottom of data ******************************** Figure 5-28 RRS Unit of Recovery information for CTG.RRM.CITGR1G.IBM.UA By selecting the Display Work IDs and the Display IDs formatted options, we are able to see the formatted XID information that is associated with the global transaction from which we took a copy of the hexadecimal GTRID (160 bytes). We are able to tie up this XID with the details written out to the JNI trace. 5. We navigate back to the RRS primary menu panel and switch to option 3 (Display/Update RRS Unit of Recovery information). Paging down on the RRS Unit of Recovery Selection panel, we paste the GTRID directly into the input field (Figure 5-29) and press Enter. Chapter 5. Gateway daemon transactions 209 RRS Unit of Recovery Selection Commands: save-Save Profile Profile Name get-Get Profile ENTER-Query . . Format ID (from 1 to 4294967295 in decimal) GTRID Pattern 00-0F 00000106 D5D837F0 00000001 00000039 10-1F 056726FA 5740F82D ADB83B28 0239F181 20-2F BC7F5B5C 30-3F 40-4F 50-5F 60-6F 70-7F BQUAL Pattern 00-0F 10-1F 20-2F Command ===> F1=Help F2=Split F3=Exit F5=ListErr F9=Swap F12=Cancel F7=Up F8=Down Figure 5-29 RRS Unit of Recovery selection using GTRID This query displays a list of all URs that have an interest in the global transaction that is represented by the GTRID (Figure 5-30). We found that RRS knew about two URs that are associated with our GTRID. By now, the first transaction leg through the Gateway daemon CITGR1G has completed and the second leg processed by Gateway daemon CITGT1G has begun. RRS Unit of Recovery List Row 1 to 2 of 2 Commands: v-View Details c-Commit b-Backout r-Remove Interest f-View UR Family S UR Identifier System Logging Group State Type Comments BDBD87EC7E5546E80000007C01020000 X1 X1 InFlight Prot BDBD88267E554A5C0000004801020000 X1 X1 InFlight Prot ******************************* Bottom of data ******************************** Figure 5-30 In-flight URs associated with global transaction 210 CICS Transaction Gateway for z/OS Version 6.1 6. We drill down into one of these URs, by entering v-View Details against the second entry to show the resource managers involved in the UR BDBD88267E554A5C0000004801020000. Example 5-9 shows that CTG.RRM.CITGT1G.IBM.UA (Gateway daemon) and DFHRXDM.A6POTGT1.IBM (CICS region) are involved in the second UR. Note that for this UR, the Gateway daemon role is also as a syncpoint coordinator or SDSRM. Example 5-9 Drilling down into the UR detail - RMs with Expressions of Interest RRS Unit of Recovery Details Row 1 to 2 of 2 Commands r-Remove Interest v-View URI Details UR identifier : BDBD88267E554A5C0000004801020000 Create time : 2005/10/10 15:41:30.755128 Comments : UR state : InFlight UR type : Prot System : X1 Logging Group : X1 SURID : N/A Work Manager Name : CTG.RRM.CITGT1G.IBM.UA Display Work IDs / Display IDs formatted Luwid . : Present Eid . . : Not Present Xid . . : Present Expressions of Interest: S RM Name Type Role CTG.RRM.CITGT1G.IBM.UA Prot SDSRM DFHRXDM.A6POTGT1.IBM Prot Participant ***************************** Bottom of data ***************************** 7. RRS offers still further detail on the status of each Resource Manager with an interest in a UR. We issued v-View URI details against the CITGT1G to see the summary of exit activity (Example 5-10). Because the transaction was still in-flight, the exit status list is largely Uncalled, but we observed how this could be useful detail when analyzing a hung transaction. Example 5-10 Drilling down into the UR detail - state of CITGT1G’s UR Interest RRS URI Details UR identifier : BDBD88267E554A5C0000004801020000 URI token . . : 7E34C1000000001E0055000155555555 RM name . . . : CTG.RRM.CITGT1G.IBM.UA Type . . . . : Prot Status . : ACTIVE Role . . . . : SDSRM State . : InFlight SURID : N/A Exit/State Status Duration Chapter 5. Gateway daemon transactions 211 BACKOUT . . COMPLETION . COMMIT . . . DSE/IN_DOUBT End_UR . . . EXIT_FAILED ONLY_AGENT . PREPARE . . STATE_CHECK . . . . . . . . . : : : : : : : : : Uncalled Uncalled Uncalled N/A Uncalled Uncalled Uncalled Uncalled Uncalled CICS CEBR command We used the CICS-supplied terminal transaction CEBR to monitor the state of our recoverable TS Queue RECIPROG. The data in each entry reflects the APPLID, data and time, CICS user ID and transaction start code written at the start of each invocation (before the delay). Example 5-11 shows the contents of the RECIPROG TS Queue at the end of the global transaction. Example 5-11 CEBR output showing committed TS Queue updates CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 10/10/05 17:40:30 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** 1 COL 1 OF 38 CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 10/10/05 17:41:31 CITGT1D D 00002 A6POTGT1 10/10/05 17:42:31 CITGT1D D 00003 A6POTGT1 10/10/05 17:43:32 CITGT1D D 00004 A6POTGT1 10/10/05 17:44:32 CITGT1D D 00005 A6POTGT1 10/10/05 17:45:32 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** 5 COL 1 OF 38 RRS Unit of Recovery log When the transaction was complete, we returned to the RRS primary menu panel and selected option 1 (Browse an RRS log stream). We then chose option 2 (RRS Unit of Recovery State) logs to see a summary of completed URs. The 212 CICS Transaction Gateway for z/OS Version 6.1 progress of the global transaction is shown in examples Example 5-12 and Example 5-13. Example 5-12 RRS Unit of Recovery State log - CITGT1G prepare and in-doubt entries READING ATR.X1.DELAYED.UR LOG STREAM X1 2005/10/10 17:46:32.578661 BLOCKID=000000000013BBB1 URID=BDBD88267E554A5C0000004801020000 LOGSTREAM=ATR.X1.DELAYED.UR PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA STATE=InPrepare EXITFLAGS=00000000 FLAGS=04000000 LUWID=PSSCX9.A6POTGT1 BD882649B2E9 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000002 X1 2005/10/10 17:46:32.583921 BLOCKID=000000000013BE0D URID=BDBD88267E554A5C0000004801020000 LOGSTREAM=ATR.X1.DELAYED.UR PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA STATE=InDoubt EXITFLAGS=00000000 FLAGS=04000000 LUWID=PSSCX9.A6POTGT1 BD882649B2E9 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000002 Example 5-12 shows the prepare and in-doubt entries for Gateway daemon CITGT1G. Similar entries were present in the log for the CITGR1G leg of the global transaction. Example 5-13 shows the commit entries for both legs of the transaction. Example 5-13 RRS Unit of Recovery log - CITGR1G and CITGT1G commit entries X1 2005/10/10 17:46:32.772517 BLOCKID=000000000013C5B1 URID=BDBD87EC7E5546E80000007C01020000 LOGSTREAM=ATR.X1.DELAYED.UR PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGR1G.IBM.UA Chapter 5. Gateway daemon transactions 213 STATE=InCommit EXITFLAGS=00800000 FLAGS=06000000 LUWID=PSSCX9.A6POTGR1 BD87EC2BB0A8 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000001 X1 2005/10/10 17:46:32.797277 BLOCKID=000000000013C855 URID=BDBD88267E554A5C0000004801020000 LOGSTREAM=ATR.X1.DELAYED.UR PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA STATE=InCommit EXITFLAGS=00800000 FLAGS=06000000 LUWID=PSSCX9.A6POTGT1 BD882649B2E9 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 00000106D5D837F00000000100000039056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000002 XA transaction abnormal termination This test demonstrates a failure scenario, using the CTGTesterCCIXA application. The configuration and application parameters are identical to those used in the scenario “XA transaction normal termination” on page 204. In this scenario, however, during the second leg of the transaction the CICS region A6POTGR1 is cancelled. The second leg of the transaction continues, making the five updates to TS Queue RECIPROG on CICS region A6POTGT1. However, at the end of the final ECI request, the prepare call to the Gateway daemon CITGR1G (an SDSRM) for transaction leg 1 fails. This leads to a complete rollback of all updated resources. 214 CICS Transaction Gateway for z/OS Version 6.1 CICS CEBR command At just over three minutes into the test scenario, we looked at the contents of the TS Queue RECIPROG on both A6POTGR1 and A6POTGT1. We could see that both queues contained updates (Example 5-14). Example 5-14 CEBR output on A6POTGR1 and A6POTGT1 after three minutes CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 11/10/05 14:52:28 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** 1 COL 1 OF 38 CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 11/10/05 14:53:28 CITGT1D D 00002 A6POTGT1 11/10/05 14:54:28 CITGT1D D 00003 A6POTGT1 11/10/05 14:55:28 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** 3 COL 1 OF 38 At this point, we cancelled CICS region A6POTGR1 with the MVS command /CANCEL CITGR1CI. RRS Archive Log The RRS archive log shows information relating to all RRS URs, including completed URs. We selected option 1 from the RRS primary menu panel and the Browse an RRS log stream menu panel was displayed. We then chose to view the detailed information of the RRS archive log. Example 5-15 shows the log entries for the XA transaction abnormal termination scenario. Example 5-15 RRS Archive log - XA transaction abnormal termination RRS/MVS LOG STREAM BROWSE DETAIL REPORT READING ATR.X1.ARCHIVE LOG STREAM X1 2005/10/11 14:58:29.007861 BLOCKID=00000000001BF925 URID=BDBEA43A7E5546E80000008201020000 JOBNAME=CITGR1G USERID=CITGR1G PARENT URID=00000000000000000000000000000000 SURID=N/A Chapter 5. Gateway daemon transactions 215 WORK MANAGER NAME=CTG.RRM.CITGR1G.IBM.UA SYNCPOINT=AtraPrp RETURN CODE=0000012D START=2005/10/11 12:58:29.007325 COMPLETE=2005/10/11 12:58:29.007448 EXITFLAGS=00000000 LUWID=PSSCX9.A6POTGR1 BEA43A704F3A 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106DA64BED6000000010000003D056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 00000106DA64BED6000000010000003D056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000001 RMNAME=CTG.RRM.CITGR1G.IBM.UA ROLE=SDSRM FLAGS=10020000 PROTOCOL=PresumeNothing StateCheck EXIT RC=Uncalled Prepare EXIT RC=Uncalled DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=00000000 EndUr EXIT RC=Uncalled ExitFailed EXIT RC=Uncalled Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled RMNAME=DFHRXDM.A6POTGR1.IBM.UA ROLE=Participant FLAGS=04010000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=Uncalled DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=Uncalled EndUr EXIT RC=Uncalled ExitFailed EXIT RC=Uncalled Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled X1 2005/10/11 14:58:30.181361 BLOCKID=00000000001BFB8F URID=BDBEA4737E554A5C0000004E01020000 JOBNAME=CITGT1G USERID=CITGT1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA SYNCPOINT=AtraBak RETURN CODE=00000000 START=2005/10/11 12:58:28.982962 COMPLETE=2005/10/11 12:58:30.180898 EXITFLAGS=00000000 LUWID=PSSCX9.A6POTGT1 BEA474198443 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000106DA64BED6000000010000003D056726FA5740F82DADB83B280239F181BC7F5B5C BQUAL= 216 CICS Transaction Gateway for z/OS Version 6.1 00000106DA64BED6000000010000003D056726FA5740F82DADB83B280239F181BC7F5B5C00000001 0000000000000000000000000002 RMNAME=CTG.RRM.CITGT1G.IBM.UA ROLE=SDSRM FLAGS=10021000 PROTOCOL=PresumeNothing StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000014 DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=00000000 EndUr EXIT RC=Uncalled ExitFailed EXIT RC=Uncalled Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled RMNAME=DFHRXDM.A6POTGT1.IBM.UA ROLE=Participant FLAGS=10000000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000000 DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=00000010 EndUr EXIT RC=Uncalled ExitFailed EXIT RC=00000010 Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled The archive log entries show the following: There are two RRS URs involved in the global transaction with the same GTRID: 00000106DA64BED6000000010000003D056726FA5740F82DADB83B280239F181BC7F5B5C The RRS work manager for the first UR (URID=BDBEA43A7E5546E80000008201020000) is the Gateway daemon CITGR1G with CTG_RRMNAME CTG.RRM.CITGR1G.IBM.UA. The RRS work manager for the second UR URID=BDBEA4737E554A5C0000004E01020000) is the Gateway daemon CITGT1G with CTG_RRMNAME CTG.RRM.CITGT1G.IBM.UA. The Gateway daemons take a syncpoint coordinator or SDSRM role for their respective URs. The CICS regions have a Participant role within the URs. Chapter 5. Gateway daemon transactions 217 The log entry for the first UR shows a prepare call (AtraPrp) to CITGR1G returns with code 0x12D. The RRS manual SA22-7616 explains the return code as follows: ATR_BACKED_OUT_OUTCOME_PENDING (0x12D) Meaning: The commit operation failed. The RRS decision was to return to the previous consistent state. However, the state of one or more of the protected resources is not known. Action: Continue normal processing for a backed out unit of recovery. The UR state is now Forgotten. This prepare failure occurs because the CICS region A6POR1CI has been cancelled. Note that CICS region A6POTGR1 will backout the ECIT mirror transaction updates during CICS emergency restart. The log entry for the second UR shows a backout call (AtraBak) to CITGT1G returns with code 0. This backout call is made because the transaction manager, WebSphere Application Server, has rolled back the global transaction following the failure of the prepare call to CITGR1G. CICS message log Inspection of the CICS message log for A6POTGT1 shows the backout of the TS Queue updates on A6POTGT1 (Example 5-16). Example 5-16 CICS log message from A6POTGT1 indicating transaction backout DFHAC2250 10/11/2005 14:58:30 A6POTGT1 The coordinator system has indicated that the current unit of work is to be backed out. Transaction ECIT running program DFHMIRS term ???? has been abnormally terminated with abend ASP3. After the restart of CICS region A6POTGR1 was complete, the CICS log showed message DFHRM0201 (Example 5-17). Example 5-17 CEBR output showing committed TS Queue updates DFHRM0201 10/11/2005 14:57:17 A6POTGR1 0 backout-failed and 1 commit-failed UOWs were reconstructed Finally, we used CEBR to check that the TS Queue updates were backed out on both CICS regions. 218 CICS Transaction Gateway for z/OS Version 6.1 5.8 Operations The enablement of XA transaction support has an impact on operational procedures for the Gateway daemon. 5.8.1 Gateway daemon shutdown and restart Before enabling XA transaction support, you should review your Gateway daemon shutdown procedures. Gateway daemon normal shutdown When the Gateway daemon is requested to perform a normal shutdown: All in-flight extended LUWs and XA transactions are allowed to complete. No new transaction requests are accepted. No new XA recover requests are accepted. XA recovery is required if an error occurs in the syncpoint process, caused by a transaction manager failure. In this case, it is necessary for the transaction manager (in our case WebSphere Application Server) to reconnect to the Gateway daemon at a later time to tell the daemon whether an XA transaction should be committed or backed out. A CTG6591I message is issued (Example 5-18) for attempts to shutdown a Gateway daemon which is currently processing in-flight extended LUWs or XA transactions. Example 5-18 Gateway daemon shutdown with outstanding XA transactions 10/12/05 13:26:42:339 [0] CTG6490I Normal shutdown of Gateway daemon started by z/OS Operator 10/12/05 13:26:42:371 [0] CTG6491I Number of active clients is 1 Gateway daemon immediate shutdown An immediate shutdown is not recommended because it terminates all work in progress and breaks any active connections. When the Gateway daemon is requested to perform an immediate shutdown: Active requests are terminated abnormally. In-flight extended LUWs are rolled back. In-flight XA transactions are rolled back. XA transactions that are in post-prepare state are not rolled back. It is not possible for the transaction manager (WebSphere Application Server) to complete the XA transaction until the Gateway daemon is restarted. Chapter 5. Gateway daemon transactions 219 5.8.2 Gateway daemon restart During restart of an XA enabled Gateway daemon, the Gateway daemon rebuilds information that is related to outstanding URs by sending a request to RRS for information related to its work manager name (CTG_RRMNAME). Important: You should not restart an XA-enabled Gateway daemon on a different LPAR if there is outstanding XA recovery to perform. 5.8.3 The ctgasi utility The CICS TG Address Space Initiation (ctgasi) utility is provided in case you need to start, stop, or refresh the CTGRRMS address space manually. You should only be required to start, stop, or refresh the address space manually if there is a need to apply maintenance (PTFs) that affect the load module hlq.SCTGLINK(CTGINIT). The post-maintenance apply procedure is : Refresh the LLA (/F LLA,REFRESH) Shutdown all Gateway daemons Shutdown all ctgmaster address spaces Issue the command ctgasi -refresh from OMVS/USS shell The operator user ID issuing the ctgasi -refresh command must have CONTROL access to the CICSTG.RRMS.SERVICE RACF FACILITY class profile. If this ID does not have this access, ctgasi returns error message CTG6209E (Example 5-19). Example 5-19 ctgasi error for insufficient authority CTG6209E ctgasi - user ID not authorized, SAF RC = 00000008 00000008 00000000 CTGRRMS keeps track of registered/de-registered address spaces (Gateway daemons or ctgmaster) and ctgasi refuses to shutdown or refresh CTGRRMS if it believes that there are active address spaces. However, if address spaces have terminated abnormally, then they might not have completed the de-registration step with CTGRRMS. In such a case, the -force option can be used to refresh or shutdown CTGRRMS even if active instances of the Gateway daemon are detected. 220 CICS Transaction Gateway for z/OS Version 6.1 5.9 Problem determination In this section, we document common transaction problems and information we learned while configuring our transaction test scenarios. For general problem determination guidance see 2.9, “Problem determination” on page 60. 5.9.1 Common transaction problems If a Gateway daemon is started, but another Gateway daemon is running with the same CTG_RRMNAME, the Gateway daemon fails to initialize. We noted the following message CTG9202E A RRS application is active using the same resource manager name('CTG.RRM.CITGS1G.IBM.UA’) If an ECI request within an XA transaction is sent to a Gateway daemon which does not have xasupport=on specified as an initialization parameter, the request fails and the global transaction rolls back. We noted the following message in the WebSphere Application Server stderr log: javax.transaction.SystemException: XAResource start association error:XAER_RMERR 5.9.2 Tips and utilities In this section, we discuss additional tips and utilities for problem determination of transaction related problems. For general tips and information about standard utilities, such as the various Gateway daemon traces, see 2.9.2, “Tips and utilities” on page 62. Identifying the transaction As different parts of a transaction are processed by different tiers of the infrastructure, the transaction is identified in different ways. Table 5-6 and the definitions that follow show the different transaction identifiers that are used by the different software components. Table 5-6 Transaction references Software Component Transaction Identifier WebSphere Application Server XID CICS Transaction Gateway XID (for XA transactions Luw_token (for extended LUWs) RRS XID UR identifier (URID) Chapter 5. Gateway daemon transactions 221 Software Component Transaction Identifier CICS TS UOW Netuowid XID An X/Open identifier (XID) acts as the identifier for a distributed unit of work. An XID consists of a Global transaction identifier (GTRID) and a Branch qualifier (BQUAL). Luw_token A token used by the CICS TG to tie together multiple DPL calls into an extended LUW. UR identifier The URID uniquely identifies the RRS unit of recovery (UR). Multiple URIDs can be involved in a single XID. UOW The CICS unit of work identifier. For transactional EXCI calls, the CICS UW references the RRS URID. Netuowid The CICS network UOW identifier which is identical for all CICS UOWs which are connected within a single CICS distributed unit of work. CICS UOW commands When investigating transaction problems, in addition to using the RRS ISPF panels as shown in our test scenarios, you might also find it useful to use the CICS CEMT INQUIRE command to inquire on CICS units of work: CEMT INQUIRE UOW This command returns information about a named unit of work, or about all the UOWs currently in the system. It displays the state of the UOW (for example, INDOUBT) and whether it is active, waiting, or shunted. Note: A shunted UOW is one awaiting resolution of an in-doubt failure, a commit failure, or a backout failure. CEMT INQUIRE UOWENQ Retrieves information about enqueues held or waited on by a UOW, or about UOWs holding or waiting on a specified enqueue. CEMT INQUIRE UOWLINK Retrieves information about connections involved in units of work. A connection can be to a remote system, or to a task-related user exit. 222 CICS Transaction Gateway for z/OS Version 6.1 For example, Figure 5-31 shows the results of a CEMT INQUIRE UOWLINK command which shows a CICS UOW with two links: a link to RRS and a link to Gateway daemon CITGT1G. I UOWLINK STATUS: RESULTS - OVERTYPE TO MODIFY Uowl(01000017) Uow(BDBEBFBF1BCA0D66) Con Lin(RRS ) Unk Rrms Ok Net(..PSSCX9.A6POTGT1....C_....) Uowl(01010047) Uow(BDBEBFBF1BCA0D66) Con Lin(CITGT1G ) Unk Irc Sys(GT1G) Net(..PSSCX9.A6POTGT1....C_....) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGT1 APPLID=A6POTGT1 TIME: 16.56.04 DATE: 10.11.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 5-31 CEMT INQUIRE TASK on A6POTGR1 with the mirror in RRSEXIT state See 6.5.2, “Test scenarios” on page 237 for an example scenario in which these commands are used to analyze an in-doubt failure. Chapter 5. Gateway daemon transactions 223 224 CICS Transaction Gateway for z/OS Version 6.1 6 Chapter 6. Gateway daemon transactions with TCP/IP load balancing In this chapter, we outline the additional configuration steps that are required to support XA transactions when operating a Gateway daemon within a TCP/IP load balanced configuration. We introduce the concept of a CICS TG group that consists of a CICS TG master process and a set of cloned Gateway daemon instances, all listening on the same TCP/IP port. The role of the CICS TG master is to coordinate transactions for the CICS TG group. We also document how we configured the CICS TG group to support transactions and explain how we tested the environment. © Copyright IBM Corp. 2006. All rights reserved. 225 6.1 Preparation For this chapter, we used WebSphere Application Server for Windows and a cloned group of Gateway daemons running on a single z/OS LPAR. For information about transactional support with a stand-alone Gateway daemon, see Chapter 5, “Gateway daemon transactions” on page 157. Windows z/OS WebSphere Application Server JSP Global transaction Servlet servlet EJB JDBC Transaction Manager CICS ECI XA JCA resource adapter Gateway daemon CICS TG Master Port TCP Datasource EXCI DB2 Resource Manager RRS CICS A Resource Manager Figure 6-1 Software components: Transaction scenario with CICS TG group Figure 6-1 shows a global transaction spanning updates to multiple resource managers, a CICS region, and a DB2 database on Windows. The global transaction is coordinated by the transaction manager, WebSphere Application Server. The CICS region is accessed via one of the Gateway daemon instances which are part of a CICS TG group. RRS is used to coordinate the resource updates on z/OS. Access to RRS, for each of the Gateway daemon instances, is controlled by the CICS TG master process. Note: In our global transaction configuration we document the setup steps for transactions which update recoverable resources in two CICS regions, but we do not document setup or testing of DB2. 226 CICS Transaction Gateway for z/OS Version 6.1 6.2 Software checklist Table 6-1 shows the levels of software that we used for our configuration. Table 6-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS 1.6 Windows 2000 SP2 CICS Transaction Gateway for z/OS V6.1 IBM WebSphere Application Server Network Deployment V6.0.2 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. CICS Transaction Server for z/OS V3.1 6.3 Transaction affinity considerations The Gateway daemon can be used with different TCP/IP load balancing technologies, such as TCP/IP port sharing and Sysplex Distributor. An example high availability configuration is shown in Figure 3-14 on page 105. In this section, we consider what restrictions apply when using TCP/IP load balancing with either extended LUWs or XA transactions. 6.3.1 Extended LUWs The first request of an Extended LUW sequence returns an LUW token to the ECI resource adapter. The LUW token is generated by the Gateway daemon which processes the request. The LUW token is Gateway daemon specific, such that any subsequent request from the resource adapter including the LUW token must be serviced by the same Gateway daemon. This creates an affinity between the resource adapter and a specific Gateway daemon instance for the duration of the extended LUW. Figure 6-2 shows the flows that occur between the CICS ECI resource adapter and a Gateway daemon during the processing of an extended LUW. The complete sequence of flows, including the ECI requests and the commit flow must be processed by the same Gateway daemon. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 227 Windows z/OS WebSphere Application Server EJB ECI request 1 CICS ECI resource Adapter Conn Factory ECI response 1 (LUW-token) ECI request 2 (LUW-token) ECI response 2 (LUW-token) Transaction Manager Gateway Daemon Instance Commit (LUW-token) Commit response Figure 6-2 Extended LUW transaction affinity If the connection from the WebSphere Application Server to the Gateway daemon is lost, for example, the Gateway daemon suffers a failure, then the next request (another ECI request or a commit flow) fails. The transaction manager by default rolls back the transaction. Unlike for a two-phase commit resource, there is no recovery from a communication failure with a one-phase commit resource. When using extended LUWs within a TCP/IP load balancing configuration, the complete set of flows for each extended LUW must be managed by a single Gateway daemon instance. 6.3.2 XA transactions The first request of an XA transaction sequence for a specific connection factory includes an Xid (X/Open identifier) that is generated by the WebSphere Application Server. Subsequent ECI requests include this Xid. There is an affinity between the resource adapter and a specific Gateway daemon instance for all ECI requests during the post-prepare phase of an XA transaction. If a subsequent ECI request for an in-flight XA transaction is received by a different Gateway daemon (for example, after a failure of the Gateway daemon instance that processed the previous ECI requests), then the request will fail with a CICS abend AITD. However, the two-phase commit syncpoint flows (prepare and commit) can be serviced by another Gateway daemon instance in the same CICS TG group. 228 CICS Transaction Gateway for z/OS Version 6.1 Figure 6-3 shows the flows that occur between the CICS ECI XA resource adapter and a Gateway daemon during the processing of an XA transaction. The CICS TG master process maintains XA transaction state information, including a table of all Xids in which the CICS TG group has an interest. The complete sequence of ECI requests must be serviced by a specific Gateway daemon. The prepare and commit flows can be serviced by other Gateway daemon instances within the same CICS TG group. z/OS Windows EJB ECI request 1 (Xid) CICS ECI XA resource Adapter Conn Factory Transaction Manager ECI response 1 ECI request 2 (Xid) CICS TG master WebSphere Application Server Gateway Daemon Instance ECI response 2 Prepare (Xid) Prepare response Commit (Xid) Xid Table Figure 6-3 XA transaction affinity Because the CICS ECI XA resource adapter supports two-phase commit processing, failures in the syncpoint process can produce in-doubt situations. This occurs if the Gateway daemon has responded positively to the prepare flow from the WebSphere Application Server but because of an application server failure, the Gateway daemon does not receive a commit flow. In this case, the WebSphere Application Server sends a recover flow when it is restarted to tell the Gateway daemon whether it should commit or backout. Transaction recovery flows can be serviced by any Gateway daemon instance in the same CICS TG group. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 229 Restriction: The Gateway daemon instances within an XA-enabled CICS TG group must be started on the same MVS image. This means that you cannot distribute ECI requests across Gateway daemons on different LPARs if the ECI request is part of an XA transaction. 6.3.3 XA-enabled CICS TG group An XA-enabled CICS TG group consists of: A CICS TG master process (ctgmaster address space) Two or more XA-enabled cloned Gateway daemons running on the same MVS image In this configuration, when processing XA transaction requests, the ctgmaster accesses RRS on behalf of each of the Gateway daemons in the group. As with a stand-alone XA-enabled Gateway daemon, the RRS registration by ctgmaster must be authorized (see “Enabling CTGRRMS services” on page 195). Therefore, during initialization, ctgmaster ensures that the CTGRRMS address space is started and available for use. If CTGRRMS is not already active on the LPAR when ctgmaster is started, then ctgmaster attempts to start it. 6.4 Configuring an XA-enabled CICS TG group In 3.7.2, “Cloned Gateway daemon connectivity tests” on page 114, we showed how to create cloned Gateway daemons in a TCP/IP port sharing configuration. In this section, we describe the additional configuration changes that we made in order to XA-enable the cloned gateway daemons. We needed to consider the following CICS TG configuration steps: Configuring and starting the CICS TG master process Configuring each Gateway daemon instance 6.4.1 CICS TG master process configuration The CICS TG master process is started by executing the USS executable ctgmaster, which can be started from a USS shell or in batch mode (via CTGBATCH). Running ctgmaster within a USS shell session is only suitable for usage within a test environment. Batch mode is the recommended runtime environment for a production system. 230 CICS Transaction Gateway for z/OS Version 6.1 Note: You can also use the z/OS supplied BPXBATCH utility to run ctgmaster in batch mode, but we recommend the CICS TG supplied utility CTGBATCH, which provides the JES logging capability. The CICS TG master process is configured using environment variables or startup switches. Switches allow the environment variables to be overridden. However, they are more suited to usage within the USS shell because the PARM string on the JCL EXEC command is limited to 100 bytes. There is no equivalent to the Gateway daemon environment variable CTGSTART_OPTS for the CICS TG master process. Table 6-2 shows the mapping between CICS TG master process environment variables and command line switches and overrides. Table 6-2 ctgmaster configuration environment variables and override switches Environment variable Override switch Value options CTG_MASTER_RRMNAME -rrmname= <CICS TG group RRM name> CTG_MASTER_INPUT -input= STDIN|CONSOLE CTG_MASTER_TRACE_ON -trace_on YES|NO CTG_MASTER_TRACE -trace= <HFS filename> CTG_MASTER_NATLANG -natlang= EN|JA|ZH CTG_XA_MAX_TRAN -xa_max_tran= 1-8192 We now consider the key configuration settings for the CICS TG master process in more detail. CTG_MASTER_RRMNAME The environment variable CTG_MASTER_RRMNAME is used to specify the name that ctgmaster uses when registering with RRS. It must match the name specified for CTG_MASTER_RRMNAME for each of the Gateway daemon instances within the CICS TG group. The name must be unique within the Sysplex. We set CTG_MASTER_RRMNAME to CTG.RRM.CITGR0G for our CICS TG group. The default value is CICSTG.DEFAULT.MASTERRRM. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 231 CTG_MASTER_INPUT During initialization, the CICS TG master process starts listening for commands entered either at the z/OS console, or from stdin. The default behavior is to listen on stdin. Because we started ctgmaster from CTGBATCH, we set CTG_MASTER_INPUT to CONSOLE. Note: Starting ctgmaster in batch mode without defining the input type to CONSOLE causes initialization to fail. Message CTG8997E reports that the stdin console is not available. See 6.6, “Operations” on page 248 for a list of commands that can be used to operate ctgmaster. CTG_XA_MAX_TRAN The Gateway daemon environment variable CTG_XA_MAX_TRAN is described in “Enabling XA support in the Gateway daemon” on page 196. When running within a CICS TG group, the Gateway daemon environment variable CTG_XA_MAX_TRAN is ignored, because it is the CICS TG master process which maintains XA transaction state for the group of Gateway daemons. We set the CTG_XA_MAX_TRAN for the CICS TG master process to the default value 8192. This allows for a collective total of 8192 concurrent XA transactions, spread across all Gateway daemons in the CICS TG group. Starting the CICS TG master process Example 6-1 shows how we customized the sample JCL in hlq.SCTGSAMP(CTGMASTR) to create the start procedure for our CICS TG master process CITGR0G. Example 6-1 CICS TG master process PROCLIB member CITGR0G //CITGR0G PROC // SET CTGHLQ='CTGV61' // SET CTGUSR='CITG.CTG610.STDENV(CITGR0G)' // SET LEOPTS='/' //MASTER EXEC PGM=CTGBATCH, // PARM='&LEOPTS./usr/lpp/cicstg/ctg610/bin/ctgmaster' //STEPLIB DD DSN=&CTGHLQ..SCTGLOAD,DISP=SHR //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //STDENV DD DSN=&CTGUSR.,DISP=SHR //* 232 CICS Transaction Gateway for z/OS Version 6.1 Example 6-2 shows how we customized the sample STDENV file hlq.SCTGSAMP(CTGMENV) to create a STDENV member for our CICS TG master process. Example 6-2 CICS TG master process STDENV member CITGR0G _BPX_SHAREAS=YES CTG_MASTER_RRMNAME=CTG.RRM.CITGR0G CTG_XA_MAX_TRAN=8192 TMPDIR=/cicstg/ctg610/config TZ=GMT-2 CTG_MASTER_INPUT=CONSOLE CTG_MASTER_TRACE=/cicstg/ctg610/trace/CITGR0G.trc CTG_MASTER_TRACE_ON=YES 6.4.2 Gateway daemon configuration In 5.6.3, “Configuring CICS TG” on page 175, we showed how we enabled XA transaction support for the Gateway daemon CITGR1G by: Enabling CTGRRMS services Specifying a unique RRM name Specifying the initialization parameter xasupport=on We followed the same steps to enable XA transaction support for the Gateway daemon CITGR2G (the second Gateway daemon in our CICS TG R group). CTG_MASTER_RRMNAME The environment variable CTG_MASTER_RRMNAME is used to specify the name that ctgmaster uses when registering with RRS. CTG_MASTER_RRMNAME must be the same for ctgmaster and for each of the Gateway daemon instances within the CICS TG group. During Gateway daemon initialization, if the environment variable CTG_MASTER_RRMNAME is defined, then this signifies that the Gateway daemon is part of a group. Gateway daemon initialization fails if a CICS TG master process with matching CTG_MASTER_RRMNAME is not active. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 233 We added entries for CTG_MASTER_RRMNAME to the STDENV files for the CITGR1G and CITGR2G Gateway daemons. Example 6-3 shows the environment variables for Gateway daemon CITGR1G. Example 6-3 CITG.CTG610.STDENV(CITGR1G) _BPX_SHAREAS=YES PATH=/bin:/usr/lpp/java142/J1.4/bin CTG_MASTER_RRMNAME=CTG.RRM.CITGR0G CTG_RRMNAME=CTG.RRM.CITGR1G.IBM.UA #CTG_XA_MAX_TRAN=1000 DFHJVPIPE=CITGR1G CICSCLI=/cicstg/ctg610/config/CITGR1G.ini ..... Example 6-4 shows the environment variables for Gateway daemon CITGR2G. Example 6-4 CITG.CTG610.STDENV(CITGR2G) _BPX_SHAREAS=YES PATH=/bin:/usr/lpp/java142/J1.4/bin CTG_MASTER_RRMNAME=CTG.RRM.CITGR0G CTG_RRMNAME=CTG.RRM.CITGR2G.IBM.UA #CTG_XA_MAX_TRAN=1000 DFHJVPIPE=CITGR2G CICSCLI=/cicstg/ctg610/config/CITGR1G.ini .... Because a Gateway daemon which is part of a CICS TG group can process extended LUWs in addition to XA transaction requests, the environment variable CTG_RRMNAME is used for RRS registration of the Gateway daemon instance. 6.5 Testing the CICS TG group configuration We started the CICS TG group by starting the ctgmaster and Gateway daemons in the following order: 1. ctgmaster 2. Gateway daemon CITGR1G 3. Gateway daemon CITGR2G We show how we tested the CICS TG group configuration using our sample J2EE application CTGTesterCCIXA. Figure 6-4 shows the basic outline of our environment. 234 CICS Transaction Gateway for z/OS Version 6.1 tx1.mop.ibm.com CITGR1G/CITGR2G CITGR1CI (applid A6POTGR1) Gateway daemon ECI Request within XA txn Windows 13101 JNI z/OS CICS EXCI Long Running Mirror ECIT WebSphere Application Server V6 JSP/Servlet CICS TG V6.1 CTGTesterCCIXA Transaction CCI Required Cam21-pc11 ctgmaster CICS ECI XA Resource Adapter ECI Request within XA txn Gateway daemon 12101 RRS CTGRRMS CICS EXCI JNI CITGT1G tx1b.mop.ibm.com Long Running Mirror ECIT CITGT1CI (applid A6POTGT1) Figure 6-4 CICS TG group scenario The CTGTesterCCIXA J2EE application contains two ECI resource references and can be used to call programs running in two CICS regions via different connection factories. The two requests are part of the same global transaction. For further details about CTGTesterCCIXA, refer to Appendix A, “Sample applications” on page 361. In this test scenario, the first connection factory is defined with port 13101 and IP name tx1.mop.ibm.com. Because the Gateway daemons CITGR1G and CITGR2G are both configured to listen on port 13101, socket connections for this connection factory will be balanced across the two Gateway daemons. CITGR1G and CITGR2G are part of the same CICS TG group , therefore, the CICS TG group master process accesses RRS on behalf of the Gateway daemons. The second connection factory is defined with port 12101 and IP address tx1b.mop.ibm.com, so that socket connections for this connection factory are processed by the stand-alone Gateway daemon CITGT1G. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 235 We used this configuration for two test scenarios: Normal termination of an XA transaction An update and commit of recoverable resources in two CICS regions within one XA transaction. The first leg of the XA transaction is processed by the CICS TG group. The second leg of the transaction is processed by CITGT1G, a stand-alone XA-enabled Gateway daemon. In-doubt failure of an XA transaction A WebSphere Application Server failure during the syncpoint processing which resulted in an in-doubt situation. We show how the in-doubt situation is resolved after the application server is restarted, and that the recover flow can be processed by either member of the CICS TG group. 6.5.1 Definitions checklist Table 6-3 and Table 6-4 list the definitions that we used to configure the scenario. Table 6-3 Definitions checklist for CICS TG group R Value CICS TG master process Port sharing Gateway daemons CICS TS hostname - tx1.mop.ibm.com - IP address - 9.100.193.122 - TCP/IP port - 13101 - Job name CITGR0G CITGR1G CITGR2G CITGR1CI APPLID - - A6POTGR1 NETNAME (DFHJVPIPE) - CITGR1G CITGR2G - CONNECTION - CTG_MASTER_RRMNAME CTG.RRM.CITGR0G CTG_RRMNAME 236 CICS Transaction Gateway for z/OS Version 6.1 GR1G CTG.RRM.CITGR0G CTG.RRM.CITGR1G.IBM.UA CTG.RRM.CITGR2G.IBM.UA Table 6-4 Definitions checklist for CICS TG T1 Value Gateway daemon CICS TS hostname tx1b.mop.ibm.com - IP address 9.100.193.121 - TCP/IP port 12101 - Job name CITGT1G CITGT1CI APPLID A6POTGT1 NETNAME (DFHJVPIPE) CITGT1G CONNECTION GT1G Note: On our system the CITGT1G Gateway daemon binds to a different TCP/IP address than the Gateway daemons within the CICS TG group, but all the Gateway daemons are actually running on the same z/OS image. In practice the CITGT1G Gateway daemon could be running on a completely different system. The CICS TG group Gateway daemon instances, however, must run on the same MVS image. 6.5.2 Test scenarios We now show the results of our two test scenarios: Normal termination of an XA transaction In-doubt failure of an XA transaction Normal termination of XA transaction We started two Web browser sessions and ran the CTGTesterCCIXA application on each Web browser. We specified WW as the input to the COMMAREA which means that the ECIPROG program writes to a recoverable TS Queue. Each Web browser session, therefore, initiates an XA transaction consisting of two ECI calls: one request to CICS region A6POTGR1 via the CICS TG group R and one request to CICS region A6POTGT1 via stand-alone Gateway daemon CITGT1G. We wanted to show in this basic test that a different Gateway daemon in the CICS TG group participates in each of the XA transactions. We checked the Gateway daemon logs, which showed that each Gateway daemon received a connection request (Example 6-5). Chapter 6. Gateway daemon transactions with TCP/IP load balancing 237 Example 6-5 Load-balancing concurrent connections to tx1.mop.ibm.com:13101 From CITGR1G stdout log: 10/21/05 16:36:17:303 [0] CTG6506I Client connected. [ConnectionManager-9] tcp@Socket[addr=9.100.199.134,port=1412,localport=13101] From CITGR2G stdout log: 10/21/05 16:36:17:081 [0] CTG6506I Client connected. [ConnectionManager-9] tcp@Socket[addr=9.100.199.134,port=1411,localport=13101] Note: The addr=9.100.199.134,port=1412 and addr=9.100.199.134,port=1411 values represent the protocol characteristics of specific connections in the WebSphere Application Server connection pool for connection factory CITGR1CI-tx1-13101-XA. See “Creating connection factories” on page 200 for information about how this connection factory was defined. We then checked the state of the TS Queues on A6POTGR1 and A6POTGT1 to confirm that each request resulted in an update to the TS Queue RECIPROG. In-doubt failure of an XA transaction In this scenario, we simulated a WebSphere Application Server failure during the syncpoint process of the XA transaction. This resulted in both the UR in RRS, and the UOW in CICS being in an in-doubt state. Note: This scenario is for illustrative purposes only. In-doubt failures are very rare in practice. While the XA transaction was in-doubt, we used the following tools to analyze the state of the transaction: ctgmaster QUERY XIDS command to query the ctgmaster process CICS CEBR transaction to query the state of the TS queues CICS CEMT INQUIRE TASK, CEMT INQUIRE UOW and CEMT INQUIRE UOWLINK commands to query the state of CICS tasks and UOWs RRS ISPF panels to query the state of RRS URs 238 CICS Transaction Gateway for z/OS Version 6.1 ctgmaster QUERY XIDS command The QUERY XIDS command is used to retrieve the list of Xids in which the CICS TG group has an interest. We executed the following MVS command: /F CITGR0G,APPL=QUERY,XIDS Example 6-6 shows the output for the QUERY XIDS command. Example 6-6 ctgmaster QUERY XIDS output CTG8998I ctgmaster command accepted. CTG8950I ctgmaster query xids result - entry 000001 UR Interest : 7E2D80800000009D0055000155555555 XID.formatID : 1463898948 XID.gtrid : 00-0F 00000107 03167364 00000001 00000001 10-1F 652EE808 ECBAEA9A B3C52EA4 110E2BA2 20-23 FCD740E7 XID.bqual : 00-0F 00000107 03167364 00000001 00000001 10-1F 652EE808 ECBAEA9A B3C52EA4 110E2BA2 20-2F FCD740E7 00000001 00000000 00000000 30-35 00000000 0001 CTG8951I ctgmaster query xids found 1 entries. This example shows that the CICS TG group has an interest in RRS UR 7E2D80800000009D0055000155555555 which has a GTRID of: 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E7 CEBR We used the CICS-supplied terminal transaction CEBR to query the state of the recoverable TS Queue RECIPROG on each CICS region. Example 6-7 shows the contents of the RECIPROG TS Queue on both CICS regions. Example 6-7 CEBR output from A6POTGR1 and A6POTGT1 during in-doubt window CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF 1 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 19/10/05 12:31:25 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF 2 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 19/10/05 12:31:26 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** Chapter 6. Gateway daemon transactions with TCP/IP load balancing 239 This shows that the mirror tasks have updated the TS Queue RECIPROG on both CICS regions. CICS tasks and UOWs We used the CEMT INQUIRE TASK to display the mirror task ECIT on each CICS region. Figure 6-5 shows that the mirror task on CICS region A6POTGT1 has ended. I TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000485) Tra(CEBR) Fac(N124) Sus Ter Pri( 001 ) Sta(TO) Use(CITGT1D ) Uow(BDC72F3041FDE708) Hty(ZCIOWAIT) Tas(0000582) Tra(CEMT) Fac(N107) Run Ter Pri( 255 ) Sta(TO) Use(CITGT1D ) Uow(BDC8945686506548) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGT1 APPLID=A6POTGT1 TIME: 12.34.37 DATE: 10.19.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-5 CEMT INQUIRE TASK for CICS region A6POTGT1 Figure 6-6 shows that the mirror task on CICS region A6POTGR1 (Task 150) is suspended in RRMSEXIT state, which means that CICS is waiting for a commit or backout flow from RRS. I TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000150) Tra(ECIT) Sus Tas Pri( 001 ) Sta(TO) Use(CITGR1D ) Uow(BDC8939F7B6C790F) Hty(RRMSEXIT) Tas(0000151) Tra(CEMT) Fac(N105) Run Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDC8943C20924C20) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 12.34.12 DATE: 10.19.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-6 CEMT INQUIRE TASK for CICS region A6POTGR1 240 CICS Transaction Gateway for z/OS Version 6.1 We then looked at the state of the CICS UOW BDC8939F7B6C790F on CICS region A6POTGR1 using the CEMT INQUIRE UOW command (Figure 6-7). INQUIRE UOW(BDC8*) STATUS: RESULTS - OVERTYPE TO MODIFY Uow(BDC8939F7B6C790F) Ind Act Tra(ECIT) Tas(0000150) Age(00000246) Ter(21 ) Netn(CITGR2G ) Use(CITGR1D ) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 12.35.35 DATE: 10.19.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-7 CEMT INQUIRE UOW for CICS region A6POTGR1 Example 6-8 shows the expanded view of the UOW. Example 6-8 CEMT INQUIRE UOW INDOUBT - detailed view INQUIRE UOW(BDC8*) RESULT - OVERTYPE TO MODIFY Uow(BDC8939F7B6C790F) Uowstate( Indoubt ) Waitstate(Active) Transid(ECIT) Taskid(0000150) Age(00000260) Termid(21) Netname(CITGR2G) Userid(CITGR1D) Waitcause() Link() Sysid() Netuowid(..PSSCX9.A6POTGR1Hl.#......) The expanded view of the UOW reveals that the UOW is in-doubt and that the Netname that is associated with the UOW is CITGR2G (the jobname of the Gateway daemon that initiated the CICS UOW). Important: When a UOW is in-doubt, CICS retains locks on the recoverable resources that the UOW has updated. This prevents further tasks from updating the resource. The locks are maintained until the in-doubt situation is resolved. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 241 From our analysis so far, we can establish that the XA transaction has failed in-doubt. The first leg of the XA transaction (processed by the CICS TG group) has resulted in a suspended task and an in-doubt UOW in CICS region A6POTGR1. Whereas the second leg of the transaction (processed by the stand-alone Gateway daemon) has completed normally. We then used the CEMT INQUIRE UOWLINK command on CICS region A6POTGR1 to investigate further, in particular, to determine whether any other systems have links to this in-doubt UOW (Figure 6-8). INQUIRE UOWLINK STATUS: RESULTS - OVERTYPE TO MODIFY Uowl(010400BC) Uow(BDC8939F7B6C790F) Con Lin(RRS ) Unk Rrms Ok Net(..PSSCX9.A6POTGR1Hl.#......) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 12.36.05 DATE: 10.19.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-8 CEMT INQUIRE UOWLINK Example 6-9 shows the expanded view of the UOWLINK. Example 6-9 CEMT INQUIRE UOWLINK - detailed view INQUIRE UOWLINK RESULT - OVERTYPE TO MODIFY Uowlink(010400BC) Uow(BDC8939F7B6C790F) Type(Connection) Link(RRS) Action( ) Role(Unknown) Protocol(Rrms) Resyncstatus(Ok) Sysid() Rmiqfy() Netuowid(..PSSCX9.A6POTGR1Hl.#......) Urid(BDC8939F7E517A5C000000BA01030000) Host() The CEMT INQUIRE UOWLINK command retrieves information about connections involved in units of work. A connection can be to a remote system, or to a task-related user exit. We can see that our in-doubt CICS UOW has a link to RRS and that the RRS URID is BDC8939F7E517A5C000000BA01030000. We use this information to query the status of the UR in the RRS ISPF panels. 242 CICS Transaction Gateway for z/OS Version 6.1 RRS ISPF panels We explain how the RRS ISPF panels can be used to retrieve information about RRS URs in “RRS ISPF panels” on page 185. The UR details for URID BDC8939F7E517A5C000000BA01030000 are shown in Figure 6-9. We see that: The UR state is in-doubt The RRS work manager for the UR is the CICS TG master process with CTG_MASTER_RRMNAME CTG.RRM.CITGR0G The section of the panel showing Expressions of Interest confirms that the CICS TG master process has a syncpoint coordinator or SDSRM (Server Distributed Syncpoint Resource Manager) role in the UR The CICS region A6POTGR1 has a Participant role in the UR RRS Unit of Recovery Details Row 1 to 2 of 2 Commands r-Remove Interest v-View URI Details UR identifier : BDC8939F7E517A5C000000BA01030000 Create time : 2005/10/19 10:31:24.853151 Comments : X UR state : InDoubt UR type : Prot System : X1 Logging Group : X1 SURID : N/A Work Manager Name : CTG.RRM.CITGR0G / Display Work IDs / Display IDs formatted Luwid . : Present Eid . . : Not Present Xid . . : Present Expressions of Interest: S RM Name Type Role CTG.RRM.CITGR0G Prot SDSRM DFHRXDM.A6POTGR1.IBM Prot Participant ******************************* Bottom of data ******************************** Command ===> F1=Help F2=Split F3=Exit F7=Up F8=Down Scroll ===> PAGE F9=Swap F12=Cancel Figure 6-9 RRS UR details for in-doubt UR Chapter 6. Gateway daemon transactions with TCP/IP load balancing 243 Finally, we show the RRS Unit of Recovery Work IDs panel for our in-doubt UR (Figure 6-10). RRS Unit of Recovery Work IDs UR identifier : BDC8939F7E517A5C000000BA01030000 Logical Unit of Work Identifier (LUWID): PSSCX9.A6POTGR1 C8939F7B6448 0001 NetID.LuName : PSSCX9.A6POTGR1 TP Instance : C8939F7B6448 SeqNum . . . : 0001 Enterprise Identifier (EID) TID : (decimal) GTID : X/Open Transactions Identifier (XID) Format ID : 001463898948 (decimal) 57415344 (hexadecimal) GTRID : 00-0F 00000107 03167364 00000001 00000001 |................| 10-1F 652EE808 ECBAEA9A B3C52EA4 110E2BA2 |..Y......E.u...s| 20-23 FCD740E7 |.P X | BQUAL : 00-0F 10-1F 20-2F 30-35 F1=Help F9=Swap 00000107 652EE808 FCD740E7 00000000 F2=Split F12=Cancel 03167364 00000001 00000001 |................| ECBAEA9A B3C52EA4 110E2BA2 |..Y......E.u...s| 00000001 00000000 00000000 |.P X............| 0001 |...... | F3=Exit F5=ListErr F7=Up F8=Down Figure 6-10 RRS Unit of Recovery Work IDs for in-doubt UR You can check that the GTRID for the UR matches that displayed by the ctgmaster QUERY XIDS command in Example 6-6 on page 239. Resolving the in-doubt transaction From our analysis, we have seen that the suspended ECIT task and in-doubt UOW in CICS region A6POTGR1 are awaiting resolution. This resolution occurs when the WebSphere Application Server is restarted. 244 CICS Transaction Gateway for z/OS Version 6.1 Note: After using RRS to list in-doubt URs, it is possible to commit or backout an in-doubt UR manually. Before doing this you should attempt to resolve the in-doubt situation by restarting the WebSphere Application Server and all resource managers which have an interest in the global transaction. In order to force an in-doubt UR to commit or to backout manually, the operator user ID must have ALTER access to the FACILITY class MVSADMIN.RRS.COMMANDS. We have also established that the ECIT task was initiated by an ECI request processed by Gateway daemon CITGR2G. In order to test the recovery capabilities of the CICS TG group, we shut down the Gateway daemon CITGR2G before restarting the WebSphere Application Server. After restarting the WebSphere Application Server, we used the following tools to analyze the state of the XA transaction: The ctgmaster QUERY XIDS command CICS CEBR transaction to query the state of the TS queues CICS CEMT INQUIRE UOW INDOUBT command to verify that the in-doubt UOW in CICS is resolved RRS archive log to view the completed global transaction from an RRS perspective ctgmaster QUERY XIDS command Example 6-10 shows the output of the QUERY XIDS command. We can see that the CICS TG group has no interest in any Xid. Example 6-10 ctgmaster QUERY XIDS output after in-doubt resolution CTG8998I ctgmaster command accepted. CTG8951I ctgmaster query xids found 0 entries. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 245 CEBR Example 6-11 shows the contents of the RECIPROG TS Queue on both CICS regions. We see the TS queue updates on both CICS regions. Example 6-11 CEBR output from A6POTGR1 and A6POTGT1 after in-doubt resolved CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF 1 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 19/10/05 12:31:25 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF 2 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 19/10/05 12:31:26 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** CEMT INQUIRE UOW INDOUBT The output of the CEMT INQUIRE UOW INDOUBT command on CICS region A6POTGR1 (Example 6-11) confirms that there are no in-doubt UOWs remaining. INQUIRE UOW INDOUBT STATUS: RESULTS - OVERTYPE TO MODIFY Uow(* ) Ind RESPONSE: 1 ERROR PF 1 HELP 3 END 5 VAR NOT FOUND SYSID=TGR1 APPLID=A6POTGR1 TIME: 12.42.17 DATE: 10.19.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 6-11 CEMT INQUIRE UOW INDOUBT after in-doubt resolution RRS Archive Log The RRS archive log shows information relating to all RRS URs, including completed URs. See “RRS Archive logs” on page 187 for more information about browsing RRS archive logs. 246 CICS Transaction Gateway for z/OS Version 6.1 Example 6-12 shows the RRS archive log entries for our resolved in-doubt transaction. Example 6-12 RRS Archive log - XA transaction in-doubt resolution X1 2005/10/19 12:32:15.788069 BLOCKID=000000000021B58B URID=BDC893A07E517DD00000006801030000 JOBNAME=CITGT1G USERID=CITGT1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA SYNCPOINT=AtraCmt RETURN CODE=00000000 START=2005/10/19 10:31:25.899029 COMPLETE=2005/10/19 10:32:15.787771 EXITFLAGS=00800000 LUWID=PSSCX9.A6POTGT1 C893A04847C0 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E7 BQUAL= 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E700000001 0000000000000000000000000002 X1 2005/10/19 12:42:06.597390 BLOCKID=000000000021B7F5 URID=BDC8939F7E517A5C000000BA01030000 JOBNAME=CITGR1G USERID=CITGR1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGR0G SYNCPOINT=AtraCmt RETURN CODE=00000000 START=2005/10/19 10:31:28.361576 COMPLETE=2005/10/19 10:42:06.597245 EXITFLAGS=00800000 LUWID=PSSCX9.A6POTGR1 C8939F7B6448 0001 TID= GTID= FORMATID=001463898948 (decimal) 57415344 (hexadecimal) GTRID= 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E7 BQUAL= 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E700000001 0000000000000000000000000001 Looking at the archive log, we are able to confirm that: The two RRS URs involved in the global transaction have the same GTRID: 00000107031673640000000100000001652EE808ECBAEA9AB3C52EA4110E2BA2FCD740E7 The RRS work manager for the first UR (URID=BDC893A07E517DD00000006801030000) is the stand-alone Gateway daemon CITGT1G with CTG_RRMNAME CTG.RRM.CITGT1G.IBM.UA. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 247 This UR was committed (SYNCPOINT=AtraCmt RETURN CODE=00000000) by the Gateway daemon CITGT1G at 12:32:15. The RRS work manager for the second UR (URID=BDC8939F7E517A5C000000BA01030000 ) is the ctgmaster for CICS TG group R with CTG_MASTER_RRMNAME CTG.RRM.CITGR0G. This UR was committed (SYNCPOINT=AtraCmt RETURN CODE=00000000) by the Gateway daemon CITGR1G 10 minutes later at 12:42:06. We know that during the time between the two commit flows, the UR BDC8939F7E517A5C000000BA01030000 was in-doubt. Important: This example clearly shows that the commit flow which resolved the in-doubt UR was processed by Gateway daemon CITGR1G, while the original ECI request was processed by Gateway daemon CITGR2G. A Gateway daemon which is an instance of a CICS TG group can process syncpoint flows on behalf of other Gateway daemons within the same group. 6.6 Operations The creation of an XA-enabled CICS TG group has an impact on the operational procedures for each Gateway daemon instance within the group. 6.6.1 CICS TG group startup and shutdown When starting up a CICS TG group, the master process must be started before the first Gateway daemon. If any Gateway daemon is started while the CICS TG master process is initializing, it suspends until the CICS TG master process initialization is complete. CICS TG master process restart During initialization, the CICS TG master process retrieves information about any URs associated with its CTG_MASTER_RRMNAME which are in-doubt. After the CICS TG master process is started, Gateway daemons which are part of the CICS TG group can start concurrently. CICS TG group normal shutdown sequence The CICS TG master process keeps track of the number of currently active Gateway daemons in the CICS TG group and, on receipt of a shutdown request, waits until all associated Gateway daemons have been shut down. If there are active Gateway daemons in the group at the time of a ctgmaster shutdown 248 CICS Transaction Gateway for z/OS Version 6.1 request, message CTG8942I is written indicating the number of active Gateway daemons (Example 6-13). Example 6-13 ctgmaster pending normal shutdown message /F CITGR0G,APPL=SHUT CTG8998I ctgmaster command accepted. 10/19/2005 13:15:52.584 [0] CTG8932I ctgmaster is being shutdown 10/19/2005 13:15:52.600 [0] CTG8942I ctgmaster is waiting on 1 associated Gateway daemons to complete shutdown When all Gateway daemons in the CICS TG group have completed shutdown, then the CICS TG master process shutdown completes and RRS de-registration takes place. While the CICS TG master process is pending shutdown, no new Gateway daemons within the CICS TG group can start. CICS TG master process immediate shutdown If the CICS TG master process receives the shutdown immediate command, it will terminate itself regardless of the number of associated active Gateway daemons. However, a log message is written indicating the number of associated Gateway daemons known to be active at the time (Example 6-14). Example 6-14 ctgmaster immediate shutdown message /F CITGR0G,APPL=SHUT,IMM CTG8998I ctgmaster command accepted. 10/19/2005 13:16:57.635 [0] CTG8943I ctgmaster immediate shutdown requested. Closing with 1 associated Gateway daemons 10/19/2005 13:16:57.694 [0] CTG8933I ctgmaster shutdown is complete Following an immediate shutdown of the CICS TG master process, the remaining active Gateway daemons continue to operate but cannot process ECI requests which are part of XA transactions. As with a normal shutdown, no new Gateway daemon instances can be started until the CICS TG master process has itself been re-started. Chapter 6. Gateway daemon transactions with TCP/IP load balancing 249 6.6.2 CICS TG group commands Table 6-5 summarizes the set of ctgmaster commands. Table 6-5 ctgmaster commands USS command (input=stdin) MVS command (input=console) /F <jobname>,APPL=….. query xids QUERY,XIDS trace on|off TRACE,ON|OFF shut|shutdown imm|immediate SHUT|SHUTDOWN[,IMM|IMMEDIATE] dump DUMP help HELP Tip: The ctgmaster command accepts commas or spaces between verbs in the APPL= command string. QUERY XIDS The QUERY XIDS command is used to retrieve the list of Xids in which the CICS TG group has an interest. Example 6-6 on page 239 shows sample output for the QUERY XIDS command. TRACE The TRACE command is used to start or stop tracing activity in the CICS TG master process dynamically. The trace destination depends on the value of CTG_MASTER_TRACE environment variable or the -trace override switch. Example 6-15 shows sample output. Example 6-15 ctgmaster TRACE command output /F CITGR0G,APPL=TRACE,ON CTG8998I ctgmaster command accepted. CTG8934I ctgmaster trace setting has been updated (state=1) /F CITGR0G,APPL=TRACE,OFF CTG8998I ctgmaster command accepted. CTG8934I ctgmaster trace setting has been updated (state=0) 250 CICS Transaction Gateway for z/OS Version 6.1 Tip: The current tracing status can be queried by omitting the ON|OFF parameter on the trace command. SHUTDOWN The SHUTDOWN command shuts down ctgmaster. See 6.6.1, “CICS TG group startup and shutdown” on page 248 for guidance on shutting down ctgmaster. DUMP Invoking the ctgmaster DUMP command does not produce an SDUMP but writes a message to stdout that contains the ctgmaster internal state data. It is intended for IBM technical support purposes. HELP The HELP command writes a message to stdout describing each of the available systems management commands. 6.7 Problem determination In this section, we describe additional tips and utilities for problem determination CICS TG group related problems. For other problem determination guidance about transaction related problems see 5.9, “Problem determination” on page 221. CTGRRMS authorization The ctgmaster user ID (CITGR0G) needs to be authorized to use CTGRRMS services. It needs UPDATE access to the CICSTG.RRMS.SERVICE RACF FACILITY class profile. You can use the following TSO command to check that the ctgmaster user ID has the appropriate access: RLIST FACILITY CTG.RRMS.SERVICE AUTHUSER Example 6-16 shows the output of this command on our system. Example 6-16 RLIST FACILITY CTG.RRMS.SERVICE AUTHUSER command output USER ---CITGTJ CITGC1G CITGR1G CITGC2G CITGR2G CITGR0G ACCESS -----ALTER UPDATE UPDATE UPDATE UPDATE UPDATE ACCESS COUNT ------ ----000000 000000 000000 000000 000000 000000 Chapter 6. Gateway daemon transactions with TCP/IP load balancing 251 CITGS0G CITGS1G CITGS2G CITGT0G CITGT1G CITGT2G UPDATE UPDATE UPDATE UPDATE UPDATE UPDATE 000000 000000 000000 000000 000000 000000 CICS in-doubt failure messages In the event of an in-doubt failure, the Recovery Manager component of CICS issues a DFHRMxxxx message. The message contains detailed information, including: The time and date of the original failure The transaction identifier and task number The netname of the remote system The operator identifier The operator terminal identifier The network-wide unit of work identifier The local unit of work identifier For examples of how to resolve in-doubt UOWs in CICS, see the CICS Intercommunication Guide, SC34-6448. 6.7.1 Tracing with ctgmaster trace Tracing with ctgmaster trace is controlled by the environment variable CTG_MASTER_TRACE. It is used to set the HFS destination for ctgmaster trace data. If undefined, trace data is written to stderr and is mapped to the STDERR DD card when running in batch mode. This trace is distinct from the Gateway daemon and JNI trace described in 2.9.3, “Tracing” on page 68. Having tracing active for the CICS TG master process does not have any impact on the performance of the Gateway daemons and does not produce a large amounts of data. 252 CICS Transaction Gateway for z/OS Version 6.1 7 Chapter 7. Transactions with WebSphere Application Server for z/OS In this chapter, we describe the transactional capabilities when the CICS Transaction Gateway for z/OS is used with WebSphere Application Server on z/OS. We document how we prepared the system and the values that we used. We also provide details on how we set up and modified the different tiers in our test environment to support transactions. In addition, we explain how we tested a global transaction involving both local and remote CICS connections from WebSphere Application Server for z/OS. © Copyright IBM Corp. 2006. All rights reserved. 253 7.1 Preparation In our testing, we used WebSphere Application Server for z/OS. For information about how to enable transactions when using WebSphere Application Server on Windows see Chapter 5, “Gateway daemon transactions” on page 157. We tested a global transaction with our sample CTGTesterCCIXA application using the cicseciXA.rar. Figure 7-1 shows how a global transaction managed by WebSphere Application Server for z/OS can span different resource managers. z/OS WebSphere Application Server JSP Global transaction RRS Java servlet Servlet CICS ECI XA CTGTesterCCIXA JCA resource JDBC adapter Transaction Manager Resource Manager EXCI CICS A Datasource DB2 Resource Manager Figure 7-1 Software components: WebSphere z/OS transaction scenarios Figure 7-1 shows a global transaction spanning updates to different resource managers, a CICS region, and a DB2 database. The global transaction is coordinated by the transaction manager, which is WebSphere Application Server for z/OS. RRS is used to coordinate the resource updates. Note: In our global transaction configuration, we document the setup steps for transactions which update recoverable resources in two CICS regions, but we do not document setup or testing of DB2. For an overview of basic transactional concepts and standards, see 5.2, “Transactions - what are they” on page 159. 254 CICS Transaction Gateway for z/OS Version 6.1 7.1.1 Software checklist Table 7-1 shows the levels of software that we used for this configuration. Table 7-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS V1.6 Windows 2000 Service Pack 4 WebSphere Application Server V6.0.1 CICS Transaction Server V3.1 CICS Transaction Gateway for z/OS V6.1 IBM SDK for z/OS Java 2 Technology Edition,V1.4.2 SR2 7.2 Resource Recovery Services Resource Recovery Services (RRS), a component of z/OS, provides a global syncpoint manager that any resource manager on z/OS can exploit. It enables transactions to update recoverable resources managed by many resource managers. For information about how RRS is used by CICS and the CICS TG, see 5.3, “Resource Recovery Services” on page 163. 7.3 Transactional support with WebSphere on z/OS Unlike the more traditional transaction managers such as CICS or IMS, WebSphere Application Server for z/OS uses RRS services as a part of its base implementation to provide two-phase commit support. WebSphere Application Server for z/OS does not start without RRS being available, and it abends in the event of an RRS failure. In addition to support for XA-capable resource managers, WebSphere Application Server for z/OS also supports resource managers that provide an RRS-enabled resource adapter. Such a resource adapter implements a specific type of transaction contract with WebSphere Application Server. This means that, in addition to the three types of transaction support defined by the JCA (see “JCA and transactions” on page 167), WebSphere Application Server for z/OS supports a fourth type, RRSTransactional. RRSTransactional is a z/OS only extension to the JCA. Chapter 7. Transactions with WebSphere Application Server for z/OS 255 A global transaction in WebSphere Application Server can use multiple resource managers that use either RRS-enabled or XA resource adapters, and WebSphere Application Server coordinates the updates across all resource managers. To do this, WebSphere Application Server generally takes an RRS SDSRM (Server Distributed Syncpoint Resource Manager) role so that it becomes the syncpoint coordinator and RRS drives the commit or backout processing for those resource managers that support RRS. Connectors for z/OS that are RRS-enabled include: CICS ECI resource adapters IMS Connect IMS JDBC DB2 for z/OS local JDBC driver WebSphere MQ JMS Some of these resource adapters support both XA and RRS modes of operation, depending on how they are configured. The RRS mode of operation is usually restricted to instances where both WebSphere Application Server and the resource manager reside on the same z/OS image. For a complete list of RRS-capable connectors, refer to Systems Programmer's Guide to Resource Recovery Services (RRS), SG24-6980. 7.3.1 Transactional support in the CICS ECI resource adapters The two CICS ECI resource adapters provided with CICS TG for z/OS V6.1 can be used with WebSphere Application Server on z/OS: cicseci.rar The CICS ECI resource adapter has the same transactional behavior as the CICS ECI resource adapter that is provided with CICS TG V6. When installed into WebSphere Application Server on z/OS, it provides two-phase commit support using RRSTransactional for local connections and single-phase commit support for remote connections. cicseciXA.rar When installed into WebSphere Application Server on z/OS, the CICS ECI XA resource adapter provides two-phase commit support by using RRSTransactional for local connections and by supporting the XAResource interface for remote connections. Restriction: For both CICS ECI resource adapters, when a local connection is used, the CICS region must reside on the same z/OS image as the WebSphere Application Server. 256 CICS Transaction Gateway for z/OS Version 6.1 Which resource adapter to use In normal circumstances, you will use a local connection from WebSphere Application Server for z/OS because a direct cross-memory connection to CICS is simpler and performs better than a remote TCP/IP connection via a Gateway daemon. Important: Using a local connection provides two-phase commit support without the additional processing overhead associated with XA transactions. When local connections are used, the transactional behavior of the ECI resource adapters is the same so either can be used. However, you might chose to use the cicseciXA resource adapter if you have plans to use remote connections because you then have greater flexibility in enabling XA transaction support for these connections in the future. There are circumstances in which you might chose to use a remote connection, for example: You do not want to install a CICS region on the same z/OS image as WebSphere Application Server. WebSphere application Server is running on z/OS.e1. Because it is not possible to install a CICS region on a z/OS.e image, you must use a remote connection. You want to connect to a CICS region on another system and, rather than routing the request via a CICS region installed on the same z/OS image and using a SNA based CICS ISC (Intersystem communication) connection, you prefer to use a TCP/IP connection. When a remote connection is used, and if you want XA transaction support for the remote connection, you must install the cicseciXA resource adapter (cicseciXA.rar). Both the ECI and ECI XA resource adapters can be deployed into WebSphere Application Server for z/OS at the same time, provided that they are both from the same release of CICS TG. 1 z/OS.e is a specially priced offering of z/OS that provides select z/OS function for the zSeries 800 system. Traditional workloads, such as CICS TS, are not licensed for z/OS.e. Chapter 7. Transactions with WebSphere Application Server for z/OS 257 7.4 Configuring for global transactions In this section, we show how we configured WebSphere Application Server for z/OS to enable a global transaction that involves two ECI connection factories: a local connection to CICS region and a remote connection to CICS region (via the Gateway daemon CITGT1G). 7.4.1 Definitions checklist Table 7-2 lists the definitions that we used to configure the scenario. Table 7-2 Definitions checklist Value WebSphere Application Server CICS region 1 Gateway daemon CICS region 2 hostname tx1.mop.ibm.com - tx1b.mop.ibm.com - IP address 9.100.193.122 - 9.100.193.121 - TCP/IP port 13880 - 12101 - Job name Daemon region: CITGRDMN Control region: CITGRS1 Servant region: CITGRS1S CITGR1CI CITGT1G CITGT1CI APPLID A6POTGR1 A6POTGT1 NETNAME CITGRS1 - CITGT1G - RRM Name BBO.CITGRC1.CITG RCTNCITGRS1.IBM - CTG.RRM.CITGT1G.I BM.UA - 7.4.2 Configuring CICS No additional CICS setup is required for global transaction support with WebSphere Application Server for z/OS over and above the transactional EXCI setup that is documented for Extended LUW support in 5.6.2, “Configuring CICS” on page 175. 258 CICS Transaction Gateway for z/OS Version 6.1 7.4.3 Configuring WebSphere To test our global transaction, we used the CTGTesterCCIXA application that contains two ECI resource references and that can be used to call programs running in two CICS regions via different connection factories. For further details about CTGTesterCCIXA, refer to Appendix A, “Sample applications” on page 361. Dealing with CICS transaction abends CICS cannot tell RRS to rollback updates to recoverable resources following an unhandled abend of the long-running mirror task. This means that, by default, if the first JCA request of a global transaction succeeds but the second JCA request fails (because of a CICS transaction abend) the updates made by the first request are committed, while the updates made by the second request cannot be committed. See “Dealing with CICS transaction abends” on page 198 for information about ways to avoid such inconsistencies, including details on the new ABENDBKOUT=YES option of the DFHXCOPT table. The CTGTesterCCIXA application is coded to issue a setRollbackOnly() on the EJB session context in the catch block for the ECIInteraction.execute() call (see Example A-1 on page 368). This ensures that the CICS abend is propagated onto WebSphere Application Server as the transaction manager and the global transaction (if one exists) is rolled back. Installing the CICS ECI XA resource adapter Because we are using a remote connection within our global transaction, we need to install the cicseciXA.rar. The process of installing the CICS ECI XA resource adapter (cicseciXA.rar) in WebSphere Application Server for z/OS is identical to that described for the ECI resource adapter (see 4.2.3, “Installing the CICS ECI resource adapter” on page 126). Chapter 7. Transactions with WebSphere Application Server for z/OS 259 Both resource adapters can be installed at the same time. Figure 7-2 shows the two resource adapters installed into our application server. Figure 7-2 Administrative Console - CICS ECI resource adapters Creating connection factories The process of creating connection factories using the CICS ECI XA resource adapter in WebSphere Application Server for z/OS is the same as described in 4.2.4, “Creating a connection factory” on page 128. We created two connection factories CITGR1CI-tx1-local and CITGR1CI-tx1b-12101-XA based upon the CICS ECI XA resource adapter. Figure 7-3 shows our connection factories in the WebSphere Administrative Console. 260 CICS Transaction Gateway for z/OS Version 6.1 Figure 7-3 Administrative Console - CICS ECI connection factories The two connection factories represent two connections to two different CICS regions; CICS region CITGR1CI is accessed with a local connection and CICS region CITGT1CI is accessed via the Gateway daemon which binds to the tx1b IP address using port 12101. Tip: Before using the XA transaction support, you should ensure that your WebSphere Application Server level contains the fix for APAR PQ99940. This APAR fixes a connection pooling performance problem following a connection failure, for example, as a result of a CICS region or Gateway daemon failure. The fix is contained in V6.0.1 of WebSphere Application Server for z/OS. Deploying the application To deploy the CTGTesterCCIXA application we followed the same procedure as is described in 4.2.6, “Deploying our sample J2EE application” on page 134. This time, we mapped two resource references using the WebSphere Administrative Console as shown in Figure 7-4. Figure 7-4 Administrative Console - mapping XA resource references Chapter 7. Transactions with WebSphere Application Server for z/OS 261 After the application was deployed, we started the application. Configure WebSphere transaction logging Prior to WebSphere Application Server V5.1, WebSphere Application Server for z/OS had no logging system of its own. It used RRS to hold information about all transactions, and on startup it retrieves this information from RRS. WebSphere Application Server V5.1 introduced an XA partner log, which holds information about XA resources accessed. When an application accesses XA resources, WebSphere Application Server stores information about the resource updates in the transaction log in order to enable XA transaction recovery. On the WebSphere Administrative Console, follow the links Application servers → server1 → Container Services → Transaction Service to a page that contains tabs for both startup configuration and runtime settings. By default, WebSphere Application Server uses the transaction log directory derived from the following pattern: install_root/tranlog/cell_name/node_name/server_name/transaction/partnerlog On z/OS, you can configure the transaction logs to be either in the HFS or in MVS logstreams. If the logs are placed in logstreams, we recommend that they be in the Coupling Facility instead of DASD logstreams. The logs (and structures, if applicable) are created in job BBOLOGSA in the WebSphere Application Server installation dialog. To specify the transaction log as a logstream, you need to specify the following for the transaction log directory: logstream://HLQ HLQ is a user-defined value from 1 to 8 characters which is specified in the WebSphere Application Server installation dialog. In our case, it is CITGRXA. In the Configuration panel, we did not specify a transaction log directory. Example 7-1 shows the default path used by WebSphere Application Server on our test system. Example 7-1 Default directory location of transaction log /CITG/CellCITGR/AppServer/profiles/default/tranlog/CITGR_cell1/CITGR_node1/CITG R_server1/transaction/partnerlog Transaction Timeout In the properties page for the Transaction Service, you can optionally change other transaction-related configuration properties. We changed the value for Total transaction lifetime timeout (default 120 seconds). This sets the number of seconds a transaction can remain inactive before it is ended by the 262 CICS Transaction Gateway for z/OS Version 6.1 Transaction Service. A value of zero (0) indicates that there is no timeout limit. Because some of our test cases sometimes involve requests which suspend in CICS, we increased this value to 1200 seconds. Configure Tivoli Performance Viewer Setting up of the Tivoli Performance Viewer for monitoring transactions is covered in section “Tivoli Performance Viewer - Transaction Manager module” on page 189. For this test scenario, we did not monitor transactions using Tivoli Performance Viewer. 7.4.4 Testing the configuration In this section, we detail how we tested the WebSphere Application Server for z/OS global transaction configuration using our sample J2EE application CTGTesterCCIXA. Figure 7-5 shows the basic outline of our environment. We used this configuration for two scenarios: Normal termination of a global transaction An update and commit of recoverable resources in two CICS regions within one global transaction (see “Global transaction normal termination” on page 266). Abnormal termination of a global transaction A CICS subsystem failure during an in-flight global transaction (see “Global transaction abnormal termination” on page 273). Chapter 7. Transactions with WebSphere Application Server for z/OS 263 z/OS tx1.mop.ibm.com WebSphere Application Server V6 CITGR1CI (applid A6POTGR1) HTML JSP ECI Request RRSTransactional Servlet CICS TG V6.1 CTGTesterCCIXA cicseciXA.rar JNI CICS Long Running Mirror ECIW EXCI CITGR_server1 RRS ECI Request within XA txn Gateway daemon 12101 CICS EXCI JNI CITGT1G tx1b.mop.ibm.com Long Running Mirror ECIT CITGT1CI (applid A6POTGT1) Figure 7-5 Global transaction scenario with WebSphere Application Server for z/OS Note: On our system the Gateway daemon binds to a different TCP/IP address than the WebSphere Application Server, but they are actually running on the same z/OS image. In practice, they could be running on two completely different systems. 264 CICS Transaction Gateway for z/OS Version 6.1 In test our configuration, we invoked our test application CTGTesterCCIXA (Figure 7-6). Figure 7-6 CTGTesterCCIXA Start global transaction Chapter 7. Transactions with WebSphere Application Server for z/OS 265 During these tests, we specified DW as the input to the COMMAREA, which means that the ECIPROG program writes to a recoverable TS Queue and then delays for one minute. The iteration count was set to 1 for transaction leg 1 and 3 for transaction leg 2. Clicking Submit results in a total of four ECI calls: one request to CICS region CITGR1CI (applid A6POTGR1) and three requests to CICS region CITGT1CI (applid A6POTGT1). This iteration give us four minutes during which to monitor the transaction state across the various components of the system. Note: This scenario is for testing purposes only. Normal transactions would not be expected to delay in this way. During the course of these scenarios, we monitored the transaction using each of the following: CICS RRS ISPF panels CEMT command RRS archive log Global transaction normal termination This test demonstrates successful update and commit of resources on two CICS regions accessed via different connection factories, within the scope of a single global transaction. The Web browser appeared to be busy for four minutes before displaying the results page that contains the COMMAREAs that are returned from the two legs of the global transaction. CICS CEMT command While the second ECI request was in-flight on A6POTGT1, we issued the CEMT INQUIRE TASK command on both CICS regions, A6POTGR1 and A6POTGT1. Figure 7-7 shows the ECIW mirror task running on A6POTGR1 as Task 519 suspended in RRMSEXIT state, which means that CICS is waiting for a commit or backout flow from RRS. Program ECIPROG on A6POTGR1 has completed at this stage but CICS is waiting for a commit flow from RRS, which occurs when the transaction manager (WebSphere Application Server) syncpoints the global transaction. 266 CICS Transaction Gateway for z/OS Version 6.1 INQUIRE TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000519) Tra(ECIW) Sus Tas Pri( 001 ) Sta(TO) Use(CITGR1D ) Uow(BDC64F4348908621) Hty(RRMSEXIT) Tas(0000520) Tra(CEMT) Fac(N123) Run Ter Pri( 255 ) Sta(TO) Use(CITGR1D ) Uow(BDC64F49EA5BE4AE) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 17.15.56 DATE: 10.17.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 7-7 CEMT INQUIRE TASK on A6POTGR1 with the mirror in RRMSEXIT state Figure 7-8 shows the ECIT mirror task running on A6POTGT1 as Task 448, suspended in ICWAIT state, due to the active EXEC CICS DELAY in ECIPROG. INQUIRE TASK STATUS: RESULTS - OVERTYPE TO MODIFY Tas(0000448) Tra(ECIT) Fac(11 ) Sus Ter Pri( 001 ) Sta(D ) Use(CITGT1D ) Uow(BDC64F7CD273C869) Hty(ICWAIT Tas(0000450) Tra(CEMT) Fac(N124) Run Ter Pri( 255 ) Sta(TO) Use(CITGT1D ) Uow(BDC64F931F317EA7) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR ) SYSID=TGT1 APPLID=A6POTGT1 TIME: 17.16.19 DATE: 10.17.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 7-8 CEMT INQUIRE TASK on A6POTGT1 with the mirror in ICWAIT state We also issued the CEMT INQUIRE EXCI command on both CICS regions. Figure 7-9 shows the EXCI information for the long-running mirror task ECIW (task 519) running on A6POTGR1, including the associated RRS UR identifier for the first leg of the transaction. The INQUIRE EXCI command provides the information to correlate an active long-running mirror task in a CICS region, with the RRS UR identifier. We see that the Exc field highlights the jobname, stepname, and the mvsid associated to the EXCI client application. In this case, the EXCI client application is job CITGRS1S (the jobname of the WebSphere Application Server servant region) which is running on the TX1 system (mvsid X1). Chapter 7. Transactions with WebSphere Application Server for z/OS 267 INQUIRE EXCI STATUS: RESULTS Exc(CITGRS1S.CITGRS1S. - X1 ) Tas(0000519) Uri(BDC64F437E517374000000D701030000) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGR1 APPLID=A6POTGR1 TIME: 17.15.09 DATE: 10.17.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 7-9 CEMT INQUIRE EXCI on A6POTGR1 showing a UR identifier Figure 7-10 shows the EXCI information for the long-running mirror task ECIT (task 448) running on A6POTGT1, including the associated RRS UR identifier for the second leg of the transaction. The EXCI client application is job CITGT1G (the jobname of the Gateway daemon). INQUIRE EXCI STATUS: RESULTS Exc(CITGT1G.CITGT1G. - X1 ) Tas(0000448) Uri(BDC64F7C7E5176E80000009001030000) RESPONSE: NORMAL PF 1 HELP 3 END 5 VAR SYSID=TGT1 APPLID=A6POTGT1 TIME: 17.16.54 DATE: 10.17.05 7 SBH 8 SFH 9 MSG 10 SB 11 SF Figure 7-10 CEMT INQUIRE EXCI on A6POTGT1 showing a UR identifier Notice that the two RRS UR identifiers are different for the two legs of the global transaction. We see in the RRS archive log (see “RRS Archive Log” on page 270) how these two different URs are tied together in a global transaction with a Global transaction identifier (GTRID) . 268 CICS Transaction Gateway for z/OS Version 6.1 Figure 7-11 shows the results of the test displayed on the Web browser. Figure 7-11 CTGTesterCCIXA global transaction normal termination Chapter 7. Transactions with WebSphere Application Server for z/OS 269 At the end of the test, the recoverable TS Queue RECIPROG on CICS region CITGR1CI (applid A6POTGR1) contained one new entry, and the recoverable TS Queue RECIPROG on CICS region CITGT1CI (applid A6POTGT1) contained three new entries (Example 7-2). Example 7-2 CEBR output showing committed TS Queue updates CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF 1 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 17/10/05 17:14:56 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF 3 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 17/10/05 17:15:56 CITGT1D D 00002 A6POTGT1 17/10/05 17:16:56 CITGT1D D 00003 A6POTGT1 17/10/05 17:17:56 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** RRS Archive Log The RRS archive log shows information relating to all RRS URs, including completed URs. We selected option 1 from the RRS primary menu panel and the Browse an RRS log stream menu panel was displayed. We then chose to view the detailed information of the RRS archive log. Example 7-3 shows the log entries for the global transaction normal termination scenario. Example 7-3 RRS Archive log - XA transaction normal termination RRS/MVS LOG STREAM BROWSE DETAIL REPORT READING ATR.X1.ARCHIVE LOG STREAM X1 2005/10/17 17:18:56.407057 BLOCKID=000000000020407E URID=BDC64F7C7E5176E80000009001030000 JOBNAME=CITGT1G USERID=CITGT1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA SYNCPOINT=AtraCmt RETURN CODE=00000000 START=2005/10/17 15:18:56.371335 COMPLETE=2005/10/17 15:18:56.406601 EXITFLAGS=00800000 LUWID=PSSCX9.A6POTGT1 C64F7CD26D28 0001 TID= GTID= FORMATID=003284271494 (decimal) C3C20186 (hexadecimal) 270 CICS Transaction Gateway for z/OS Version 6.1 GTRID= BDC64F4342ADA7450000002400000007BD915E37E48D782D0000012C000000010964C17A BQUAL= BDC64F4342ADA7450000002400000007BD915E37E48D782D0000012C000000010964C17A00000001 000001C800000002000000000001 RMNAME=CTG.RRM.CITGT1G.IBM.UA ROLE=SDSRM FLAGS=10021000 PROTOCOL=PresumeNothing StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000014 DistSp EXIT RC=Uncalled Commit EXIT RC=00000000 Backout EXIT RC=Uncalled EndUr EXIT RC=Uncalled ExitFailed EXIT RC=Uncalled Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled RMNAME=DFHRXDM.A6POTGT1.IBM ROLE=Participant FLAGS=10001000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000000 DistSp EXIT RC=Uncalled Commit EXIT RC=00000010 Backout EXIT RC=Uncalled EndUr EXIT RC=Uncalled ExitFailed EXIT RC=00000010 Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled X1 2005/10/17 17:18:56.416139 BLOCKID=00000000002042E8 URID=BDC64F437E517374000000D701030000 JOBNAME=CITGRS1 USERID=CITGRCRU PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=BBO.CITGRC1.CITGRCTNCITGRS1.IBM SYNCPOINT=AtraPrp RETURN CODE=00000000 START=2005/10/17 15:18:56.383242 COMPLETE=2005/10/17 15:18:56.416012 EXITFLAGS=00840000 LUWID=PSSCX9.A6POTGR1 C64F43488898 0001 TID= GTID= FORMATID=003284271494 (decimal) C3C20186 (hexadecimal) GTRID= BDC64F4342ADA7450000002400000007BD915E37E48D782D0000012C000000010964C17A BQUAL= BDC64F4342ADA7450000002400000007BD915E37E48D782D0000012C000000010964C17A00000001 000001C800000002000000000002 RMNAME=DFHRXDM.A6POTGR1.IBM ROLE=Participant FLAGS=10001000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000000 DistSp EXIT RC=Uncalled Chapter 7. Transactions with WebSphere Application Server for z/OS 271 Commit EXIT RC=00000010 Backout EXIT RC=Uncalled EndUr EXIT RC=Uncalled ExitFailed EXIT RC=00000010 Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled RMNAME=BBO.CITGRC1.CITGRCTNCITGRS1.IBM FLAGS=10041000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000000 DistSp EXIT RC=Uncalled Commit EXIT RC=00000000 Backout EXIT RC=Uncalled EndUr EXIT RC=00000000 ExitFailed EXIT RC=00000000 Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled ROLE=SDSRM The archive log entries show the following: There are two different RRS URs involved in the global transaction, but they have the same GTRID: BDC64F4342ADA7450000002400000007BD915E37E48D782D0000012C000000010964C17A Note: An X/Open identifier (XID) acts as the identifier for a distributed unit of work. An XID consists of a Global transaction Identifier (GTRID) and a Branch qualifier (BQUAL). RRS supports XIDs and WebSphere Application Server for z/OS uses the XID to link the two URs. If you check the GTRID and BQUAL for both URs, you will see that they are the same. The RRS work manager for the first UR (URID=BDC64F7C7E5176E80000009001030000) is the Gateway daemon CITGT1G with Work Manager Name CTG.RRM.CITGT1G.IBM.UA. The Gateway daemon has a syncpoint coordinator or SDSRM (Server Distributed Syncpoint Resource Manager) role for this UR. The RRS work manager for the second UR (URID=BDC64F437E517374000000D701030000) is the WebSphere Application Server CITGRS1 with Work Manager Name BBO.CITGRC1.CITGRCTNCITGRS1.IBM. WebSphere Application Server has the SDSRM role for this UR. The CICS regions DFHRXDM.A6POTGT1.IBM and DFHRXDM.A6POTGR1.IBM have a Participant role within the URs. 272 CICS Transaction Gateway for z/OS Version 6.1 The RRS archive log also highlights the two types of 2PC protocol used by RRS: Presumed nothing When the UR is in an in-prepare state, RRS logs information about the resource manager. If the resource manager fails, RRS presumes nothing and during resource manager restart RRS is able to return log information when the resource manager calls the Retrieve_UR_Interest service. RRS tells the resource manager that the UR is to be backed out. This protocol is used between RRS and the Gateway daemon. Presumed abort When the UR is in an in-prepare state, RRS does not log any information about the resource manager. If the resource manager fails, RRS presumes that it’s resources are backed out. This protocol is used between RRS and CICS. Note: The RRS exit call return codes are not always easy to interpret. For example, in this case the CICS regions return x’10’ from the commit exit, meaning ATRX_FORGET. CICS is telling RRS to forget its interest in this UR. For more detailed information about RRS exit return codes, refer to z/OS MVS Programming: Resource Recovery, SA22-7616. In this example, we see clearly that both an RRSTransactional resource manager (CICS ECI local connection) and an XA resource manager (CICS ECI XA remote connection) can be involved in a global transaction. WebSphere Application Server provides the overall transaction management and uses a combination of RRSTransactional and XA protocols to manage the global transaction. Global transaction abnormal termination This test scenario demonstrates a failure scenario, using the CTGTesterCCIXA application. The configuration and application parameters are identical to those used in the scenario “Global transaction normal termination” on page 266. In this scenario, however, during the second leg of the transaction the CICS region A6POTGR1 is cancelled. The second leg of the transaction continues, making the three updates to TS Queue RECIPROG on CICS region A6POTGT1. However, at the end of the final ECI request, the syncpoint prepare call to CICS region A6POTGR1 for transaction leg 1 fails. This leads to a complete rollback of all updated resources. Chapter 7. Transactions with WebSphere Application Server for z/OS 273 CICS CEBR command At just over three minutes into the test scenario, we looked at the contents of the TS Queue RECIPROG on both A6POTGR1 and A6POTGT1. We could see that both queues contained updates (Example 7-4). Example 7-4 CEBR output on A6POTGR1 and A6POTGT1 after three minutes CEBR TSQ RECIPROG SYSID TGR1 REC 1 OF 1 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGR1 18/10/05 09:54:49 CITGR1D D ************************* BOTTOM OF QUEUE ***************************** CEBR TSQ RECIPROG SYSID TGT1 REC 1 OF 1 COL 1 OF 38 ENTER COMMAND ===> ************************** TOP OF QUEUE ******************************* 00001 A6POTGT1 18/10/05 09:55:50 CITGT1D D 00002 A6POTGT1 18/10/05 09:56:50 CITGT1D D ************************* BOTTOM OF QUEUE ***************************** At this point, we cancelled CICS region A6POTGR1 with the MVS command /CANCEL CITGR1CI. After the ECIPROG program on A6POTGT1 ended, WebSphere Application Server attempts to commit the global transaction. At this time, the application server is unable to communicate with the failed CICS region A6POTGR1 so the global transaction must be rolled back. Example 7-12 shows how this transaction rollback is reported on the Web browser. 274 CICS Transaction Gateway for z/OS Version 6.1 Figure 7-12 CTGTesterCCIXA global transaction abnormal termination RRS Archive Log The RRS archive log shows us information that is related to the failed transaction. We selected option 1 from the RRS primary menu panel and the Browse an RRS log stream menu panel was displayed. We then chose to view the detailed information of the RRS archive log. Chapter 7. Transactions with WebSphere Application Server for z/OS 275 Example 7-5 shows the log entries for the XA transaction abnormal termination scenario. Example 7-5 RRS Archive log - XA transaction abnormal termination RRS/MVS LOG STREAM BROWSE DETAIL REPORT READING ATR.X1.ARCHIVE LOG STREAM X1 2005/10/18 09:58:50.705010 BLOCKID=00000000002084EC URID=BDC72EFB7E517A5C0000002201030000 JOBNAME=CITGT1G USERID=CITGT1G PARENT URID=00000000000000000000000000000000 SURID=N/A WORK MANAGER NAME=CTG.RRM.CITGT1G.IBM.UA SYNCPOINT=AtraBak RETURN CODE=00000000 START=2005/10/18 07:58:50.663694 COMPLETE=2005/10/18 07:58:50.704442 EXITFLAGS=00000000 LUWID=PSSCX9.A6POTGT1 C72EFB37FEBB 0001 TID= GTID= FORMATID=003284271494 (decimal) C3C20186 (hexadecimal) GTRID= BDC72EC1458FC440000000240000000ABD915E37E48D782D0000012C000000010964C17A BQUAL= BDC72EC1458FC440000000240000000ABD915E37E48D782D0000012C000000010964C17A00000001 000001C800000002000000000001 RMNAME=CTG.RRM.CITGT1G.IBM.UA ROLE=SDSRM FLAGS=10021000 PROTOCOL=PresumeNothing StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000014 DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=00000000 EndUr EXIT RC=Uncalled ExitFailed EXIT RC=Uncalled Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled RMNAME=DFHRXDM.A6POTGT1.IBM ROLE=Participant FLAGS=10000000 PROTOCOL=PresumeAbort StateCheck EXIT RC=Uncalled Prepare EXIT RC=00000000 DistSp EXIT RC=Uncalled Commit EXIT RC=Uncalled Backout EXIT RC=00000010 EndUr EXIT RC=Uncalled ExitFailed EXIT RC=00000010 Completion EXIT RC=Uncalled OnlyAgent EXIT RC=Uncalled 276 CICS Transaction Gateway for z/OS Version 6.1 The archive log entries show the following: A single entry for just one RRS UR: BDC72EFB7E517A5C0000002201030000 The RRS work manager for the UR is the Gateway daemon CITGT1G with CTG_RRMNAME CTG.RRM.CITGT1G.IBM.UA. The Gateway daemon has an SDSRM role and the CICS region A6POPTGT1 has a Participant role within the UR. The log entry shows that the backout call (AtraBak) to CITGT1G returns with code 0. This backout call is made because the transaction manager, WebSphere Application Server for z/OS, has rolled back the global transaction following the failure to prepare the CICS region CITGR1G. Note that there is no log entry for the prepare failure because the two-phase commit protocol used for a local connection between WebSphere Application Server for z/OS and the CICS region A6POTGR1 is the Presumed Abort protocol. RRS does not log any information about CICS; if CICS fails, RRS presumes that it’s resources are backed out. After the restart of CICS region A6POTGR1 was complete, the CICS log showed message DFHRM0201 (Example 7-6). Example 7-6 CEBR output showing committed TS Queue updates DFHRM0201 10/18/2005 12:10:34 A6POTGR1 0 backout-failed and 1 commit-failed UOWs were reconstructed. Finally, we used CEBR to check that the TS Queue updates on both CICS regions were backed out. The TS Queue updates made by the ECIT mirror transaction on A6POTGT1 were backed out when the transaction was rolled back. The TS Queue updates made by the ECIW mirror transaction are backed out during the emergency restart of A6POTGR1. 7.5 Problem determination In this section, we document common transaction problems and information that we learned while configuring our transaction test scenario with WebSphere Application Server for z/OS. For general problem determination guidance see 2.9, “Problem determination” on page 60. Chapter 7. Transactions with WebSphere Application Server for z/OS 277 Abend ARXC If a Transactional EXCI is received by a CICS region which has the system initialization parameter RRMS=NO specified, the request fails and the global transaction rolls back. We noted the following message in the CICS message log: A6POTGR1 Transaction ECIW abend ARXC in program DFHXMAB term 31. Updates to local recoverable resources will be backed out. EXCI job = CITGRS1S.CITGRS1S. - X1. 7.5.1 Tracing In order to enable JNI trace to analyze transaction problems, you need to specify the CTG_JNI_TRACE and CTG_JNI_TRACE_ON environment variables using the WebSphere Administrative Console. See 4.7.1, “Tracing” on page 152 for information about enabling JNI trace with WebSphere Application Server for z/OS. 278 CICS Transaction Gateway for z/OS Version 6.1 Part 4 Part 4 Security Management In this part, we describe the different security capabilities of the CICS Transaction Gateway for z/OS. We provide detailed information about how we configured secure interoperation between WebSphere Application Server and CICS. We cover two different topologies: Running WebSphere Application Server on Windows Running WebSphere Application Server on z/OS We discuss the configuration of security within WebSphere Application Server, the Gateway daemon, and CICS. © Copyright IBM Corp. 2006. All rights reserved. 279 280 CICS Transaction Gateway for z/OS Version 6.1 8 Chapter 8. Gateway daemon security In this chapter, we outline the main CICS and RACF security functions that can be used with the CICS Transaction Gateway for z/OS. We start with an introduction to CICS security and then we discuss how the basic CICS security mechanisms can be used to secure CICS programs which are called via the Gateway daemon on z/OS. We document how we prepared the system and the settings that we used. We explain how we tested the environment using the CICS TG sample ECI application EciB1 and our own sample J2EE application CTGTesterCCI. We also give details of some of the common security related problems you might see when using the CICS TG for z/OS. © Copyright IBM Corp. 2006. All rights reserved. 281 8.1 Preparation In our testing, we connected from Java client applications running on WebSphere Application Server for Windows to CICS via the Gateway daemon on z/OS. In this chapter, we concentrate on how to configure security within and between each tier (Figure 8-1). Windows z/OS Java application EciB1 WebSphere Application Server V6 HTML CICS TS V3.1 CICS TG V6.1 TCP/IP EJB CTGTesterCCI CCI Gateway daemon EXCI CICS TG V6.1 CICS ECI resource adapter CICS program MRO/IRC user ID authentication (user ID + password) authorization RACF Figure 8-1 Software components: Gateway daemon security For information about how to enable SSL connections to the Gateway daemon, see Chapter 9, “Enabling SSL with the Gateway daemon” on page 309. 8.1.1 Software checklist Table 8-1 shows the levels of software that we used for this configuration. Table 8-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS V1R6 Windows 2000 SP4 CICS Transaction Gateway for z/OS V6.1 IBM WebSphere Application Server Network Deployment V6.0.2 with security enabled IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. CICS Transaction Server for z/OS V3.1 Before starting the security tests with our CTGTesterCCI J2EE application, we enabled security for our WebSphere Application Server. Our security configuration was as follows: 282 CICS Transaction Gateway for z/OS Version 6.1 Global security Java 2 security User registry Enabled Not enabled Local OS For details on enabling security in WebSphere, see the WebSphere Application Server V6 information center (section on Setting Up and Enabling Security). 8.1.2 Definitions checklist Table 8-2 lists the definitions that we used to configure the security scenarios. Table 8-2 Definitions checklist Value WebSphere Gateway daemon CICS TS hostname cam21-pc3 tx1.mop.ibm.com - IP address (dhcp) 9.100.193.122 - TCP/IP port 9080 11101 - Job name - CITGS1G CITGS1CI APPLID - - A6POTGS1 NETNAME - CITGS1G - CONNECTION - - GS1G Table 8-3 lists the user IDs that we used in our configuration. Table 8-3 User ID checklist Value Gateway daemon CICS TS CICS region user ID - CITGS1CI Gateway daemon user ID CITGS1G - Flowed user ID to which we want to permit access - CITGSD Flowed user ID to which we want to deny access - CITGNW 8.2 Introduction to CICS and CICS TG security CICS uses the z/OS System Authorization Facility (SAF) to route authorization requests to an external security manager (ESM) to perform all its security Chapter 8. Gateway daemon security 283 checks. You could use any suitable ESM. However, because the RACF product from IBM is the most commonly used, we refer to it for the remainder of this book. For complete information about CICS security, refer to the CICS RACF Security Guide, SC33-1701. Every CICS region requires certain special user IDs to be established and also uses certain user IDs when receiving inbound requests from other systems. These user IDs are as follows: Region user ID This is the user ID under which the CICS started task runs. Default user ID This is used when users do not explicitly sign on, and should be given very low authorization. It is specified in the SIT parameter DFLTUSER. Flowed user ID This is any user ID that is flowed in an ISC or MRO request, and includes user IDs flowed in ECI and EPI requests from Java applications. Link user ID This is a user ID associated with the connecting system. It can be defined in the SESSIONS definition. It is used in link security and to determine if connected systems are equivalent. Authentication of CICS users and access control is normally determined by RACF. Once authenticated, the user passes through transaction and resource security checks. If the request has been forwarded from another system, then surrogate and intercommunication security checks are also performed. The different types of CICS security are explained briefly in the sections that follow. Transaction security CICS uses transaction security to control a user’s permission to start a transaction. CICS performs a transaction security check even if no user has signed on. Users who do not sign on can use only those transactions that are authorized to the CICS default user ID. Usually, the transactions to which this user ID has access are very limited. Resource security CICS provides a further (optional) level of security by controlling access to individual resources, which include programs, files, and other resources. There are no special implications for resource security with the CICS TG and so this subject is not addressed any further in this chapter. 284 CICS Transaction Gateway for z/OS Version 6.1 Surrogate user security CICS performs surrogate user security checking to ensure that one user is authorized to act for another user. For example, a surrogate check is performed to ensure that a Gateway daemon is authorized to initiate work on behalf of a user ID flowed in an ECI request. Intercommunication security Intercommunication security in CICS is concerned with incoming requests for access to CICS resources. There are three fundamentally different intercommunication security checks that are performed when a Gateway daemon forwards a request to a CICS region: Bind security This verifies that the system wishing to connect (bind) to CICS is authorized to do so. The Gateway daemon is given authority to bind to a CICS region. User security This causes a check to be made against the flowed user ID when an inbound request attaches the transaction in CICS. This behavior is defined in the ATTACHSEC parameter on the CONNECTION definition. For EXCI requests from the Gateway daemon, this should always be IDENTIFY, meaning that the flowed user ID is identified but not verified (a password check is not done by CICS). Link security This is an additional level of authorization checking that can apply to the attached transaction. A specific user ID (the link user ID) can be thought of as the user ID associated with the server which is passing the request to CICS. In our case, this is the user ID of the Gateway daemon started task. If link security is enabled, the link user ID, like the flowed user ID, must be authorized to access all transactions and resources invoked as a result of the request. In this chapter, we discuss how these security checks are configured when using the Gateway daemon on z/OS. For information about how these apply when securing connections from WebSphere Application Server for z/OS see Chapter 10, “Security with WebSphere Application Server for z/OS” on page 339. Chapter 8. Gateway daemon security 285 8.3 Configuring CICS and CICS TG security The commands listed in the following sections are in addition to the basic configuration necessary for normal functioning of the CICS TG for z/OS. Before you implement any security in your environment, we recommend that you set up and test a non-secure environment. You can information about the following in Chapter 2, “Installing and configuring the CICS TG” on page 25: Setting up a user ID for the Gateway daemon started task Defining programs and libraries as program controlled Allowing the Gateway daemon access to program controlled libraries The following steps are necessary to secure access to our CICS programs: Configuring CICS security Configuring MRO bind time security Enabling CICS TG password checking Configuring security for CICS CONNECTION and SESSIONS definitions Configuring link security Configuring the flowed user ID Permitting access to the mirror transaction Configuring surrogate security Configuring CICS security Our CICS region CITGS1CI (with APPLID A6POTGS1) was configured with security prefixing and transaction security active using the following parameters: IRCSTRT=YES SECPRFX=YES SEC=YES XDCT=NO XFCT=NO XPCT=NO XPPT=NO XTRAN=YES XCMD=NO Note: We used security prefixing (SECPRFX=YES) in our CICS region, which prevents our RACF security profiles from affecting other CICS regions. This parameter can be quite useful in a production environment, because it means all security profiles are unique to an individual region. However, conversely it can mean more work for the security administrator because more profiles must be defined. 286 CICS Transaction Gateway for z/OS Version 6.1 Configuring MRO bind security MRO bind security prevents unauthorized attached MRO regions from starting transactions in a CICS region. It is implemented using two different DFHAPPL FACILITY class profiles. To connect to a CICS region, the Gateway daemon user ID requires the following permissions: Update access to the RACF FACILITY class profile DFHAPPL.<DFHJVPIPE>, where <DFHJVPIPE> is the EXCI pipe name as defined in the CICS TG environment variable DFHJVPIPE and in the NETNAME parameter in the CICS CONNECTION definition. If a generic EXCI connection is used, there is no pipe name and this does not apply. Read access to the RACF FACILITY class profile DFHAPPL.<APPLID>, where <APPLID> is the APPLID of the CICS region in question. Note: Use of MRO bind-time security is optional and if these DFHAPPL profiles are not defined, then any MRO connected system will be able to connect to your CICS region. We activated MRO bind security for our configuration as follows: 1. We gave update access to the RACF FACILITY class profile DFHAPPL.<DFHJVPIPE> using the following RACF command: RDEFINE FACILITY (DFHAPPL.CITGS1G) UACC(NONE) PERMIT DFHAPPL.CITGS1G CLASS(FACILITY) ID(CITGS1G) ACCESS(UPDATE) SETROPTS RACLIST(FACILITY) REFRESH In our configuration, CITGS1G is the user ID of the Gateway daemon started task and also the name that we used for the DFHJVPIPE environment variable. Note: We defined all of our security profiles with a UACC(NONE), meaning a universal access of none. Thus, by default all users were denied access, and permissions were then granted for each user ID as required. 2. We gave read access to the RACF FACILITY class profile DFHAPPL.<APPLID> with the following RACF command: RDEFINE FACILITY (DFHAPPL.A6POTGS1) UACC(NONE) PERMIT DFHAPPL.A6POTGS1 CLASS(FACILITY) ID(CITGS1G) ACCESS(READ) SETROPTS RACLIST(FACILITY) REFRESH In our configuration, A6POTGS1 is the APPLID of the CICS region. Tip: The SETROPTS command refreshes the FACILITY class. It must be run each time you modify the DFHAPPL FACILITY class profiles. Chapter 8. Gateway daemon security 287 Enabling CICS TG password checking To enable the CICS TG to authenticate each user ID and password flowed on an ECI request, the environment variable AUTH_USERID_PASSWORD must be set in the CICS TG environment variables. We set this variable in our CITG.CTG610.STDENV(CITGS1G) PDS member (Example 8-1), which we supplied as input to our Gateway daemon using the STDENV DD card. Example 8-1 CICS TG environment variables _BPX_SHAREAS=YES CICSCLI=/cicstg/ctg610/config/CITGS1G.ini COLUMNS=80 CTG_JNI_TRACE=/cicstg/ctg610/trace/CITGS1G.jnitrc CTG_JNI_TRACE_ON=YES CTG_RRMNAME=CTG.RRM.CITGS1G.IBM.UA DFHJVPIPE=CITGS1G DFHJVSYSTEM_00=A6POTGS1-Primary server DFHJVSYSTEM_01=A6POTGS2-Secondary server PATH=/bin:/usr/lpp/java142/J1.4/bin TMPDIR=/cicstg/ctg610/trace TZ=GMT-2 AUTH_USERID_PASSWORD=YES JAVA_PROPAGATE=NO Notes: If you set AUTH_USERID_PASSWORD to YES you must also set _BPX_SHAREAS to YES and JAVA_PROPAGATE to NO. If you include AUTH_USERID_PASSWORD with any value, YES will be assumed. For any value other than YES, you get the following warning message: CTG6134W Environment variable ‘AUTH_USERID_PASSWORD’ is set, but not to the value ‘YES’; user authentication will be enabled See 8.5.1, “Testing security with Java program EciB1” on page 296 for how we tested CICS TG password checking. 288 CICS Transaction Gateway for z/OS Version 6.1 CONNECTION and SESSIONS definitions In order for our CICS region to use the user ID flowed in the EXCI call from the Gateway daemon, we set the parameter ATTACHSEC=IDENTIFY in the EXCI connection definition in our CICS region, as shown in Figure 8-2. OVERTYPE TO MODIFY CEDA ALter CONnection( GS1G ) CONnection : GS1G Group : GS1G DEscription ==> CONNECTION IDENTIFIERS Netname ==> CITGS1G INDsys ==> CONNECTION PROPERTIES ACcessmethod ==> IRc PRotocol ==> Exci Conntype ==> Specific SECURITY SEcurityname ==> ATtachsec ==> Identify BINDPassword : BINDSecurity ==> No Usedfltuser ==> No CICS RELEASE = 0640 Vtam | IRc | INdirect | Xm Appc | Lu61 | Exci Generic | Specific Local | Identify | Verify | Persistent | Mixidpe PASSWORD NOT SPECIFIED No | Yes No | Yes SYSID=TGS1 APPLID=A6POTGS1 Figure 8-2 CICS CONNECTION definition IDENTIFY means that CICS uses the flowed user ID in the EXCI request, but does not expect a password to be flowed with the request because this should be checked by the Gateway daemon itself. The default is ATTACHSEC=LOCAL, which means that the flowed user ID is not supplied by the connecting server. Configuring link security The link user ID is the user ID associated with the Gateway daemon which is passing the request to CICS. If link security is enabled, the link user ID, like the flowed user ID, must be authorized to access all transactions and resources invoked as a result of the request. The link user ID can be preset on the EXCI SESSIONS definition using the USERID parameter. For EXCI requests, link security works as follows: 1. If the link user ID is the same as the CICS region user ID, then the systems are deemed equivalent and no link security authorization is performed. Chapter 8. Gateway daemon security 289 2. If the link user ID is preset as something other than the CICS region user ID, then this user ID must be authorized to access all transactions and resources invoked as a result of the request. 3. If the link user ID is not preset, then the user ID of the connected region (that is, the user ID of the Gateway daemon started task) is the link user ID and this user ID must be authorized to access all transactions and resources invoked as a result of the request. We decided to use preset link security. This has the advantage that it avoids the overhead of EXCI sign-on for each request that is passed to CICS. When running any application that connects to CICS through the EXCI (such as the Gateway daemon), CICS writes one DFHSN1400/DFHSN1500 message pair to the CICS joblog for each request. By default, this message pair is written for each ECI request. When the link user ID is preset, however, EXCI sign-on is done when the CICS CONNECTION is installed but not for each ECI request. It is also possible to suppress EXCI sign-on messages by using the CICS user exit XMEOUT. An assembler sample of this program can be found in SDFHSAMP(DFH$SXP1). To enable preset link security, we set the USERID parameter in the SESSIONS definition to be CITGS1G, which is the CICS TG started task user ID. Our SESSIONS definition is shown in Figure 8-3. OVERTYPE TO MODIFY CEDA ALter Sessions( GS1G ) Sessions : GS1G Group : GS1G DEscription ==> SESSION IDENTIFIERS Connection ==> GS1G SESSName ==> NETnameq ==> MOdename ==> SESSION PROPERTIES Protocol ==> Exci MAximum ==> 000 , 000 RECEIVEPfx ==> 1 RECEIVECount ==> 260 PRESET SECURITY USERId ==> CITGS1G CICS RELEASE = 0640 Appc | Lu61 | Exci 0-999 1-999 SYSID=TGS1 APPLID=A6POTGS1 Figure 8-3 CICS SESSIONS definition 290 CICS Transaction Gateway for z/OS Version 6.1 We recommend that you specify ATTACHSEC=IDENTIFY and use non-equivalent systems so that security checks are also performed against a a link user ID. Table 8-4 lists the different settings for link security and ATTACHSEC and how they interoperate. Table 8-4 Attach security settings with an EXCI connection from the Gateway daemon Equivalent systems? Link user ID = CICS region user ID Link user ID not = CICS region user ID ATTACHSEC LOCAL IDENTIFY LOCAL IDENTIFY Link user ID check NO NO YES YES Flowed user ID check NO YES NO YES User ID mirror transaction runs under in CICS CICS default user ID Flowed user ID Link user ID Flowed user ID In our configuration, the link user ID CITGS1G is not equivalent to the CICS region user ID CITGS1CI and ATTACHSEC IDENTIFY is defined. The security checks that apply are therefore those listed in column 4 of Table 8-4. Configuring the flowed user ID The flowed user ID is the identity that is passed on the ECI request. It often represents the identity of the end user of the application. When using the base CICS TG classes, the user ID and password can be specified as parameters when constructing an ECIRequest object (see 8.5.1, “Testing security with Java program EciB1” on page 296). When using the JCA, the user ID and password can be specified in a variety of different ways (see 8.5.2, “Testing security with J2EE application CTGTesterCCI” on page 299). In our tests we flow the user ID CITGSD. Because the CICS TG runs as a shell process under UNIX System Services, any user ID that it tries to authenticate with RACF, such as a user ID flowed with an ECI request, must have an OMVS segment defined in its RACF profile. See OS/390® UNIX System Services Planning, GA22-7800 for more information. Permitting access to the mirror transactions An EXCI request received by CICS from the Gateway daemon attaches a mirror transaction. Therefore, we needed to authorize our flowed user ID CITGSD and our link user ID CITGS1G to our private mirror transaction ECIT. We issued the following RACF commands: REDFINE TCICSTRN CITGS1CI.ECIT UACC(NONE) PERMIT CITGS1CI.ECIT CL(TCICSTRN) ID(CITGS1G) ACCESS(READ) PERMIT CITGS1CI.ECIT CL(TCICSTRN) ID(CITGSD) ACCESS(READ) SETROPTS RACLIST(TCICSTRN) REFRESH Chapter 8. Gateway daemon security 291 Tip: For our examples, we permitted access to single users (CITGSD and CITGS1G). In a production environment, you might create a group of users who require common access. When you build a group, you permit access to a group so that user access can be controlled by the group to which a user belongs, rather than by individual permissions. This method simplifies the security definitions that are required. Configuring surrogate security In order for the EXCI to be able to switch the security context of the EXCI request to the flowed user ID, the correct SURROGAT security profiles must be defined. The user ID of the EXCI client (in our case the Gateway daemon) requires read access to the SURROGAT class profile <USERID>.DFHEXCI, where <USERID> is the flowed user ID in the EXCI request. We issued the following commands to authorize our gateway daemon user ID CITGS1G to switch to the user ID CITGSD which is flowed in the ECI request: RDEFINE SURROGAT CITGSD.DFHEXCI UACC(NONE) OWNER(CITGSD) PERMIT CITGSD.DFHEXCI CLASS(SURROGAT) ID(CITGS1G) ACCESS(READ) It is possible to disable surrogate security by either reassembling the EXCI options table DFHXCOPT, with SURROGAT=NO, or by using a RACF surrogate profile with universal READ access such as: RDEFINE SURROGAT *.DFHEXCI UACC(READ) OWNER(CITGSD) Note: If you assemble DFHXCOPT into a new library you should ensure that it is defined in the environment variable EXCI_OPTIONS and also that this library is program controlled. If it is not program controlled you will get the following error message: ICH420I PROGRAM DFHXCOPT FROM LIBRARY CITG.CITGS1G.USERLOAD CAUSED THE ENVIRONMENT TO BECOME UNCONTROLLED 292 CICS Transaction Gateway for z/OS Version 6.1 8.4 Configuring JCA security The JCA has specific support for enabling secure access from a J2EE application to an Enterprise Information System (EIS) such as CICS. Both container-managed sign-on and component-managed sign-on are supported. When deploying a J2EE component such as an EJB, the deployer must set the res-auth element in the deployment descriptor for each resource reference in order to indicate which authentication method is being used. Container-managed security If you use container-managed security, the J2EE application server is responsible for flowing security context to the EIS. You must set the res-auth deployment descriptor element to Container. The application deployer is then able to set up the authentication information (user ID and password) when he deploys the application. In WebSphere Application Server V5, authentication data can be specified in the connection factory by using a JAAS authentication alias. In WebSphere Application Server V6, this should be done using a J2C authentication data entry mapped to specific application resources (see 8.4.1, “J2C authentication data” on page 294). Note: JAAS authentication aliases are deprecated in WebSphere Application Server V6 and are supported for compatibility reasons. Component-managed security For component-managed security, the application is responsible for flowing security context to the EIS. You must set the res-auth deployment descriptor element to Application. Note We do not cover component-managed security in our JCA security test scenarios as we recommend that you leave security management to the container. Chapter 8. Gateway daemon security 293 Within our application EAR file, the EJB archive CTGTesterCCIEJB.jar contains file META-INF/ejb-jar.xml. This deployment descriptor file contains a resource-ref for the ECI reference. We set the res-auth for this resource to Container (Example 8-2). Example 8-2 EJB deployment descriptor res-auth setting <resource-ref id="ResourceRef_1"> <description>CICS ECI Resource adapter</description> <res-ref-name>ECI</res-ref-name> <res-type>javax.resource.cci.ConnectionFactory</res-type> <res-auth>Container</res-auth> </resource-ref> 8.4.1 J2C authentication data J2C authentication data entries are used by resource adapters such as JDBC data sources and the CICS ECI resource adapters. An authentication data entry includes the following information: Alias The name of the authentication data entry. When configuring the resource adapter, the administrator can specify which authentication data to choose using the corresponding alias. User ID A user identity known in the target user registry (in our case a RACF user ID). Password The password of the user ID. Note: The password is encoded in the WebSphere configuration repository. We created a J2C authentication data entry as follows: 1. In the Administrative Console Welcome panel, we clicked Global security. In the right-side panel of the screen, we selected JAAS configuration → J2C authentication data. 294 CICS Transaction Gateway for z/OS Version 6.1 2. We then clicked New and defined our authentication alias as shown in Figure 8-4. Figure 8-4 J2C authentication alias 3. We clicked OK and saved the changes. In 8.5.2, “Testing security with J2EE application CTGTesterCCI” on page 299, we show how we modified the resource authentication method for the ECI resource reference in our CTGTesterCCI application in order to use this authentication data entry. Important: The authentication of user ID and password in CICS TG for z/OS is optional (based on the setting of the environment variable AUTH_USERID_PASSWORD). So even if you define a J2C authentication alias the password will only be verified by the Gateway daemon if AUTH_USERID_PASSWORD is set. Chapter 8. Gateway daemon security 295 If your application requires a more dynamic authentication data mapping function, you can develop your own J2C mapping module. For example, the mapping module could use the WSSubject.getRunAsSubject method to retrieve the subject that represents the identity of the current running WebSphere thread (the identity of the current running thread is known as the RunAs identity). The mapping module is then able to specify that this user ID should be used for the EIS connection. 8.4.2 Setting user ID and password on the ECIConnectionSpec It is also possible for the application code itself to specify the user ID and password. We designed the CTGTesterCCI application in such a way that if user ID or password information is specified on the input screen it uses this information in an ECIConnectionSpec when it gets a connection from the connection pool. This is shown in Example 8-3. Example 8-3 Setting user ID and password on ECIConnectionSpec ECIConnectionSpec connSpec = new ECIConnectionSpec(); connSpec.setUserName("CITGSD"); connSpec.setPassword("PASSWRD"); Connection conn = connfac.getConnection(connSpec); Note: If container managed sign-on is being used and a J2C authentication alias has been defined, the user ID and password that is set in the J2C authentication alias overrides those that are set on the ECIConnectionSpec. 8.5 Testing the configuration After we configured the different software components we tested our configuration with two different Windows application clients: The sample Java test application EciB1 supplied with the CICS TG Our sample J2EE application CTGTesterCCI 8.5.1 Testing security with Java program EciB1 The aim of this basic security scenario is to verify that CICS TG user ID password authentication is working correctly. Figure 8-5 shows the outline of our environment. The platform of the Java client in our scenario is not important for 296 CICS Transaction Gateway for z/OS Version 6.1 our test purposes and could have equally been deployed on any other platform (such as UNIX System Services or Linux). W indows z/O S 9.100.193.122 CICS TS V3.1 CITG S1G C ITG S1C I Java application EciB1 G ateway daem on 11101 TCP/IP EC01 M RO/IRC EXCI user ID authentication (user ID + passw ord) authorization RACF Figure 8-5 CICS TG for z/OS security test environment - EciB1 In order to test our secure CICS TG for z/OS environment, we chose to use the EciB1 synchronous ECI sample. The runtime class for this is provided by the CICS TG in the ctgsamples.jar and the source is also provided in the \samples\java\com\ibm\ctg\samples\eci directory. To test from our Windows system, it was necessary to have a Java Development Kit (JDK) installed on the system. We then copied the ctgclient.jar and ctgsamples.jar files to the client machine and set the classpath to contain these two JAR files. Example 8-4 shows the results of testing EciB1 from a Windows workstation. Example 8-4 Testing z/OS EXCI security with EciB1 sample C:\>java com.ibm.ctg.samples.eci.EciB1 tcp://tx1.mop.ibm.com 11101 CICS TG Basic ECI Sample 1 Usage: java com.ibm.ctg.samples.eci.EciB1 [Gateway Url] [Gateway Port Number] To enable client tracing, run the sample with the following Java option: -Dgateway.T.trace=on The address of the Gateway has been set to TCP://tx1.mop.ibm.com Port:11101 CICS Servers Defined: 1. A6POTGS1 -Primary server 2. A6POTGS2 -Secondary server Choose Server to connect to, or q to quit: Chapter 8. Gateway daemon security 297 1 Enter your CICS User ID: CITGSD Enter your CICS Password: elfr1ede Program EC01 returned with data:Hex: 32332f30392f30352031323a31333a31310 ASCII text: 23/09/05 12:13:11 EciB1 is written so that it makes a call to the specified CICS region initially without a user ID and password, and then if it receives a security error, it then prompts you for a user ID and password. It also allows trace to be activated by specifying -Dgateway.T.trace=on. This traces the CICS TG calls made in the Java application and is particularly useful in showing the user ID and COMMAREA flowed to and from the Gateway daemon. When we ran EciB1 with trace on, we saw the input on the first ECI call to the Gateway daemon specifying A6POTGS1 as the CICS region. As seen in Example 8-5, the user ID at offset 57 is not set (low values), because we have not been prompted for it as yet. Example 8-5 EciB1 object/COMMAREA on initial call (created with -Dgateway.T.trace=on) S-C: S-C: S-C: S-C: S-C: CCL6603I: CCL6603I: CCL6603I: CCL6603I: CCL6603I: # # # # # 00000: 00016: 00032: 00048: 00064: 47 00 00 00 00 61 00 00 53 00 74 00 00 43 00 65 00 01 53 00 00 00 00 43 00 60 00 00 50 00 00 00 00 4A 00 00 03 00 41 00 00 45 00 34 00 00 43 00 00 2A 00 49 00 00 2A 01 00 00 00 2A 00 00 00 00 2A 00 00 00 00 2A 00 00 00 00 2A 00 60 00 00 2A Gate.@.......... ........ECI....` ................ .A6POTGS1....... .........******* In Example 8-6, we see the second ECI request to the Gateway daemon, after we had been prompted for a user ID and password. This time the user ID (CITGSD) is contained in the ECI request, and the password is masked (********) for security. Example 8-6 EciB1 object/COMMAREA after ID and password input (created with -Dgateway.T.trace=on) S-C: S-C: S-C: S-C: S-C: 298 CCL6603I: CCL6603I: CCL6603I: CCL6603I: CCL6603I: # # # # # 00000: 00016: 00032: 00048: 00064: 47 00 00 00 00 61 00 00 53 00 74 00 00 43 00 65 00 01 53 00 00 00 00 43 00 40 00 00 50 00 00 00 00 4A 00 00 03 00 41 00 00 45 00 34 00 CICS Transaction Gateway for z/OS Version 6.1 00 43 00 63 2A 00 49 00 69 2A 01 00 00 63 2A 00 00 00 73 2A 00 00 00 72 2A 00 00 00 73 2A 00 60 00 32 2A Gate.@.......... ........ECI....` ................ .A6POTGS1CITGSD. .........******* 8.5.2 Testing security with J2EE application CTGTesterCCI The aim of this security scenario is to flow a user ID and password from WebSphere Application Server for Windows to a CICS region via the Gateway daemon running on z/OS. Figure 8-6 shows the basic outline of our environment. For details of our WebSphere Application Server configuration see 8.4, “Configuring JCA security” on page 293. We test the setting of user ID and password information in the CTGTesterCCI application itself and we also show how we configured a J2C authentication data entry. Windows z/OS WebSphere Application Server V6 HTML EJB CTGTesterCCI CCI CICS TG V6.1 CICS ECI resource adapter CICS TS V3.1 CITGS1CI CITGS1G TCP/IP Gateway daemon authentication (user ID + password) ECIPROG MRO/IRC EXCI user ID RACF authorization Figure 8-6 CICS TG for z/OS security test environment - CTGTesterCCI We performed the following sequence of tests: 1. For the initial test of our configuration, we invoked the sample CTGTesterCCI J2EE application (Figure 8-7) to call the ECIPROG COBOL program in CICS. We specified a valid user ID password combination. ECIPROG returns a COMMAREA which contains the user ID under which the CICS task runs. For further details about our sample application refer to Appendix A, “Sample applications” on page 361. If user ID or password information is specified on the input screen, the CTGTesterCCI application uses this information in an ECIConnectionSpec when it gets a connection from the connection pool. Chapter 8. Gateway daemon security 299 Figure 8-7 CTGTesterCCI input with user ID and password specified Figure 8-8 shows the results of our test. The output from ECIPROG shows that the request ran successfully in our secure CICS region CITGS1CI, and that the CICS task ran under the user ID CITGSD. 300 CICS Transaction Gateway for z/OS Version 6.1 Figure 8-8 CTGTesterCCI results with user ID and password specified 2. We then repeated the test but this time we specified an invalid password on the CTGTesterCCI input screen. This test failed, as shown in Example 8-7, because our Gateway daemon is enabled for user ID password authentication. Example 8-7 STDERR error for invalid user ID and password from CTGTesterCCI CTG6808E Authorize userid/password with RACF. User ID = CITGSD, Return code = -1, errno = 111, errno2 = 0X90C0000 Chapter 8. Gateway daemon security 301 3. For our final test, we wanted to use the J2C Authentication alias that we defined in Figure 8-4 on page 295. We needed to modify the resource authentication method for the ECI resource reference in the CTGTesterCCI application. In the Administrative Console Welcome panel, we clicked Applications → Enterprise Applications, then CTGTesterCCI. In the right-side panel of the screen, we selected Map resource references to resources where we performed the following actions: a. We selected the JNDI name eis/CITGS1CI-tx1-11101 for our existing connection factory. b. We then scrolled down the page and selected the EJB module CTGTesterCCIEJB in our J2EE application. c. We scrolled back up to the top of the page and clicked Apply. d. We then checked Use default method as the authentication method and we selected our authentication alias CICS-alias as the authentication data entry to be used for this resource reference, and we clicked Apply. The final result is shown in Figure 8-9. In the bottom panel, the resource reference ECI in the EJB CTGTesterEJBCCI is mapped to the JNDI name eis/CITGS1CI-tx1-11101, and that Cam21-Pc3Node01/CICS-alias is defined as the authentication data for this resource. Thus, any use of the resource reference ECI in our J2EE application now results in the user ID CITGSD and its defined password being passed to our Gateway daemon. 302 CICS Transaction Gateway for z/OS Version 6.1 Figure 8-9 Administrative Console - mapping resource reference to authentication alias Chapter 8. Gateway daemon security 303 We repeated our test, but this time we did not specify user ID and password on the CTGTesterCCI input screen (Figure 8-10). Figure 8-10 CTGTesterCCI with no user ID and password specified 304 CICS Transaction Gateway for z/OS Version 6.1 Figure 8-11 shows the results of this test. The CICS program again runs successfully with the user ID CITGSD although this time the authentication information has been taken from the J2C authentication alias. Figure 8-11 CTGTesterCCI with no user ID and password specified Note: If user ID password checking is not enabled for the Gateway daemon, it is normally necessary to establish a trust relationship between the WebSphere Application Server and the Gateway daemon so that the application server can be trusted to flow an already authenticated user ID. Solutions such as SSL client authentication and virtual private networks (VPN) can be used to establish such a trust relationship. We show how we enabled SSL client authentication between WebSphere Application Server and the Gateway daemon in 9.4, “Configuring client authentication” on page 329. Chapter 8. Gateway daemon security 305 8.6 Problem determination In this section, we document common security problems and information we learned while configuring our Gateway daemon security scenarios. For general problem determination guidance see 2.9, “Problem determination” on page 60. 8.6.1 Common security problems Security errors are reported normally to the Java client with the return code -27(ECI_ERR_SECURITY_ERROR). These errors can be caused by several different situations, including those described in this section. Password verification errors If Gateway daemon user ID authentication fails because the password is not valid, the Gateway daemon writes a message to STDERR (Example 8-8). Example 8-8 STDERR error for user ID password verification failure CTG6808E Authorize userid/password with RACF. User ID = CITGNW, Return code = -1, errno = 111, errno2 = 0X90C0000 If user ID authentication fails because no user ID is specified, the Gateway daemon writes a similar message to STDERR (Example 8-9). Example 8-9 STDERR error for user ID not specified failure CTG6808E Authorize userid/password with RACF. User ID = , Return code = -2, errno = 0, errno2 = 0 Surrogate security errors If the Gateway daemon does not have the required access to flow a specific user ID (CITGNW) to CICS, you will receive the RACF error shown in Figure 8-10. Example 8-10 RACF message of Surrogate check failure for Gateway daemon ICH408I USER(CITGS1G) GROUP(CITG) NAME(CITG CTG GATEWAY DAEMON ) CITGNW.DFHEXCI CL(SURROGAT) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) The request has failed because the Gateway daemon user ID CITGS1G does not have access to the SURROGAT class profile CITGNW.DFHEXCI. 306 CICS Transaction Gateway for z/OS Version 6.1 Bind security errors If the Gateway daemon does not have the required access to bind to the CICS connection, you will see the RACF error shown in Figure 8-11 in the syslog. Example 8-11 RACF message of MRO bind security failure - DFHAPPL.<DFHJVPIPE> ICH408I USER(CITGS1G ) GROUP(CITG ) NAME(CITG CTG GATEWAY DAEMON) DFHAPPL.CITGS1G CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(UPDATE ) ACCESS ALLOWED(NONE ) If the Gateway daemon does not have the required access to bind to the CICS region, you will see the RACF error shown in Figure 8-12. Example 8-12 RACF message of MRO bind security failure - DFHAPPL.<APPLID> ICH408I USER(CITGS1G ) GROUP(CITG ) NAME(CITG CTG GATEWAY DAEMON) DFHAPPL.A6POTGS1 CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) If you do not see RACF messages but you still suspect an MRO bind security problem, you should enable JNI trace. A JNI trace for an MRO bind security check is shown in Example 8-13. Example 8-13 JNI trace of MRO bind security failure - DFHAPPL.<APPLID> EXCI function error. Function Call = 3, Response = 16, Reason = 609, Subreason field-1 = 0x10800b0, subreason field-2 = 0x00, Cics_Rc = -9 Security attach failures for the mirror transaction If the user ID that is flowed in the ECI request does not have READ access to the TCICSTRN class profile that controls access to the specified mirror transaction, then return code -27 (ECI_ERR_SECURITY_ERROR) is reported to the Java client. If this is the case, then error message ICH408I is also written to the CICS job log. For details on configuring the required profiles, refer to “Permitting access to the mirror transactions” on page 291. Chapter 8. Gateway daemon security 307 8.6.2 Tips and utilities In this section, we discuss additional tips and utilities for problem determination of security related problems. For general tips and information about standard utilities, such as the various Gateway daemon traces, see 2.9.2, “Tips and utilities” on page 62. Because there are several software tiers involved in this configuration, you will find security error messages in different locations: 308 CICS MSGUSR DD This is where to look for CICS security errors, for example, if the link or flowed user ID does not have authority to execute the mirror transaction. JES SYSLOG General RACF security errors are written here. Search for the message ICH408. Gateway daemon trace This provides an indication of the errors the Gateway daemon itself encounters, for example, password verification errors. JNI trace This provides tracing of the EXCI calls made by the Gateway daemon JNI module libctgjni.so, and can be very useful in problem determination of security problems. CICS Transaction Gateway for z/OS Version 6.1 9 Chapter 9. Enabling SSL with the Gateway daemon In this chapter, we outline the support that is provided by the CICS TG for z/OS for encrypted Secure Sockets Layer (SSL) connections. We show how we implemented SSL connections, using the Java Secure Sockets Extension (JSSE) SSL protocol handler, from remote Java client applications. We tested the configuration using a RACF generated key ring and certificates on the z/OS server machine and JSSE generated keystores and certificates on the client workstation. We provide details about: The creation of key rings and certificates using RACF Cipher suites Configuring server authentication Configuring client authentication Finally, we explain how we tested the environment using the CICS TG sample ECI application EciB1 and our own sample J2EE application CTGTesterCCI. © Copyright IBM Corp. 2006. All rights reserved. 309 9.1 Preparation In our testing, we connected from Java client applications running on WebSphere Application Server for Windows to CICS via the Gateway daemon on z/OS. We concentrate on how to configure SSL connections between the Java client applications and the Gateway daemon (Figure 9-1). Windows z/OS Java application EciB1 WebSphere Application Server V6 HTML SSL Gateway daemon EJB CTGTesterCCI CCI CICS TS V3.1 CICS TG V6.1 CICS TG V6.1 CICS ECI resource adapter SSL server authentication EXCI MRO/IRC CICS program SSL client authentication RACF Server certificate Server CA's certificate Client certificates Figure 9-1 Software components: Gateway daemon SSL scenarios For more information about how to enable non-SSL connections to the Gateway daemon, see Chapter 3, “Gateway daemon connections” on page 77. For information about other Gateway daemon security features, see Chapter 8, “Gateway daemon security” on page 281. 9.1.1 Software checklist Table 9-1 shows the levels of software that we used for this configuration. Table 9-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS V1R6 Windows 2000 SP4 CICS Transaction Gateway for z/OS V6.1 IBM WebSphere Application Server Network Deployment V6.0.2 IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. JSSE V1.42 JSSE V1.42 SSL V3 Protocol CICS Transaction Server for z/OS V3.1 310 CICS Transaction Gateway for z/OS Version 6.1 9.1.2 Definitions checklist Table 9-2 lists the definitions that we used to configure the scenario. Table 9-2 Definitions checklist Value WebSphere Gateway daemon CICS TS hostname cam21-pc11 tx1.mop.ibm.com - IP address (dhcp) 9.100.193.122 - SSL port 8059 - Job name CITGS1G CITGS1CI APPLID A6POTGS1 NETNAME CITGS1G CONNECTION GS1G 9.2 Secure Sockets Layer Secure Sockets Layer (SSL) is a protocol designed to create a secure connection to a server using public key encryption, and to protect the data integrity and privacy as the data is transferred over the connection. CICS TG supports the Java Secure Socket Extension (JSSE) implementation of SSL which provides 128 bit and 256 bit encryption. JSSE is supplied as part of the IBM SDK for z/OS, Java 2 Technology Edition, V1.4.2 SR2. With previous versions of CICS TG you were able to use SystemSSL which is specific to z/OS and the Java-based SSLight. JSSE is the only supported option for providing SSL with CICS TG V6 and later releases. If migrating from V5 to V6 see 9.5.1, “SystemSSL to JSSE migration” on page 335. Important: SystemSSL and SSLight are not supported by CICS TG V6 and later releases. The SSL Handshake Protocol consists of two phases: Server authentication In the first phase, the server responds to a client’s request by sending its certificate and cipher preferences. The client then generates a master key which it encrypts with the server’s public key and then transmits the encrypted master key to the server. The server authenticates itself to the client by returning a message authenticated with keys derived from the master key. Chapter 9. Enabling SSL with the Gateway daemon 311 Subsequent data is encrypted and authenticated with keys derived from the master key. Client authentication In the second optional phase, the server requests that a client identifies itself during the SSL handshake by providing its client certificate. Client authentication can only be requested by the server. Note: CICS TG supports both server and client authentication. When setting up an SSL environment you have a choice between self-signed certificates and CA signed certificates: Self-signed certificates A self-signed certificate is an identity certificate that is signed by its creator, so the creator is verifying that the certificate is valid. Certificating authority (CA) signed certificates CA signed certificates are created by a user organization and sent to a CA to be signed. Before signing a certificate the CA will verify that the organization requesting the certificate is actually who they claim to be, so the CA is verifying that the certificate is valid. In the examples in this chapter, we use self-signed certificates as they are more easily and quickly generated. In a production environment, it is expected that CA signed certificates are used because they provide a more secure solution. 9.2.1 JSSE Java Secure Sockets Extension (JSSE) is a Java implementation of SSL common across all platforms. JSSE implements SSL 3.0 and TLS 1.0 (Transport Layer Security) algorithm types as Java 2 standard extensions. It provides the socket factories that allow for ease of programming, if the provided defaults are taken. JSSE is now the strategic SSL implementation for CICS TG and supports the following: 1. Cryptographic handshakes on z/OS using an IBMJCE4758 Peripheral Component Interconnect (PCI) card. 2. Management tools to generate keystores and manage certificates: keytool hwkeytool ikeyman 312 a command line interface for maintaining keystores a command line interface for maintaining hardware keys a GUI interface for maintaining keystores CICS Transaction Gateway for z/OS Version 6.1 3. The ability to store digital certificates in a RACF database. Restriction: At this time CICS TG does not support Transport Layer Security (TLS). 9.2.2 Cipher suites IBM SDK 1.4.2 JSSE supports a wide range of cipher suites. Example 9-1 provides an explanation of the cipher suite naming convention. Example 9-1 Cipher suite components SSL_RSA_WITH_RC4_128_SHA RSA is a type of key exchange cipher RC4_128 is the cipher for data encryption SHA is the hashing algorithm The key exchange cipher is used for the initial handshake. The most widely used key exchange cipher is RSA. This type of cipher is referred to as asymmetric as it uses public and private keys. The cipher for data encryption (also known as the data encryption cipher) is used to secure the data which is passed over the SSL connection after the handshake has completed. Possible data encryption ciphers are AES, DES, Triple DES, RC2 and RC4 (which have 40,128 and 256-bit variants). These ciphers are referred to as symmetric as they use a single key for encryption and decryption. Symmetric key or secret key algorithms can be further classified as either block ciphers or stream ciphers. Block ciphers (such as DES) operate on the data in blocks while stream ciphers (such as RC4) operate on the data one bit at a time. The hashing algorithm is a one way method of producing a hash number from a message. A slight change in the message produces a completely different hash number (a 128 bit fingerprint) and is used to ensure a message has not been tampered with. The client and the server both generate a hash number from the data transmitted using the hashing algorithm. If the hash numbers do not match the integrity of the message has been compromised. Data integrity is provided by one of the following hashing algorithms (also known as message digest algorithms); SHA1, SHA2, SHA3, SHA5, MD2 and MD5. Cipher suite names ending in _SHA indicate that the SHA1 hashing algorithm is being used. SSL can use signature algorithms SHA1 with RSA, SHA1 with DSA, MD5 with RSA and MD2 with RSA. Chapter 9. Enabling SSL with the Gateway daemon 313 Tips: CICS TG V6 supports a range of different cipher suites and allows you to specify which ciphers to use. When deciding this you should consider the following performance criteria. 1. You can significantly reduce the number of SSL handshakes when connecting from WebSphere Application Server by using JCA connection pooling. 2. RSA encryption and decryption processing during the SSL handshake can be off-loaded using crypto engines (PCICA, PCIXCC or CEX2C). 3. Larger keys (512/1024/2048) for key exchange ciphers have a higher handshake cost. 4. Choose appropriate bulk data ciphers. It is generally accepted that: – MD5 is faster than SHA but is less secure – RC4 normally has a lower CPU cost than DES in software – RC4 stronger ciphers (128-bit versus 40-bit) do not require more CPU cycles – SHA, DES and AES can often be implemented efficiently in hardware 9.2.3 Hardware cryptography You can use hardware cryptography with the CICS TG to offload the SSL handshake to a cryptographic device. This provides extra security and improved performance. Note: At present JSSE supports hardware cryptography only for handshakes and not for data encryption. 9.2.4 The hwkeytool command You can use the hwkeytool command that is provided as part of the IBM Java SDK in much the same way that we used the keytool command to create a keystore and manage certificates (see 9.3.1, “Creating a keystore on z/OS using keytool” on page 316). Extra parameters are available to specify how the key is stored on the cryptographic device and how it is to be used. Example 9-2 shows an example of hwkeytool. Example 9-2 hwkeytool hwkeytool -genkey -alias server1 -keyalg RSA -storetype JCE4785KS -dname “cn=tx1.mop.ibm.com,o=IBM,c=FR” -keystore ctgv61.server.HWkeystore -keypass antigone -storepass antigone -hardwaretype CLEAR -hardwareusage KEYMANAGEMENT 314 CICS Transaction Gateway for z/OS Version 6.1 The additional parameters used with hwkeytool are: -hardwaretype The type of key pair that will be generated, this is either CLEAR (the default), PKDS (stored in the Public Key Data set and encrypted with the co-processor master key) or RETAINED (not recommended for z/OS). -hardwareusage This parameter sets the usage of the key pair being generated. Possible options are KEYMANAGEMENT (default for RSA) or SIGNATURE (default for DSA). -hardwarekey Specifies that the key pair is to be deleted from the hardware storage as well as from the keystore. The default is to delete from the keystore only. Tips: A keystore created by hwkeytool must contain a personal certificate and the CA certificate used to sign it. If the personal certificate is self-signed (generated with the -selfcert parameter) first export the certificate and then import it into the same keystore with a different alias. If you are warned when you are importing the certificate back into the keystore that it already exists, type Y to confirm that you want to import it. Secure key processing (where the key is stored in the PKDS) offers greater security over clear key processing by hiding sensitive key data but can require additional hardware and result in a processing overhead. For further information about hardware cryptography, we recommend the following: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1535 http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100647 9.2.5 Java configuration for using hardware cryptography Java Cryptography Extension (JCE) 1.2.1 provides a framework and implementations for encryption, key generation, key agreement and Message Authentication Code (MAC) algorithms. The IBMJCE4758 implementation extends JCE to add hardware cryptography functions using the IBM Common Cryptographic Architecture (CCA) interfaces. Chapter 9. Enabling SSL with the Gateway daemon 315 To use the hardware cryptographic functions of IBMJCE4758 and hwkeytool, take the following steps: 1. Update the java.security file located in the /<java-home>/lib/security directory so that it contains the following lines: security.provider.1=com.ibm.crypto.hdwrCCA.provider.IBMJCE4758 security.provider.2=com.ibm.crypto.provider.IBMJCE 2. Copy the unrestricted policy files from /<java-home>/demo/jce/policy-files/unrestricted to /<java-home>/lib/security. Note: After you have configured your JVM to be able to use hardware cryptography, you will not be able to use keytool. If you wish to revert back to using keytool, you need to remove the line that references IBMJCE4758 from the java.security file. 9.3 Configuring server authentication In this section, we describe how we: 1. Created a keystore on z/OS using the Unix System Services (USS) shell keytool command. 2. Created a key ring and certificates on z/OS using RACF and used the ikeyman tool on Windows to create a keystore and to import a server certificate. 3. Configured CICS TG to support SSL. 4. Tested our configuration. 9.3.1 Creating a keystore on z/OS using keytool In this section, we describe how we created a keystore and the certificates required for server authentication using the keytool command. Note: We show how we created a keystore on z/OS using the keytool command for completeness. However, we recommend the use of RACF for storing keys and certificates (see 9.3.2, “Creating a key ring and certificates on z/OS using RACF” on page 319). 316 CICS Transaction Gateway for z/OS Version 6.1 A keystore can be created and a self-signed certificate generated on z/OS using the USS shell keytool command as shown in Example 9-3. Example 9-3 Keytool command to create a keystore and generate a certificate keytool -genkey -alias server -keysize 1024 -dname “cn=tx1.mop.ibm.com,o=IBM,c=FR” -keystore ctgv61.server.jks -keypass antigone -storepass antigone -keyalg RSA The options shown with the keytool command are as follows: -genkey Generates a key pair and puts the public key into a self-signed certificate. -alias Specifies the unique name that is used to refer to the key pair and certificate within the keystore. -dname Specifies the X.500 distinguished name to be associated with the alias. This is used as the issuer and subject fields of the self-signed certificate. The distinguished name consists of a number of fields separated by commas. -keystore The keystore location. -keypass The password used to protect the private key. -storepass The password used to protect the keystore. -keyalg Specifies the algorithm to be used to generate the key pair. Note: We found that -keypass and -storepass must be the same in order for the keystore to be used successfully by the Gateway daemon. We used the keytool command with the -list and -v parameters to check the contents of the certificate (Example 9-4). Example 9-4 Keytool -list command keytool -list -v -keystore ctgv61.server.jks -storepass antigone Example 9-5 shows the results of the keytool -list command. Example 9-5 Output from keytool -list -v command Keystore type: jks Keystore provider: IBMJCE Your keystore contains 1 entry Alias name: server Creation date: Sep 20, 2005 Chapter 9. Enabling SSL with the Gateway daemon 317 Entry type: keyEntry Certificate chain length: 1 Certificate[1]: Owner: CN=tx1.mop.ibm.com, O=IBM, C=FR Issuer: CN=tx1.mop.ibm.com, O=IBM, C=FR Serial number: 43303dec Valid from: 9/20/05 6:50 PM until: 12/19/05 6:50 PM Certificate fingerprints: MD5: 19:2E:2C:78:E8:31:D9:BE:EC:21:6F:BC:FC:8F:80:2F SHA1: 5A:E0:04:14:ED:69:1A:12:F6:D0:19:CA:B7:CA:7B:65:E6:E1:D8:E1 The next step was to export the self-signed certificate from the server keystore to a binary file named servercert.der. Example 9-6 shows the keytool -export command was used to do this. Example 9-6 Keytool -export command keytool -export -alias server -keystore ctgv61.server.jks -storepass antigone -keypass antigone -file servercert.der After successfully processing this command, keytool responds with the following message: Certificate stored in file <servercert.der> Configuring Windows The exported certificate was transferred to the client workstation using FTP (in binary mode). We then imported the certificate servercert.der into a client keystore ctgv61.client2.jks using the command shown in Example 9-7. Example 9-7 Import of a server certificate using keytool on Windows keytool -import -alias server -keystore ctgv61.client2.jks -storepass antigone -keypass antigone -file servercert.der Example 9-8 shows the output from the keytool command. Example 9-8 Output from a keytool import command on Windows Owner: CN=tx1.mop.ibm.com, O=IBM, C=FR Issuer: CN=tx1.mop.ibm.com, O=IBM, C=FR Serial number: 43303dec Valid from: 9/20/05 6:50 PM until: 12/19/05 5:50 PM Certificate fingerprints: MD5: 19:2E:2C:78:E8:31:D9:BE:EC:21:6F:BC:FC:8F:80:2F SHA1: 5A:E0:04:14:ED:69:1A:12:F6:D0:19:CA:B7:CA:7B:65:E6:E1:D8:E1 Trust this certificate? [no]: y Certificate was added to keystore 318 CICS Transaction Gateway for z/OS Version 6.1 9.3.2 Creating a key ring and certificates on z/OS using RACF In this section, we describe how we created a key ring and the certificates that are required for server authentication using RACF: 1. To create a self-signed Certificate Authority (CA) certificate, we used the following RACDCERT command: RACDCERT CERTAUTH GENCERT SUBJECTSDN(OU(‘CTG V61 Local CA’) O(‘IBM’) C’FR’)) KEYUSAGE(CERTSIGN) WITHLABEL(‘CTG V61 CA’) 2. We then created a personal certificate signed by the CA certificate generated in the previous step: RACDCERT ID(CITGS1G) GENCERT SUBJECTSDN(OU(‘CTG V61 Personal’) O(‘IBM’) C(‘FR’)) WITHLABEL(‘CTG V61 Personal’) SIGNWITH(CERTAUTH LABEL(‘CTG V61 CA’)) 3. We created a key ring (CTGV61ring) using the command: RACDCERT ADDRING(CTGV61ring) ID(CITGS1G) 4. We used the following commands to put the CA and personal certificates in the key ring: RACDCERT ID(CITGS1G) CONNECT(CERTAUTH LABEL(‘CTG V61 CA’) RING(CTGV61ring) USAGE(CERTAUTH)) RACDCERT ID(CITGS1G) CONNECT(LABEL(‘CTG V61 Personal’) RING(CTGV61ring) DEFAULT USAGE(PERSONAL) Attention: When creating your key ring ensure that the ID parameter specified on the RACDCERT ADDRING command is the user ID used for the Gateway daemon started task (CITGS1G in our case). 5. To complete this part of the configuration, we exported the CA certificate in Base64 X.509 format using the following RACDCERT command: RACDCERT CERTAUTH EXPORT(LABEL(‘CTG V61 CA’)) DSN(CITGSD.CERTAUTH.X509) FORMAT(CERTB64) Note: The RACF RACDCERT command can also be used for generating a key ring and certificates to be used by a cryptographic device. For more information, see The z/OS Security Server (RACF) Command Reference, SA22-7687. Chapter 9. Enabling SSL with the Gateway daemon 319 Configuring Windows We used FTP as shown in Example 9-9 to send the exported certificate (in ASCII format) to our client machine. Example 9-9 How to use FTP to send an exported CA certificate to a client workstation C:\SDA_CTG_V61>ftp tx1.mop.ibm.com Connected to tx1.mop.ibm.com. 220-FTPD11 IBM FTP CS V1R6 at TX1.MOP.IBM.COM, 07:49:41 on 2005-09-27. 220 Connection will close if idle for more than 5 minutes. User (tx1.mop.ibm.com:(none)): citgsd 331 Send password please. Password: 230 CITGSD is logged on. Working directory is "CITGSD.". ftp> ascii 200 Representation type is Ascii NonPrint ftp> get "CERTAUTH.X509" 200 Port request OK. 125 Sending data set CITGSD.CERTAUTH.X509 250 Transfer completed successfully. ftp: 898 bytes received in 0.00Seconds 898000.00Kbytes/sec. ftp> quit 221 Quit command received. Goodbye. After completing the FTP transfer we continued the SSL configuration on our Windows workstation. This can be done either with keytool or with ikeyman (the IBM key management tool). Both of these tools are provided with the IBM Java SDK. On our workstation, these tools were found in the directory: C:\Program Files\IBM\Java142\jre\bin After opening a command prompt window we set our PATH variable and typed ikeyman to invoke the iKeyMan GUI: set PATH=C:\Program Files\IBM\Java142\jre\bin\;%PATH% ikeyman 320 CICS Transaction Gateway for z/OS Version 6.1 We used iKeyMan to create a keystore called ctgv61.client.jks (with password antigone), and we then added the CERTAUTH.X509 file to the keystore as shown in Figure 9-2. Figure 9-2 iKeyMan - Add a CA certificate to the keystore from a X509 file Chapter 9. Enabling SSL with the Gateway daemon 321 We clicked OK to proceed, and we were then prompted to enter a label for the certificate. Figure 9-3 shows our CA certificate in the client workstation keystore. Figure 9-3 iKeyMan - After adding CA certificate 322 CICS Transaction Gateway for z/OS Version 6.1 See 9.3.4, “Testing server authentication” on page 325 for information about how we tested SSL server authentication from a java client using the ctgv61.client.jks keystore. 9.3.3 Configuring CICS TG In order to use the RACF key ring that we created in 9.3.2, “Creating a key ring and certificates on z/OS using RACF” on page 319, we needed to configure our Gateway daemon to use SSL. We specified the keyring=<RACF keyring name> and esmkeyring parameters in the SSL protocol parameter list used by our Gateway daemon. Example 9-10 shows the SSL parameter list in the Gateway configuration file (CITGS1G.ini). Example 9-10 SSL protocol parameters for using a RACF key ring [email protected]=com.ibm.ctg.server.SslHandler [email protected]=port=8059;\ connecttimeout=2000;\ idletimeout=600000;\ pingfrequency=60000;\ clientauth=off;\ keyring=CTGV61ring;\ esmkeyring; Tip: We found that the esmkeyring parameter must be specified as the last parameter. If this was not the case, SSL initialization failed with the following error message: CTG6525E Unable to start handler for the ssl: protocol. [java.lang.IllegalArgumentException: keyring=] Limiting available cipher suites The cipher suites which are available for use by CICS TG can be limited by using the ciphersuites parameter. The protocol definition to limit SSL to one cipher suite (SSL_RSA_WITH_RC4_128_SHA) is shown in Example 9-11. Example 9-11 Ciphersuites parameter of SSL protocol [email protected]=com.ibm.ctg.server.SslHandler [email protected]=port=8059;\ connecttimeout=2000;\ idletimeout=600000;\ pingfrequency=60000;\ clientauth=off;\ ciphersuites=SSL_RSA_WITH_RC4_128_SHA;\ keyring=CTGV61ring;\ esmkeyring; Chapter 9. Enabling SSL with the Gateway daemon 323 Note: The parameter ciphersuites=128 bitonly is obsolete. If this parameter is coded, it is replaced by a ciphersuites setting of SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_DSS_WITH_R C4_128_SHA. During startup, the Gateway daemon writes a CTG8401I message listing which ciphers are enabled. Cipher suite negotiation then takes place during the SSL handshake phase. At this point both sides of the connection agree on a cipher suite that will be used. The Gateway daemon writes a connection message showing which cipher suite is being used (Example 9-12). Example 9-12 Gateway daemon message confirming cipher suite usage 19:12:46:516 ssl: S-C: Client socket[addr=9.100.199.140/9.100.199.140,port=1099,localport=8059] connected using cipher suite SSL_RSA_WITH_RC4_128_SHA If you do not use the ciphersuites parameter to limit the number of cipher suites the content of the CTG8401I message written at gateway daemon startup will be as shown in Example 9-13. Example 9-13 CTG8401I message when the ciphersuites parameter is not used CTG8401I The following ciphers are enabled: SSL_RSA_WITH_RC4_128_MD5 SSL_RSA_WITH_RC4_128_SHA SSL_RSA_WITH_AES_128_CBC_SHA SSL_RSA_WITH_AES_256_CBC_SHA SSL_RSA_WITH_DES_CBC_SHA SSL_RSA_FIPS_WITH_DES_CBC_SHA SSL_RSA_WITH_3DES_EDE_CBC_SHA SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA SSL_DHE_RSA_WITH_AES_128_CBC_SHA SSL_DHE_RSA_WITH_AES_256_CBC_SHA SSL_DHE_RSA_WITH_DES_CBC_SHA SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_AES_128_CBC_SHA SSL_DHE_DSS_WITH_AES_256_CBC_SHA SSL_DHE_DSS_WITH_RC4_128_SHA SSL_DHE_DSS_WITH_DES_CBC_SHA SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_RSA_EXPORT_WITH_RC4_40_MD5 SSL_RSA_EXPORT_WITH_DES40_CBC_SHA SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5 SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA SSL_RSA_WITH_NULL_MD5 324 CICS Transaction Gateway for z/OS Version 6.1 SSL_RSA_WITH_NULL_SHA SSL_DH_anon_WITH_AES_128_CBC_SHA SSL_DH_anon_WITH_AES_256_CBC_SHA SSL_DH_anon_WITH_RC4_128_MD5 SSL_DH_anon_WITH_DES_CBC_SHA SSL_DH_anon_WITH_3DES_EDE_CBC_SHA SSL_DH_anon_EXPORT_WITH_RC4_40_MD5 SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA Of the SSL cipher suites that are listed in message CTG8401I, the following considerations apply: The SSL_DHE_xxx cipher suites require the use of a DSA encrypted key so key rings generated with RSA will not work. To use DSA to generate the key use the keytool command with the -keyalg DSA option. By default the Java Cryptography Extension (JCE) is shipped with limited strength ciphers. To use 256-bit Advanced Encryption Standard (AES) encryption algorithms you must apply unlimited jurisdiction policy files. These are placed in the <java_home>/lib/security directory. Note: Details on upgrading the security policy file for the IBM SDK can be found at the following Web site: http://www.ibm.com/developerworks/java/jdk/security/142/ Although the CTG8401I message lists anonymous ciphers as being enabled, the SSL_DH_anon_xxx cipher suites will not work because the JSSE trust manager does not support anonymous connections. If you try to use them you will get a handshake failure. 9.3.4 Testing server authentication After we created the RACF key ring and configured SSL support for our Gateway daemon we tested our configuration with two different Windows application clients: The sample Java test application EciB1 supplied with the CICS TG Our sample J2EE application CTGTesterCCI deployed on WebSphere Application Server (see “Testing SSL server authentication with CTGTesterCCI” on page 327) Chapter 9. Enabling SSL with the Gateway daemon 325 Testing SSL server authentication with Java program EciB1 To test our SSL setup we used the Java client application program EciB1 which is contained in the jar file ctgsamples.jar. We ran EciB1 with the SSL Keyring and SSL Password options, the command and output are shown in Example 9-14. Example 9-14 Command to execute EciB1 and the output produced C:\SDA_CTG_V61>java com.ibm.ctg.samples.eci.EciB1 ssl://tx1.mop.ibm.com 8059 ctgv61.client.jks antigone CICS Transaction Gateway Basic ECI Sample 1 Usage: java com.ibm.ctg.samples.eci.EciB1 [Gateway URL] [Gateway Port Number] [SSL Keyring] [SSL Password] To enable client tracing, run the sample with the following Java option: -Dgateway.T.trace=on The address of the Gateway has been set to ssl://tx1.mop.ibm.com Port:8059 CICS Servers Defined: 1. A6POTGS1 -Primary server 2. A6POTGS2 -Secondary server Choose Server to connect to, or q to quit: 1 Enter your CICS User ID: citgsd Enter your CICS Password: elfr1ede Program EC01 returned with data:Hex: 30342f31302f30352031323a33363a35330 ASCII text: 04/10/05 12:36:53 Because we had specified connectionlogging=on for the Gateway daemon, we found confirmation that SSL was being used in the Gateway daemon STDOUT 326 CICS Transaction Gateway for z/OS Version 6.1 log (Example 9-15). Note that the connection is to the Gateway daemon SSL port 8059. Example 9-15 CICS TG log showing a connection to the Gateway daemon port 8059 10/11/05 12:32:09:714 [0] CTG6506I Client connected. [ConnectionManager-0] ssl@Socket[addr=9.100.199.140/9.100.199.140,port=3041,localport=8059] Testing SSL server authentication with CTGTesterCCI To demonstrate the use of SSL between WebSphere Application Server for Windows and the Gateway daemon we repeated our test scenario with the CTGTesterCCI J2EE application. JCA connection pooling with SSL connections With SSL security enabled there is an additional performance overhead incurred by the security processing, in particular, during the initial handshake negotiation. If JCA connection pooling is used this overhead can be significantly reduced (see 3.2.2, “Creating connection factories” on page 82 for details on connection pooling). Important: JCA connection pooling significantly reduces the performance overhead of SSL connections. Because an SSL connection takes longer to set up than a non-SSL connection, the connection pool timeout settings should be reviewed at the time that SSL is being implemented. Ideally the connection pool should configured such that a minimum number of connections are terminated and re-established during normal operation. With SSL connections it might be preferable to have longer timeout values resulting in unused connections persisting longer rather than incurring the cost of starting new connections. Enabling an SSL connection from WebSphere Application Server We needed to make the CERTAUTH.X509 certificate (created in 9.3.2, “Creating a key ring and certificates on z/OS using RACF” on page 319) available to our WebSphere Application Server. To do this we: 1. Transferred the CERTAUTH.X509 server certificate to the WebSphere Application Server machine in ASCII mode using FTP. 2. Imported the CERTAUTH.X509 certificate to the WebSphere Application Server DummyServerKeyFile.jks using iKeyMan. We used the same process that we describe in “Configuring Windows” on page 320. Chapter 9. Enabling SSL with the Gateway daemon 327 Important: We used the WebSphere Application Server default keystore for our tests. However, never use this dummy file in a production environment or any time that sensitive data is being transmitted. In the WebSphere Administration Console in Resource Adapters → ECIResourceAdapter → J2C connection factories → CITGS1CI-tx1-11101 → Custom properties, we updated the following: ConnectionURL to specify the SSL protocol KeyRingClass to point to the DummyServerKeyFile.jks KeyRingPassword to be WebAS PortNumber to be the Gateway daemon’s SSL port, 8059 The connection factory custom properties are displayed in Figure 9-4. Figure 9-4 Administrative Console - SSL connection factory custom properties We stopped and restarted WebSphere Application Server to pick up the changes to the connection factory and tested with the CTGTesterCCI application as shown in 8.5.2, “Testing security with J2EE application CTGTesterCCI” on page 299. Because we had specified connectionlogging=on for the Gateway daemon, we were able to confirm that SSL was being used by checking the connection message in the Gateway daemon STDOUT log (see Example 9-15 on page 327). 328 CICS Transaction Gateway for z/OS Version 6.1 9.4 Configuring client authentication In client authentication the server sends a request to the client to identify itself. The client authenticates itself by returning the client’s digital signature and its public-key certificate. In this section we describe: 1. 2. 3. 4. How we created a self-signed client certificate How we exported the client personal certificate CICS TG configuration for client authentication How we tested our configuration 9.4.1 Creating the client certificate on a Windows client machine The first step to set up client authentication is to create a client certificate. We did this using the iKeyMan option Create New Self-Signed Certificate as shown in Figure 9-5. Figure 9-5 iKeyMan - Create a self-signed client certificate Chapter 9. Enabling SSL with the Gateway daemon 329 9.4.2 Extracting a client personal certificate After creating the new self-signed client certificate we used the iKeyMan extract option to extract the certificate to a Base64 ASCII file (Figure 9-6). Figure 9-6 iKeyMan - extract client certificate to a file After extracting the client certificate to file clientcr.arm, we transferred it by FTP (in ASCII format) to our z/OS machine (Example 9-16). Example 9-16 FTP of a client certificate C:\SDA_CTG_V61>ftp tx1.mop.ibm.com Connected to tx1.mop.ibm.com. 220-FTPD11 IBM FTP CS V1R6 at TX1.MOP.IBM.COM, 15:01:26 on 2005-10-11 220 Connection will close if idle for more than 5 minutes. User (tx1.mop.ibm.com:(none)): citgsd 331 Send password please. Password: 230 CITGSD is logged on. Working directory is "CITGSD.". 330 CICS Transaction Gateway for z/OS Version 6.1 ftp> ascii 200 Representation type is Ascii NonPrint ftp> put "clientcr.arm" 200 Port request OK. 125 Storing data set CITGSD.CLIENTCR.ARM 250 Transfer completed successfully. ftp: 706 bytes sent in 0.00Seconds 706000.00Kbytes/sec. ftp> quit 221 Quit command received. Goodbye. To complete our setup for client authentication we used the RACDCERT command to add the client certificate to the RACF digital key ring class and to connect it to our RACF key ring. Example 9-17 shows the commands to do this. Example 9-17 RACDCERT commands to add and connect a client certificate RACDCERT CERTAUTH ADD(‘CITGSD.CLIENTCR.ARM’) TRUST WITHLABEL(‘CTG V61 Client Certificate’) RACDCERT ID(CITGS1G) CONNECT(CERTAUTH LABEL(‘CTG V61 Client Certificate’) RING(CTGV61ring) USAGE(CERTAUTH)) 9.4.3 Configuring CICS TG To activate client authentication in the Gateway daemon we set the parameter clientauth=on in the SSL protocol parameter list (Example 9-18). Example 9-18 SSL protocol parameters [email protected]=com.ibm.ctg.server.SslHandler [email protected]=port=8059;\ connecttimeout=2000;\ idletimeout=600000;\ pingfrequency=60000;\ clientauth=on;\ ciphersuites=SSL_RSA_WITH_RC4_128_SHA;\ keyring=CTGV61ring;\ esmkeyring; If tracing is active, the Gateway daemon writes the following message during initialization: SslHandler: + Client Authentication enabled for ssl: protocol Chapter 9. Enabling SSL with the Gateway daemon 331 Mapping client certificates to RACF user IDs Optionally with SSL client authentication, client certificates can be mapped to RACF user IDs. To map a certificate to a RACF user ID the certificate must first be associated with a RACF user ID, using one of the following methods: The RACF command RACDCERT This method stores client certificates in the RACF database, and associates a user ID with them. This is an excellent way to create a one-to-one mapping between client certificates and user IDs, but it does not scale well in the case where, for example, large numbers of certificates need to be mapped to a small number of user IDs. RACF certificate name filtering This uses rules, to allow many certificates to be assigned a single user ID with one profile. The com.ibm.ctg.util.RACFUserid class can be used to map an ECI request to a RACF user ID, based on the distinguished name in the SSL client certificate. The class has the following methods: setCertificate(byte[] clientCertificateData) getRACFUserid() Create a RACFUserid as follows: RACFUserid myUseridObject = new RACFUserid(myCertificate); This creates an object and automatically populates it with certificate data. When the object has been created, the getRACFUserid() method can be used to make a native RACF call to determine the user ID associated with the certificate data. If successful, it returns a string containing the user ID. The SSLServerCompression.java class in the <install_path>/samples/java/com/ibm/ctg/samples/security subdirectory shows an example of how to use the RACFUserid class. For more information about the com.ibm.ctg.util.RACFUserid class, refer to CICS Transaction Gateway: Programming Guide. 9.4.4 Testing client authentication After we configured the different software components, we tested our client authentication configuration with two different Windows application clients: The sample Java test application EciB1 supplied with the CICS TG Our sample J2EE application CTGTesterCCI 332 CICS Transaction Gateway for z/OS Version 6.1 Testing SSL client authentication with Java program EciB1 We ran EciB1 with the SSL Keyring and SSL Password options, as shown in Example 9-14 on page 326. In this case, the Java client is requested to send the client certificate that we created in the keystore ctgv61.client.jks. Testing SSL client authentication with CTGTesterCCI To demonstrate the use of SSL client authentication between WebSphere Application Server for Windows and the Gateway daemon, we repeated our test scenario with the CTGTesterCCI J2EE application. To demonstrate that we had correctly configured client authentication for the Gateway daemon we first tried to run CTGTesterCCI before adding a client certificate to the keystore used by the application server. Figure 9-7 shows the error that we received when attempting to execute the CTGTesterCCI application. Figure 9-7 CTGTesterCCI Error with clientauth=on but no Client Certificate We also noted the messages shown in Example 9-19 in STDERR. Example 9-19 STDERR - SSL client authentication error messages ssl: S-C: Client Socket[addr=9.100.199.140/9.100.199.140,port=1154,localport=8059] failed to connect, handshake failure ssl: .SslHandler:javax.net.ssl.SSLHandshakeException: handshake failure ssl: .SslHandler: at com.ibm.jsse.bv.a(Unknown Source) ssl: .SslHandler: at com.ibm.jsse.bv.startHandshake(Unknown Source) ssl: .SslHandler: at com.ibm.ctg.server.JSSEServerSocket.accept(JSSEServerSocket.java:109) ssl: .SslHandler: at com.ibm.ctg.server.SocketHandler.run(SocketHandler.java:666) ssl: .SslHandler: at java.lang.Thread.run(Thread.java:568) To enable client authentication we: 1. Created a self-signed Client certificate on the WebSphere Application Server machine in the DummyServerKeyFile.jks using iKeyMan, using the same process as in 9.4.1, “Creating the client certificate on a Windows client machine” on page 329. 2. Extracted the self-signed client certificate as clientws.arm using iKeyMan and transferred to z/OS in ASCII mode using FTP as was done in 9.4.2, “Extracting a client personal certificate” on page 330. 3. Imported to our server keystore on z/OS using commands similar to those in Example 9-17 on page 331. Chapter 9. Enabling SSL with the Gateway daemon 333 We ran the CTGTesterCCI application and it completed successfully. Asserted Identity It is possible to use SSL client authentication as a way of establishing a trust relationship between the WebSphere Application Server and the Gateway daemon so that the application server can be trusted to flow an asserted user ID as a security credential. In this scenario user ID password checking would not be enabled in the Gateway daemon. We repeated the test shown in 8.5.2, “Testing security with J2EE application CTGTesterCCI” on page 299 passing user ID CITGSD but this time we did not specify a password. We found that if a password is not specified on the ECIInteractionSpec the specified user ID is not flowed and the target CICS program runs with the default CICS user ID (Figure 9-8). Figure 9-8 CTGTesterCCI - User ID with no password To get the result that we expected we specified the user ID CITGSD and non-null password (any password is allowed in this instance because the Gateway daemon was not configured to verify passwords). The CICS program then ran with the specified user ID (CITGSD) (Figure 9-9). Figure 9-9 CTGTesterCCI - User ID with non-null password Note: APAR PK17700 (CICS TG for Multiplatforms) and PK19503 (CICS TG for z/OS) have been raised to change the behavior of the CICS ECI resource adapters to allow a user ID to be specified on the ECIInteractionSpec without specifying a password. 334 CICS Transaction Gateway for z/OS Version 6.1 9.5 Migration considerations JSSE is the only supported implementation of the SSL protocol with CICS TG V6 and above. If a previous version of the CICS TG was configured to use SSLight or SystemSSL, you must take steps to migrate the associated resources to a JSSE configuration. 9.5.1 SystemSSL to JSSE migration In this section, we give a brief outline of the steps required to migrate SystemSSL certificates to JSSE. To migrate a certificate with private keys perform the following steps: Use the SystemSSL tool gskkyman to export the certificate in Base64 PKCS#12 format Use the iconv command to convert the file from EBCDIC to ISO8859 format Use keytool to import the certificate and private key from a PKCS#12 file to a keystore To migrate a certificate without private keys: Use gskkyman to export the certificate to a binary ASN.1 DER encoded file Use keytool to import the certificate and private key from a ASN.1 DER file to a keystore See the CICS Transaction Gateway z/OS Administration Version 6.1 guide for detailed information about migrating to JSSE. Chapter 9. Enabling SSL with the Gateway daemon 335 9.6 Problem determination In this section, we document common SSL problems and information that we learned while configuring our SSL test scenarios: The example shown in Example 9-20 shows the error messages sent to the workstation during client authentication when the client certificate had not been connected to the RACF key ring: Example 9-20 Handshake failure, missing client certificate in RACF key ring java.io.IOException: CTG6651E Unable to connect to the Gateway daemon. [address = tx1.mop.ibm.com, port = 8059] [javax.net.ssl.SSLHandshakeException: handshake failure] at com.ibm.ctg.client.SslJavaGateway.open(SslJavaGateway.java:211) at com.ibm.ctg.client.JavaGateway.open(JavaGateway.java:526) at com.ibm.ctg.client.JavaGateway.<init>(JavaGateway.java:205) at com.ibm.ctg.samples.eci.EciB1.main(EciB1.java:132) Caused by: javax.net.ssl.SSLHandshakeException: handshake failure If we omitted a KeyRingClass definition in the connection factory after specifying the SSL protocol in “Testing SSL server authentication with CTGTesterCCI” on page 327 we received the CTG9674E error message shown in Figure 9-10. Figure 9-10 Server Authentication, no keyfile in connection factory After we had defined the KeyRingClass if we failed to import the CERTAUTH.X509 certificate into the ServerKeyFile.jks, we received the CTG9630E error message shown in Figure 9-11. Figure 9-11 Server Authentication, no certificate in keyfile 336 CICS Transaction Gateway for z/OS Version 6.1 Apart from the obvious case of when the key ring does not exist the message shown in Example 9-21 can occur if the user ID associated with the key ring by the RACDCERT ID(userid) ADDRING(ringname) command does not match the region userid of the CICS TG. Example 9-21 Example 9-21Key ring not found CTG6525E Unable to start handler for the ssl: protocol. [java.lang.Exception: java.io.IOException: R_datalib (IRRSDL00) error: profile for ring not found (8, 8, 84)] 9.6.1 Tracing When diagnosing SSL security problems, we used a java security trace. We set this using the CTGSTART_OPTS environment variable as shown in Example 9-22. Example 9-22 Java security trace CTGSTART_OPTS=-j-Djava.security.auth.debug=all Chapter 9. Enabling SSL with the Gateway daemon 337 338 CICS Transaction Gateway for z/OS Version 6.1 10 Chapter 10. Security with WebSphere Application Server for z/OS In this chapter, we outline the main CICS and RACF security functions that can be implemented when the CICS TG is used with WebSphere Application Server for z/OS. We start with a look at how the security mechanisms that we introduced in Chapter 8, “Gateway daemon security” on page 281 can be used to secure connections from WebSphere Application Server for z/OS. We document how we prepared the system and the settings that we used. We also explain how we tested the environment using our sample J2EE application CTGTesterCCI. We provide details of some of the common security related problems that you might see when using the CICS TG with WebSphere Application Server for z/OS. © Copyright IBM Corp. 2006. All rights reserved. 339 10.1 Preparation In our testing, our sample J2EE application is deployed on WebSphere Application Server for z/OS and connects to CICS using the CICS ECI resource adapter. In this chapter, we concentrate on how to configure security within and between each tier when a local connection is made between the WebSphere application server and the CICS region (Figure 10-1). User ID Password or Client Certificate HTTP(S) z/OS WebSphere Application Server, Version 6 JSP CICS TS C O M M A R E A Servlet CICS TG V6.1 CTGTesterCCI CCI EXCI CICS ECI resource adapter COBOL application Authorization checks Authentication and authorization checks RACF Figure 10-1 Software components: WebSphere security on z/OS For general information about configuring local connections from WebSphere Application Server on z/OS, see Chapter 4, “Connecting from WebSphere Application Server for z/OS” on page 121. It is also possible to connect from WebSphere Application Server on z/OS in remote mode to a Gateway daemon. In this case the security implications are similar to those discussed in Chapter 8, “Gateway daemon security” on page 281. 340 CICS Transaction Gateway for z/OS Version 6.1 10.1.1 Software checklist Table 10-1 shows the levels of software that we used for this configuration. Table 10-1 Software checklist Windows 2000 z/OS Internet Explorer V6.0 z/OS V1.6 Windows 2000 Service Pack 4 WebSphere Application Server V6.0.1 CICS Transaction Server V3.1 CICS Transaction Gateway for z/OS V6.1 IBM SDK for z/OS Java 2 Technology Edition,V1.4.2 SR2 10.1.2 Definitions checklist Table 10-2 lists the definitions that we used to configure the scenario. Table 10-2 Definitions checklist: WebSphere Application Server for z/OS Value WebSphere Application Server for z/OS hostname tx1.mop.ibm.com TCP/IP port 11880 Job names Daemon region: CITGSDMN Control region: CITGSS1 Servant region: CITGSS1S APPLID CICS TS CITGS1CI A6POTGS1 NETNAME CITGSS1 CONNECTION GSS1 Private mirror ECIW Chapter 10. Security with WebSphere Application Server for z/OS 341 Table 10-3 lists the user IDs that we used in our configuration. Table 10-3 User ID checklist Value Gateway daemon CICS region user ID WebSphere default user ID CICS TS CITGS1CI CITGSSRU Flowed user ID to which we wish to permit access CITGSD Flowed user ID to which we wish to deny access CITGNW 10.2 Configuring CICS and CICS TG security The basic CICS security checks were introduced in Chapter 8, “Gateway daemon security” on page 281. Here we discuss how these security checks apply to a WebSphere z/OS configuration. We describe the following tasks: Configuring CICS security Configuring MRO bind time security Enabling CICS TG password checking Configuring security for CICS CONNECTION and SESSIONS definitions Configuring link security Permitting access to the mirror transaction Configuring surrogate security Configuring CICS security Our CICS region, CITGS1CI (with APPLID A6POTGS1), was configured with security prefixing and transaction security as described in “Configuring CICS security” on page 286. Configuring MRO bind security MRO bind security prevents unauthorized attached MRO regions from starting transactions in a CICS region, and as such applies to a WebSphere servant region. It is implemented using two different DFHAPPL FACILITY class profiles. 342 CICS Transaction Gateway for z/OS Version 6.1 To connect to a CICS region, the WebSphere servant user ID requires the following permissions: Update access to the RACF FACILITY class profile DFHAPPL.<DFHJVPIPE>, where <DFHJVPIPE> is the EXCI pipe name as defined in the CICS TG environment variable DFHJVPIPE and in the NETNAME parameter in the CICS CONNECTION definition. If a generic EXCI connection is used, there is no pipe name and this does not apply. Read access to the RACF FACILITY class profile DFHAPPL.<APPLID>, where <APPLID> is the APPLID of the CICS region in question. Note: Use of MRO bind-time security is optional and if these DFHAPPL profiles are not defined, then any MRO connected system will be able to connect to your CICS region. We enabled MRO bind security with the following commands: 1. We gave update access to the RACF FACILITY class profile DFHAPPL.<DFHJVPIPE> to our WebSphere servant region user ID CITGSSRU, using the following RACF commands: RDEFINE FACILITY (DFHAPPL.CITGSS1) UACC(NONE) PERMIT DFHAPPL.CITGSS1 CLASS(FACILITY) ID(CITGSSRU) ACCESS(UPDATE) SETROPTS RACLIST(FACILITY) REFRESH In our configuration, CITGSSRU is the user ID of the WebSphere servant region, and CITGSS1 is the name that we used for the DFHJVPIPE environment variable. 2. We gave read access to the RACF FACILITY class profile DFHAPPL.<APPLID> to our WebSphere servant region user ID CITGSSRU, using the following RACF command: RDEFINE FACILITY (DFHAPPL.A6POTGS1) UACC(NONE) PERMIT DFHAPPL.A6POTGS1 CLASS(FACILITY) ID(CITGSSRU) ACCESS(READ) SETROPTS RACLIST(FACILITY) REFRESH In our configuration, A6POTGS1 is the APPLID of the CICS region. Enabling CICS TG password checking It is possible to enable the CICS TG password checking by setting the environment variable AUTH_USERID_PASSWORD as a WebSphere variable with the WebSphere Administration Console. However, for a local mode connection it is likely that the user ID has already been authenticated by the WebSphere Application Server and CICS TG password checking is not normally required. Chapter 10. Security with WebSphere Application Server for z/OS 343 With WebSphere Application Server installed on z/OS, it is possible to configure a traditional z/OS authentication model. Authentication takes place at the entry point to z/OS (which for a J2EE application will be normally in an HTTP Server or in the WebSphere Application Server ) and after authentication the security identity (normally a RACF user ID) flows with the work request as it runs in different parts of the system. We did not set the AUTH_USERID_PASSWORD environment variable. CONNECTION and SESSIONS definitions In order for our CICS region to use the user ID flowed in the EXCI call from the WebSphere servant region, we set the parameter ATTACHSEC=IDENTIFY in the EXCI connection definition in our CICS region, as shown in Figure 10-2. OVERTYPE TO MODIFY CEDA ALter CONnection( GSS1 ) CONnection : GSS1 Group : GSS1 DEscription ==> CONNECTION IDENTIFIERS Netname ==> CITGSS1 INDsys ==> CONNECTION PROPERTIES ACcessmethod ==> IRc PRotocol ==> Exci Conntype ==> Specific SECURITY SEcurityname ==> ATtachsec ==> Identify BINDPassword : BINDSecurity ==> No Usedfltuser ==> No CICS RELEASE = 0640 Vtam | IRc | INdirect | Xm Appc | Lu61 | Exci Generic | Specific Local | Identify | Verify | Persistent | Mixidpe PASSWORD NOT SPECIFIED No | Yes No | Yes SYSID=TGS1 APPLID=A6POTGS1 Figure 10-2 CICS CONNECTION definition Link security This is an additional level of authorization checking that can apply to the attached transaction. A specific user ID (the link user ID) can be thought of as the user ID associated with the server which is passing the request to CICS. In our case, this is the user ID of the WebSphere servant region. If link security is enabled, the link 344 CICS Transaction Gateway for z/OS Version 6.1 user ID, like the flowed user ID, must be authorized to access all transactions and resources invoked as a result of the request. The link user ID can be preset on the EXCI SESSIONS definition using the USERID parameter. For EXCI requests, link security works as follows: 1. If the link user ID is the same as the CICS region user ID, then the systems are deemed equivalent and no link security authorization is performed. 2. If the link user ID is preset as something other than the CICS region user ID, then this user ID must be authorized to access all transactions and resources invoked as a result of the request. 3. If the link user ID is not preset, then the user ID of the connected region (that is, the user ID of the WebSphere servant region) is the link user ID and this user ID must be authorized to access all transactions and resources invoked as a result of the request. We used preset link security. This has the advantage that it avoids the overhead of EXCI sign-on for each request that is passed to CICS. To enable preset link security, we set the USERID parameter in the SESSIONS definition to be CITGSSRU, which is the WebSphere servant region user ID. Our SESSIONS definition is shown in Figure 10-3. OVERTYPE TO MODIFY CEDA ALter Sessions( GSS1) Sessions : GSS1 Group : GSS1 DEscription ==> SESSION IDENTIFIERS Connection ==> GSS1 SESSName ==> NETnameq ==> MOdename ==> SESSION PROPERTIES Protocol ==> Exci MAximum ==> 000 , 000 RECEIVEPfx ==> 1 RECEIVECount ==> 260 PRESET SECURITY USERId ==> CITGSSRU CICS RELEASE = 0640 Appc | Lu61 | Exci 0-999 1-999 SYSID=TGS1 APPLID=A6POTGS1 Figure 10-3 CICS SESSIONS definition Chapter 10. Security with WebSphere Application Server for z/OS 345 Table 8-4 on page 291 lists the different settings for link security and ATTACHSEC and how they interoperate. Configuring the flowed user ID The flowed user ID is the identity that is passed on the ECI request. It often represents the identity of the end user of the application. When using the base CICS TG classes, the user ID and password can be specified as parameters when constructing an ECIRequest object. When using the JCA with WebSphere Application Server for z/OS, it is possible for an authenticated user ID to be automatically passed to CICS (see 10.3, “Configuring JCA security” on page 347). In our tests we flow the user ID CITGSD. Permitting access to the mirror transactions An EXCI request received by CICS from the WebSphere Application Server for z/OS attaches a mirror transaction. Therefore, we needed to authorize our flowed user ID CITGSD and our link user ID CITGSSRU to the private mirror transaction ECIW that we are using. We issued the following RACF commands: RDEFINE TCICSTRN CITGS1CI.ECIW UACC(NONE) PERMIT CITGS1CI.ECIW CL(TCICSTRN) ID(CITGSSRU) ACCESS(READ) PERMIT CITGS1CI.ECIW CL(TCICSTRN) ID(CITGSD) ACCESS(READ) SETROPTS RACLIST(TCICSTRN) REFRESH Tip: For our examples, we permit access to single users (CITGSD and CITGSSRU). In a production environment, you will probably create a group of users requiring common access. After a group is built, you will permit access to a group. This permits user’s access to be controlled by the group to which they belong, rather than by individual permissions. This simplifies the security definitions required. Configuring surrogate security In order for the EXCI to be able to switch the security context of the EXCI request to the flowed user ID, the correct SURROGAT security profiles must be defined. The user ID of the EXCI client (in our case the WebSphere servant region) requires read access to the <USERID>.DFHEXCI SURROGAT class profile, where <USERID> is the flowed user ID in the EXCI request. We issued the following commands to permit our WebSphere servant region user ID CITGSSRU to switch to the user ID CITGSD which is flowed in the ECI request: RDEFINE SURROGAT CITGSD.DFHEXCI UACC(NONE) OWNER(CITGSD) PERMIT CITGSD.DFHEXCI CLASS(SURROGAT) ID(CITGSSRU) ACCESS(READ) 346 CICS Transaction Gateway for z/OS Version 6.1 It is possible to disable surrogate security by either reassembling the EXCI options table DFHXCOPT, with SURROGAT=NO, or by using a RACF surrogate profile with universal READ access such as: RDEFINE SURROGAT *.DFHEXCI UACC(READ) OWNER(CITGSD) 10.3 Configuring JCA security The JCA has specific support for enabling secure access from a J2EE application to an Enterprise Information System (EIS) such as CICS. Both container-managed sign-on and component-managed sign-on are supported, as described in 8.4, “Configuring JCA security” on page 293. When using container-managed sign-on, the application deployer is able to set up the authentication information (user ID and password) when he deploys the application. In WebSphere Application Server V6, this can be done using a J2C authentication data entry (see 8.4.1, “J2C authentication data” on page 294). When using WebSphere Application Server for z/OS, the container can also derive the identity from the currently executing Java thread. This is known as thread identity support. 10.3.1 Thread identity support Thread identity support allows WebSphere Application Server for z/OS to automatically pass the user ID of the Java thread to CICS when using an ECI resource adapter. The setting of the thread user ID is dependent on the RunAs policy for the J2EE application. If RunAs is set to Caller, and the user of the Web application has authenticated with the WebSphere Application Server, then thread identity support enables the caller’s identity to be propagated to CICS automatically. Thread identity support enables WebSphere Application Server to behave in a way that traditional z/OS address spaces behave. Once the user ID has been authenticated, that user ID flows with any work done within the z/OS system. This is similar to the way that MRO requests in CICS are managed, for example. Thread identity support is enabled when: WebSphere Global security is enabled and the active user registry is defined as Local OS (normally this means RACF). A local connection is being used between WebSphere Application Server and CICS. Container-managed security is being used (the res-auth deployment descriptor is set to Container). Chapter 10. Security with WebSphere Application Server for z/OS 347 The resource reference is not mapped to a J2C authentication data entry and the connection factory does not specify a JAAS Authentication Alias. Note: JAAS authentication aliases are deprecated in WebSphere Application Server V6 and are supported for compatibility reasons. In order to enable thread identity support, we needed to consider the following configuration and setup tasks: Enabling WebSphere Global Security Creating a connection factory Updating the deployment descriptors for the sample application CTGTesterCCI Defining a security role Deploying the sample application Enabling WebSphere Global Security To enable Global Security for our WebSphere Application Server, we first started the server using the command: S CITGSCR,JOBNAME=CITGSS1,ENV=CITGSC1.CITGSN1.CITGSS1 Note: Before making changes to the WebSphere Application Server security settings, we took a copy of the security.xml file in /usr/lpp/zWebSphere/V6R0/config/cells/BaseApplicationServerCell. This allows us to back out the changes if we are unable to connect to the WebSphere Application Server Administration Console. We entered the following URL in the Web browser to access the WebSphere Admin console and logged on with our user ID CITGSADM: http://tx1.mop.ibm.com:11880/admin To configure WebSphere Application Server Global Security, we did the following: 1. Clicked Security → Global Security → Authentication Mechanisms → LTPA. 2. Filled in our password and confirmed it by entering it again. We then clicked Apply and saved our configuration change. 3. Clicked Security → Global Security. Selected Enable global security and deselected Enforce Java 2 Security. Set Active Protocol as CSI and SAS and Active Authentication Mechanism as LTPA. 348 CICS Transaction Gateway for z/OS Version 6.1 4. Set Active User Registry as Local OS. Figure 10-4 shows the Global Security definitions. Figure 10-4 Administrative Console: define Global security 5. We clicked Apply, Save, and then restarted the J2EE server. 6. We connected to the Administrative Console again and noted that the server redirected us to the console URL accessed via the SSL port: https://tx1.mop.ibm.com:11843/ibm/console/logon.jsp 7. We received a certificate warning and clicked Yes to proceed. We were then required to specify a password for the WebSphere administration user ID CITGSADM. Chapter 10. Security with WebSphere Application Server for z/OS 349 Creating a connection factory We defined a connection factory called CITGS1CI-tx1-local using the same procedure as described in 4.2.4, “Creating a connection factory” on page 128. Figure 10-5 shows the custom properties for the connection factory. Figure 10-5 Administrative Console - connection factory custom properties Updating the application deployment descriptors We used Rational Application Developer to update the following deployment descriptors for the CTGTesterCCI sample application: Defining a security constraint Setting the Res-auth for the resource reference to Container Setting RunAs for the EJB methods to Caller Defining a security constraint We added a security constraint to the CTGTesterCCI application. If Global Security is enabled, and a security constraint is set for a particular resource, the resource is secured and users of the application must authenticate themselves. 350 CICS Transaction Gateway for z/OS Version 6.1 Within the application EAR file, the Web application archive CTGTesterCCIWeb.jar contains file WEB-INF/web.xml. We added the security constraint as shown in Example 10-1. Example 10-1 Web application deployment descriptor security constraint <security-constraint> <web-resource-collection> <web-resource-name>CTGTesterCCI Web resource </web-resource-name> <description></description> <url-pattern>/*</url-pattern> <url-pattern>/index.jsp</url-pattern> </web-resource-collection> <auth-constraint> <description>Role required to run CTGTesterCCI</description> <role-name>CTGTEST</role-name> </auth-constraint> <user-data-constraint> <transport-guarantee>NONE</transport-guarantee> </user-data-constraint> </security-constraint> <login-config> <auth-method>BASIC</auth-method> <realm-name>WASWebContainer</realm-name> </login-config> <security-role> <description>Security role required in order to run CTGTesterCCI application</description> <role-name>CTGTEST</role-name> </security-role> This security constraint specifies that the Web application user must: Login to the J2EE server using basic authentication (<auth-method>BASIC</auth-method>) Be authorized to role CTGTEST (<role-name>CTGTEST</role-name>) Chapter 10. Security with WebSphere Application Server for z/OS 351 Setting the Res-auth Within the application EAR file, the EJB archive CTGTesterCCIEJB.jar contains file META-INF/ejb-jar.xml. This deployment descriptor file contains a resource-ref for the ECI reference. We set the res-auth for this resource to Container (Example 10-2). Example 10-2 EJB deployment descriptor res-auth setting <resource-ref id="ResourceRef_1"> <description>CICS ECI Resource adapter</description> <res-ref-name>ECI</res-ref-name> <res-type>javax.resource.cci.ConnectionFactory</res-type> <res-auth>Container</res-auth> </resource-ref> Setting RunAs A RunAs setting on a servlet or EJB affects the Java thread identity of methods invoked on a subsequent call. WebSphere Application Server RunAs policy allows three choices in assigning the Java thread identity for the current request: Caller uses the caller’s identity for the method selected and to propagate it to any subsequent methods invoked or J2EE resources accessed. This is the default behavior. Server indicates that the method should assume the identity of the WebSphere servant region. Role the application assembler selects the name of a security role (we look at roles in a few moments) that is defined in the application. Authorization is performed by checking that the principal name has been assigned to one of the required security roles. In our tests, we used the default RunAs setting (Caller) for all methods of the sample J2EE application CTGTesterCCI. Defining a security role Security roles are used to control access to J2EE resources such as servlets and EJBs. The name of the roles are defined in the application deployment descriptor, for example, in Example 10-1 on page 351 we defined a role-name CTGTEST. There are two basic ways to control role access with WebSphere Application Server for z/OS: Using SAF EJBROLE profiles Using WebSphere bindings 352 CICS Transaction Gateway for z/OS Version 6.1 We defined the EJBROLE CITGS.CTGTEST and authorized the user ID CITGSD to the EJBROLE. Note: CITGS is the security domain prefix for our WebSphere Application Server. The specification of a security domain prefix affects the specific EJBROLE profiles used by WebSphere Application Server for z/OS. This enables you to deploy the same application on different cells, but have different user to role mappings if desired. We used RACF to create the EJBROLE class and authorize the CITGSD user ID to the role with the following commands: RDEFINE EJBROLE CITGS.CTGTEST UACC(NONE) PERMIT CITGS.CTGTEST CLASS(EJBROLE) ID(CITGSD) ACCESS(READ) SETROPTS RACLIST(EJBROLE) REFRESH Deploying the sample application After having used RAD to modify the deployment descriptors of the application, we exported the application and created a new EAR called CTGTesterCCI_Sec.ear. We then followed the same process as described in 4.2.6, “Deploying our sample J2EE application” on page 134 to deploy the application to the WebSphere Application Server. During the deployment process, we mapped the ECI resource reference to the JNDI name of the CITGS1CI-tx1-local connection factory. 10.4 Testing the configuration We wanted to test that thread identity support could be used to automatically flow an authenticated user ID from WebSphere Application Server to CICS. 10.4.1 Testing thread identity support We closed all the Web browser sessions to ensure that no basic authentication headers were cached, and we invoked the test application CTGTesterCCI using the following URL: http://tx1.mop.ibm.com:11880/CTGTesterCCIWeb/index.jsp Chapter 10. Security with WebSphere Application Server for z/OS 353 The Web browser basic authentication pop-up window was displayed (Figure 10-6). We specified user ID CITGSD and a valid password. We noted that if we specified an invalid password the basic authentication login was re-displayed. Figure 10-6 CTGTesterCCI basic authentication login We clicked OK, and the CTGTesterCCI JSP was displayed. We did not specify a user ID and password on the CTGTesterCCI input screen (Figure 10-7). Figure 10-7 CTGTesterCCI input parameters - no user ID and password specified 354 CICS Transaction Gateway for z/OS Version 6.1 The CTGTesterCCI application ran successfully. ECIPROG returns a COMMAREA which contains the user ID under which the CICS task runs. The output from ECIPROG shows that the request ran successfully in our secure CICS region CITGS1CI, and that the CICS task ran under the user ID CITGSD (Figure 10-8). CITGSD is the user ID that we used to login to the WebSphere Application Server. It is automatically passed to CICS by the application server as a result of the unique thread identity support that is provided by WebSphere Application Server for z/OS. Figure 10-8 CTGTesterCCI results with thread identity support enabled In order to test that an unauthorized user is not able to access the CTGTesterCCI application, we repeated the test but this time we logged in with an unauthorized user ID. We closed all the Web browsers on the machine and ran the CTGTesterCCI application again, specifying user ID CITGNW and a valid password when prompted. Figure 10-9 shows that the user is not authorized to run the application. Chapter 10. Security with WebSphere Application Server for z/OS 355 Figure 10-9 Web browser CTGTesterCCI authorization failure We also noted the following RACF EJBROLE authorization message in the syslog (Example 10-3). Example 10-3 SYSLOG - EJBROLE access not allowed ICH408I USER(CITGNW ) GROUP(CITG ) CITGS.CTGTEST CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) In this case, the user CITGNW has authenticated successfully with the WebSphere Application Server but the EJBROLE check has failed because CITGNW does not have READ access to the EJBROLE CITGS.CTGTEST. 356 CICS Transaction Gateway for z/OS Version 6.1 10.5 Problem determination In this section, we document common security problems and information we learned while configuring our WebSphere z/OS security scenarios. For additional tips and utilities for problem determination of WebSphere z/OS connection related problems, see 2.9.2, “Tips and utilities” on page 62. Surrogate security errors If the WebSphere servant region does not have the required access to flow a specific user ID (CITGNW) to CICS, you will receive the RACF error shown in Example 10-4. Example 10-4 Surrogate check failure for WebSphere servant region user ID ICH408I USER(CITGSSRU) GROUP(CITGSCFG) NAME(WAS APPSVR SR CITGSD.DFHEXCI CL(SURROGAT) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) ) The request has failed because the WebSphere servant region user ID CITGSSRU does not have access to the SURROGAT class profile CITGSD.DFHEXCI. Bind security errors If the WebSphere servant region does not have the required access to bind to the CICS connection, you will receive the RACF error shown in Figure 10-5. Example 10-5 RACF message of MRO bind security failure - DFHAPPL.DFHJVPIPE ICH408I USER(CITGSSRU) GROUP(CITGSCFG) NAME(WAS APPSVR SR DFHAPPL.CITGSS1 CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(UPDATE ) ACCESS ALLOWED(NONE ) ) If the WebSphere servant region does not have the required access to bind to the CICS region, you will receive the RACF error shown in Figure 10-6. Example 10-6 RACF message of MRO bind security failure - DFHAPPL.APPLID ICH408I USER(CITGSSRU) GROUP(CITGSCFG) NAME(WAS APPSVR SR DFHAPPL.A6POTGS1 CL(FACILITY) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE ) ) Chapter 10. Security with WebSphere Application Server for z/OS 357 358 CICS Transaction Gateway for z/OS Version 6.1 Part 5 Part 5 Appendixes In this part, we describe the sample applications that are provided with this book. We also provide further details on data conversion and EXCI tracing. © Copyright IBM Corp. 2006. All rights reserved. 359 360 CICS Transaction Gateway for z/OS Version 6.1 A Appendix A. Sample applications In this appendix, we give details of our two sample applications: CTGTesterCCI CTGTesterCCI uses the J2EE CCI classes and the CICS ECI resource adapter to flow an ECI request to CICS. It uses a JSP/servlet/EJB design and the EJB session bean has a single JCA resource reference. CTGTesterCCIXA The CTGTesterCCIXA application is based on CTGTesterCCI but it is intended to be used with the CICS ECI XA resource adapter. It contains two JCA resource references and can be used to flow two ECI requests to different CICS regions, such that the two calls can be committed within the same global transaction. Both of these applications are designed for the testing of connectivity from the application server to CICS, either with a local connection or through the Gateway daemon. As such, all the CICS dependencies, such as COMMAREA size, input, encoding and length, are externalized to enable you to test different configurations. The applications are not designed for production use, because such externals are unlikely to be exposed in real-life customer applications. © Copyright IBM Corp. 2006. All rights reserved. 361 The CTGTesterCCI application The CTGTesterCCI application contains a Java servlet and a session bean that we used in various test scenarios. It is intended to provide a simple enterprise application that can be used out of the box to test ECI connectivity from any CICS Transaction Gateway to any configured CICS region using the CICS ECI resource adapter. It can be used to test a local connection from WebSphere Application Server to CICS or a remote connection via a Gateway daemon. This sequence of events is illustrated in Figure A-1. WebSphere Application Server servlet HTML JSP CTGTesterCCIServlet Web browser CICS TS session bean CICS ECI CTGTesterCCICCI resource adapter CCI Gateway daemon CICS program Figure A-1 Architecture of CTGTesterCCI All input parameters can be specified as HTTP query strings or HTML form parameters, and all output is through text-based HTML. It has the following features: Setting of the CICS program name Setting of the COMMAREA input data, length and codepage Optional setting of user ID, password and mirror transaction name Iteration count for repeated execution of the CICS program Optional setting of application CICS TG Java client trace CTGTesterCCI contains a single JCA resource reference. When deploying the application (see 3.2.3, “Deploying the sample J2EE application” on page 89), the resource reference must be mapped to a connection factory. The connection factory contains the CICS APPLID and the connection details of the Gateway daemon that will be used. In addition, the connection factory can contain other optional information, such as security credentials that is used for the connection. 362 CICS Transaction Gateway for z/OS Version 6.1 Application overview The following is the sequence of events that occur when the end user interacts with the CTGTesterCCI application through a Web browser, illustrated in Figure 3-16 on page 109: 1. The user opens the Web application using the following URL and enters the input parameters for the JCA request: http://localhost:9080/CTGTesterCCIWeb/index.jsp 2. The user then clicks a button on a Web page that submits a form to the servlet. 3. The servlet receives the request for action and calls a method on the remote interface of the CTGTesterCCI session bean. 4. The session bean uses the CICS ECI resource adapter to call the CICS program. 5. The CICS ECI resource adapter returns data from the CICS program to the session bean. 6. The session bean returns the output data from the CICS program back to the servlet. 7. The servlet forwards to a JSP, which displays the response to the user. This sequence of events is illustrated in Figure A-2. CTGTesterCCI servlet 1 welcome JSP 2 CTGTesterCCIServlet 3 session bean CTGTesterCCI 6 7 Web browser success results JSP exception exception JSP 5 4 CICS ECI resource adapter Figure A-2 Flow of CTGTesterCCI Appendix A. Sample applications 363 Static HTML Form The HTML form index.jsp is stored in the WebContent folder of the CTGTesterCCIWeb package and can be viewed using Rational Application Developer. The HTML form consists of a number of fields where data for the application is entered. Table A-1 shows the fields on the form. Table A-1 Fields in the HTML form, CTGTesterCCI Field Name Purpose CICS program name funcName Program on CICS to call COMMAREA input commareaInput COMMAREA data COMMAREA length commareaLength Length of the COMMAREA Encoding encoding Encoding to convert data to before sending and after receiving User ID username Username to flow on the ECI request Password password Password to flow on the ECI request Mirror transaction mirror Transaction to use on the CICS server Iterations® iterations The number of times to run the CICS program Application trace appTrace Whether to enable CICS TG Java client trace Mandatory details Optional details The action of the HTML form is to call the servlet CTGTesterCCIServlet, using the POST method to send the parameters. Then the parameters are sent in the HTTP request body. Because an HTTP POST request is intended to affect some sort of change on the Web server, a Web browser prompts to resubmit the data if the user clicks Refresh on the results page. Conversely, the GET method could be used, which sends the parameters on the HTTP request URL. Because HTTP GETs are not intended to alter anything on the Web server, the results page of such a request can be refreshed without a prompt being shown. 364 CICS Transaction Gateway for z/OS Version 6.1 CTGTesterCCIServlet The Servlet converts the HTML FORM user data into a format which the Session EJB can use. The Session EJB facade masks how or where the request is actually processed, providing separation between presentation and business logic. Figure A-3 summarizes the CTGTesterCCIServlet code. CTGTesterCCIServlet HTML FORMS DATA processRequest() Map2Details() eciReq1 Lookup ejbHome JNDI Namespace ejbHome.create() ejb.execute(eciReq1) ejb.execute() results.jsp/exception.jsp Figure A-3 CTGTesterCCIServlet logic overview The following gives a brief description of the main methods of CTGTesterCCIServlet. processRequest() - 1 The CTGTesterCCIServlet doGet() and doPost() methods both pass on the HTML form data for the request (whichever convention was used) to the processRequest() method. The processRequest() method then creates an ECIRequestDetails object called eciReq1 (an encapsulation of the user input data) and calls MapRequest2Details() to populate it. MapRequest2Details() The MapRequest2Details() method takes the HttpServerletRequest object (effectively the input) and an ECIRequestDetails object (effectively the output) together with an optional suffix string. The field names, shown in the central column of Figure A-1 are combined with the suffix input string to identify name/value pairs contained in the HttpServerletRequest data. The value for each field is copied into the corresponding EciRequestDetails class variable, performing any type conversions from the user input that are required. Appendix A. Sample applications 365 Note: CTGTesterCCIServlet only uses a single ECI request, and so the suffix field is set to an empty string. The usage of the suffix string is clearer in the context of the CTGTesterCCIXAServlet code (see “CTGTesterCCIXAServlet” on page 374). processRequest() - 2 Once the request data as been setup in eciReq1, processRequest() obtains a reference to CTGTesterCCIHome via a lookup on the initial context for the EJB reference ejb/CTGTesterCCI. The CTGTesterCCIHome.create() method is used to create an instance of the CTGTesterCCI EJB remote interface. The local CTGTesterCCI instance is called tester. The ejb reference is defined in the Web Deployment Descriptor (Figure A-4). <ejb-ref id="EjbRef_1"> <description></description> <ejb-ref-name>ejb/CTGTesterCCI</ejb-ref-name> <ejb-ref-type>Session</ejb-ref-type> <home>itso.cics.eci.j2ee.testercci.CTGTesterCCIHome</home> <remote>itso.cics.eci.j2ee.testercci.CTGTesterCCI</remote> </ejb-ref> Figure A-4 CTGTesterCCI EJB resource reference definition Having gathered together all of the details for the ECI request, and created an instance of the Session EJB CTGTesterCCI remote interface, processRequest() finally invokes the execute() method on the CTGTesterCCI bean instance, passing in the EciRequestDetails object, eciReq1. 366 CICS Transaction Gateway for z/OS Version 6.1 Session EJB CTGTesterCCI Figure A-5 shows the interactions between the calling servlet and session EJB CTGTesterCCI. CTGTesterCCIServlet CTGTesterCCIHome.create() CTGTesterCCI.execute() Invoke appropriate JSP CTGTesterCCI ejbCreate() lookup ("java:comp/env/ECI") create connection factory execute(eciReq1) flowRequest(eciReq1) get CCI connection get CCI interaction copy eciReq1 to EciInteractionSpec loop # iterations interaction.execute() return COMMAREA or throw exception Figure A-5 CTGTesterCCI EJB logic overview The following gives a brief description of the main methods of CTGTesterCCI. ejbCreate() The EJB life cycle method ejbCreate() is used to create a conection factory. We create an InitialContext object and use it to look up a connection factory using the JNDI name java:comp/env/ECI. This is a local JNDI name that will be mapped to the actual JNDI name of the connection factory at deploy time. To indicate that we want such a mapping, we create a resource reference in the EJB deployment descriptor with the name ECI. execute() The only method available on the remote interface of the CTGTesterCCI EJB is the execute() method. This method and supporting logic is implemented by class CTGTesterCCIBean. The EciRequestDetails and JavaStringRecord classes also provide supporting function. Appendix A. Sample applications 367 Note: The transaction attribute is set in the assembly descriptor section of the EJB deployment descriptor for the execute() method to control the transactional behavior of the EJB and associated JCA request (see “Updating the EJB deployment descriptor” on page 176). In the version of CTGTesterCCI provided with the Redbook, the transaction attribute is set to NotSupported. flowEciRequest() The flowEciRequest() method creates a connection by calling the getConnection() method. The getConnection()method creates an EciConnectionSpec and sets security details if specified on the JSP input page. Method flowEciRequest() then creates an interaction by calling getInteraction() and sets up the ECIInteractionSpec object (eSpec) based on the other input parameters entered on the JSP input page (CICS program, COMMAREA length and mirror transaction name). Finally, it enters a loop, based on the iteration limit entered on the JSP input page, issuing the execute() method on the interaction (eciInt). The execute() method takes the JavaRecordString object jsr as a parameter twice, once for COMMAREA input and once for COMMAREA output. The iteration loop continues for as long as each request completes normally and until the iteration limit is reached, each time overwriting the output COMMAREA of the previous iteration. Ultimately, the invocation of flowEciRequest() either returns a JavaStringRecord, representing the COMMAREA returned by the last iteration (if successful) or re-throws a ResourceException caught on the EciInteraction.execute() call. Catching and handling a CICS transaction abend The execute() method might throw a ResourceException, which has several sub-classes. For example, in the event of a CICS transaction abend a CICSTxnAbendException is thrown by the CICS ECI resource adapter. If this exception occurs, we issue a setRollbackOnly() on the EJB session context in the catch block for the execute() call. This ensures that the CICS abend will be propagated onto WebSphere Application Server as the transaction manager and the local transaction (if one exists) will be rolled back. Example A-1 shows this logic. Example: A-1 Catching and handling a transaction abend public class CTGTesterCCIBean implements SessionBean { ... private JavaStringRecord flowEciRequest(ECIInteractionSpec eSpec, 368 CICS Transaction Gateway for z/OS Version 6.1 EciRequestDetails eciRD, String cfRef) throws Exception { ... // make the call try { eciInt.execute(eSpec, jsr, jsr); } catch (ResourceException re) { // output the stacktrace and re-throw it back to the client re.printStackTrace(); // Depending upon the kind of exception thrown, we may decide // to abandon the current transaction by calling setRollbckOnly() // on the current context. If this request is not part of a local // or global transaction, there will be a second exception to // catch (IllegalStateException). If so, we will log the exception // details, but otherwise absorb it. // In the case of a CICSTxnAbendException, we will write a log // message and rollback any current transaction. if (re instanceof com.ibm.connector2.cics.CICSTxnAbendException ) { System.err.println("CTGTesterCCIBean.flowEciRequest: CICS ABEND detected."+ " Connection Factory="+targetCF.toString()); try { mySessionCtx.setRollbackOnly(); } catch(IllegalStateException ise) { ise.printStackTrace(); } } else { // Write generic failure log message System.err.println("CTGTesterCCIBean.flowEciRequest: ResourceException detected."+ " Connection Factory="+targetCF.toString()); } // clean up interaction and connection references dropConnection(eciConn, eciInt); eciInt = null; eciConn = null; // throw exception back to caller ResourceException re2 = new ResourceException(re.getMessage()+ " ConnectionFactory="+targetCF.toString()); re2.initCause(re); throw re2; } } Note: The setRollbackOnly command shown in Example A-1 is not necessary if the option ABENDBKOUT=YES is set in the DFHXCOPT table (see “ABENDBKOUT option in DFHXCOPT” on page 165). Appendix A. Sample applications 369 JSPs The application uses two JSPs to format and display the results. One of two scenarios can occur; the request completes successfully, or an exception occurs inside the CICS ECI resource adapter or the application. The JSPs are stored in the CTGTesterCCIWeb WebContent folder and can be viewed using Rational Application Developer. results.jsp The results.jsp processes successful completion of a request. The JSP defines the title of the HTML page and the style sheet used to format the HTML in the header. The JSP defines the JavaBeans™ components that the page uses to format and display the results (Figure A-6). <jsp:useBean <jsp:useBean <jsp:useBean <jsp:useBean <jsp:useBean <jsp:useBean class="java.lang.String" class="java.lang.String" class="java.lang.String" class="java.lang.String" class="java.lang.String" class="java.lang.String" id="results" scope="request"/> id="commareaInput" scope="request"/> id="encoding" scope="request"/> id="defaultResults" scope="request"/> id="hexResults" scope="request"/> id="requestTime" scope="request"/> Figure A-6 JavaBeans components used by results.jsp These components are all in the request scope and are all String objects. Their IDs correspond to the names of the attributes we set in the HttpRequest in our servlet. They contain the result of the CICS request made by the servlet. The page then includes our common JSP header, which displays the information specified in the initial form, such as trace level and number of iterations. The rest of the page then outputs the values of the request attributes using the scriptlet format <%= JavaBean id %> (Figure A-7). In our application, the JavaBean component encoding contains the encoding used to convert the input COMMAREA into bytes, and is inserted into the HTML output with the JSP scriplet <%= encoding %>. 370 CICS Transaction Gateway for z/OS Version 6.1 <TABLE BGCOLOR=white CELLPADDING=0 width="624"> <TBODY> <TR ALIGN=LEFT> <TH width="121">Results</TH> <TH width="497">COMMAREA</TH> </TR> <TR> <TD width="121">Input:</TD> <TD BGCOLOR=yellow width="497"><%= commareaInput %></TD> </TR> <TR> <TD width="121">Output using <%= encoding %>:</TD> <TD BGCOLOR=yellow width="497"><%= results %></TD> </TR> <TR> <TD width="121">Output in HEX:</TD> <TD BGCOLOR=yellow width="497"><%= hexResults %></TD> </TR> <TR> <TD width="121">Request time (ms):</TD> <TD BGCOLOR=yellow width="497"><%= requestTime %></TD> </TR> </TBODY> </TABLE> <TABLE BGCOLOR=white><TBODY> <TR><TH>Request succeeded.</TH></TR> </TBODY></TABLE> Figure A-7 Outputting the results into the HTML The results page outputs the COMMAREA data converted to a String using the JVM default encoding, the encoding specified in the form and in a hexadecimal representation. error.jsp The JSP error.jsp is used when an error occurs. It does not display the output COMMAREA. Instead, it displays the contents of the exception that occurred and any errors found in the processing of the form parameters by the servlet. Appendix A. Sample applications 371 The CTGTesterCCIXA application The CTGTesterCCIXA application is based on CTGTesterCCI and provides a simple enterprise application that can be used out of the box to test the XA capabilities of the CICS TG V6.1 ECI XA resource adapter. It can be used to test connections to two different CICS regions (Figure A-8). WebSphere Application Server V6 servlet HTML JSP Web browser CICS TG V6.1 Gateway daemon CICS TS CICS program CTGTesterCCIXAServlet CICS TG V6.1 CICS TS session bean CICS ECI XA CCI CTGTesterCCIXA resource CCI adapter Gateway daemon CICS program Figure A-8 Architecture of CTGTesterCCIXA The pair of JCA requests can form two legs of an XA transaction. Application overview The CTGTesterCCIXA application is an extension of CTGTesterCCI. The main differences to CTGTesterCCI are that the CICS ECI XA resource adapter must be used and two JCA calls are made rather than the single call made by CTGTesterCCI. CTGTesterCCIXA contains two JCA resource references. When deploying the application (see “Deploying our sample application” on page 200), the resource references must be mapped to two connection factories. The following is the sequence of events that occur when the end user interacts with the CTGTesterCCIXA application through a Web browser, illustrated in Figure 5-25 on page 203. 1. The user opens the Web application using the URL and enters the input parameters for the two JCA requests: http://localhost:9080/CTGTesterCCIXAWeb/index.jsp 372 CICS Transaction Gateway for z/OS Version 6.1 2. The user then clicks a button on a Web page that submits a form to the servlet. 3. The servlet receives the request for action and calls a method on the remote interface of the CTGTesterCCIXA session bean. 4. The session bean uses the CICS ECI XA resource adapter to call two CICS programs, possibly via two different Gateway daemons. 5. The CICS ECI XA resource adapter returns data from the CICS programs to the session bean. 6. The session bean returns the output data from the CICS programs back to the servlet. 7. The servlet forwards to a JSP, which displays the response to the user. Static HTML Form The HTML form index.jsp is stored in the WebContent folder of the CTGTesterCCIXAWeb package and can be viewed using Rational Application Developer. The HTML form contains the same fields used in CTGTesterCCIWeb (see Table A-1 on page 364), but includes a numeric suffix to distinguish the sets of parameters. Table A-2 shows the fields on the form. Table A-2 Fields in the HTML form, CTGTesterCCI Field Name Purpose CICS program name funcName{1+2} Program on CICS to call COMMAREA input commareaInput{1 +2} COMMAREA data COMMAREA length commareaLength {1+2} Length of the COMMAREA Encoding encoding{1+2} Encoding to convert data to before sending and after receiving User ID username{1+2} Username to flow on the ECI request Password password{1+2} Password to flow on the ECI request Mirror transaction mirror{1+2} Transaction to use on the CICS server Iterations iterations{1+2} The number of times to run the CICS program Application trace appTrace (common field) Whether to enable CICS TG Java client trace Mandatory details Optional details Appendix A. Sample applications 373 The action of the HTML form is to call the servlet CTGTesterCCIXAServlet, using the POST method to send the parameters. CTGTesterCCIXAServlet The Servlet converts the HTML FORM user data into a format which the Session EJB can use. The Session EJB facade masks how or where the request is actually processed, providing separation between presentation and business logic. Figure A-9 summarizes the CTGTesterCCIXAServlet code. CTGTesterCCIXAServlet HTML FORMS DATA processRequest() MapRequest2Details() eciReq1 MapRequest2Details() eciReq2 lookup ejbHome JNDI Namespace ejbHome.create() ejb.execute(eciReq1,eciEeq2) ejb.execute() results.jsp/exception.jsp Figure A-9 CTGTesterCCIXAServlet logic overview The following gives a brief description of the main methods of CTGTesterCCIXAServlet: processRequest() - 1 The CTGTesterCCIXAServlet doGet() and doPost() methods both pass on the HTML form data for the request (whichever convention was used) to the processRequest()method. processRequest() creates a pair of ECIRequestDetails objects eciReq1 and eciReq2, and calls MapRequest2Details() twice to populate them, using the suffix parameter to distinguish the sets of name/value pairs. MapRequest2Details() The MapRequest2Details() method takes a HttpServletRequest object (effectively the input), an ECIRequestDetails object (effectively the output) together with an optional suffix string. The field names, shown in the central column of Table A-2 are combined with the suffix input string to identify name/value pairs contained in the HttpServletRequest data. The value for 374 CICS Transaction Gateway for z/OS Version 6.1 each field is copied into the corresponding EciRequestDetails class variable, performing any type conversions from the user input that are required. MapRequest2Details() is invoked with the suffix 1 to populate eciReq1, and then again with suffix 2 to populate eciReq2. processRequest() - 2 When the request data as been setup in objects eciReq1 and eciReq2, processRequest() obtains a reference to CTGTesterCCIHome via a lookup on the initial context for EJB reference ejb/CTGTesterCCIXA. The CTGTesterCCIXAHome.create() method is used to create an instance of the CTGTesterCCIXA EJB remote interface. The local CTGTesterCCIXA instance is called tester. The ejb reference is defined in the Web Deployment Descriptor. Having gathered together all of the details for the pair of JCA requests, and created an instance of the Session EJB CTGTesterCCIXA remote interface, processRequest() finally invokes the execute() method on the CTGTesterCCIXA bean instance, passing in both EciRequestDetails objects, eciReq1 and eciReq2. Session EJB CTGTesterCCIXA Figure A-10 shows the interactions between the calling servlet and session EJB CTGTesterCCIXA. CTGTesterCCIXAServlet CTGTesterCCIXA EJB CTGTesterCCIHome.create() ejbCreate() lookup ("java:comp/env/ECIAXA") create connection factory1 lookup ("java:comp/env/ECIAXA") create connection factory1 CTGTesterCCI.execute() execute(eciReq1,eciReq2) flowRequest(eciReq1) get CCI connection1 get CCI interaction1 copy eciReq1 to eSpec1 loop # iterations1 interaction1.execute() flowRequest(eciReq2) get CCI connection2 get CCI interaction2 copy eciReq2 to eSpec2 loop # iterations2 interaction2.execute() Invoke appropriate JSP return commareaPair or throw exception Figure A-10 CTGTesterCCIXA EJB logic overview Appendix A. Sample applications 375 We give a brief description of the main methods of CTGTesterCCIXA in the following sections. ejbCreate() The EJB life cycle method ejbCreate() is used to create two conection factories. We create an InitialContext object and use it to look up the two connection factories using the JNDI names java:comp/env/ECIAXA and java:comp/env/ECIAXB. These are the local JNDI names that will be mapped to the actual JNDI name of the connection factories at deploy time. To indicate we want such a mapping, we create two resource references in the EJB deployment descriptor with the names ECIAXA and ECIBXA (Figure A-11). <resource-ref id="ResourceRef_1126776812792"> <description>CICS XA ECI Resource adapter to CICS region A</description> <res-ref-name>ECIAXA</res-ref-name> <res-type>javax.resource.cci.ConnectionFactory</res-type> <res-auth>Container</res-auth> <res-sharing-scope>Shareable</res-sharing-scope> </resource-ref> <resource-ref id="ResourceRef_1126872415486"> <res-ref-name>ECIBXA</res-ref-name> <res-type>javax.resource.cci.ConnectionFactory</res-type> <res-auth>Container</res-auth> <res-sharing-scope>Shareable</res-sharing-scope> </resource-ref> Figure A-11 CTGTesterCCIXA EJB Deployment Descriptor execute() The only method available on the remote interface of the CTGTesterCCIXA EJB is the execute() method. This method and supporting logic is implemented by class CTGTesterCCIXABean. The EciRequestDetails, commareaPair, and JavaStringRecord classes also provide supporting function. Note: The transaction attribute is set in the assembly descriptor section of the EJB deployment descriptor for the execute() method to control the transactional behavior of the EJB and associated JCA requests. In the version of CTGTesterCCIXA provided with this book, the transaction attribute is set to Required. 376 CICS Transaction Gateway for z/OS Version 6.1 The execute() method calls flowEciRequest() twice: The first request processes eciReq1, using ECIInteractionSpec eSpec1 and the resource reference ECIAXA The second request processes eciReq2, using ECIInteractionSpec eSpec2 and the resource reference ECIBXA Note: The first request completes all of its iterations before the second request starts its first iteration. The final difference to the CTGTesterCCI EJB execute() method is the return object. The CTGTesterCCIXA.execute() method returns the two COMMAREAS in a commareaPair object which consists of a pair of String objects. flowEciRequest() The flowEciRequest() method is unchanged from CTGTesterCCI. Refer to “flowEciRequest()” on page 377. Catching and handling a CICS transaction abend The handling of CICS transaction abends is unchanged from CTGTesterCCI. Refer to “Catching and handling a CICS transaction abend” on page 368. JSPs The application uses two JSPs to format and display the results. One of two scenarios can occur; the request completes successfully, or an exception occurs inside the CICS ECI XA resource adapter or the application. The JSPs are stored in the CTGTesterCCIXAWeb WebContent folder and can be viewed using Rational Application Developer. The only difference to the CTGTesterCCIWeb version of results.jsp is the inclusion of an additional COMMAREA for the second JCA request. Appendix A. Sample applications 377 378 CICS Transaction Gateway for z/OS Version 6.1 B Appendix B. DFHCNV and CICS data conversion Because WebSphere Application Server runs on ASCII based platforms and CICS runs on an EBCDIC based platform, the need arises for code page conversion to be performed on data passed between J2EE applications and CICS programs. In this appendix, we explain how the data that is passed to a CICS program on an ECI call can be converted correctly between ASCII and EBCDIC. © Copyright IBM Corp. 2006. All rights reserved. 379 Data conversion We discuss the following data conversion considerations: DFHCNV Code page aware Java programs with DFHCNV Code page aware Java programs without DFHCNV DFHCNV and the mirror program ECI applications use the facilities of the CICS mirror program to link to the specified user program, passing the COMMAREA for input and output. The CICS mirror program (DFHMIRS) invokes the services of the data conversion program (DFHCCNV) to perform the necessary code page conversion of the COMMAREA (Figure B-1). Distributed platform or z/OS z/OS CICS TS V3.1 DFHCCNV CICS TG V6 ECI Mirror program DFHMIRS Figure B-1 ECI data conversion 380 CICS Transaction Gateway for z/OS Version 6.1 C O M M A R E A Target application program If DFHCCNV finds a conversion template in the DFHCNV table whose resource name (RNAME) matches the target program name, it performs code page conversion for the COMMAREA associated with the ECI request. Example B-1 shows the DFHCNV table entry that we used for the program EC01 in our IVP test. Example: B-1 DFHCNV entry DFHCNV TYPE=ENTRY,RTYPE=PC,RNAME=EC01,USREXIT=NO, SRVERCP=037,CLINTCP=850 DFHCNV TYPE=SELECT,OPTION=DEFAULT DFHCNV TYPE=FIELD,OFFSET=0,DATATYP=CHARACTER,DATALEN=32767, LAST=YES The SRVERCP represents the EBCDIC code page in which the data is stored within CICS. The CLINTCP is the default code page for client requests. The DATALEN is the maximum length of the COMMAREA you require to be translated. Code page aware Java programs All Java strings are stored in Unicode, a double byte character set which is similar to ASCII. The trailing byte maps to the ASCII code point for the common ASCII characters, for example, the character “A” represented by the ASCII code point X ‘41’ is represented in Unicode by X ‘0041’. The COMMAREA flowed to CICS in an ECI request must be a byte array (composed of single byte characters). Typically, within Java, the getBytes() method on the String object is used to convert a String to a byte array, and similarly, the default String constructor is used to convert a byte array into a String. Unless an encoding is specified on these calls, the conversion from Unicode to the byte array will be performed using the default platform encoding. On Intel or UNIX platforms, this encoding will usually default to U.S. ASCII (ISO 8859-1); however, within the z/OS JVM, it could also be 1047 (an extended 037 EBCDIC code page). Note: Since the arrival of WebSphere Application Server V5 on z/OS, the default JVM encoding has been U.S. ASCII (ISO 8859-1). This allows EJB components that rely on the JVM behavior instead of explicit specification of encoding to execute unchanged when ported to WebSphere Application Server for z/OS. Appendix B. DFHCNV and CICS data conversion 381 Because Unicode and ASCII share the first 256 code points, String to ASCII byte array conversion performs well, as it just involves removal of the high-order byte. Example B-2 shows an example of this technique using the ECI Sample JavaStringRecord class provided by the CICS TG to perform the data conversion from Unicode to an ASCII byte array. Example: B-2 Code page-aware Java application, ASCII COMMAREA Context ic = new InitialContext(); cxfn = (ConnectionFactory) ic.lookup("java:comp/env/eis/ECICICS"); Connection cxn= cxnf.getConnection(); Interaction ixn= cxn.createInteraction(); ECIInteractionSpec ixnSpec= new ECIInteractionSpec(SYNC_SEND_RECEIVE,"ECIPROG"); JavaStringRecord jsr = new JavaStringRecord("8859_1"); jsr.setText("DATA1"); ixn.execute(ixnSpec, jsr, jsr); ixn.close(); cxn.close(); This technique means that the COMMAREA character data always flows to CICS in ASCII, and so a DFHCNV table entry is required in CICS to convert the COMMAREA from ASCII to EBCDIC. Numeric data If you wish to flow numeric data to CICS from a Java application then it will be necessary to convert all integer values into a byte array before they can be passed to CICS. The description of how to do this is beyond the scope of this book, but further details are provided in Appendix B of Java Connectors for CICS: Featuring the J2EE Connector Architecture, SG24-6401. Code page aware Java programs without DFHCNV. An alternative to passing an ASCII COMMAREA to CICS and having the ASCII data converted to EBCDIC is to create an EBCDIC byte array, using the code page IBM037 as the encoding. This removes the need for the DFHCNV usage in CICS but can increase the cost within the Java code, because conversion from Unicode to EBCDIC in Java requires a table lookup rather than simply removing the high-order byte. 382 CICS Transaction Gateway for z/OS Version 6.1 Example B-3 shows a Java code example. Example: B-3 Code page-aware Java application — EBCDIC COMMAREA Context ic = new InitialContext(); cxfn = (ConnectionFactory) ic.lookup("java:comp/env/eis/ECICICS"); Connection cxn= cxnf.getConnection(); Interaction ixn= cxn.createInteraction(); ECIInteractionSpec ixnSpec= new ECIInteractionSpec(); ixnSpec.setInteractionVerb(ixnSpec.SYNC_SEND_RECEIVE); ixnSpec.setFunctionName("ECIPROG"); JavaStringRecord jsr = new JavaStringRecord("IBM037"); jsr.setText("DATA1"); ixn.execute(ixnSpec, jsr, jsr); ixn.close(); cxn.close(); Note: The CTGTesterCCI application allows you to specify different code page encodings. Appendix B. DFHCNV and CICS data conversion 383 384 CICS Transaction Gateway for z/OS Version 6.1 C Appendix C. EXCI Trace This appendix provides information about using EXCI trace to diagnose CICS TG problems. © Copyright IBM Corp. 2006. All rights reserved. 385 EXCI trace The EXCI trace captures details of all EXCI calls which the Gateway daemon makes to CICS. Exception trace entries are always written to a trace table in the Gateway daemon address space. A more detailed trace can be obtained by coding TRACE=2 in DFHXCOPT as shown in Example C-1. It can be viewed by dumping the address space and formatting using the IPCS VERBEXIT supplied with CICS. The process for viewing an EXCI trace is outlined in the manual CICS Transaction Gateway for z/OS Administration, SC34-6672. It is repeated here for clarity. To obtain the EXCI trace: 1. Use the following command to display all OMVS tasks: /D OMVS,A=ALL Example: C-1 CICS TG ASID displayed using /D OMVS,A=ALL CITGR1G CITGR1G LATCHWAITPID= 006D 16777742 0 CMD=CTGBATCH 1 1F---- 09.58.25 10.028 Tip: The /D OMVS,A=<asid> command displays details of the process running in the address space. 2. Find the Gateway address space, and note the address space ID (ASID). 3. Enter the DUMP command with a suitable comment. For example: /DUMP COMM=(CICS TG DEMO DUMP) 4. Reply to the message with the ASID as follows: /R 86,ASID=6D,END where 86 is the message number for the reply and 6D is the ASID. 5. The system dumps as shown in Example C-2. Note the dump name created. In our example, the dump name was SYS1.DUMP.D050923.T141658.X1.S00007. Example: C-2 Dumping the Gateway address space in SDSF IEE600I REPLY TO 86 IS;ASID=6D,END IEA794I SVC DUMP HAS CAPTURED: 506 DUMPID=008 REQUESTED BY JOB (*MASTER*) DUMP TITLE=CICS TG DEMO DUMP IEF196I IGD100I 6614 ALLOCATED TO DDNAME SYS00023 DATACLAS ( ) IEF196I IEF285I SYS1.DUMP.D050927.T110610.X1.S00008 CATALOGED IEF196I IEF285I VOL SER NOS= MOXWK0. IEA611I COMPLETE DUMP ON SYS1.DUMP.D050927.T110610.X1.S00008 510 386 CICS Transaction Gateway for z/OS Version 6.1 6. Go into IPCS and select option 0. Enter the dump name as shown in Figure C-1. ------------------------- IPCS Default Values --------------------------------Command ===> You may change any of the defaults listed below. The defaults shown before any changes are LOCAL. Change scope to GLOBAL to display global defaults. Scope ==> BOTH (LOCAL, GLOBAL, or BOTH) If you change the Source default, IPCS will display the current default Address Space for the new source and will ignore any data entered in the Address Space field. Source Address Message Message Display ==> DSNAME('SYS1.DUMP.D050927.T110610.X1.S00008') Space ==> Routing ==> NOPRINT TERMINAL Control ==> CONFIRM VERIFY FLAG(WARNING) Content ==> NOMACHINE REMARK REQUEST NOSTORAGE SYMBOL Press ENTER to update defaults. Use the END command to exit without an update. Figure C-1 Selecting the dump data set in IPCS 7. Format the dump using the CICS TS V3.1 trace formatter by going to option 6 within IPCS and entering VERBX DFHPD640 ‘TR=1’ (the last parameter requests an abbreviated trace). Pressing Enter on this screen formats the dump. IPCS asks whether you want to use summary dump data. Reply Y as shown in Figure C-2. Appendix C. EXCI Trace 387 Figure C-2 3270 ISPF screen capture 8. On the resulting IPCS formatted dump output, you now need to find the EXCI trace table. To do this, enter the command FIND ‘TRACE TABLE’. Example C-3 shows the output from our job. Note the EXIT ECI_PARAMETERS_OUTPUT line. The return code of -3 equates to an ECI_ERR_NO_CICS. Example: C-3 IPCS dump trace table INTERNAL TRACE TABLE XCI XCI XCI XCI XCI XCI XCI XCI XCI XCI EXCI EXCI EXCI EXCI EXCI EXCI EXCI EXCI EXCI EXCI EX EX EX EX EX EX EX EX EX EX 03ED 03EE 8000 8003 8003 8004 0439 043B 8021 8010 ECI ECI JVDLL JVDLL JVDLL JVDLL ????? ????? JVDLL JVDLL ENTRY EXIT ENTRY DATA DATA DATA ????? ????? DATA *EXC* ECI_SERVER_LIST_ENTRY_PARAMETERS 20 ECI_SERVER_LIST_EXIT_PARAMETERS 0 ECI_PARAMETERS_PASSED Worker-9,1 CONVERTED_PARAMETER Worker-9,jstrServer,A6POTGR1 CONVERTED_PARAMETER Worker-9,jstrProgram,EC01 INBOUND_COMMAREA Worker-9,18,0F9440D8 **_POINT_ID_NOT_RECOGNISED_** **_POINT_ID_NOT_RECOGNISED_** FIRST_REQUEST_ON_TCB_ADDRESS 008CAE88 ERROR_RESPONSE_RECEIVED 3,-3 The CICS TG trace entries are documented in Chapter 11, Problem Determination, of the CICS Transaction Gateway for z/OS Administration, SC34-6672. 388 CICS Transaction Gateway for z/OS Version 6.1 EXCI trace and GTF It is possible to use GTF to analyze both CICS and EXCI traces. This tracing is well suited to in-depth error analysis because it provides you can interleave both EXCI and CICS trace entries. It does require, however, that you start and initialize the GTF process in order to collect the data. Example C-4 gives a brief explanation of how we debugged a simple problem with the CICS TG for z/OS using GTF tracing. In this error scenario, the call using EciB1 failed with ECI_ERR_TRANSACTION_ABEND (-7) and AEI0 due to the disabling of the invoked program EC01 within CICS. Note: The principal advantage of GTF tracing is that the EXCI and CICS trace entries can be combined in the same output, which can help considerably in problem determination. However, the Gateway daemon and JNI trace cannot be written to GTF and must be analyzed separately. The steps in this scenario are as follows: You must enable both EXCI and CICS trace entries to be written to GTF. The DFHXCOPT table controls EXCI tracing and activates an EXCI trace. You specify TRACE=2 and GTF=ON in the DFHXCOPT table (Example C-4). Example: C-4 EXCI options table, DFHXCOPT DFHXCO TYPE=CSECT, TIMEOUT=0, TRACE=2, TRACESZE=4096, DURETRY=30, TRAP=OFF, GTF=ON, MSGCASE=MIXED, CICSSVC=0, CONFDATA=SHOW, SURROGCHK=NO, END DFHXCOPT No Timeout Trace Entries Trace table size Retry SDUMPS for 30 seconds DFHXCTRA - OFF GTF - ON Mixed case messages EXCI will obtain CICS SVC number Show user commarea data in trace Perform e-user check The customized DFHXCOPT must be in the STEPLIB. This can be ensured by specifying the library containing the DFHXCOPT table in the EXCI_OPTIONS environment variable in STDENV. Restart the Gateway daemon to pick up the changed DFHXCOPT. Appendix C. EXCI Trace 389 CICS GTF tracing can be activated using the CETR transaction (Figure C-3) by setting the GTF Trace Status to STARTED. We also reduced the CICS trace entries to IS=1-2 and PC=1 to capture only MRO and program control trace entries. This can be achieved using PF4 on the CETR screen. CETR CICS Trace Control Facility TGR1 A6POTGR1 Type in your choices. Item Choice Possible choices Internal Trace Status Internal Trace Table Size ===> ===> STARTED 1024 K STArted, STOpped 16K - 1048576K Auxiliary Trace Status Auxiliary Trace Dataset Auxiliary Switch Status ===> ===> ===> STOPPED A NO STArted, STOpped, Paused A, B NO, NExt, All GTF Trace Status ===> STARTED STArted, STOpped Master System Trace Flag Master User Trace Flag ===> ===> ON ON ON, OFf ON, OFf When finished, press ENTER. PF1=Help 3=Quit 4=Components 5=Ter/Trn 6=JVM 9=Error List Figure C-3 Turning on CICS GTF tracing using CETR To activate GTF you need to start your GTF started task. The JCL for our started task is shown in Example C-5. We used the command /S GTF2.EXTRC,ID=CITGCA to pass the token EXTRC and user ID CITGCA. 390 CICS Transaction Gateway for z/OS Version 6.1 Example: C-5 GTF2 proc //GTF2 //* // //DSN // //* //GTF // //IEFRDER // // //SYSPRINT //SYSLIB PROC MEMBER=GTFPARM,ID=DEFAULT EXEC PGM=IEFBR14 DD DISP=(MOD,DELETE),DSN=&ID..GTF.TRACE, UNIT=SYSDA,SPACE=(TRK,(1)) EXEC PGM=AHLGTF,PARM='MODE=EXT,DEBUG=NO,TIME=YES,BLOK=4M', REGION=40M DD DSN=&ID..GTF.TRACE, DISP=(NEW,CATLG),UNIT=SYSDA,SPACE=(CYL,(250)), DCB=(LRECL=4092,BLKSIZE=4096,RECFM=VB) , DD SYSOUT=H DD DSN=SYS1.PARMLIB1(&MEMBER),DISP=SHR The SYS1.PARMLIB1(GTF) member referenced in Example C-5 defines the following options for GTF: TRACE=SYSM,USR,TRC,DSP,PCI,SRM,RNIO When GTF has initialized, you need to reply U to the initialization message (Example C-6). Example: C-6 Reply to GTF commands R 99,U IEE600I REPLY TO 99 IS;U U AHL906I THE OUTPUT BLOCK SIZE OF 4096 WILL BE USED FOR OUTPUT 838 DATA SETS: CITGCA.GTF.TRACE AHL080I GTF STORAGE USED FOR GTF DATA: 839 GTFBLOCK STORAGE 4096K BYTES (BLOK= 4096K) PRIVATE STORAGE 1024K BYTES (SIZE= 1024K) SADMP HISTORY 40K BYTES (SADMP= 40K) SDUMP HISTORY 40K BYTES (SDUMP= 40K) ABEND DUMP DATA 0K BYTES (ABDUMP= 0K) AHL031I GTF INITIALIZATION COMPLETE GTF tracing is now active and trace entries from the Gateway daemon and the CICS region will be captured by GTF. Once the problem you wish to trace has been re-created, terminate the GTF started task using the token specified on initialization, for example, we issued the command /P EXTRC. Go to IPCS option 0 and set the name of the GTF trace data which has been created by GTF. In our case the data set was called CITGCA.GTF.TRACE. Appendix C. EXCI Trace 391 Select PF3 to return to the IPCS main menu, select option 6 and enter the command: GTF USR(F6C) This produces the formatted trace output for the EXCI and CICS trace entries. An abbreviated output from the formatted trace (Example C-7), showing the ECI return code -7 caused by the our PGMIDERR condition. Example: C-7 Formatted GTF EXCI and CICS trace entries EX 8003 JVDLL DATA - CONVERTED_PARAMETER JAVA_TID(Worker-0) NAME(jstrServer) VALUE(A6POTGR1) EX 8003 JVDLL DATA - CONVERTED_PARAMETER JAVA_TID(Worker-0) NAME(jstrProgram) VALUE(EC01 ) EX 1000 XCPRH ENTRY - OPEN_PIPE TOKEN(0F6C8140) TO CICS(A6POTGR1) BY USER(CITGR1G ) EX 2001 XCPRH EVENT IRP_CONNECT TO CICS(A6POTGR1) BY USER(CITGR1G ) EX 1000 XCPRH ENTRY - OPEN_PIPE TOKEN(0F6C8140) TO CICS(A6POTGR1) BY USER(CITGR1G ) EX 1000 XCPRH ENTRY - DPL TO PROGRAM(EC01 ) ON CICS(A6POTGR1) USING PIPE(0F6C8140) BY USER(CITGR1G ) EX 2004 XCPRH EVENT SWITCH_REQUEST SENT TO CICS(A6POTGR1) BY USER(CITGR1G ) EX 2005 XCPRH DATA DATA SENT ON PIPE(0F6C8140) BY USER(CITGR1G ) AP DD18 CRNP EVENT - DEQUEUE WORK ELEMENT TYPE (NEW CONNECT WITH INPUT RECEIVED) TIMESTAMP (BDAE180BD69185A7) SCCB AT 7F4A2E80 SYSTEM CITGR1G EX 0802 XCPRH *EXC* - PGMIDERR_DETECTED - PROGRAM(EC01 ) ON CICS(A6POTGR1) BY USER(CITGR1G ) EX 8011 JVDLL *EXC* - DPL_REQUEST_ERROR ECI_RC(-7) EX 1000 XCPRH ENTRY - CLOSE_PIPE TOKEN(0F6C8140) TO CICS(A6POTGR1) BY USER(CITGR1G ) EX 2002 XCPRH EVENT IRP_DISCONNECT FROM CICS(A6POTGR1) BY USER(CITGR1G ) 392 CICS Transaction Gateway for z/OS Version 6.1 D Appendix D. Additional material This appendix describes the additional material to which this redbook refers that you can download from the Internet. Locating the Web material The Web material associated with this redbook is available in softcopy format on the Internet from the redbooks Web server. Point your Web browser to: ftp://www.redbooks.ibm.com/redbooks/SG247161 Alternatively, you can go to the redbooks Web site at: http://www.redbooks.ibm.com Select Additional Materials and open the file that corresponds with the redbook form number SG247161. © Copyright IBM Corp. 2006. All rights reserved. 393 Using the Web material The additional Web material that accompanies this redbook includes the following files: File name SG247161.zip Description Zipped Code Samples The SG247161.zip file includes the three EAR files which we used in our test scenarios. Table D-1 provides a short description of each EAR file. Table D-1 EAR files EAR files Description CTGTesterCCI.ear Includes the CTGTesterCCI application code that we used in our connectivity tests in chapters 3 and 4 and in our security tests in chapters 8 and 9. CTGTesterCCIXA.ear Includes the CTGTesterCCIXA application code that we used in our transaction tests in chapters 5, 6, and 7. CTGTesterCCI_Sec.ear Includes the CTGTesterCCI application code with deployment descriptors that define a security constraint. We used this EAR in our Thread Identity Support test scenario in Chapter 10. For further details about the sample J2EE applications see Appendix A, “Sample applications” on page 361. System requirements for downloading the Web material There are no specific system requirements for your workstation to download the supplied materials. However, to be able to use the supplied samples, you need to adhere to product specifications as described throughout this book. How to use the Web material Create a subdirectory (folder) on your workstation, and unzip the contents of the Web material zipped file into this folder. 394 CICS Transaction Gateway for z/OS Version 6.1 Abbreviations and acronyms AOR Application-Owning Region IDE API Application Programming Interface Integrated development environment IRC inter-region communication Advanced Program-to-Program Communications ISC Inter-System Communication ITSO International Technical Support Organization ASCII American Standard Code for Information Interchange J2EE Java 2 Enterprise Edition CCI Common Client Interface JAR Java Archive CICS Customer Information Control System JCA J2EE Connector Architecture JDBC Java Database Connectivity APPC CICS TG CICS Transaction Gateway JDK Java Developer’s Kit DPL Distributed Program Link JNDI EAR Enterprise Application Archive Java Naming and Directory Interface™ EBCDIC Extended Binary Coded Decimal Interchange Code JNI Java Native Interface JSP JavaServer™ Page ECI External Call Interface JSSE EIS Enterprise Information Systems Java Secure Sockets Extension JVM Java Virtual Machine EJB Enterprise JavaBeans™ LPAR Logical Partition EPI External Presentation Interface LUW Logical Unit of Work MRO Multi Region Operation ESI External Security Interface PEM ESM External Security Manager Password Expiration Management EXCI External CICS Interface RAR Resource Adapter Archive FTP File Transfer Protocol RRMS GTF Generalized Trace Facility Recoverable Resource Management Services GTF Generalized Trace Facility RRS Resource Recovery Services HFS Hierarchical File System SAF System Authorization Facility HTML Hypertext Transfer Protocol SNA HTTP Hypertext Markup Language Systems Network Architecture IBM International Business Machines SSL Secure Sockets Layer UOW Unit of Work UR Unit of Recovery © Copyright IBM Corp. 2006. All rights reserved. 395 USS Unix System Services VIPA Virtual IP Addressing XCF Cross Coupling Facility XID X/Open identifier 396 CICS Transaction Gateway for z/OS Version 6.1 Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook. IBM Redbooks For information about ordering these publications, see “How to get IBM Redbooks” on page 398. Note that some of the documents referenced here might be available in softcopy only. CICS Transaction Gateway V5 The WebSphere Connector for CICS, SG24-6133 Revealed! Architecting e-business Access to CICS, SG24-5466 WebSphere for z/OS V6 Connectivity Handbook, SG24-7064 Java Connectors for CICS: Featuring the J2EE Connector Architecture, SG24-6401 Other publications These publications are also relevant as further information sources: CICS External Interfaces Guide, SC33-1944 CICS Intercommunication Guide, SC34-6448 CICS Transaction Gateway for z/OS Administration, SC34-6672 CICS Transaction Server for z/OS Installation Guide V3.1, GC34-6326 Program Directory for CICS Transaction Gateway for z/OS, GI10-2587 The z/OS Security Server (RACF) Command Reference, SA22-7687 WebSphere Application Server for z/OS V6.0.1, Installing your application serving environment, GA22-7957 z/OS V1R6 UNIX System Services Command Reference, SA22-7802 z/OSV1R2.0 MVS Setting up a Sysplex, SA22-7625 z/OS V1R6 UNIX System Services Planning, GA22-7800 z/OS MVS Programming: Resource Recovery, SA22-7616 © Copyright IBM Corp. 2006. All rights reserved. 397 Online resources These Web sites and URLs are also relevant as further information sources: CICS and CICS Transaction Gateway library http://www.ibm.com/software/htp/cics/library/ WebSphere Application Server library http://www.ibm.com/software/webservers/appserv/was/library/index.html IBM Technical Sales Support Site http://www.ibm.com/support/techdocs/atsmastr.nsf/Web/TechDocs How to get IBM Redbooks You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications, and Additional materials, as well as order hardcopy Redbooks or CD-ROMs at this Web site: ibm.com/redbooks Help from IBM IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services 398 CICS Transaction Gateway for z/OS Version 6.1 Index Symbols _BPX_SHAREAS environment variable 38 A Abend ARXC 278 abend code ECIP 183 S806 61 ABENDBKOUT, EXCI option 166, 198, 259, 369 address space 31, 97, 137, 149, 164, 195, 386 balance incoming socket open requests 103 required sharing 32 Address Space Initiation utility (ctgasi) 220 Advanced Encryption Standard (AES) 313 Advanced Program Function (APF) 33 AIX 4 Anynet LU6.2/IP 17 APARS PK17426 166 PK17427 166 PK17700 334 PQ92943 97, 137 PQ99940 200, 261 APPC connection 21 Application Server Network Deployment V6.0.2 79, 310 V6.0 59 ASCII text 50, 298, 326 AUTH_USERID_PASSWORD 288, 295, 343–344 Authorized Program Facility (APF) 197 Automatic Restart Management (ARM) 51 B basic authentication 351, 353–354 bind security 285, 287, 307, 342–343, 357 C CA certificate 315 ccf2.jar 30 CEDA ALTER TRANS 101, 141 CEMT INQUIRE EXCI command 185, 267 © Copyright IBM Corp. 2006. All rights reserved. CEMT INQUIRE UOW command 241 INDOUBT 245 changing a password 10 checklist software 26, 78–79, 123, 159, 227, 255, 282, 310, 341 checklist, definitions 27, 79, 123, 174, 236, 258, 283, 311, 341 CICS ECI resource adapter connection factory 83 transactional capabilities 167 CICS ECI resource adapter (cicseci.rar) 59, 78, 122, 167, 227, 256, 294, 334, 340, 361 CICS ECI XA resource adapter (cicseciXA.rar) 169–170, 193, 199 CICS EPI resource adapter (cicsepi.rar) 10 CICS TG configuring 33 configuring security 286, 342 ctginstall 27 installing running ctginstall 27 master process 197, 225 master process CITGR0G 232 master process configuration 230 master process environment variable 231 master process immediate shutdown 249 master process initialization 248 master process shutdown 249 multiple CICS regions 100 overview 4 password checking 286, 288, 342 previous versions 311 product executables 33 product file 27 security tasks 30 SMP/E installation 27 STDERR message 62 STDOUT log 116 STDOUT output 46 transactional support 45, 165–166 user ID password authentication 296 XA resource adapter 194 399 CICS TG for Multiplatforms 4–5 CICS TG for z/OS V6.1 6 CICS Transaction Gateway. See CICS TG. CICS Transaction Server. See CICS TS. CICS TS 7, 26–27, 77, 79, 121, 123, 159, 165, 236, 257, 282–283, 310–311, 341 CICS Universal Client V6 5 CICSCLI environment variable 38 cicseci.rar 9, 14–15 cicseciXA.rar 9, 15 cicsepi.rar 14 cicsj2ee.jar 30 CLASSPATH environment variable 39 Client authentication 309 CICS TG configuration 329 client certificate 312 one-to-one mapping 332 COBOL 4 COLUMNS environment variable 39 command CEMT INQUIRE EXCI 185, 267 CEMT INQUIRE UOW 241 RACDCERT 319 COMMAREA 8, 46, 96, 144, 183, 237, 266, 298, 355, 361, 380 COMMAREA length 72, 187, 364 commit optimization 17 Common Client Interface (CCI) 9, 12 Common Connector Framework (CCF) 9 Common Cryptographic Architecture (CCA) 315 Communications Server for Linux 21 configuration file 29, 33, 36 default name 36 configuration tool 59 configuring CICS TG 33 CONNECTION definition 43 connection factory 82, 124, 128, 169, 228, 235, 258–259, 293, 328, 348, 362 actual JNDI name 367 CICS regions 198 connection pool 108 custom properties 350 General Properties panel 84 JNDI name 135 KeyRingClass definition 336 socket connections 235 TraceLevel parameter 119 connection management contract 13 400 connection manager thread 92–93 connection pool 87–88, 132, 296, 327 property 87, 131 connection pooling 17–18, 20 connector.jar 30 contract connection management 13 security 13 transaction management 13 contracts, system-level 13 creating a Gateway daemon configuration file 36 creating an HFS data set 33 creating an STDENV file 37 creating the started task JCL 42 credentials 17 CTG.INI 36, 58 CTG_JNI_TRACE 152–153, 278 CTG_RRMNAME 175, 188, 196, 208, 217, 220–221 CTG_RRMNAME environment variable 39 ctgadmin 6 ctgasi utility 195 ctgclient.jar 30 ctgd 6 ctgenvvar 29 ctgenvvarsamp 29 ctginstall running 28 CTGRRMS 194, 230 ctgsamp.ini 29 ctgsamples.jar 30 ctgserver.jar 30 ctgstart 29 CTGTesterCCI application 115, 136, 158, 295–296, 328, 333, 350, 362, 383 end user interacts 363 password information 299 CTGTesterCCI input screen 301, 354 CTGTesterCCIXA application 199–200, 237, 254, 361, 372 D data conversion 46, 380 definitions checklist 27, 79, 123, 174, 236, 258, 283, 311, 341 DFHJVPIPE environment variable 39, 61, 287, 343 CICS Transaction Gateway for z/OS Version 6.1 DFHJVSYSTEM environment variable 40 DFHMIRS, CICS mirror program 380 DFHXCURM 100, 106 diagnosing problems 385 distributed program link (DPL) 8, 160 Domain Name Services (DNS) 95 DSA 313 dumping the Gateway address space 386 dynamic trace 69, 71 E EC01, COBOL example 46, 50 ECI password, verify 10 ECI request 9, 48, 93, 140, 166, 227, 285, 332, 346, 361, 381 complete sequence 229 Gateway process sequence 206 JNI trace 206 specific Gateway daemon instance 228 XID information 206 ECI sample 297 ECI_ERR_NO_CICS 50, 60, 70, 388 ECI_ERR_RESOURCE_SHORTAGE 116 ECI_ERR_SECURITY_ERROR 306–307 ECI_ERR_SYSTEM_ERROR 116 EJB container 171 Enterprise Information System (EIS) 11, 167, 293, 347 environment variable 37, 42, 92, 137, 231 _BPX_SHAREAS 38 CICSCLI 38 CLASSPATH 39 COLUMNS 39 CTG_RRMNAME 39 DFHJVPIPE 39, 61, 287, 343 DFHJVSYSTEM 40 PATH 40 search order 41 STEPLIB 40 TMPDIR 40 TZ 40 EPI 9 EPI support classes 10 EPIRequest 9 ESI 10 password, change 10 EXCI connection 43 EXCI interface 7 EXCI options table 98, 166, 292, 347 EXCI pipe 44, 96, 131 maximum number 108 maximum potential number 101 EXCI request 98, 148, 285, 345 flowed user ID 289 limited workload balancing 100 security context 292 EXCI trace 73 EXCI_LOADLIB 61 EXCI_OPTIONS 292, 389 extended LUW 178, 227 JNI trace 189 extended LUWs TCP/IP load balancing 227 External Call Interface. See ECI. External Presentation Interface.See EPI. External Security Interface. See ESI. external security manager (ESM) 10, 283 F FACILITY class, RACF 31, 115, 196, 220, 245, 251, 287, 342–343 files ccf2.jar 30 cicseci.rar 9, 14–15 cicseciXA.rar 9, 15 cicsepi.rar 14 cicsj2ee 30 connector.jar 30 ctgclient.jar 30 ctgsamp.ini 29 ctgsamples.jar 30 ctgserver.jar 30 guide.jar 30 psk.jar 30 STDENV 37 flowed user ID 283–286, 289, 291–292, 308, 342, 345–346 formatting traces, IPCS 392 G Gateway daemon 5, 18, 35, 56, 79, 82, 84, 126, 166–167, 169, 225, 227, 257, 261, 285, 309, 319, 361, 373, 386 cloned group 226 configuration files 36 Index 401 connection details 362 dummy DD statement 65 ECI requests 230 EXCI connection 105 RACF permissions 56 worker threads 109 Gateway daemon configuration file 36 Gateway trace 68 Generalized Trace Facility (GTF) 68 get-use-cache model 14 global transaction 157, 226, 245, 253, 258, 361 Configuring 258 full support 168 group configuration 234 group Gateway daemon instances 237 group master process access 235 group normal shutdown sequence 248 group RRM name 231 guide.jar 30 H hardware cryptography 314 HFS data set, creating 33 HFS data set, mounting 34 HFS directory 39, 80 high availability configuration 105 HiperSockets 21 HP-UX 4 hwkeytool 312, 314–316 RunAs policy 347 secure access 293 Java client trace 362 Java Cryptography Extension 315 Java Development Kit 63, 297 Java Secure Sockets Extension (JSSE) 309 JCA 6 authentication entry 17 Connection Pool 131, 190 overview 11 security 296 Version 1.5 13 with different topologies 15 JDBC 13 JNDI name 135, 179, 353, 367 JNI trace 68, 71, 152–153, 184, 186, 206, 278, 307 K key ring 309 personal certificates 319 keystore 314, 316–317 keytool 316–317 L IBM SDK 26, 79, 123, 159, 227, 255, 282, 310, 341 security policy file 325 ikeyman 312, 316, 320 IMS JDBC 256 in-doubt 162, 213, 222–223, 229, 236–239, 241–246, 248, 252 initconnect, Gateway daemon parameter 94, 108 initworker, Gateway daemon parameter 94 Install Shield Multiplatform 6 Intersystem communication (ISC) 257 IPCS, formatting traces 392 Language Environment (LE) 66 last-participant support 17 last-resource commit optimization 17 lazy connection association 14 lazy transaction enlistment 14 link pack area (LPA) 138 Link user ID 284–285, 344 Linked List Area (LLA) 196 Linux 4 Linux on zSeries 21 Linux, RHEL V3 5 local connection 122, 126, 256, 340, 361 local mode of operation 11 local transaction 80, 176, 178, 183, 189, 193, 368 LocalTransaction interface 168 LOGONLIM 97–98, 108–109, 117, 137 LPAR 97, 165, 226 Luw_Token 187 J M J2EE Connector Architecture. See JCA. Java 2 Platform Enterprise Edition 9, 77, 90, 121, 379 maxconnect, Gateway daemon parameter 93–94, 108 maxworker, Gateway daemon parameter 93–95, I 402 CICS Transaction Gateway for z/OS Version 6.1 108 Message Authentication Code (MAC) 315 mounting an HFS data set 34 MRO bind security 115, 287, 307, 342–343 OMVS 63, 386 overview of CICS TG 4 remote mode of operation 10 resource adapter 12, 14, 22, 60, 79, 82–83, 127, 167, 174, 227, 255, 294, 328 resource manager logs information 273 resource manager (RM) 45, 158, 161, 226, 254–255 Resource Recovery Management Service (RRMS) 45 Resource Recovery Service (RRS) 39, 163, 255 resource reference 135, 174, 200, 259, 293, 302, 348, 361 RRMS, SIT parameter 45, 139, 175 RRMSEXIT state 204, 240, 266 RRS archive log 186, 245, 266 P S parallel sysplex 103–104, 149 password, changing 10 password, verity 10 PATH environment variable 40 Performance Monitoring Infrastructure (PMI) 146 Peripheral Component Interconnect (PCI) 312 private mirror 86, 123 protocol handler 91 psk.jar 30 sample application 110, 158, 178, 299, 348, 361 sample ECI application EciB1 281, 309 sample J2EE application 78, 122, 124, 182, 263, 339 CTGTesterCCI 80, 107, 142, 281, 309 CTGTesterCCIXA 201, 234 scripts ctgenvvarsamp 29 ctginstall 28 ctgstart 29 SCTGLINK library 33 SDSF log 48 Secure Sockets Layer (SSL) 309 security 6, 283, 286, 342 configuring CICS TG 286, 342 considerations 30 tasks 30 security contract 13 security credentials 17 security management 17, 19, 21 self-signed certificate 312 public key 317 subject fields 317 Servant region 123, 258, 341 SESSIONS definition 43–45, 57, 97, 101, 109, 116, 139, 284, 286, 342 USERID parameter 290 SETROPTS RACLIST 287, 343 setting permissions 35 SLES 9, Linux 5 SNA network connections 17 N NETNAME, CONNECTION definition 27, 56, 79, 123, 174, 194, 236–237, 258, 283, 311, 341 netstat, TCP/IP test tool 49, 62 nonames, Gateway daemon parameter 94–95 O Q Quality of Service (QOS) 103, 149 R RACDCERT command 319 RACF BPX.FILEATTR.PROGCTL FACILITY class 31 BPX.SERVER FACILITY class 31 TCICSTRN class 291, 307, 346 RACF error 306, 357 RACF security tasks 30 RACF user ID 294, 332, 344 Rational Application Developer (RAD) 177, 350, 364, 370, 373, 377 RECEIVECOUNT 44, 97, 137 recoverable resource 158, 182, 226, 254 committed and backed-out updates 182 region userid 337 remote connection 122, 256, 362 XA transaction support 257 XAResource interface 256 Index 403 software checklist 26, 78–79, 123, 159, 227, 255, 282, 310, 341 SSL client authentication 305, 332 client authentication error message 333 client certificate 332 connection 6, 282, 309–310, 399 JCA connection pooling 327 performance overhead 327 keyring 50, 86, 326 private keys 7 protocol 5 handler 309 parameter list 323 static trace 68, 71 STDENV DD card 41, 288 statement 38 STDENV file 37 STEPLIB environment variable 40 subreason field-1 60, 192, 307 subreason field-2 60, 192, 307 surrogate security 292, 306, 346–347, 357 syncpoint coordinator 166, 243, 256 Sysplex Distributor 60, 103, 149, 227 System Authorization Facility (SAF) 283 system initialization table (SIT) 139 system-level contracts 13 systems administration tool 6 systems management 6 Systems Network Architecture (SNA) 16 SystemSSL 59, 311, 335 EXCI trace 68, 73 Gateway trace 68 JNI trace 68, 71 static trace 68, 71 TRANCLASS 101, 140 transaction attribute 168, 173, 368 possible values 172 transaction management 17, 19–20 transaction management contract 13 transaction manager 158, 160, 226, 254, 368 global transaction 204 Transactional EXCI 100, 165, 258 Transport Layer Security (TLS) 312 TS Queue RECIPROG 238 two-phase commit 19–20, 100, 126, 157, 159, 161–163, 168–169, 228–229, 255–257, 277 TZ environment variable 40 T W TCP protocol 5 TCP/IP 16 TCP/IP connection 91, 257 TCP/IP port 27, 79, 123, 174, 225, 258, 283, 341 cloned Gateway daemons 230 sharing 103, 150, 227 TCP62 protocol 17 Tivoli Performance Viewer 145, 189, 263 TMPDIR environment variable 40 topologies 15 tracing creating a data set 33 dynamic trace 69, 71 Web browser 126, 183, 237, 266, 348, 363, 393 CTGTesterCCIXA application 237 WebSphere Application Server 11, 13, 16, 18, 77, 100, 121, 163 WebSphere servant region EXCI call 344 user ID 343 Windows 2000 4 Windows 2003 4 Windows XP 4 work manager 164, 243, 270 worker thread 49, 92–93 connection manager threads 93 initial number 94 404 U Unit of Recovery (UR) 164, 238 unit-of-work (UOW) 159, 238 user ID 30, 112, 126, 191, 284–285, 319, 343, 362 CTGTesterCCI input 300 USS program control 31 V verifying a password 10 virtual private networks 19 VPN 19 VTAM 17 CICS Transaction Gateway for z/OS Version 6.1 Java client disconnects 94 maximum number 95 X X/Open identifier (XID) 206, 228, 272 XA transaction 7, 39, 105, 157–158, 225, 257, 372 ECI request 221 in-doubt failure 236 Normal termination 237 post-prepare phase 228 Z zSeries File System (ZFS) 33 Index 405 406 CICS Transaction Gateway for z/OS Version 6.1 CICS Transaction Gateway for z/OS Version 6.1 (0.5” spine) 0.475”<->0.875” 250 <-> 459 pages Back cover ® CICS Transaction Gateway for z/OS Version 6.1 Install and configure the Gateway on z/OS Connect from WebSphere on Windows and z/OS Enable global transaction support and SSL security The CICS Transaction Gateway (CICS TG) is used widely to provide access to CICS programs from Java environments. It provides the IBM implementation of the J2EE Connector Architecture (JCA). This IBM Redbook introduces the new facilities of the CICS TG for z/OS V6.0 and V6.1, which provide improvements in the areas of transactional integration, systems management, performance, security, and ease of use. Included in this book are a set of configuration scenarios for a variety of different deployment topologies for integrating J2EE applications on WebSphere Application Server with CICS TS for z/OS. This book provides step-by-step explanations about how to configure connections between each tier of the test environment. We show you how to enable support for global transactions which span resources on different systems. We also cover the configuration of CICS TG security, including the use of JSSE for establishing secure SSL connections to the CICS TG. INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. For more information: ibm.com/redbooks SG24-7161-00 ISBN 0738496227