* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Aster Client Guide - Information Products
Survey
Document related concepts
Microsoft Access wikipedia , lookup
Oracle Database wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Concurrency control wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Functional Database Model wikipedia , lookup
Ingres (database) wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Versant Object Database wikipedia , lookup
Relational model wikipedia , lookup
Clusterpoint wikipedia , lookup
ContactPoint wikipedia , lookup
Transcript
Aster Client Guide Release Number AC 5.10 April 2013 The product or products described in this book are licensed products of Teradata Corporation or its affiliates. Teradata, Active Data Warehousing, Active Enterprise Intelligence, Applications-Within, Aprimo, Aprimo Marketing Studio, Aster, BYNET, Claraview, DecisionCast, Gridscale, MyCommerce, Raising Intelligence, Smarter. Faster. Wins., SQL-MapReduce, Teradata Decision Experts, "Teradata Labs" logo, "Teradata Raising Intelligence" logo, Teradata ServiceConnect, Teradata Source Experts, "Teradata The Best Decision Possible" logo, The Best Decision Possible, WebAnalyst, and Xkoto are trademarks or registered trademarks of Teradata Corporation or its affiliates in the United States and other countries. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. Axeda is a registered trademark of Axeda Corporation. Axeda Agents, Axeda Applications, Axeda Policy Manager, Axeda Enterprise, Axeda Access, Axeda Software Management, Axeda Service, Axeda ServiceLink, and Firewall-Friendly are trademarks and Maximum Results and Maximum Support are servicemarks of Axeda Corporation. Data Domain, EMC, PowerPath, SRDF, and Symmetrix are registered trademarks of EMC Corporation. GoldenGate is a trademark of Oracle. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the United States and other countries. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, RACF, Tivoli, and z/OS are registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. LSI is a registered trademark of LSI Corporation. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are registered trademarks of Microsoft Corporation in the United States and other countries. NetVault is a trademark or registered trademark of Quest Software, Inc. in the United States and/or other countries. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. Oracle, Java, and Solaris are registered trademarks of Oracle and/or its affiliates. QLogic and SANbox are trademarks or registered trademarks of QLogic Corporation. Red Hat is a trademark of Red Hat, Inc., registered in the U.S. and other countries. Used under license. SAS and SAS/C are trademarks or registered trademarks of SAS Institute Inc. SPARC is a registered trademark of SPARC International, Inc. Symantec, NetBackup, and VERITAS are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Unicode is a registered trademark of Unicode, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED ON AN "AS-IS" BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. IN NO EVENT WILL TERADATA CORPORATION BE LIABLE FOR ANY INDIRECT, DIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS OR LOST SAVINGS, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please email: [email protected]. Any comments or materials (collectively referred to as "Feedback") sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback. Copyright © 2000-2013 by Teradata Corporation. All Rights Reserved. Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Conventions Used in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Typefaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 SQL Text Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Command Shell Text Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Contacting Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 About Teradata Aster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 About This Document. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Chapter 1: Install Aster Database Tools and Utilities . . . . . . . . 11 Install the Aster Database Cluster Terminal (ACT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Obtain ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Install ACT on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Install ACT on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Install ACT on MacOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Install ACT on Solaris. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Chapter 2: Aster Database Cluster Terminal (ACT) . . . . . . . . . . . 15 ACT Quick Start. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Install ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Launch ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Launch ACT on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Launch ACT on Linux or Solaris. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Launch ACT on Mac. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Launch ACT Directly on the Queen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Log In to ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Startup Parameters for ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Using the “on-error-stop” Option in ACT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Use a Configuration File to Pass ACT Startup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 22 How to Use ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Issue SQL Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Aster Client Guide 3 Exit ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Page Through Query Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Throttle Query Results in ACT and Aster Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 ACT Utility Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Repeat Previously Typed Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Tab Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 ACT Commands (at the SQL Prompt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Database Parameters Set with \set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Common SSL Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Setting Parameters on the Queen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Encrypting Communications from the Queen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Using Single Sign-on (SSO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Scenario 1: Queen Provides a Self-Signed Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Scenario 2: Client Must Have a CA-Signed Copy of the Server’s Certificate . . . . . . . . . . 38 Scenario 3: Client CA-signed Certificate Must Match the Queen Certificate . . . . . . . . . . 39 Scenario 4: Encrypting Communication from the Queen to the Client . . . . . . . . . . . . . . 40 Troubleshooting ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 ACT Connection Hangs When Using SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Misleading Error Message Reports Problem With a Role Instead of With a User . . . . . . 41 Chapter 3: Connect Using Database Drivers . . . . . . . . . . . . . . . . . . . . 43 General Tips for Connecting Clients to Aster Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Recommended Character Set Is UTF-8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 When Querying System Tables with ODBC, Set AUTOCOMMIT to 'OFF'. . . . . . . . . . . 44 AIX Client Dependent Libaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 ODBC Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 ODBC Driver Manager Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Obtain the ODBC Driver Package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Install ODBC on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Install ODBC on Linux, Solaris, or MacOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Configure Driver Manager on Linux and AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Install and Configure the unixODBC Driver Manager on Solaris . . . . . . . . . . . . . . . . . . . 52 Set Up ODBC for Perl Connectivity on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Set up ODBC for PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 ODBC Usage Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 JDBC Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Aster JDBC Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Differences from the Legacy JDBC Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Before You Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Install the JDBC Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4 Aster Client Guide Use the JDBC Driver in a Java Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Parameters for Connecting through JDBC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Configuring the JDBC Log Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Behavior and Performance Settings for JDBC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 The Copy, Install File, Uninstall File, and Download File Commands . . . . . . . . . . . . . . . 65 Using Client-Side Cursors in JDBC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Test JDBC Connect Program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Configure Aster Database SQL Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 SQL Behavior Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Setting the SQL Behavior Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Syntax for ODBC Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Teradata Wallet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Wallet Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 TD Wallet Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Download TD Wallet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Install and Configure TD Wallet on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Install and Configure TD Wallet on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 SSL Security for Aster Database-Client Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 SSL-Related Files and Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 SSL Settings Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Common SSL Configuration Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Encrypting Data Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Adding AD-Based SSO Authentication to SSL-Secured Aster Database . . . . . . . . . . . . . . 84 How to Set Configuration Parameters on the Queen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 How to Set Configuration Parameters on the Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Creating Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Process SQL Statements in JDBC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Process a Simple Query in JDBC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 JDBC Troubleshooting and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Connect Reporting Tools to Aster Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Connect Aqua Data Studio to Aster Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Connect MicroStrategy to Aster Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Chapter 4: Tools for .NET Environments . . . . . . . . . . . . . . . . . . . . . . . . . 97 Installing the Aster Database ADO.NET Driver (nClusterDNProvider) . . . . . . . . . . . . . . . . . 97 Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Installing nClusterDNProvider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Sample Program to Demonstrate the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Performing SSIS Data Loading with nClusterDNProvider . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Aster Client Guide 5 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Sample Program for ADO.NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Features Demonstrated in the Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Steps to Build and Test the Sample Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Source Code of Program to Demonstrate nClusterDNProvider . . . . . . . . . . . . . . . . . . . 113 Possible Exceptions and Resolutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Chapter 5: Data Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Best Practices for Data Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Loading Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Scenario 1: Pre-Production Data Loading with Aster Database . . . . . . . . . . . . . . . . . . . . 122 Scenario 2: Loading in a Production Environment with Aster Database . . . . . . . . . . . . 123 Loading Best Practices Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Aster Database Loader Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Argument Flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Exit Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Install the Loader Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Load Data with the Loader Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Load from Multiple Files Using a Map File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Hints for Successful Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Error Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Troubleshoot Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Running Multiple Loaders Concurrently . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Load stalls upon cancellation or failure encountered during a load . . . . . . . . . . . . . . . . 143 Load fails on UNIQUE or PRIMARY KEY violation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Invalid Input Syntax Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Single quote character must be escaped when using the -q option . . . . . . . . . . . . . . . . . 143 Using the -C option with uppercase or special characters . . . . . . . . . . . . . . . . . . . . . . . . 144 Uppercase characters are passed as lowercase if not escape quoted . . . . . . . . . . . . . . . . . 144 Chapter 6: Export and Load Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Aster Database Export Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Argument Flags for Exporter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Installing Aster Database Export . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6 Aster Client Guide Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Aster Client Guide 7 8 Aster Client Guide Preface This guide provides instructions for users and administrators of Aster Client, version AC 5.10. If you’re using a later version, you must download a newer edition of this guide! The following additional resources are available: • Aster Database upgrades, clients and other packages: http://downloads.teradata.com/download/tools • Documentation for existing customers with a Teradata @ Your Service login: http://tays.teradata.com/ • Documentation that is available to the public: http://www.info.teradata.com/ Conventions Used in This Guide This document assumes that the reader is comfortable working in Windows and Linux/UNIX environments. Many sections assume you are familiar with SQL. This document uses the following typographical conventions. Typefaces Command line input and output, commands, program code, filenames, directory names, and system variables are shown in a monospaced font. Words in italics indicate an example or placeholder value that you must replace with a real value. Bold type is intended to draw your attention to important or changed items. Menu navigation and user interface elements are shown using the User Interface Command font. SQL Text Conventions In the SQL synopsis sections, we follow these conventions: Aster Client Guide • Square brackets ([ and ]) indicate one or more optional items. • Curly braces ({ and }) indicate that you must choose an item from the list inside the braces. Choices are separated by vertical lines (|). • An ellipsis (...) means the preceding element can be repeated. 9 Contacting Technical Support • A comma and an ellipsis (, ...) means the preceding element can be repeated in a commaseparated list. • In command line instructions, SQL commands and shell commands are typically written with no preceding prompt, but where needed the default Aster Database SQL prompt is shown: beehive=> Command Shell Text Conventions For shell commands, the prompt is usually shown. The $ sign introduces a command that’s being run by a non-root user: $ ls The # sign introduces a command that’s being run as root: # ls Contacting Technical Support For further assistance, contact Teradata technical support. Support Portal: http://tays.teradata.com/ Email: [email protected] Telephone: +1-650-273-5599 About Teradata Aster Teradata Aster provides data management and advanced analytics for diverse and big data, enabling the powerful combination of cost-effective storage and ultra-fast analysis of relational and non-relational data. Teradata Aster is a division of Teradata and is headquartered in San Carlos, California. For more information, go to http://www.asterdata.com About This Document This is the “Aster Client Guide,” version AC 5.10, edition 1. This edition covers Aster Database Clients version AC 5.10 and was published April 17, 2013 10:35 pm. Document revision history: December 2012: 5.0.3 April 2013: AC 5.10 Aster Client Guide 10 Install Aster Database Tools and Utilities CHAPTER 1 This section explains how to install various utilities that complement your Aster Database installation. • Install the Aster Database Cluster Terminal (ACT) (page 11) • Install the Loader Tool (page 133) Install the Aster Database Cluster Terminal (ACT) ACT is a terminal-based query tool that connects with Aster Database. ACT lets you type queries, see the results, and save them to a file. Alternatively, you can source your queries from a file. ACT supports connection via SSL and SSO and is available for Windows, Linux, Solaris and Mac. Follow the instructions for your OS: • “Obtain ACT” on page 11 • “Install ACT on Windows” on page 12 • “Install ACT on Linux” on page 12 • “Install ACT on MacOS” on page 13 • “Install ACT on Solaris” on page 13 Obtain ACT ACT is installed automatically on the queen during the installation of Aster Database. By default, the install directory is /home/beehive/clients/act. To launch ACT from the queen, see “Launch ACT Directly on the Queen” on page 17. If you want to install ACT on another machine, get the file for the client operating system and copy it to the computer you'll use to query your database. You may obtain the file in one of two ways: • Aster Client Guide To get the newest package, download it from http://downloads.teradata.com/download/tools 11 Install Aster Database Tools and Utilities Install the Aster Database Cluster Terminal (ACT) • On your queen node, you can find the installers in the directory /home/beehive/ clients_all/<your_client_OS>. Install ACT on Windows To install ACT on Windows: 1 Obtain ACT. 2 Uncompress the tar file and place the executable act.exe in a directory on your client machine. Teradata Aster recommends that you create a directory, C:\Program Files\asterdata and place the act.exe executable there. 3 Set permissions to make the file executable: • In Windows File Explorer, navigate to find the ACT executable file. • Right-click on it, select Properties, click Security. • In the To change permissions section, click Edit. • Click the name of the group who can use the application, and tick the checkbox for Allow/Read & Execute in the bottom of the window. • 4 Click OK and click OK again. Test your ACT installation like this: • From the windows Start menu, choose Run..., type cmd, and click OK. • At the command line, change directories to the folder holding act.exe. • Type act --help at the command line. The help page should be printed to the command line. • To connect to your Aster Database, type act -h <ip address of your queen> • When prompted, type the database username and password (The default user/ password credentials are beehive/beehive) The <dbuser>=> prompt indicates you've logged in successfully (i.e. beehive=> for the default user). Type \q to quit. • See the section “Launch ACT” on page 16 for information on running ACT. Install ACT on Linux To install ACT on Linux: 1 Obtain ACT. 2 Uncompress the tar file and place the executable in a directory on your client machine. Teradata Aster recommends that you use the directory, /home/beehive/clients. 3 Change the file’s permissions to make it executable: $ chmod 755 act Tip! ACT for Linux requires glibc version 2.6.18 or higher. If you do not have glibc version 2.6.18 or higher, you must use the IP address instead of the hostname for the -h flag when running ACT. To check the version of glibc, issue the command ldd --version. 12 Aster Client Guide Install Aster Database Tools and Utilities Install the Aster Database Cluster Terminal (ACT) 4 See the section “Launch ACT” on page 16 for information on running ACT. Install ACT on MacOS To install ACT on MacOS: 1 Obtain ACT. 2 Double-click the .dmg file. A new icon with a name similar to the .dmg file name appears on your desktop. 3 If a new Finder window doesn’t automatically appear, double click on the new icon that has appeared on your desktop. A new Finder window appears. 4 Find the application’s icon in the Finder window. Drag and drop it into your Applications directory. 5 See the section “Launch ACT” on page 16 for information on running ACT. Install ACT on Solaris To install ACT on Solaris: 1 Obtain ACT. 2 Use pkgadd to install ACT. Here is an example, using a SunOS 5.10 operating system: $ pkgadd -d nCluster_AC 5.10_client_SunOS_5.10.pkg all 3 Aster Client Guide See the section “Launch ACT” on page 16 for information on running ACT. 13 Install Aster Database Tools and Utilities Install the Aster Database Cluster Terminal (ACT) 14 Aster Client Guide Aster Database Cluster Terminal (ACT) CHAPTER 2 This section explains how to use Aster Database Cluster Terminal (ACT) to query and manage databases. ACT is a terminal-based query tool that connects with Aster Database. ACT lets you connect to the database (optionally using SSO and/or SSL), type queries, issue them to Aster Database, and get query results. Alternatively, you can source your queries from a file. ACT can return your query results to the command line or to a file, which makes it useful for extracting data. Meta-commands and shell-like features are provided to facilitate writing scripts and automating tasks. Tip! Beginning with ACT version 4.6, the ACT client cannot connect to versions of Aster Database prior to version 4.6. If you attempt to connect to a pre-4.6 version of Aster Database with a 4.6 or later version of ACT, you will see an error message indicating that there is a version mismatch between Aster Database and the client. You should obtain the version of ACT that matches the version of Aster Database to which you are attempting to connect. The following sections explain how to launch and use ACT: • ACT Quick Start (page 15) • Install ACT (page 16) • Launch ACT (page 16) • Startup Parameters for ACT (page 18) • How to Use ACT (page 24) • ACT Commands (at the SQL Prompt) (page 30) • Troubleshooting ACT (page 40) ACT Quick Start You run ACT from the command line. The launch command is: $ act -d <db name> -h <hostname> -U <username> [-w <password>] [argument flags] where Aster Client Guide 15 Aster Database Cluster Terminal (ACT) Install ACT • db name is your database name, • hostname is the name or IP address of the queen, • username is your SQL login name, • password is your SQL password in Aster Database (this is optional; ACT prompts you for a password if you do not pass a -w parameter), and • argument flags are any of the parameter/value combinations listed in “Startup Parameters for ACT” on page 18. For example: $ act -d emea_sales -h 10.48.58.100 -U mjones Tip! When using SSO (single sign-on), the -U and-w options are not used, because the username and password are passed directly to the host via SSO. To log in to the default database that is provided in your installation, type this, replacing the IP address with the hostname or IP address of your Aster Database queen: $ act -d beehive -h 10.42.52.100 -U beehive -w beehive To see a list of ACT command line arguments, type: $ act --help Install ACT You can install ACT on Linux, Windows, Solaris, or Mac. Launch ACT See the appropriate section below for instructions on launching ACT: • “Launch ACT on Windows” on page 17 • “Launch ACT on Linux or Solaris” on page 17 • “Launch ACT on Mac” on page 17 • “Launch ACT Directly on the Queen” on page 17 Tip! On an Aster Database where LDAP authentication is enabled, if during logon an ACT user gets the error message: 'ERROR: An internal error has occurred.', make sure the username is present in Aster Database with proper privileges. • 16 “Log In to ACT” on page 18 Aster Client Guide Aster Database Cluster Terminal (ACT) Launch ACT Launch ACT on Windows 1 Make sure ACT is installed on your client machine. 2 Open a command prompt. 3 Change directories to the folder which contains the act.exe executable. 4 Log In to ACT. Launch ACT on Linux or Solaris 1 Make sure ACT is installed on your client machine. 2 Ensure that your path includes the directory where ACT is installed. 3 Log In to ACT. Tip! ACT for Linux requires glibc version 2.6.18 or higher. If you do not have glibc version 2.6.18 or higher, you must use the IP address instead of the hostname for the -h flag when running ACT. To check the version of glibc, issue the command ldd --version. Launch ACT on Mac 1 Make sure ACT is installed on your client machine and that its directory is in your path. 2 Open a terminal. 3 Log In to ACT. Launch ACT Directly on the Queen Your Aster Database queen also contains an installation of the ACT client. To run it there, you’ll need a user account on the queen machine, and you’ll need an SSH client on your workstation. To run the ACT on the queen: 1 Open a SSH connection to the queen. If you do not have a user account on the queen, ask the machine’s administrator for one. Tip! If you need an SSH client, do one of the following: • Install the OpenSSH client as explained on the OpenSSH homepage at http://www.openssh.org/ • or, for Windows only, install the PuTTY SSH client. 2 Change directories to the directory where ACT is installed (by default, /home/beehive/ clients. 3 Log in to ACT: $ act -d <db name> -U <username> -w <password> [argument flags] Note that if you do not provide the hostname using -h, ACT defaults to the localhost. For details on the command line options, see “Startup Parameters for ACT” on page 18 Aster Client Guide 17 Aster Database Cluster Terminal (ACT) Startup Parameters for ACT Log In to ACT 1 Run ACT by typing a command like: act -d <db name> -h <hostname> -U <username> [-w <password>] [argument flags] For details on the command line options, see “Startup Parameters for ACT” on page 18 2 Provide your database password by: • adding –w <password> to the ACT login string, or • omitting the -w argument and providing your database password at the prompt. 3 Choose a database by adding -d <database name> to the ACT login string. If -d is not used, ACT places you in the system database (with the default name “beehive”). 4 You will see a welcome message, followed by the database prompt, which shows the database name, followed by “=>”. For example: Welcome to act AC 5.10, the Aster Database Terminal. beehive=> Startup Parameters for ACT ACT takes a variety of startup parameters when you launch it. (Don’t confuse these with ACT’s at-the-SQL-prompt flags, which you can read about in “ACT Commands (at the SQL Prompt)” on page 30.) You can pass these parameters at the command-line (see “Using the “on-error-stop” Option in ACT” on page 22) or in a configuration file or “Use a Configuration File to Pass ACT Startup Parameters” on page 22). The same startup parameter cannot be repeated in a single invocation of ACT, or an error will be returned. To list the command line parameters, type act --help. 18 Aster Client Guide Aster Database Cluster Terminal (ACT) Startup Parameters for ACT Descriptions of each parameter are below: Table 2 - 1: Summary of the most common command-line parameters for ACT Flag Description -d [ --dbname ] DBNAME Specify database name to connect to (default: “beehive”). -h [ --host ] HOSTNAME Aster Database server host (default: “localhost”). -U [ --username ] NAME Aster Database username (default: “beehive”). Not used with SSO. -l (the letter “ell”) List available databases, then exit. [ --list-databases ] Tip! Note the default values for the connection parameters. If you do not specify the parameters -d (database name), -h (hostname), -U (username), and/or -p (port) in the connect string, ACT will use the default values. The default values are: • “beehive” for database name • “localhost” for hostname • “beehive” for username, and • “2406” for port. If -w is not used, ACT will prompt for a password. Table 2 - 2: General-purpose command-line parameters for ACT Flag Description -d [ --dbname ] DBNAME Specify database name to connect to (default: “beehive”). --config-file FILENAME Lads startup parameters from a configuration file specified by FILENAME. See “Use a Configuration File to Pass ACT Startup Parameters” on page 22 for more information. -c [ --single-command ]COMMAND Run only single command (SQL or internal) and exit. For example: act -c "COPY MyTable FROM stdin;" < myDataFile.dat -f [ --input-file ] FILENAME Execute commands from file, then exit. Run a SQL script. -1 (the numeral “one”) Execute command file as a single transaction. [ --single-transaction ] -l (the letter “ell”) List available databases, then exit. [ --list-databases ] Aster Client Guide -? [ --help ] Show command line help, then exit. -V [ --version ] Output version information, then exit. 19 Aster Database Cluster Terminal (ACT) Startup Parameters for ACT Table 2 - 2: General-purpose command-line parameters for ACT (continued) Flag Description --on-error-stop Enables the “on-error-stop” option, by default this option is disabled. See “Using the “on-error-stop” Option in ACT” on page 22 for more information. or -E Table 2 - 3: Input- and output-related command-line parameters for ACT Flag Description -a [ --echo-script-input ] Echo all input from script. -e [ --echo-all-input ] Echo commands sent to server. -o [ --redirect-query-results ] FILENAME Send query results to file (or | pipe). Table 2 - 4: SSL and SSO related command-line parameters for ACT 20 Flag Description --enable-ssl Enables Secure Socket Layer (ssl) support. Must be used if any of the other SSL/SSO arguments are used. --ssl-encrypt-reads SSL Encrypt Reads. Must be used if secureWrites=true on the server. Conversely, must not be used if secureWrites=false on the server. See How to Set Configuration Parameters on the Queen (page 85) for information on how to set the secureWrites parameter on the server. --ssl-self-signed-peer Indicates that ACT will connect to a Queen which will provide a self-signed certificate. --ssl-private-key-path PATH The SSL Private Key Path indicates where the private key is stored on the client (ACT) machine. --ssl-certificate-path PATH The SSL Certificate Path indicates where the certificate is stored on the client (ACT) machine. --ssl-trusted-ca-dir DIRECTORY When using a chain of certificates rather than a single certificate, use the SSL Trusted CA Dir to set the directory on the client machine where the chain of trusted certificates is stored. --ssl-trusted-ca-file FILENAME Use SSL Trusted CA Filename to provide the location of the signed copy of the server’s certificate on the client machine. --ssl-cert-filetype ARG SSL Certificate File Type (use 1 for PEM; 2 for ANS1; default: 0). --enable-sso Enables Single sign-on (SSO) support. Aster Client Guide Aster Database Cluster Terminal (ACT) Startup Parameters for ACT Table 2 - 4: SSL and SSO related command-line parameters for ACT (continued) Flag Description --gss-lib-path PATH For Linux, sets the GSS shared library path (default on linux is /opt/guest/lib32 or /opt/guest/lib64). Ignored on Windows. Tip! The SSL settings in ACT have interdependencies, and in most cases they rely on the SSL settings on the queen. See Common SSL Configurations (page 36). Table 2 - 5: Output format-related command-line parameters for ACT Flag Description -q [ --quiet ] Run quietly and do not print messages, only query output. Use this for clean query output. Often used with the -c flag. -t [ --print-rows-only ] Print rows only. -x [ --expanded ] Turn on expanded table output. -A [ --unaligned ] Turn on unaligned table output -F [ --field-separator ] ARG Set field separator (default: '|') Table 2 - 6: Connection-related command-line parameters for ACT Flag Description -h [ --host ] HOSTNAME Aster Database queen hostname or IP address (default: "localhost"). Note that ACT supports glibc version 2.6.18 or higher. If you do not have glibc version 2.6.18 or higher, you must use the IP address instead of the hostname. To check the version of glibc, issue the command ldd --version. When using SSO, you should specify a fully qualified hostname using the -h option, as in the example: <hostname>.<domain>.<com|org etc>. If only the hostname is used with SSO, ACT will append the local domain name before attempting to look up the host. Using an IP address with -h is not supported with SSO. Aster Client Guide -p [ --port ] PORT Aster Database server port (default: "2406"). -U [ --username ] USERNAME Aster Database username (default: "beehive"). -w [ --password ] PASSWORD Aster Database password. This parameter is optional; ACT will prompt for a password if you do not pass a -w parameter. Not used with SSO. 21 Aster Database Cluster Terminal (ACT) Startup Parameters for ACT Using the “on-error-stop” Option in ACT The “on-error-stop” option can be used to stop ACT if an error occurs while running queries. It discontinues executing the multi-statement SQL or the SQL file. When the “on-error-stop” option is set, ACT halts query processing and exits as follows: • If ACT is in interactive mode, it returns to the ACT prompt when meeting an error. • If ACT is executed in the command line, it exits with status “3” when meeting an error. Note! Unless the “-f” or “-c” options are used, ACT is always in interactive mode once it is launched. The “\set” command can be used at the command line at any time. For more details on the \set syntax for the “on-error-stop” option, see “Database Parameters Set with \set” on page 35. Setting “on-error-stop” in the command line To enable the “on-error-stop” option (by default this option is disabled) from the command prompt, use either of the following parameters: • --on-error-stop • -E For example, to enable the “on-error-stop” in a running session of ACT, type the following at the ACT prompt: beehive$ ./act -h 153.65.197.120 -U beehive -w beehive -E -f test_queries.sql beehive$ ./act -h 153.65.197.120 -U beehive -w beehive --on-error-stop -f test_queries.sql Use a Configuration File to Pass ACT Startup Parameters Specifying all the necessary startup parameters for ACT on the command line can become cumbersome, especially when using SSL. Because of this, Teradata Aster has provided a way to specify startup parameters in a configuration file, to streamline starting ACT. On Windows and Linux/UNIX-based operating systems, ACT looks for the configuration file in a specific location and loads it automatically. Alternatively, the configuration file may be invoked using the --config-file parameter when launching ACT. All of the startup parameters in ACT are supported in the config file, except for the following: • --config-file • -V [ --version ] • -? [ --help ] To use a configuration file, first create a text file of startup parameters. The following rules apply when creating the config file: 1 Lines starting with a # character are ignored (considered as comments). 2 Blank lines are ignored (including lines containing just spaces). 3 Parameters are entered using the format flagname: value 22 Aster Client Guide Aster Database Cluster Terminal (ACT) Startup Parameters for ACT where flagname is same as the name of the command line flag without the preceding hyphens (--) and value is the flag value as it would be provided on the command line. Note that the short notations of flags are not supported. For example: host: <ip> will work but the following: h: <ip> will not work. 4 Flags which do not take any argument on the command line should be given a value of either true or false. 5 Flag names are case-sensitive. 6 If the config file includes invalid flag names or repeated entries, ACT will not launch, and an error will display. 7 If the config file includes the “on-error-stop” option with the parameters set to enable this option, ACT will stop if an error occurs while running SQL queries. See Set “on-errorstop” in the ACT config file (page 23) for information on setting this option. Set “on-error-stop” in the ACT config file To enable the “on-error-stop” option, add it to the ACT config file: "actconfig.ini" and set the value to "true". Set the value to "false" to turn off this option. By default, this feature is disabled or “off ”. The following is an example from of a config file for ACT: # ACT configuration file example # Contains setting to enable on-error-stop host: 10.10.10.10 dbname: sampledb username: beehive on-error-stop: true The default location for the config file on Linux/UNIX-based systems is $(HOME)/ .actconfig. For Windows, the default location for the config file is %HOMEPATH%/ actconfig.ini. Upon launching, ACT will look for the config file in the default location, and if found, will use it by default. Command line flags can be specified when starting ACT, in addition to the flags in the configuration file. If a flag is present in the config file and specified at the command line as well, the value on the command line will override the value in the config file. ACT will notify the user that the command line flag was used upon startup. The following is an example of a config file for ACT: # ACT configuration file example # Contains settings for connecting securely to a specific host and database host: 10.10.10.10 dbname: sampledb username: sampleuser # SSL settings Aster Client Guide 23 Aster Database Cluster Terminal (ACT) How to Use ACT enable-ssl: true ssl-self-signed-peer: true ssl-encrypt-reads: false To start ACT, explicitly using the config file, issue a command like this example: $ act --config-file /home/beehive/.act_ssl_config To start ACT, explicitly using the config file and also specifying an additional parameter to redirect query results to a file for this session only, issue a command like this example: $ act --config-file /home/beehive/.act_ssl_config -o /home/beehive/ query_results_file How to Use ACT Issue SQL Queries To use ACT to run a SQL query against Aster Database, you can do one of the following: • Enter the query at the ACT command line • Run the query from a file However, before you can run queries, you must first launch ACT, as described in “Launch ACT” on page 16. For example, to run queries against the retail_sales database, run this command on the Queen to launch ACT: [root@localhost ~]# act -d retail_sales –U beehive –w beehive Welcome to act AC 5.10, the Aster nCluster Terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help with act commands \g or terminate with semicolon to execute query \q to quit retail_sales=> To list the tables in the database, enter \d at the ACT prompt (in this case, retail_sales=>). For example: retail_sales=> \d List of relations Schema | Name | Type | Owner --------+--------------+-------+--------public | customer_dim | table | beehive public | date_dim | table | beehive public | geo_dim | table | beehive public | product_dim | table | beehive public | region_dim | table | beehive public | sales_fact | table | beehive public | store_dim | table | beehive (7 rows) 24 Aster Client Guide Aster Database Cluster Terminal (ACT) How to Use ACT Run queries from the command line To control how ACT handles errors when running single or multi-line SQL queries from the command line, see “Using the “on-error-stop” Option in ACT” on page 22. To run a single-line SQL query from the command line: 1 Type the query at the ACT prompt. Note! The query must end with a semicolon. 2 Press Enter. For example, to list all the rows of the customer_dim table of the retail_sales database and order the results based on gender, enter this command at the ACT command-line prompt: retail_sales=> select customer_id, gender, city_id from customer_dim order by gender; customer_id | gender | city_id -------------+--------+--------743 | F | 46 2711 | F | 124 744 | F | 66 To run a multi-line SQL query from the command line: 1 Type the first line of the query, then press Enter. The ACT prompt changes from => to -> (for example, retail_sales->). 2 Enter the remaining lines. You can press Ctrl-C at any time to abandon this command mode without executing the query. 3 On the last line, type a semicolon at the end of the line, then press Enter to run the query. For example: retail_sales=> select customer_id, gender, city_id retail_sales->from customer_dim retail_sales->order by gender; or retail_sales=> select customer_id, gender, city_id retail_sales->from customer_dim retail_sales->order by gender retail_sales->; Run queries from files To control how ACT handles errors when running SQL queries from a file, see “Use a Configuration File to Pass ACT Startup Parameters” on page 22. To run a query stored in a file: 1 Make sure the filename ends with .sql. The query does not have to end with a semicolon. 2 At the prompt, enter this command: \i filename Aster Client Guide 25 Aster Database Cluster Terminal (ACT) How to Use ACT For example, consider this query stored in the file myQuery.sql: SELECT * FROM customer_dim ORDER BY gender To run this query, enter this command at the ACT command prompt: retail_sales=> \i myQuery.sql You can also run this query using the –f flag of the act command. For example: [root@localhost ~]# act -d retail_sales -w beehive -f myQuery.sql Workaround for Multibyte Characters ACT does not accept multibyte characters from the SQL prompt. For example, the following statement will produce a syntax error: beehive=> create database db The workaround to input multibyte characters is as follows: 1 2 Input the multibyte characters from an SQL file instead of at the SQL prompt: a Create a file containing the SQL to insert the multibyte data (for example, insert_file.sql). b Execute it by issuing \i insert_file.sql. Or, use the SQL command buffer in ACT: a Input the data using the query buffer, by issuing the ACT \e command, which brings up the default text editor. b Type in the SQL, including the multibyte characters. c Execute the SQL by exiting the text editor (for example by typing :wq if your default text editor is vi.) Exit ACT To quit ACT, type \q and hit <Enter>. Page Through Query Results When ACT returns results to the screen, it (by default) prints one page of results at a time. Hit the space bar to display the next page of results, hit <Enter> to display just one more line of results, and type “q” to quit looking at results and return to the SQL prompt. When you get to the last page of results, ACT displays the text, “(END)”. Type “q” to return to the SQL prompt. Throttle Query Results in ACT and Aster Database To reduce the memory footprint of ACT and other Aster Database clients, you can set a fetchcount that constrains the number of rows returned at one time when you select from a large table. You can limit the total number of rows returned by using fetch-limit. Additionally, you can make your query results stream more efficiently by using the server-side cursors feature. In 26 Aster Client Guide Aster Database Cluster Terminal (ACT) How to Use ACT ACT, you can set these parameters using \set, and in other clients you can typically set them in your data source definition or parameters file. Let’s look at fetch-count first. Reduce Memory Use With Fetch-Count The purpose of using fetch-count is to reduce the memory footprint of ACT on the server. Therefore, this type of configuration would normally be done by an administrator, or at least the setting would be made under an administrator’s guidance. To set a fetch-count, you specify a value for the fetch-count parameter in ACT. This causes ACT to fetch rows in sets, with each set containing only the user-specified number of rows (or fewer rows). ACT uses a fetch count by default, i.e. even when fetch-count is not set explicitly. The fetch-count (number of rows per fetch) should always be set to greater than 0, The default value is 1024 rows. To set the fetch-count in a running session of ACT, use the \set command to set the fetch-count parameter. To do this, type the following at the ACT prompt: \set fetch-count n where n is the maximum number of rows ACT should return at a time. To enforce the fetch-count, ACT uses server side cursors to fetch results, which can help prevent the memory footprint of ACT from growing too large. Note that to the user, the results returned will not be different when using fetch-count. The purpose is simply to reduce the memory footprint of ACT on the server. Limit Caching With Server-Side Cursors Setting use-server-cursors has the same effect as declaring a server-side cursor with the CURSOR syntax in SQL. When server-side cursors are activated, one batch of data is retrieved at a time. This lets the queen avoid caching the entire result set and lets the workers continue performing computations while the queen is retrieving rows. As the result set becomes available from a worker, the queen sends it to the client. Depending on the type of query, results can start streaming back to the client immediately (e.g. SELECT * WHERE...) or the results may have to be computed in their entirety (e.g. GROUP BY) before streaming to the client. Regardless of whether the results begin to stream to the client before the query has finished executing, server-side cursors can improve performance significantly for large result sets. If you want to start streaming results of a query as soon as possible, you can benefit from setting use-server-cursors and specifying a fetch-count. By default, use-server-cursors is set to 0 (off). Example of Fetch-Count With Server-Side Cursors The following steps are an example of how to use server-side cursors and fetch-count together: 1 Enable server cursors by issuing the following command inside ACT (a setting of 1 turns cursors “on” and a setting of 0 means “off ”): beehive=> \set use-server-cursors 1 2 Aster Client Guide Set a fetch-count of 100: 27 Aster Database Cluster Terminal (ACT) How to Use ACT beehive=> \set fetch-count 100 3 Issue the following SQL query: SELECT A,B,COUNT(*) FROM lineitem GROUP BY a, b; This will actually cause the following statements to be run: BEGIN; DECLARE x FETCH 100 FETCH 100 ... ... FETCH 100 CLOSE x; END; CURSOR FOR SELECT A,B,COUNT(*) FROM LINEITEM GROUP BY A,B; FROM x; // 100 FROM x; // 200 FROM x; // until all the rows have been fetched Aster Database reports all the fetched rows as the row count for this query. Set the Maximum Number of Rows Returned With Fetch-Limit To limit the total number of rows returned by each query, set the fetch-limit in ACT. fetchlimit (the total number of rows fetched for a query) can be set to any value. A value less than 0 implies fetch all rows. A value greater than 0 implies fetch-limit rows in total. The default value is -1 (all rows). Tip! When fetch-limit is used, the total row count returned for the query will be the total row count returned by the query or the row count specified by fetch-limit, whichever is smaller. For example, if a query normally returns 35,453 rows, but you have specified a fetch limit of 1000, the query will return 1000 rows (and it will display “1000 rows returned”). There will be no indication that there were in fact 35,453 rows that would have been returned had you not had a fetch limit set. In ACT, you set the fetch-limit by typing: \set fetch-limit n where n is the maximum number of rows a single query will be allowed to return. The SQL LIMIT Clause vs. Fetch-Limit and Fetch-Count Setting a fetch-limit is an alternative to specifying a LIMIT clause explicitly in your SELECT query, with an important difference in how the query will be executed, which can affect performance. fetch-limit is an ACT feature which controls how many rows are fetched by the queen from the workers and by the client from the queen. ACT stops fetching results from the cursor after the specified limit is reached. LIMIT is a SQL clause which controls how many rows are computed on the worker(s) or the queen. The following examples illustrate the difference between using fetch-limit and using a LIMIT clause in the SQL query: Consider the statement: SELECT * FROM FOO, BAR where FOO.A = BAR.A; Example 1: Using fetch-limit and fetch-count To issue the query in ACT with a fetch-count of 100 and a fetch-limit of 1000: 28 Aster Client Guide Aster Database Cluster Terminal (ACT) How to Use ACT \set fetch-count 100 \set fetch-limit 1000 SELECT * FROM FOO, BAR where FOO.A = BAR.A; Here is how the query will execute: 1 ACT opens a cursor on the query. 2 The queen activates Parallel Cursors and passes the query SELECT * FROM FOO, BAR where FOO.A = BAR.A; to the workers. 3 Each worker then computes the entire equi-join as dictated by the FOO.A = BAR.A constraint. 4 The queen fetches results from workers in batches of 100 rows, until a limit of 1000 is reached. 5 ACT fetches results from the queen in batches of 100 rows, until a limit of 1000 is reached. Example 2: Using a LIMIT clause in the SQL query To issue the same query in ACT using the SQL LIMIT clause to limit results to 1000 rows: SELECT * FROM FOO, BAR where FOO.A = BAR.A LIMIT 1000; Here is how the query will execute: 1 ACT issues the SQL statement to the queen. 2 The queen planner identifies LIMIT 1000 and pushes down the following query to each of the workers: SELECT * FROM FOO, BAR where FOO.A = BAR.A LIMIT 1000; 3 Each worker computes the equi-join as dictated by the FOO.A = BAR.A LIMIT 1000 constraint. This allows for optimizations where the entire equi-join computation need not be done at each worker. 4 The queen fetches results from the workers. 5 ACT fetches results from the queen. The difference between the two approaches occurs in step 3 of both examples. Using the SQL LIMIT clause (Example 2), the workers are allowed to compute with the constraint of LIMIT 1000, whereas in Example 1 they have to compute the entire equi-join. So, as you can see, fetch-limit via server-side cursors does not translate into the workers doing a LIMIT 1000 on their individual slice of data. Therefore, if the use case calls for it, an Aster Database power-user should be aware that using the SQL LIMIT clause can speed up query execution dramatically in Aster Database. ACT Utility Commands ACT also offers many utility commands that provide assistance and carry out useful utility operations such as uploading files and changing output options. These commands start with a backslash character, and we describe them in the section, “ACT Commands (at the SQL Prompt)” on page 30. Aster Client Guide 29 Aster Database Cluster Terminal (ACT) ACT Commands (at the SQL Prompt) Repeat Previously Typed Commands Hit the up arrow on your keyboard to toggle through your history of previously typed commands. To edit a previously typed command, just use the usual combination left arrow, right arrow, delete, and backspace keys to make your edits, and type <Enter> to finish editing the line. If needed, type “;” to issue the command. To list your command history on Linux, type \s. Tip! If you receive an “Error writing history to file.” error on Linux when attempting to view command history with \s, check that the current Linux user has permissions to write to the current working directory. Tab Completion The UNIX/Linux version of ACT can tab-complete SQL commands and table names that you type. Tab-completion is not available in the Windows version of ACT. To use this feature, type the first couple of letters of a command and hit the <Tab> key. If the completion is unambiguous, ACT completes the command. If ACT doesn’t complete the command, hit <Tab> again and ACT prints all the possible completions. Using the list as a reference, type enough additional characters to unambiguously identify the desired command or table, and hit <Tab> again to complete it. Here are a few common uses of tab completion: • To complete common SQL commands. For example, type “se” and hit <Tab> to type SELECT. • To list various ACT utility commands. For example, type “\”and hit <Tab> to show all the commands, or type “\d” and hit <Tab> to show all the commands that start with “d”. • To complete a table name. For example, type “SELECT * FROM sa” and hit <Tab> to complete the table name or hit <Tab> twice show all the table names that start with “sa”. You can also list the names of all tables in the database by typing “SELECT * FROM ” (note the trailing space) and hitting <Tab> twice. ACT Commands (at the SQL Prompt) The ACT SQL prompt accepts a number of ACT-specific commands that you issue by typing a backslash followed by a character or combination of characters and arguments. Do not type a semicolon to conclude these commands! Hitting <Enter> executes the command. Note! Don’t confuse these with the ACT command-line startup parameters (also known as “command line flags”), which you type at the shell command line when you launch ACT. For a list of those, see “Startup Parameters for ACT” on page 18. We describe the SQL-prompt commands in the tables that follow: 30 • General-purpose utility commands in ACT (page 31) • Environment settings in ACT (page 31) • Query Buffer Commands in ACT (page 32) • Input/output-related commands in ACT (page 32) Aster Client Guide Aster Database Cluster Terminal (ACT) ACT Commands (at the SQL Prompt) • Installed-function and installed-file management commands in ACT (page 32) • Informational commands in ACT (page 33) • Formatting-related commands in ACT (page 34) Table 2 - 7: General-purpose utility commands in ACT Command Description \? prints help for ACT commands. \c[onnect] HOST PORT \c[onnect] HOST \c[onnect] \c[onnect] DBNAME USER DBNAME USER DBNAME USER DBNAME change login credentials and/or connect to a new database. The parameters must be specified in the order shown, with a space before each, and parameters may not be skipped. In other words, if only one parameter is specified, it is understood to be DBNAME; if a second parameter is also specified, it is understood to be USER; and so on. \cd [DIR] change the current working directory. \copyright show ACT usage and distribution terms. \h help with SQL commands. \h [SQL command name] help with syntax of the specified SQL command, * for all commands. \g or terminate with a semicolon (;) to execute query. \q quit ACT. \! [command] execute command in shell or start interactive shell. \password change the password for the current user. Table 2 - 8: Environment settings in ACT Aster Client Guide Command Description \info display current environment settings. \set display current ACT parameter settings. \set param-name [param-value] set ACT parameter setting param-name to value paramvalue. (For example, “\set fetch-count 500” tells ACT to fetch no more than 500 rows at a time when selecting.) If no parameter value is supplied, displays the current setting for the specified parameter. \timing [on|off] toggle or set timing of commands. \pager [on|off] toggle or set to use pager to enable paging through large result sets. 31 Aster Database Cluster Terminal (ACT) ACT Commands (at the SQL Prompt) Table 2 - 9: Query Buffer Commands in ACT Command Description \e [FILE] edit the query buffer (or file) with external editor. On most systems, this launches your default text editor. When you save and exit the editor, the edited statement is passed back to ACT for running. \g [FILE] send query buffer to server (and results to file or | (pipe character)). \p show the contents of the query buffer. \r reset (clear) the query buffer. \w FILE write query buffer to file. Table 2 - 10: Input/output-related commands in ACT Command Description \echo [STRING] write string to query output stream (see \o below). \i [FILE] execute SQL commands from SQL script file. (Run an SQL script.) \o [FILE] redirect all query results to file or | (pipe character). \o Type \o with no argument to stop sending results to a file and resume sending them to the ACT shell. \s [FILENAME] display command history in Linux (optionally, print history to a file specified by FILENAME) Note that query history includes only the first 2048 characters of each query. Table 2 - 11: Installed-function and installed-file management commands in ACT Command Description \dF list installed files, SQL-MapReduce functions, and other functions in the current schema. Use a regular expression as an argument to display a subset of the available functions. For example, to view all installed functions in the database, issue: \dF *.* where the first asterisk means "all schemas" and the second means "all functions and files." Alternatively, you can examine the system views. \dF+ show details for all installed files, SQL-MapReduce functions, and other functions in the current schema. For each function, the output shows the name, schema, owner, upload time, and MD5 Hash fingerprint of the function. Use a regular expression as an argument to display a subset of the available functions. For Example, type \dF+ *.* to show details for functions and files in all schemas in the database. Alternatively, you can examine the system views. 32 Aster Client Guide Aster Database Cluster Terminal (ACT) ACT Commands (at the SQL Prompt) Table 2 - 11: Installed-function and installed-file management commands in ACT (continued) Command Description \dE show all the installed SQL-MR functions for which the current user has privileges. Use a regular expression as an argument to display a subset of the available functions. Shows function name, schema, owner, function version and creation time. \install <FILE> [[<SCHEMA>/ ]<FILE_ALIAS> ] install the file or SQL-MapReduce function in Aster Database. The file must be available on the file system where ACT is running. Note that the database user running this command must have permission to install files and functions in the specified schema. You cannot install two files or functions with the same name. If attempting to do this, you must follow these steps: • remove the existing file or function • install the new file or function • grant the appropriate privileges on the file or function. There is a limit of 238MB on the size of the file to be installed. If you try to install a larger file, you will see an error like: ERROR: row text exceeds limit of 238MB ... Note that when installing larger files, the queen may run out of memory. The queen needs available memory of approximately eight times the size of the file to be installed, in order to encode, buffer and copy the file. \download [[<SCHEMA>/ ]<FILE_ALIAS> ] <FILE> download the specified installed file or function (identified by its FILE or FILE_ALIAS) to the machine where ACT is running. Note that the database user running this command must have permission to download files and functions from the specified schema. \remove [[<SCHEMA>/ ]<FILE_ALIAS> ] remove from the cluster the file or SQL-MapReduce function specified by its FILE_ALIAS. Note that the database user running this command must have permission to remove files and functions from the specified schema. Table 2 - 12: Informational commands in ACT Aster Client Guide Command Description \d list all tables, indexes and views in the current schema. \d [PATTERN} describe table or index. \dt list all tables in the current schema. \dt [PATTERN} print schema, name, type, and owner of a table or tables. To see tables in a custom schema, type \dt schemaname.* \dv list views. \dv [PATTERN] describe view. \di list indexes. 33 Aster Database Cluster Terminal (ACT) ACT Commands (at the SQL Prompt) Table 2 - 12: Informational commands in ACT (continued) Command Description \di [PATTERN} describe index. \dg list groups. \dg [PATTERN} describe group. \du list users. \du [PATTERN} describe user. \dn list schemas. \dn [PATTERN} describe schema. \l list all databases. \extl host=hostname_or_IP [option_name=option_value, …] lists all databases on an external system. \extd host=hostname_or_IP database=dbname [option_name=option_value, …] lists all tables in a database or describe a table on an external system. Tip! In Aster Database 5.0, for the \extd command in ACT, if the optional user argument is not specified, the command will fail on any but the default database. The error message is not specific about what caused the command to fail. The workaround is to always specify the argument user when issuing \extd. Tip! ACT uses the schema search path (search_path) for the database user when displaying lists of tables, veiws and indexes. The schema search path defaults to the schema search path for the current user in the database. To set the search_path from ACT, issue the following command: beehive=> SET session search_path TO <schema>; Note that multiple schemas are not supported. If multiple schemas are listed in the search_path, the first schema listed will be used. To display the current search_path type: beehive=> SHOW search_path; Note that you may also set the search_path on the server. Alternatively, you can specify the schema to use when issuing commands by following the command with a schema qualified reference. This example shows how to display information on all tables in the schema “myschema”: \dt myschema.* Table 2 - 13: Formatting-related commands in ACT 34 Command Description \a toggle between unaligned and aligned output mode. \f [STRING] show or set field separator for unaligned query output. \t [on|off] show only rows (off by default). Aster Client Guide Aster Database Cluster Terminal (ACT) Database Parameters Set with \set Table 2 - 13: Formatting-related commands in ACT (continued) Command Description \x [on|off] set or toggle expanded output mode ON and OFF. With expanded output mode turned on, each record is split into rows, with one row for each value, and each new record is introduced with a text label in the form, --[ RECORD 37 ]---. This can help make wide tables readable on a small screen, and is very useful if you’re trying to read EXPLAIN output. Note that in expanded mode, the number of rows is not returned at the end of the table. Because of this, when querying a table with no rows, you will simply see the ACT prompt again. Database Parameters Set with \set The syntax to use the \set command is: \set [ name [ value [ ... ] ] ] Table 2 - 14: Database parameters Parameter Description auto-commit [1|0] When set to 1 (the default, on), each SQL command is automatically committed upon successful completion. When set to 0 (off), you may manually commit your changes after each transaction or series of transactions by issuing the COMMIT command, or undo changes by issuing ROLLBACK. If you do not issue the COMMIT command, all transactions that occurred since the last COMMIT will rollback automatically. fetch-count [int] To limit the number of rows returned at a time. ACT uses a fetch count by default (i.e. even when fetch-count is not set explicitly.) The fetch-count (number of rows per fetch) should always be set to greater than 0, The default value is 1024 (1024 rows). fetch-limit [int] To set the maximum number of rows returned per query. A value less than 0 implies fetch all rows. A value greater than 0 implies fetchlimit rows in total. The default value is -1 (all rows). use-server-cursors [1|0] When set to 1, sets the server to use cursors (useful when the result set is very large). When set to 0 (the default, off), sets the server to not use cursors. on-error-stop [1|0] By default, this feature is disabled or set to off = 0. When set to 1 (or “on”) ACT will stop and exit if it meets an error during SQL query processing. The following are ACT exit messages: • EXIT_SUCCESS = 0 means ACT finished processing normally. • EXIT_FAILURE = 1 means an error occurred, such as "file not found" in the “-f ” option. • EXIT_USER = 3 means an error occurred in a sql script and the option “on-error-stop” was on or enabled. Aster Client Guide 35 Aster Database Cluster Terminal (ACT) Common SSL Configurations Common SSL Configurations This section presents common SSL configuration scenarios and shows how to set up each one when using ACT. For more general information on setting up SSL, see SSL Security for Aster Database-Client Communications (page 77). In addition to the server settings, the command line argument --enable-ssl must be used when invoking ACT. If you attempt to use an SSL argument (--ssl-self-signed-peer, for example) without --enable-ssl, a warning message will display. In addition, secureMuleServer=true must be set on the server if any SSL command line arguments are used in ACT. The common SSL scenarios are: • Scenario 1: Queen Provides a Self-Signed Certificate (page 37) • Scenario 2: Client Must Have a CA-Signed Copy of the Server’s Certificate (page 38) • Scenario 3: Client CA-signed Certificate Must Match the Queen Certificate (page 39) • Scenario 4: Encrypting Communication from the Queen to the Client (page 40) Setting Parameters on the Queen To set up any of the following SSL configurations, the parameter secureMuleServer=true must be set on the queen. In addition, each scenario includes some additional SSL settings that must be made on the queen. See How to Set Configuration Parameters on the Queen (page 85) for information on how to set these parameters. Encrypting Communications from the Queen By default, communications from the client to the queen (for example, queries submitted to the database) are SSL encrypted, and query results travel in the clear. To encrypt query results, follow the steps in Scenario 4: Encrypting Communication from the Queen to the Client in addition to the steps to set up SSL. Using Single Sign-on (SSO) For information on setting up SSO on the queen, see Adding AD-Based SSO Authentication to SSL-Secured Aster Database (page 84). Note that when using SSO, the -U and -w options are not used, because the username and password are passed directly to the host via SSO. Also, when using SSO, you should specify a fully qualified hostname using the -h option, as in the example: <hostname>.<domain>.<com|org etc>. If only the hostname is used, ACT will append the local domain name before attempting to look up the host. Using an IP address with -h is not supported with SSO. The following is an example of a configuration file that uses SSO: # ACT configuration file example # Contains settings for connecting securely to a specific host and database host: saturn.asterdata.com 36 Aster Client Guide Aster Database Cluster Terminal (ACT) Common SSL Configurations dbname: sampledb username: sampleuser # SSL settings enable-ssl: true ssl-self-signed-peer: true # SSO settings enable-sso: true Scenario 1: Queen Provides a Self-Signed Certificate In this scenario, the server provides a self-signed certificate to the client at the time of authentication. The certificate on the queen will be signed by the server itself, as opposed to by a CA (certification authority). In such a configuration, any client can establish an SSL-secured connection, provided the user authenticates successfully with Aster Database. Queen-Side Settings Make the following settings on the queen (note that these are the default settings for a new Aster Database installation): • disallowPeerWithoutCertificates=false • sslCertificatePath=/home/beehive/certs/server.cert • sslPrivateKeyPath=/home/beehive/certs/server.key • sslFileType=1 (A value of “1” means SSL_FILETYPE_PEM.) • Ensure that secureWrites is set to false • Ensure that secureMuleServer is set to true • There is no need to set the trustedCAPath and trustedCAFileName parameters. Client-Side Settings Use the following command line arguments when executing ACT: • --enable-ssl • --ssl-self-signed-peer Or use a config file similar to the following: # ACT configuration file example # Contains settings for connecting securely to a specific host and database host: 10.10.10.10 dbname: sampledb username: sampleuser # SSL settings enable-ssl: true ssl-self-signed-peer: true Note that the Client need not have a copy of the server's certificate. Do not use the other SSL settings such as --ssl-trusted-ca-file, or --ssl-trusted-ca-path. Aster Client Guide 37 Aster Database Cluster Terminal (ACT) Common SSL Configurations Scenario 2: Client Must Have a CA-Signed Copy of the Server’s Certificate In this scenario, ACT (the client) needs a CA-signed copy of the queen’s certificate. The location of this certificate can be specified with --ssl-trusted-ca-file, or as a chain of trusted certificates using the --ssl-trusted-ca-dir parameter. For more specific information on using a certificate chain, see Queen-Side SSL Parameters (page 79). Queen-Side Settings Make the following settings on the queen: • disallowPeerWithoutCertificates=false • sslCertificatePath=/home/beehive/certs/server.cert • sslPrivateKeyPath=/home/beehive/certs/server.key • sslFileType=1 (A value of "1" means SSL_FILETYPE_PEM. A value of “2” means SSL_FILETYPE_ASN1.) • Ensure that secureWrites is set to false. • Ensure that secureMuleServer is set to true. • There is no need to set the trustedCAPath and trustedCAFileName parameters. Client-Side Settings Do the following: Copy the queen's public key (self-signed certificate), /home/beehive/certs/server.pem, to the client. For this example, we will assume the client will store the public key as /home/ jbloggs/certs/server.pem. Use the following command line arguments when executing ACT: • --enable-ssl • --ssl-trusted-ca-file server.pem • --ssl-trusted-ca-dir /home/jbloggs/certs/ Or use a config file similar to the following: # ACT configuration file example # Contains settings for connecting securely to a specific host and database host: 10.10.10.10 dbname: sampledb username: sampleuser # SSL settings enable-ssl: true ssl-trusted-ca-file: server.pem ssl-trusted-ca-dir: /home/jbloggs/certs/ Because --ssl-self-signed-peer is not specified, the connection can be made only when the server provides a CA signed certificate. The identical CA signed certificate must already exist on the client. However, when --ssl-self-signed-peer is used, the server is able to supply the certificate at the time of connection and nothing is required on the client. 38 Aster Client Guide Aster Database Cluster Terminal (ACT) Common SSL Configurations Scenario 3: Client CA-signed Certificate Must Match the Queen Certificate This scenario presents a stricter regime, where the queen only accepts connections from clients that provide a CA-signed certificate, for which a copy already exists on the queen at the time of connection. In other words, clients cannot connect using a self-signed certificate, nor can they connect using a copy of the queen's public key. The identical signed certificate must exist on the queen as well. Setting the server flag disallowPeerWithoutCertificates = true is what forces the client to provide a certificate in order to connect. Queen-Side Settings Do the following: 1 Get the root certificate of the CA (certificate authority) that signed your client's certificate. Save the root certificate on the queen. For this example, we will save it as /home/ beehive/certs/client.pem on the queen. 2 Make the following settings on the queen: • disallowPeerWithoutCertificates=true • trustedCAFileName=/home/beehive/certs/client.pem • sslCertificatePath=/home/beehive/certs/server.cert • sslPrivateKeyPath=/home/beehive/certs/server.key • sslFileType=1 (A value of "1" means SSL_FILETYPE_PEM. A value of “2” means SSL_FILETYPE_ASN1.) • There is no need to set the trustedCAPath parameter if you use a single root certificate for all clients. • Ensure that secureWrites is set to false. • Ensure that secureMuleServer is set to true. Variation: If your client's certificates were not all signed by the same CA, then you must set Aster Database to recognize all the CA root certificates used to sign you clients' certificates, like so: 1 Save the root certificates of all the signing CAs on the queen. 2 Set trustedCAPath to point to the directory that contains the root certificates. For example: • 3 trustedCAPath=/home/beehive/certs Un-set the queen configuration parameter, trustedCAFileName, by setting it to no value at all. For example: • trustedCAFileName= Client-Side Settings Use the following command line arguments when executing ACT. For this example, we will assume the client will store the certificate as /home/jbloggs/certs/client.cert and the key as /home/jbloggs/certs/client.key: • Aster Client Guide --enable-ssl 39 Aster Database Cluster Terminal (ACT) Troubleshooting ACT • --ssl-certificate-path /home/jbloggs/certs/client.cert • --ssl-private-key-path /home/jbloggs/certs/client.key • --ssl-cert-filetype 1 (A value of "1" means SSL_FILETYPE_PEM. A value of “2” means SSL_FILETYPE_ASN1.) Or use a config file similar to the following: # ACT configuration file example # Contains settings for connecting securely to a specific host and database host: 10.10.10.10 dbname: sampledb username: sampleuser # SSL settings enable-ssl: true ssl-certificate-path: /home/jbloggs/certs/client.cert ssl-private-key-path: /home/jbloggs/certs/client.key ssl-cert-filetype: 1 Scenario 4: Encrypting Communication from the Queen to the Client By default, only the queries issued by ACT to the queen are encrypted. It is also possible to encrypt the communication from the queen to ACT. Note that these changes can be applied to all of the previous scenarios. To do this, make the following changes in addition to the settings necessary to enable SSL. Queen-Side Settings Make the following setting on the queen: • secureWrites=true Even though you may have set secureWrites=false when setting up SSL, set it to true now in order to enable encryption of communication from the queen. This permits setting up the SSL connection before enabling two way encryption, if desired. Client-Side Settings Use the following command line arguments when executing ACT: • --ssl-encrypt-reads Or add the following entry to the config file: • ssl-encrypt-reads: 1 Troubleshooting ACT ACT Connection Hangs When Using SSL When ACT is configured to use SSL, the queen must also be set up to use SSL. If the queen is not setup for SSL and ACT tries to connect using SSL, the connection will fail. An error is not 40 Aster Client Guide Aster Database Cluster Terminal (ACT) Troubleshooting ACT returned; the ACT client just hangs. If you experience this, check to ensure that SSL settings on the queen and in ACT match. Misleading Error Message Reports Problem With a Role Instead of With a User When attempting to connect to a database as a user who does not have connect privileges on that database, the error incorrectly reports that the problem is with a role instead of with the user. This example illustrates the error message: beehive=> create user kris with password 'beehive'; CREATE USER beehive=> create database foo; CREATE DATABASE beehive=> \c foo kris; Password for "kris": act: ERROR: role "kris" does not have connect privileges on this database. Previous connection kept Aster Client Guide 41 Aster Database Cluster Terminal (ACT) Troubleshooting ACT 42 Aster Client Guide Connect Using Database Drivers CHAPTER 3 Aster Database provides Open Database Connectivity (ODBC) and Java Database Connectivity (JDBC) drivers for connecting business intelligence (BI) tools. This section explains how to install and use the Aster Database drivers. General tips: • General Tips for Connecting Clients to Aster Database (page 44) Setting up database drivers: • ODBC Driver (page 44) • JDBC Driver (page 59) • Tools for .NET Environments (page 97): ADO.NET driver (also known as Aster Database DNProvider driver) for SSIS, SSAS, SSRS, and .NET Using database drivers: • Process SQL Statements in JDBC (page 90) • JDBC Troubleshooting and Limitations (page 92) • Configure Aster Database SQL Settings (page 71) Security for database drivers: • Teradata Wallet (page 73) • SSL Security for Aster Database-Client Communications (page 77) Connecting BI and query tools: • Aster Client Guide Connect Reporting Tools to Aster Database (page 92) 43 Connect Using Database Drivers General Tips for Connecting Clients to Aster Database General Tips for Connecting Clients to Aster Database Recommended Character Set Is UTF-8 For all tools that you use to connect to databases in Aster, you will typically want to set the default character set to UTF-8. This is particularly important if you plan to use special characters (for example, German letters ä and ß) in a char, varchar, or text column, and it is particularly important if you are connecting from a Windows-based machine. For example, if you will use an SSH client (for example, putty) to run ACT or ncluster_loader, make sure you set the SSH client’s default character set to UTF-8. When Querying System Tables with ODBC, Set AUTOCOMMIT to 'OFF' Cursors cannot be declared and used with Aster Database system tables. The default mode of autocommit-on with the ODBC driver declares cursors. Therefore, the ODBC driver should always query system tables by turning AUTOCOMMIT to 'off '. AIX Client Dependent Libaries For Aster AIX Clients, the following dependent libraries are required. Table 3 - 1: Dependent LIbraries for AIX Clients Required Library ODBC Aster Loader and Export Tools libstdc++.a (gcc) 4.6.1 4.6.1 libgcc_s.a (gcc) 4.6.1 4.6.1 ODBC Driver Teradata Aster provides a standard ODBC driver for Aster Database. The following list shows all the supported operating systems with the corresponding ODBC driver package name: 44 • Windows 32-bit: nClusterODBCInstaller_i386.msi • Windows 64-bit: nClusterODBCInstaller_x64.msi • Linux 32-bit: clients-odbc-linux32.tar.gz • Linux 64-bit: clients-odbc-linux64.tar.gz • Solaris Sparc 32: clients-odbc-solaris-sparc.tar.gz • Solaris i386: clients-odbc-solaris-x86.tar.gz • Mac OS x86: clients-odbc-mac.tar.gz • AIX Power PC 32-bit: clients-odbc-aix.tar.gz Aster Client Guide Connect Using Database Drivers ODBC Driver • AIX Power PC 64-bit: clients-odbc-aix.tar.gz The Aster Database ODBC driver may change in any Aster Database release. For this reason, with each new edition of Aster Database, you should reinstall the driver and recompile your applications that include the driver. ODBC Driver Manager Compatibility The Aster Database ODBC driver requires a driver manager. Different platforms use different driver managers. Here is a list of platforms and their supported ODBC driver managers: • Windows: Microsoft Driver Manager which is part of the Windows operating system. • Linux: Driver Manager, which is bundled and automatically installed with the Aster Database ODBC driver. • AIX: Driver Manager, which is bundled and automatically installed with the Aster Database ODBC driver. • Solaris: unixODBC driver manager • MacOS: iODBC driver manager The following diagram illustrates the path the ODBC connection takes from an application to Aster Database: Figure 1: Path of an ODBC connection Linux/AIX ODBC Application Driver Manager Supplied by Aster Database Aster Database ODBC Driver Aster Database Driver Manager Supplied by Microsoft Aster Database ODBC Driver Aster Database unixODBC Driver Manager (Open Source) Aster Database ODBC Driver Aster Database iODBC Driver Manager (Open Source) Aster Database ODBC Driver Aster Database Windows ODBC Application Solaris ODBC Application Mac OS X ODBC Application Obtain the ODBC Driver Package First, you’ll need to get the appropriate Aster Database ODBC driver package for your operating system. To obtain the package, download it from one of these places onto your client machine in the directory where you will install it: Aster Client Guide • To get the newest package, download it from http://downloads.teradata.com/download/tools • On your queen node, you can find the installers in the directory /home/beehive/ clients_all/<your_client_OS>. 45 Connect Using Database Drivers ODBC Driver Next, see the section below that applies to your platform: • Install ODBC on Windows (page 46) • Install ODBC on Linux, Solaris, or MacOS (page 48) • Install ODBC on AIX (page 50) Install ODBC on Windows Follow the steps below to install the Aster Database ODBC driver on Windows. Driver Manager Support Windows includes Driver Manager, which allows ODBC applications to use the correct driver. Installation Procedure 1 Verify that Microsoft Visual C++ 2008 Redistributable Package (x86) is installed on the system where you want to install the driver. Note that Microsoft also offers newer versions of the package, such as Microsoft Visual C++ 2010 Redistributable Package (x86). Teradata Aster has not tested compatibility with these later versions! If the supported version is not installed: a Download it from Microsoft, choosing the version that fits your architecture: • 32-bit • 64-bit 2 Obtain the ODBC Driver Package. 3 If you are upgrading, use the Windows Add/Remove Programs tool to uninstall the old Aster ODBC version. 4 Double-click on the .msi file to install it. A setup wizard walks you through the installation of Aster Database ODBC as one of the available data sources on your computer. 46 5 From the Windows Control Panel, double-click Administrative Tools to open the Administrative Tools window. 6 Double-click the Data Sources (ODBC) option to open the ODBC Data Source Administrator dialog box. 7 Click the System DSN tab. 8 Click Add to open the Create New Data Source dialog box. Aster Client Guide Connect Using Database Drivers ODBC Driver 9 Select the Aster ODBC Driver data source from the list. 10 Click Finish. The Aster Database Login window appears. 11 In the Aster Database Login window, enter the following information: • Data Source: Use this field to give this database connection an easy-to-recognize name. • Server: The hostname or IP address of your Aster Database queen. • Port: The port on which your Aster Database queen listens for client connections. The default is 2406. • Database: The name of the database in Aster Database you want to connect to. Default system database is beehive. Aster Client Guide • Username: Database user name. • Password: Database user’s password. • Fetch Count: See “Throttle Query Results in ACT and Aster Database” on page 26. 47 Connect Using Database Drivers ODBC Driver • SSL Settings: See “SSL Security for Aster Database-Client Communications” on page 77. • SSO Settings: See “Adding AD-Based SSO Authentication to SSL-Secured Aster Database” on page 84. • enable_quoted_identifiers: See “Quoted-Identifier Handling” on page 71. • enable_backslash_escapes: See “Escape Character Handling” on page 71. • Map NUMERIC/DECIMAL to DOUBLE: Teradata Aster recommends that you turn this feature on. • Bytea As Varchar: The login window does not provide a check box for specifying that values in a column of datatype bytea should be retrieved in a character representation (as opposed to the default binary representation). However, you can set this later in the Windows registry. See “Optional ODBC Setting for bytea-Stored Data” on page 57. 12 Click OK to save the data source information. 13 Click Test Connection to test the connectivity to the database. If the connection is successful, the Aster Database Login window closes automatically. 14 If this window does not close automatically, click OK. Your ODBC setup is complete. Now, in your applications that will query Aster Database, you may connect to Aster Database as an ODBC data source. Install ODBC on Linux, Solaris, or MacOS To install the Aster Database ODBC driver on Linux, Solaris, or MacOS, follow the instructions below. Driver Manager Support The Aster Database ODBC driver for Linux comes with Driver Manager, which is installed automatically when you install the Aster Database ODBC driver. Driver Manager allows ODBC applications to use the correct driver (in this case, the Aster Database ODBC driver) to connect to Aster Database. The Driver Manager bundled with Aster Database ODBC supports these Linux platforms: • Linux 32-bit • Linux 64-bit For Solaris, use unixODBC driver manager and for MacOS, use iODBC manager. Prerequisites 1 48 The Aster Database ODBC driver requires the following libgcc version, depending on your operating system: • Linux and Solaris: libgcc 3.4.6 • Mac OS 10.5: libgcc 4.2 • Mac OS 10.6: libgcc 4.5 Aster Client Guide Connect Using Database Drivers ODBC Driver 2 On Solaris and MacOS systems, the Aster Database ODBC driver requires that you install a driver manager: • On Solaris, use the unixODBC driver manager, version 2.2.12. Instructions are available here: “Install and Configure the unixODBC Driver Manager on Solaris” on page 52. • On MacOS, use the iODBC driver manager, version 3.52.3. This driver and its documentation are available from http://www.iodbc.org. Installation Procedure Install the Aster Database ODBC driver as shown below. Warning! For MacOS, the Aster Database ODBC Driver supports only MacOS version 10.6 or earlier. 1 Extract the ODBC package into the directory where you want to install the driver. Once extracted, you will see the directory stage that includes the driver. 2 Change to the Aster Database driver directory: # cd stage/clients-odbc-<your_client_os> 3 Edit your library path environment variable, to add the Aster ODBC library directory to it. The Aster ODBC library directory has the path <install location>/stage/clientsodbc-<your_client_os>/Libs. The library path variable is: • LD_LIBRARY_PATH on Linux and Solaris • DYLD_LIBRARY_PATH on MacOS Edit the appropriate environment settings file to do this (for example, edit the ~/.bashrc file if you want to set it for the current user on a typical Linux environment). To set it for the current session only, type the command shown below, substituting DYLD_LIBRARY_PATH for LD_LIBRARY_PATH on MacOS: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/stage/clientsodbc-<your_client_os>/Libs 4 Add or edit the ODBCSYSINI environment variable, setting it to the directory where your ODBC connection settings files (odbc.ini and odbcinst.ini) will reside. To follow this example, let’s assume we are working as user “mjones” and will save the configuration files to our home directory /home/mjones. export ODBCSYSINI=/home/mjones 5 Check that the Aster Database ODBC driver library can find all its dependencies. Assuming we have installed in /usr/local/lib, we would type (on Linux or Solaris): # cd /usr/local/lib # ldd stage/clients-odbc-linux64/ODBCDriver/ libAsterDriver_unixODBC.so On MacOS, we would type: # cd /usr/local/lib Aster Client Guide 49 Connect Using Database Drivers ODBC Driver # otool -L stage/clients-odbc-linux64/ODBCDriver/ libAsterDriver_unixODBC.dylib If a “not found” message does not appear, then all the required libraries have been linked. 6 Choose the next step, depending on your operating system: • Configure Driver Manager on Linux and AIX • Install and Configure the unixODBC Driver Manager on Solaris • For MacOS, go to http://www.iodbc.org to download and configure iODBC driver manager. Install ODBC on AIX Aster Database ODBC driver for AIX supports these platforms: • AIX 5.3 (32-bit and 64-bit) • AIX 6.1 (32-bit and 64-bit) See also “AIX Client Dependent Libaries” on page 44. Driver Manager Support The Aster Database ODBC driver for AIX comes with Driver Manager, which is installed automatically when you install the Aster Database ODBC driver. No other ODBC driver manager is supported on AIX. Driver Manager allows ODBC applications to use the correct driver (in this case, the Aster Database ODBC driver) to connect to Aster Database. Installation Procedure Install the Aster Database ODBC driver on AIX as shown below. 1 Obtain the ODBC Driver Package. 2 Extract the bundle for the ODBC driver: a Unzip the file: $ gunzip clients-odbc-aix.tar.gz b Untar the file: $ tar -xvf clients-odbc-aix.tar Next Step: Configure Driver Manager on Linux and AIX. Configure Driver Manager on Linux and AIX Use the following process to configure the Driver Manager that is installed with the Aster ODBC Driver on Linux and AIX: 1 Find the Driver Manager configuration files under the extracted directory: $ cd stage/clients-odbc-<your_client_os>/DataDirect/setup 2 Copy the odbcinst.ini, odbc.ini, and aster.ini files from the /setup directory to your home directory: $ cp -p * ~ 3 50 Change to your home directory: Aster Client Guide Connect Using Database Drivers ODBC Driver $ cd 4 Make backups of the files you moved: $ cp -p aster.ini aster.ini.backup $ cp -p odbc.ini odbc.ini.backup $ cp -p odbcinst.ini odbcinst.ini.backup 5 Make the following edits to aster.ini: a Set DriverManagerEncoding to UTF-8. b Set ODBCInstLib to <InstallDir>/DataDirect/lib/odbcinst.so, replacing <InstallDir> with the folder where the driver is installed. For example: [driver] DriverManagerEncoding=UTF-8 ODBCInstLib = <InstallDir>/DataDirect/lib/odbcinst.so ErrorMessagesPath=<InstallDir>/ErrorMessages DSILogging=0 6 Modify odbc.ini as follows: a Change the DSN configuration parameters SERVER, UID, PWD, DATABASE and PORT. [ODBC Data Sources] ... ... asterdsn=AsterDriver [ODBC] ... ... [asterdsn] Driver=<InstallDir>/DataDirect/lib/libAsterDriver.so SERVER=192.206.82.100 PORT=2406 DATABASE=beehive UID=beehive PWD=beehive b Add this item to the [ODBC] section of odbc.ini: InstallDir=<InstallDir>/DataDirect 7 8 Add the <InstallDir>/Libs directory to: • LD_LIBRARY_PATH for Linux, or • LIBPATH for AIX PowerPC. Export the following values, where <directory_path> is the path to the directory where the files odbc.ini and odbcinst.ini reside: export ODBCHOME=<directory_path> export ODBCINI=$ODBCHOME/odbc.ini export ODBCINST=$ODBCHOME/odbcinst.ini 9 Edit the odbcinst.ini file, as shown in this example: [ODBC Drivers] ... ... AsterDriver=Installed [ODBC Translators] OEM to ANSI=Installed [ODBC] Aster Client Guide 51 Connect Using Database Drivers ODBC Driver #This section must contain values for DSN-less connections #if no odbc.ini file exists. If an odbc.ini file exists, #the values from that [ODBC] section are used. [AsterDriver] Driver=<InstallDir>/DataDirect/lib/libAsterDriver.so IconvEncoding=UCS-4LE Install and Configure the unixODBC Driver Manager on Solaris On Solaris, Teradata supports using the unixODBC driver manager. This section describes how to install and configure unixODBC. Install the unixODBC driver manager If you already have the unixODBC manager installed, skip this section and proceed to “Configure the unixODBC driver manager” on page 52. 1 Download the unixODBC driver manager v 2.2.12, from http://www.unixodbc.org/ unixODBC-2.2.12.tar.gz 2 Install unixODBC by running the following commands: # CFLAGS="-DSIZEOF_LONG=8" ./configure --prefix=/usr \ --sysconfdir=/etc/unixodbc \ --enable-fdb \ --disable-gui && make 3 Wait while it builds and installs. 4 Working as root user, run this: # make install && find doc -name "Makefile*" -exec rm {} \; && chmod 644 doc/{lst,ProgrammerManual/Tutorial}/* && install -v -m755 -d /usr/share/doc/unixODBC-2.2.12 && cp -v -R doc/* /usr/share/doc/unixODBC-2.2.12 5 Check the installation by typing the following command. This prints version information and lists the names and locations of the configuration files you will need to install or set up. # odbcinst -j If your unixODBC installation is not working properly, check the instructions at http:// for help. www.unixodbc.org/ Next Step: Proceed to the next section, Install and Configure the unixODBC Driver Manager on Solaris. Configure the unixODBC driver manager If you have chosen to use the unixODBC Driver Manager instead of the bundled Driver Manager, configure it as follows: 52 Aster Client Guide Connect Using Database Drivers ODBC Driver 1 Get the templates for the ODBC connection settings files. Copy these files from the Aster Database driver’s Setup directory to the user’s home directory. The files you need are aster.ini, odbc.ini and odbcinst.ini: # # # # cd cp cp cp /usr/local/lib/stage/clients-odbc-linux64/Setup odbc.ini ~ odbcinst.ini ~ aster.ini ~/.aster.ini Note that we have also renamed the aster.ini file, adding a dot at the beginning of the file name. You must do this. 2 Edit the .aster.ini file as follows: a Set DriverManagerEncoding to UTF-32. b Set ODBCInstLib to <unixODBCDir>/lib/libodbcinst.so, replacing <unixODBCDir> with the directory where the driver is installed: c Set the ErrorMessagesPath to point to the ErrorMessages subdirectory in the Aster Database driver directory. For this example, we edit the last line in the file to read: ErrorMessagesPath=/usr/local/lib/stage/clients-odbc-linux64/ ErrorMessages Tip! At this point, you can run “odbcinst -j” to find out where the ODBC driver expects to find its configuration files. 3 In a text editor, edit the odbc.ini file, making the following changes: a Set SERVER to the hostname or IP address of your Aster Database queen. b Set PORT to 2406, the standard port on which your Aster Database queen listens for client connections. c Set DATABASE to the name of the database in Aster Database you want to connect to. d Optionally, you may set UID and PWD to your Aster Database SQL username and password, respectively. e Finally, Teradata Aster recommends that you add the setting, NumericAndDecimalAsDouble=1. f If you retrieve bytea-stored data through the ODBC driver, you can specify whether values in a column of datatype bytea will be retrieved in a character representation, or in the default binary representation. To have the ODBC driver to retrieve values in character representation, add the setting, ByteaAsVarchar=1 to your odbc.ini; if you leave it unset, the driver preserves the binary output representation of bytea data. g Optionally, you can set a number of other database connection behavior settings. These include enable_quoted_identifiers (see “Quoted-Identifier Handling” on page 71), enable_backslash_escapes: See “Escape Character Handling” on page 71. For this example, we set the contents of odbc.ini to read: [ODBC Data Sources] Aster Data ODBC for nCluster DSN=AsterDriver Aster Client Guide 53 Connect Using Database Drivers ODBC Driver [Aster Data ODBC for nCluster DSN] Driver=AsterDriver SERVER=10.50.52.100 PORT=2406 DATABASE=beehive NumericAndDecimalAsDouble=1 # UID=<USERNAME> # PWD=<PASSWORD> Tip! You can have multiple data sources. The name “Aster Data ODBC for nCluster DSN” in the odbc.ini file is just a default name that Teradata Aster has given to the sample data source. You can rename this source and add more, as shown in this example: [ODBC Data Sources] my_1st_source=AsterDriver my_2nd_source=AsterDriver [my_1st_source] Driver=AsterDriver SERVER=10.50.52.100 ... [my_2nd_source] Driver=AsterDriver SERVER=10.42.43.100 ... 4 In a text editor, edit the odbcinst.ini file, setting the Driver parameter to the Aster Database driver directory path. For this example, we set the contents of odbcinst.ini to read: [AsterDriver] Driver=/usr/local/lib/stage/clients-odbc-linux64/ODBCDriver/ libAsterDriver_unixODBC.so IconvEncoding=UCS-4LE On MacOS, it will look like: [AsterDriver] Driver=/usr/local/lib/stage/clients-odbc-linux64/ODBCDriver/ libAsterDriver_unixODBC.dylib IconvEncoding=UCS-4LE 5 The installation and configuration are now complete. Test the unixODBC Driver Manager Once the driver is installed and configured, you should test it: The unixODBC driver manager comes with a tool called “isql” that you can use to test the driver. You can test your connection now by connecting to Aster Database using the isql tool. 1 Type the command as shown below. Here, we put the data source name in single quotes because it contains spaces. The “beehive” and “beehive” are the default username and password in a new Aster Database installation: # isql -v 'Aster Data ODBC for nCluster DSN' beehive beehive 54 Aster Client Guide Connect Using Database Drivers ODBC Driver 2 A “Connected!” message indicates that the driver is working correctly: +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> 3 Set-up is complete. Troubleshooting If after installation you cannot connect: 1 Find the library libodbcinst.so and note its path. 2 Set the LD_LIBRARY_PATH environment variable so that it includes the directory that contains libodbcinst.so. For example, if the library is in /usr/lib64, then you will type: export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH Set Up ODBC for Perl Connectivity on Linux To install the Aster Database ODBC driver and configure your environment to allow Perl scripts to connect to the database through the driver, follow these steps. These instructions assume you have installed Perl on your workstation, and that the Aster Database queen is reachable on the network. Follow these steps to provide an Aster Database connection your Perl scripts can use: 1 Set up your /etc/resolv.conf and /etc/nsswitch.conf for accessing the Aster Database queen. 2 Install and Aster Database ODBC. Follow the instructions in “Install ODBC on Linux, Solaris, or MacOS” on page 48. 3 Make sure that Driver Manager is properly configured. Follow the instructions in “Configure Driver Manager on Linux and AIX” on page 50. 4 The server connection information is set in the odbc.ini file, as explained earlier in this chapter. On Windows, the server connection information is set using the ODBC Data Source Administrator tool as explained in “Set up ODBC for PHP” on page 56. 5 In /root type $ perl -eshell -MCPAN 6 Get the latest packages and install them by entering: $ install Bundle::CPAN $ install DBI 7 Rebuild and install DBD::ODBC. $ export LD_LIBRARY_PATH=/home/beehive/toolchain/x86_64-unknownlinux-gnu/unixODBC-2.2.12/lib Aster Client Guide 55 Connect Using Database Drivers ODBC Driver $ export PATH=$PATH:/home/beehive/toolchain/x86_64-unknown-linuxgnu/unixODBC-2.2.12/bin $ which odbc_config /home/beehive/toolchain/x86_64-unknown-linux-gnu/unixODBC-2.2.12/ bin/odbc_config $ odbc_config --cflags -DHAVE_UNISTD_H -DHAVE_PWD_H -DHAVE_SYS_TYPES_H -DHAVE_LONG_LONG -DSIZEOF_LONG=8 $ perl -eshell -MCPAN cpan[1]> force install DBD::ODBC 8 Run odbcinst -j to see where the .ini files are being picked up: $ odbcinst -j 9 Run the following Perl script to check your installation. If everything is set correctly, it will run without an error. If you encounter problems, check the data source name (“DSN”) in the connect statement. In the example below, the username and password of the database are “beehive” and “beehive”: #!/usr/bin/perl use DBI; use DBD::ODBC; # Connect to ODBC DSN and turn off AutoCommit my $dbh = DBI->connect('dbi:ODBC:nc',"beehive","beehive", {AutoCommit => 0}); $dbh->do("BEGIN"); $dbh->do("set random_page_cost to '4'"); $dbh->do("set enable_seqscan to 'off'"); $dbh->disconnect; Set up ODBC for PHP To set up PHP to work with ODBC, first follow all of the instructions to set up ODBC on Linux: “Install ODBC on Linux, Solaris, or MacOS” on page 48. As described in the instructions, you should use unixODBC 2.2.12 and build it with the CFLAG, SIZEOF_LONG=8. The same will apply to building PHP when compiled to be used with unixODBC. Here are the supported versions: • PHP 5.4.10 • Apache 2.2.23 PHP To set up PHP: 1 Make sure Apache is installed and make note of the directory. For this example, we will assume Apache is installed at /usr/local/apache 2 Ensure that unixODBC has been installed as described above. 3 Download the source for PHP 5.4.10 and extract it to the desired directory. The following setup instructions should be used for PHP: $ CFLAGS="-DSIZEOF_LONG=8" ./configure --with-apxs2=/usr/local/ apache/bin/apxs --with-zlib --with-unixODBC --with-pdo-odbc=unixODBC 56 Aster Client Guide Connect Using Database Drivers ODBC Driver $ make && make install $ cp -p .libs/libphp5.so /usr/local/apache/modules Apache and PHP Apart from the typical PHP-Apache setup, there are two things to keep in mind: 1 Ensure that the Linux system is set up to find the libraries that the ODBC driver requires. a Add the <INSTALL-DIR>/Libs and <INSTALL-DIR>/ODBCDriver paths to the /etc/ ld.so.conf file, where <INSTALL-DIR> is the location for the ODBC installation. b The following commands should be executed to refresh the ld cache and restart the Apache web server: $/sbin/ldconfig $/etc/init.d/apachectl restart 2 To make sure that the ODBCSYSINI environment variable is available when PHP calls are made, use the following line in your PHP code: #given ODBCSYSINI is to be set to /etc then: putenv("ODBCSYSINI=/etc") Tip! You should not set up your own PHP or use /etc/init.d/apachectl on the queen for your own web pages. ODBC Usage Notes Optional ODBC Setting for bytea-Stored Data If you retrieve bytea-stored data through the ODBC driver, you can specify whether values in a column of datatype bytea will be retrieved in a character representation, or in the default binary representation. To set this up you must set the ByteaAsVarchar setting. Setting ByteaAsVarchar=1 instructs the ODBC driver to retrieve values in character representation; leaving it unset preserves the binary output representation. • To set this up on Linux, Solaris, or MacOS, see the discussion of the odbc.ini file in “Configure the unixODBC driver manager” on page 52. • To set this up on Windows, see below. Bytea-Stored Data Settings on Windows See the section that applies to your operating system and driver: • For a 64-bit ODBC driver running on 64-bit Windows, or for a 32-bit driver on 32-bit Windows: Set the flag by adding it to the Windows registry entry for the DSN. Using a registry editor, add this line to the registry, taking care to first replace <DSN-NAME> with your Data Source name: Aster Client Guide 57 Connect Using Database Drivers ODBC Driver [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\<DSN-NAME>] "ByteaAsVarchar"="1" • For a 32-bit ODBC driver running on 64-bit Windows Set the flag by adding it to the Windows registry entry for the DSN. Using a registry editor, add this line to the registry, taking care to first replace <DSN-NAME> with your Data Source name: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI\<DSN-name>] "ByteaAsVarchar"="1" Avoid using bind_param with SQL_DECIMAL When using the bind_param method call with decimal fields, you should not explicitly specify SQL_DECIMAL as the parameter bind type. If you do this, and the parameter value has a fractional digital part, the factional part changes to 0 after applying bind_param. The DBD::ODBC bind_param implementation makes this conversion when specifying SQL_DECIMAL/SQL_NUMERIC as the parameter bind type. Note that bind_param works fine when SQL_DECIMAL is not explicitly specified, so you should avoid this when passing decimal fields. Bind columns without specifying a SQL datatype Use the Perl API without specifying a SQL datatype the Aster datatypes, as in the example: $rc = $sth->bind_col($column_number, \$var_to_bind); Avoid specifying the SQL datatype to bind to the column, as in: $rc = $sth->bind_col($column_number, \$var_to_bind, $bind_type); This will work fine when binding boolean and character columns with DBI::SQL_CHAR, but for the other datatypes, this will cause two issues: 1 If you specify a numeric SQL type in bind_col, corrupt data may be returned. Numeric types include SQL_INTEGER, SQL_SMALLINT, SQL_DOUBLE, SQL_REAL, SQL_NUMERIC/SQL_DECIMAL, SQL_BIT, SQL_TIME, SQL_TIMESTAMP, SQL_DATE, and SQL_DATETIME. 2 If you specify a SQL_VARCHAR type in bind_col, an error may be returned. The error looks like: • On Unix ODBC: “DBD::ODBC::st fetch failed: [unixODBC][Driver Manager]Invalid application buffer type (SQL-HY003)” • On Windows ODBC: “DBD::ODBC::st fetch failed: [Microsoft][ODBC Driver Manager] Program type out of range (SQL-HY003)” This issue has been observed with the following software versions: 58 • Perl 5.8.9 • DBD-ODBC 1.3.1 • DBI-1.616 Aster Client Guide Connect Using Database Drivers JDBC Driver JDBC Driver JDBC is an API for the Java programming language that provides methods for querying and updating data in a relational database. The Aster Database JDBC driver enables your Java applications and reporting tools to retrieve data directly from Aster Databases. The Aster Database JDBC driver is a Type 4 JDBC driver that implements the JDBC 3 specification. • Aster JDBC Driver (page 59) • Differences from the Legacy JDBC Driver (page 60) • Before You Start (page 60) • Install the JDBC Driver (page 61) • Use the JDBC Driver in a Java Application (page 61) • Parameters for Connecting through JDBC (page 62) • Configuring the JDBC Log Settings (page 63) • Behavior and Performance Settings for JDBC (page 63) • Using Client-Side Cursors in JDBC (page 67) • Test JDBC Connect Program (page 69) Aster JDBC Driver This version of the JDBC driver, introduced in Aster Client release 5.0.3, adds these capabilities to those that were already supported in the legacy JDBC driver: Aster Client Guide • Ability to handle larger data sets than the legacy JDBC driver; doesn’t try to load whole result set onto the client in all cases • Support for multibyte characters in data and user metadata UTF-8 encoding • Support for Single Sign-On (SSO) authentication • Support for Secure Socket Layers (SSL) encrypted communications • Support customized connection parameters • Support configuring statement FetchSize to limit the data being returned • Support server side cursors • Support specifying maximum number of rows • Support for these commands: • Copy—Moves data between Aster Database tables and a remote client (from and to a file) via the connection between the client and the server. • Install File—Installs the data file or SQL-MapReduce function in the specified Aster Database schema. • Uninstall File—Removes the file or SQL-MapReduce function from Aster Database. • Download File—Downloads the specified installed file or function. 59 Connect Using Database Drivers JDBC Driver Differences from the Legacy JDBC Driver These are important differences between this JDBC driver and the legacy (pre-5.0.3) JDBC driver: • JDBC only supports scrollable forward cursors. Only the con.createStatement (ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY) and connection.createStatement() functions are supported. The Other parameter values like ResultSet.TYPE_SCROLL_INSENSITIVE and ResultSet.CONCUR_UPDATABLE are not supported. • The executeBatch() function throws a SQL exception (java.sql.SQLException). • The result set cannot be accessed after an explicit commit. • After running PreparedStatement.executenatch(), you must set all bind values or else the driver throws an error. • The java.sql.PreparedStatement.setObject() function does not throw exceptions for invalid java.sql.Types.TINYINT values. Instead, the function wraps the values around. This behavior complies with the Java standard. • ResultSetMetaData cannot be accessed after a ResultSet reset. • The driver cannot establish a connection to the database when there are white space characters to the left of the connection string. • The register serverConnect to database using JDBC in Aqua Data Studio (ADS) as register server. Expand the tree view when connection is lost. Register server will prompt an error. • The DatabaseMetaData.getBestRowIdentifier() function is not supported. • The ResultSet.getArray() function is not supported. • You can call com.asterdata.ncluster.Driver.ASTER_BUILD_VERSION() to get JDBC version information. For detailed version information, check the MANIFEST.MF file. Before You Start The SSO support in the JDBC driver is based on Quest Authentication service. You must acquire the SSO license from Quest in order to take advantage of SSO support in this client. This support is for connecting to Aster Database only, and not for connecting through the Teradata Connector or SQL-H. Prerequisites The JDBC driver supports these versions of the Java JDK: 60 • Oracle JDK 1.5 • IBM JDK 1.6 Aster Client Guide Connect Using Database Drivers JDBC Driver Install the JDBC Driver Follow these steps to install the Aster Database JDBC driver: 1 Obtain the package that contains the Aster Database JDBC driver . You can do this in one of these ways: • To get the newest package, download it from http://downloads.teradata.com/download/ tools • 2 On your queen node, you can find the installers in the directory /home/beehive/ clients_all/<your_client_OS>. Unzip the ZIP package. The resulting folder contains multiple JAR files. 3 Copy the JAR files to a location in the classpath of the application that uses the driver. Use the JDBC Driver in a Java Application Follow these steps to integrate the Aster Database JDBC driver into your Java application: 1 Set the CLASSPATH environment variable to include the paths of the JAR files extracted from the Aster Database JDBC driver package. • Use the Java -classpath flag at the command line to add the paths to the CLASSPATH environment variable. For example, if you place the JAR files in /usr/jars, then add the path like this: java -classpath /usr/my_program /usr/jars/ or • Add the paths to the driver to the CLASSPATH environment variable. For example: export CLASSPATH=/usr/jars/:$CLASSPATH 2 In your application: a Import the java.sql package like this: import java.sql.*; b Load the driver using the Class.forName() method: Class.forName("com.asterdata.ncluster.Driver"); c Define the username, password, and url (including the host, port, and database) parameters, which are needed to connect to the database. For example: String username = "user"; String password = "password"; String url = "jdbc:ncluster://myhost:2406/database"; For more information about the URL format, see “Required Parameters” on page 62. d Get a Connection instance from JDBC using the DriverManager.getConnection() method: try { Connection conn = DriverManager.getConnection(url, username, password); } Aster Client Guide 61 Connect Using Database Drivers JDBC Driver catch (SQLException ex) { // could not connect } NOTE: The getConnection method throws a “No driver available SQLException” exception if CLASSPATH does not contain the path for the JDBC driver, or if the parameters are incorrect. Parameters for Connecting through JDBC Required Parameters To establish a connection to an Aster Database using the Aster Database JDBC driver, you must provide the driver with the URL to use to connect to the database. The URL has this format: jdbc:ncluster://<Host:Port>/<Database>?enable_backslash_escapes=<on_or off>&enable_quoted_identifiers=<on_or off> For example: jdbc:ncluster://192.65.197.90:2406/ beehive?enable_backslash_escapes=on&enable_quoted_identifiers=on The URL needs three parameters to connect to an Aster Database: Table 3 - 2: Parameters in URL to connect to an Aster Database Parameter Required? Description Host Optional The name of the server where the database resides. To specify an IPv6 address, enclose the this parameter in square brackets. For example: jdbc:ncluster://[::1]:2406/nCluster Default is localhost. Port Optional The port number that the database server is listening on. Default is 2406. Database Required The database name. enable_backslash_escapes Optional Sets the queen parameter enable_backslash_escapes for the current session. The two possible values are: • on—Enables this parameter on the queen (default). • off—Disables this parameter on the queen. enable_quoted_identifiers Optional Sets the queen parameter enable_quoted_identifiers for the current session. The two possible values are: • on—Enables this parameter on the queen (default). • off—Disables this parameter on the queen. In addition, to the URL, you must also provide the username and password needed to access the Aster Database, which you can get from your Aster Database administrator. 62 Aster Client Guide Connect Using Database Drivers JDBC Driver Optional Parameters You can set the Autocommit and fetch_count settings for the connection in the URL by adding the autocommit and fetch_count parameters. See “Frequently Used JDBC Settings” on page 63. If your application will query large tables, you should set autocommit to false, and you should declare a fetch_count for the connection. By doing this, you enable the connection to use distributed cursors for improved performance. You can also use the NumericAndDecimalAsDouble parameter to map NUMERIC and DECIMAL type columns to SQL_DOUBLE. When you set this parameter, its value will be stored in a connection context. This example shows how to use this parameter to decode the row header: if ((sqlType == Types.NUMERIC || sqlType == Types.DECIMAL) && inSettings.connectionSettings_.numericAsDouble_) { sqlType = Types.DOUBLE; } Configuring the JDBC Log Settings To enable and configure JDBC logging: 1 Create a file named simba.properties and add it to the folder containing the JDBC driver. 2 In the file, define the LogLevel and LogPath properties. • LogLevel—(Optional) Specifies the log level: OFF, FATAL, ERROR, WARNING, INFO, DEBUG, and TRACE. • LogPath—(Optional) Specifies the path to the log file (AsterJDBC.log). If this property is not defined, the JDBC driver stores the log file in the same folder where the JDBC driver is located. For example: LogLevel=DEBUG LogPath=D:\\logs Behavior and Performance Settings for JDBC When using the Aster Database JDBC driver, you make most behavior and performance settings in your SQL code using the SET command. You can also use a number of standard JDBC driver methods for setting behavior and performance options, such as setAutoCommit() and setPrepareThreshold(). The scope of these variables are limited to the scope of the transaction. After the commit() call, the transaction variables revert to their original values. In order to limit the transaction variables to a particular query, the variables need to be saved prior to being set and once the query is executed the variables must be set back to their original values. Frequently Used JDBC Settings • autocommit: Boolean value that sets the autocommit behavior for the connection, true to turn on autocommit (each statement, by default concluded with a semicolon, is run immediately in the database) or false for off (you must COMMIT your transaction to run Aster Client Guide 63 Connect Using Database Drivers JDBC Driver it). You can set this in the connection URL with the autocommit parameter (for example, jdbc:ncluster://10.80.50.100:2406/beehive?autocommit=false) or in the Java code for your connection with Connection.setAutoCommit(). The default is true. Unsupported JDBC Settings Most JDBC option-setting methods in the Aster Database driver exhibit the standard JDBC behavior, but there are exceptions. Please note: • setReadOnly(): Not supported; Aster Database does not allow changing connection type to read-only. • setTransactionIsolation(): Not supported; Aster Database does not allow changing transaction isolation levels. Example Code Snippet That Sets Performance Tuning Variables This example (PerformanceTuning.java) shows you how you can set some performance tuning variables for a particular transaction: package userguide; import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; public class PerformanceTuning { public static void main(String[] args) { String username = "db_superuser"; String password = "db_superuser"; String url = "jdbc:ncluster://153.65.197.125:2406/beehive"; Connection conn = null; try { conn = DriverManager.getConnection(url, username, password); // Remember the current autocommit state boolean autoCommit = conn.getAutoCommit(); // Turn off auto commits conn.setAutoCommit(false); // create a statement with transaction variables set // which will be used for this query Statement stmt = conn.createStatement(); stmt.executeUpdate("SET enable_hashjoin = 'false'"); stmt.executeUpdate("SET enable_mergejoin = 'false'"); stmt.executeUpdate("SET enable_nestloop='true'"); stmt.executeUpdate("SET random_page_cost = '4.0'"); Statement statement = conn.createStatement(); String query="select * from knn_test"; ResultSet resultSet = statement.executeQuery(query); // Process the result set while (resultSet.next()) { // do something } 64 Aster Client Guide Connect Using Database Drivers JDBC Driver // commit the transaction conn.commit(); resultSet.close(); stmt.close(); // Set the autocommit state back conn.setAutoCommit(autoCommit); } catch (SQLException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } In the example above, the scope of the SET variables is limited to the commands between the autoCommit(false) and commit() lines. The Copy, Install File, Uninstall File, and Download File Commands Version 5.0.3 of the JDBC driver adds support for these commands: • Copy (page 65) • Install File (page 65) • Uninstall File (page 66) • Download File (page 67) For more information about these commands, see the section describing SQL commands in the Aster Database User Guide. Copy This command moves data between Aster Database tables and a remote client (from and to a file) via the connection between the client and the server. This is an example: public void copyCommandExamples() { String stmt1 = "COPY simba TO 'd:\\simba.txt' DELIMITER as ','"; String stmt2 = "COPY simba TO 'd:\\simba.csv' with csv QUOTE AS '@';"; try { Statement s = conn_.createStatement(); s.execute(stmt1); s.execute(stmt2); s.close(); } catch (Exception e) { e.printStackTrace(); fail(e.getMessage()); } } Install File This command installs the data file or SQL-MapReduce function in the specified Aster Database schema. Aster Client Guide 65 Connect Using Database Drivers JDBC Driver INSTALL FILE filename [[schema/]alias] This is an example: public void testInstallWithSchema() { String stmt1 = "install file 'd:\\odbcng.tar.gz' "; String stmt2 = "install file 'd:\\odbcng.tar.gz' uploadfile/'odbcng.zip'"; String stmt3 = "install file 'd:\\abc.txt' uploadfile/'abc.txt'"; try { conn_.setAutoCommit(false); Statement s = conn_.createStatement(); s.executeUpdate(stmt1); s.executeUpdate(stmt2); s.executeUpdate(stmt3); conn_.commit(); s.close(); } catch (Exception e) { try { conn_.rollback(); } catch (SQLException e1) { e1.printStackTrace(); } fail(e.getMessage()); } } Uninstall File This command removes the file or SQL-MapReduce function from Aster Database. UNINSTALL FILE 'filename' from schema schemaname This is an example: public void testUninstall() { String stmt1 = "UNINSTALL file 'abc.txt' from schema uploadfile ;"; String stmt2 = "UNINSTALL file 'odbcng.zip' from schema uploadfile ;"; String stmt3 = "UNINSTALL file 'odbcng.tar.gz' ;"; try { conn_.setAutoCommit(false); Statement s = conn_.createStatement(); s.executeUpdate(stmt1); s.executeUpdate(stmt2); s.executeUpdate(stmt3); conn_.commit(); s.close(); } catch (Exception e) { try { conn_.rollback(); } catch (SQLException e1) { e1.printStackTrace(); } fail(e.getMessage()); } } 66 Aster Client Guide Connect Using Database Drivers JDBC Driver Download File This command downloads the specified installed file or function. DOWNLOAD FILE [[schema/]alias] filename This is an example: public void download_file_examples() { String stmt1 = "DOWNLOAD file uploadfile/'abc.txt' 'd:\\download-abc.txt'"; String stmt2 = "DOWNLOAD file uploadfile/'odbcng.zip' 'd:\\download-uploadfileodbcng.zip'"; String stmt3 = "DOWNLOAD file 'odbcng.tar.gz' 'd:\\download-public-odbcng.zip'"; try { Statement s = conn_.createStatement(); s.executeUpdate(stmt1); s.executeUpdate(stmt2); s.executeUpdate(stmt3); s.close(); } catch (Exception e) { fail(e.getMessage()); } } Using Client-Side Cursors in JDBC To give you control over the latency of initial query results, the Aster Database JDBC driver supports client-side cursors. Client-side cursors let you avoid retrieving all the result rows for a query at once. Instead, you specify the number of rows that should be retrieved to the client at a time. When that set is exhausted, the next page of rows is retrieved by repositioning the cursor. Observe the following guidelines when working with cursors: 1 Turn off autocommit mode for the Connection object. See “Using Cursors in Your Code”, below. 2 The ResultSet object that receives the output of your Statement cannot be a scrollable ResultSet. That is, it must have the type, ResultSet.TYPE_FORWARD_ONLY. This is the default, so you need not rewrite your code. 3 If your application will query large tables, you should make sure your ResultSet can use distributed cursors. To do this, a make sure the ResultSet is not updatable (ResultSet.CONCUR_READ_ONLY) b make sure it’s not scrollable (ResultSet.TYPE_FORWARD_ONLY) c make sure it does not have HOLD enabled (ResultSet.CLOSE_CURSORS_AT_COMMIT). d Also, the application should connect with autocommit set to false and the connection must have a specified fetch_count. Tip! When working with ResultSets of type ResultSet.TYPE_FORWARD_ONLY, you cannot scroll backwards, nor can you jump to any location in the ResultSet other than the next row. Aster Client Guide 67 Connect Using Database Drivers JDBC Driver 4 In the Statement object, you must pass a single query, not multiple queries strung together with semicolons. 5 You must set the statement’s fetch_count using the Statement.setFetchSize(int rows) command. This instructs the driver to fetch the specified number of rows at a time from the database. If the fetch size is not set, the driver fetches the full set of rows that match the query. 6 To help ensure a quick response when a page of rows is exhausted, the JDBC driver, by default, pre-fetches and caches ten pages of results from the database. A page is one fetch_count worth of rows. You can set the number of pages to be pre-fetched, or you can disable pre-fetching if desired. Using Cursors in Your Code The driver fetches fetch_count number of rows (as set with Statement rows)) from Aster Database and exposes them to the database application as a “rows” data structure. The default fetch_count is zero, so you must set a fetch_count if you wish to use cursors. With ResultSet.next(), your application iterates over the rows data structure. When all rows in the rows data structure have been processed, the next call to next() will force the fetching a new set of fetch_count number of rows, and so on. To use cursors, set the fetch size of your SQL statement using the setFetchSize(int rows) method. Later, setting the fetch_count back to 0 will cause all rows to be retrieved when the query runs (the default behavior). Tip! In the example below, we set the Autocommit setting and the FetchSize setting using the setAutoCommit() and setFetchSize() methods, but you also have the option of setting these in the JDBC connection parameters when you make the connection. See “Frequently Used JDBC Settings” on page 63. Example That Uses Client-Side Cursors Assuming a Connection conn, you can set the FetchSize like this: // Remember the current autocommit state boolean autoCommit = conn.getAutoCommit(); // Make sure autocommit is off conn.setAutoCommit(false); Statement st = conn.createStatement(); // Turn use of the cursor on by setting FetchSize, expressed in rows st.setFetchSize(50); ResultSet rs = st.executeQuery("SELECT * FROM mytable"); // Set the autocommit state back conn.setAutoCommit(autoCommit); while (rs.next()) { System.out.print("a row was returned."); } rs.close(); // Close the statement. st.close(); 68 Aster Client Guide Connect Using Database Drivers JDBC Driver Disabling Cursors Cursors are enabled by default. To turn them off, set the FetchSize to zero. Assuming a statement st and a ResultSet rs, you would do this as shown here: st.setFetchSize(0); rs = st.executeQuery("SELECT * FROM mytable"); while (rs.next()) { System.out.print("many rows were returned."); } Test JDBC Connect Program This program can be used as a template for your JDBC connectivity: // JDBC Test Program import java.sql.Connection; import java.sql.DatabaseMetaData; import java.sql.DriverManager; import java.sql.SQLException; import java.sql.Statement; import java.sql.ResultSet; public class jdbctest { static String userid="beehive", password = "beehive"; // Change the IP address to the node you want to connect to. static String url = "jdbc:ncluster://10.60.3.100:2406/beehive"; //2.0 static Connection con = null; public static void main(String[] args) throws Exception { Connection con = getJDBCConnection(); if(con!= null){ try { int count = 0; System.out.println("Got Connection."); DatabaseMetaData meta = con.getMetaData(); System.out.println("getDriverName: "+meta.getDriverName()); System.out.println("getDriverVersion: "+meta.getDriverVersion()); System.out.println("getDriverMajorVersion: "+meta.getDriverMajorVersion()); System.out.println("getDriverMinorVersion: "+meta.getDriverMinorVersion()); System.out.println("getDatabaseProductName: "+meta.getDatabaseProductName()); System.out.println("getDatabaseProductVersion: "+meta.getDatabaseProductVersion()); System.out.println("getIdentifierQuoteString: "+meta.getIdentifierQuoteString()); System.out.println("\ngetSQLKeywords: "+meta.getSQLKeywords()); System.out.println("\ngetNumericFunctions: "+meta.getNumericFunctions()); System.out.println("\ngetStringFunctions : "+meta.getStringFunctions()); System.out.println("getSystemFunctions : "+meta.getSystemFunctions()); System.out.println("getTimeDateFunctions : "+meta.getTimeDateFunctions()); System.out.println("getSearchStringEscape : "+meta.getSearchStringEscape()); System.out.println("getExtraNameCharacters : "+meta.getExtraNameCharacters()); System.out.println("getCatalogTerm : "+meta.getCatalogTerm()); Aster Client Guide 69 Connect Using Database Drivers JDBC Driver System.out.println("getCatalogSeparator : "+meta.getCatalogSeparator()); System.out.println("getURL : "+meta.getURL()); System.out.println("getUserName : "+meta.getUserName()); System.out.println("getMaxCursorNameLength : "+meta.getMaxCursorNameLength()); System.out.println("getMaxSchemaNameLength : "+meta.getMaxSchemaNameLength()); System.out.println("getMaxProcedureNameLength : "+meta.getMaxProcedureNameLength()); System.out.println("getMaxCatalogNameLength : "+meta.getMaxCatalogNameLength()); System.out.println("getMaxColumnsInIndex : "+meta.getMaxColumnsInIndex()); System.out.println("supportsSubqueriesInComparisons : "+meta.supportsSubqueriesInComparisons()); System.out.println("getMaxConnections : "+meta.getMaxConnections()); System.out.println("getMaxColumnsInTable : "+meta.getMaxColumnsInTable()); System.out.println("isReadOnly : "+meta.isReadOnly()); System.out.println("\ngetCatalogs:"); ResultSet res = meta.getCatalogs(); while (res.next()) { System.out.println(res.getString(1)); } System.out.println("\ngetTables:"); res.close(); res = meta.getTables(null,null,"%",null); while (res.next()) { System.out.println(res.getString(1) +res.getString(2) +res.getString(3) +res.getString(4)); } System.out.println(""); res.close(); res = meta.getTableTypes(); System.out.println("\ngetTableTypes:"); while (res.next()) { System.out.println(res.getString(1)); } res.close(); Statement stmt=con.createStatement(); ResultSet rs = stmt.executeQuery("select count(*) from page_views"); while (rs.next()) { System.out.println(rs.getInt(1)); } rs.close(); stmt.close(); con.close(); } catch(SQLException ex) { System.err.println("SQLException: " + ex.getMessage()); } }else{ System.out.println("Could not Get Connection"); } } public static Connection getJDBCConnection(){ try { Class.forName("com.asterdata.ncluster.Driver"); } catch(java.lang.ClassNotFoundException e) { 70 Aster Client Guide Connect Using Database Drivers Configure Aster Database SQL Settings System.err.print("ClassNotFoundException: "); System.err.println(e.getMessage()); } try { con = DriverManager.getConnection(url,userid, password); } catch(SQLException ex) { System.err.println("SQLException: " + ex.getMessage()); } return con; } } Configure Aster Database SQL Settings This section explains how to set parameters that customize the behavior of Aster Database SQL. SQL Behavior Parameters Escape Character Handling enable_backslash_escapes: Set to “on” (the default), Aster Database allows backslash escape strings in your constants. The escape strings are C-like backslash escape sequences, each introduced with a backslash character (\). Set to “off ”, Aster Database does not recognize backslash escape strings in your constants unless the constant is prefixed with a letter E before its opening single quote. In any constant not prefixed with the letter E, a backslash is interpreted simply as a backslash character. Quoted-Identifier Handling enable_quoted_identifiers: This setting affects the way Aster Database processes strings enclosed in double-quote characters ("..."). With enable_quoted_identifiers='on' (the default), Aster Database follows the standard behavior of interpreting each double-quoted string as an identifier (a column, table, schema, or function name). With enable_quoted_identifiers='off', each double-quoted string is interpreted as a literal string constant. Any printable character may be represented in a double-quoted string. Setting the SQL Behavior Parameters At the SQL Prompt You can set these parameters from the SQL prompt by executing SET commands in the form: SET SESSION <parameter name> = {'<param setting>'}; For example: SET SESSION enable_backslash_escapes = {'on'|'off'}; and SET SESSION enable_quoted_identifiers = {'on'|'off'}; Aster Client Guide 71 Connect Using Database Drivers Configure Aster Database SQL Settings In the Aster Database ODBC Driver The Aster Database ODBC-NG driver allows you to set Aster Database SQL behavior parameters like, for example, enable_backslash_escapes. The connection parameters are: • enable_backslash_escapes={on|off} • enable_quoted_identifiers={on|off} These can be set in the DSN configuration (i.e., the odbc.ini file or Windows registry). Alternatively, a connecting application may set these options in the connection string when connecting without a registered DSN. Syntax for ODBC Commands This section lists some ODBC commands for which the syntax differs from the native Aster Database syntax. Note that comments and parameter markers are not allowed in these commands. However, the prepare/execute model is supported. COPY FROM/TO Semantics and overall flow of execution for COPY is the same as for the Aster Database Loader Tool. See “Aster Database Loader Tool” on page 125. Syntax COPY table [(column list)] FROM <quoted file name> ...COPY attributes or COPY table [(column list)] TO <quoted file name> ...COPY attributes The COPY command accepts a quoted file name and streams the data into or out of Aster Database using the Aster Database Loader Tool protocol for maximum throughput. INSTALL The INSTALL command is similar to the ACT command “\install <FILE> [[<SCHEMA>/ ]<FILE_ALIAS>]” on page 33, and supports SQL-MR security semantics. Syntax INSTALL FILE <quoted file name> [[<schema>/]<file alias>] • The schema name must be quoted if it contains spaces or mixed case. • The file alias must be quoted. UNINSTALL The UNINSTALL command supports SQL-MR security semantics. Syntax UNINSTALL FILE <quoted file name> [[<schema>/]<file alias>] 72 • The schema name must be quoted if it contains spaces or mixed case. • The file alias must be quoted. Aster Client Guide Connect Using Database Drivers Teradata Wallet DOWNLOAD The DOWNLOAD command is similar to the ACT command “\download [[<SCHEMA>/ ]<FILE_ALIAS>] <FILE>” on page 33, and supports SQL-MR security semantics. Syntax DOWNLOAD FILE <quoted file name> [[<schema>/]<file alias>] • The schema name must be quoted if it contains spaces or mixed case. • The file alias must be quoted. Teradata Wallet Teradata Wallet (TD Wallet) is a software utility that provides the users of an Aster Database client system with full and unrestricted access to their stored database passwords on that system, while at the same time protecting those passwords from being exposed in scripts. Each user on an Aster Database client has a TD Wallet. The TD Wallet securely stores the passwords that the user adds to the wallet. However, a user cannot access the passwords stored in the wallets of other users. Rather than using the passwords in scripts, you can use their corresponding names defined in your wallet. The ACT, JDBC, and ODBC drivers support TD Wallet. Wallet Contents A wallet contains a set of <name, value> pairs. These pairs consist of Unicode character sequences. Table 3 - 3: Wallet contents Item Description name The name of this password entry. This is the name you use to reference the password in scripts without exposing it. We recommend using meaningful names to help you determine what the password is used for. The name is case-sensitive. value The password. The password is not exposed in your scripts. Only the name of this password is exposed. The example in Table 3 - 4 shows entries in the TD Wallet of user client system user jdoe. Table 3 - 4: The Teradata Database passwords of user jdoe Aster Client Guide Name Value pwd_for_beehive_db s4t#gp6s_#4 pwd_for_customers_db nsdho_34f 73 Connect Using Database Drivers Teradata Wallet Table 3 - 4: The Teradata Database passwords of user jdoe Name Value pwd_for_clickstream_db oc_m_3nd234 TD Wallet Commands TD Wallet provides these commands: Table 3 - 5: TD Wallet commands Command Description tdwallet add name Adds an item to the wallet. When you run this command, tdwallet prompts you for the value component of the name/value pair. tdwallet addsk name Adds a string with the specified name (saved-key). tdwallet del name Deletes the specified item from the wallet. tdwallet list Lists the contents of the wallet. tdwallet chgpwd Changes or establishes the password of the wallet. tdwallet suppwd Supplies the password of the wallet. tdwallet forgetpwd Forgets the password of the wallet. tdwallet chgsavkey Changes or establishes the saved-key. tdwallet version Displays the version information for tdwallet. tdwallet help [topic] ... Displays the help information for tdwallet. Download TD Wallet TD Wallet support has been added in the back-end client connector layer, both the Java and C++ connectors. All the Aster Database clients based on the back-end client connector can use it. To use TD Wallet in Aster Database, you must first download it from http:// downloads.teradata.com/download/tools. The supported version of TD Wallet is version 14.00. Note that this version of TD Wallet does not currently support MacOS. Install and Configure TD Wallet on Linux ACT 1 Install TD Wallet. 2 Add a name/value pair to the wallet. $ ./tdwallet add mypassword Enter desired value for the string named "userpass": String named "userpass" added. 3 74 Install the latest ACT for Aster Database. Aster Client Guide Connect Using Database Drivers Teradata Wallet 4 Create the symbolic link “tdwalletdir” in the directory where ACT is installed. For example, these commands add the symbolic link: cd /home/beehive/work/multibranch/build/bin ln -s /opt/teradata/client/tdwallet tdwalletdir JDBC 1 Install TD Wallet. 2 Add a name/value pair to the wallet. $ ./tdwallet add mypassword Enter desired value for the string named "userpass": String named "userpass" added. 3 Get the JDBC Driver and tdwalletJNI.so. 4 Copy tdwalletJNI.so into /home/beehive/work/multibranch/builds/build-main/lib/. $ javac -classpath /home/beehive/work/multibranch/builds/build-main/ lib/noarch-aster-jdbc-driver.jar com/test/testjdbc/tdwallet.java $ java -classpath .:/home/beehive/work/multibranch/builds/build-main/ lib/noarch-aster-jdbc-driver.jar -Djava.library.path=/home/beehive/ work/multibranch/builds/build-main/lib/ com.test.testjdbc You can also find the tdwalletJNI.so file in the same directory as noarch-aster-jdbcdriver.jar in these packages: 5 • clients-platform-version-reversion.tar.gz • clients_all/platform... Create the symbolic link “tdwalletdir” in the directory where tdwalletJNI.so located is located. For example, these commands add the symbolic link: cd /home/beehive/work/multibranch/builds/build-main/lib/ ln -s /opt/teradata/client/tdwallet tdwalletdir Install and Configure TD Wallet on Windows ACT 1 Install TD Wallet. 2 Add a name/value pair to the wallet. E:\>tdwallet add mypassword Enter desired value for the string named "mypassword": String named "mypassword" added. 3 Use the TD Wallet password instead of the real password to log in to Aster Database. For example: E:\asterclientWIN\win64\bin>act.exe -h 192.65.197.130 -U beehive -w $tdwallet(mypassword) Welcome to act 05.10.00.00, the Aster nCluster Terminal. ... Aster Client Guide 75 Connect Using Database Drivers Teradata Wallet ODBC 1 Install TD Wallet. 2 Add name/value pairs to the wallet. 3 Install the latest ODBC Driver for Aster Database. 4 Use the TD Wallet password ($tdwallet(mypassword)) instead of the real password to log in to Aster Database. $tdwallet(mypassword) JDBC 1 Install TD Wallet. 2 Add name/value pairs to the wallet. E:\>tdwallet add mypassword Enter desired value for the string named "mypassword": String named "mypassword" added. 3 Get the latest JDBC Driver, and make the dependent library libtdwalletJNI.dll available in the library path. E:\>javac -classpath e:\asterclient\lib\noarch-aster-jdbc-driver.jar com\test\testjdbc\tdwallet.java E:\>java -classpath e:\asterclient\lib Djava.library.path=e:\asterclient\lib com.test.testjdbc 4 Use the TD Wallet password ($tdwallet(mypassword)) instead of the real password to log in to Aster Database. Usage After you install TD Wallet you can use the $tdwallet directive in place of your password. The syntax if the directive is: $tdwallet(name) where name is the name of a password entry in the TD Wallet. TD Wallet Password Processing Rules When TD Wallet processes $tdwallet directives, it follows these rules: 1 If specified, process name as follows: • 76 Replace “\)” with “)” Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications • Replace “\$” with “$” • Replace “\\” with “\” 2 Query the corresponding current user’s wallet for an item with a name matching the result of the processing in step 1. 3 Replace name with the value of the item found by the query in step 2. Example of TDWallet Use in SQL-MR queries In this query, TD Wallet replaces $tdwallet(abc) with the corresponding password stored on the client TD Wallet. select * from cfilter( on (select 1) partition by 1 database('beehive') userid('beehive') PASSWORD('$tdwallet(abc)') inputtable('cfilter_test') outputTable('cfilter_test1') inputColumns('item') joinColumns('tranid') droptable('true')); SSL Security for Aster Database-Client Communications Aster Database provides the option to use SSL to secure communications between Aster Database ODBC clients and the Aster Database queen. By default, an ODBC client running on Windows or Linux sends all of its communications to Aster Database over an SSL-encrypted connection. By default, results and other data returned from Aster Database to the client are NOT encrypted. This default configuration can be changed to suit your needs, as explained in the sections that follow. Port Number The Aster Database queen port number for SSL connections is the same as the regular client connection port: 2406. Port 2406 is multiplexed to support both secure sockets layer (SSL) connections and unencrypted connections. Aster Client Guide 77 Connect Using Database Drivers SSL Security for Aster Database-Client Communications Contents of This Section This section contains: • SSL-Related Files and Settings (page 78) • SSL Settings Reference (page 79) • Common SSL Configuration Scenarios (page 80) • Encrypting Data Traffic (page 83) • Adding AD-Based SSO Authentication to SSL-Secured Aster Database (page 84) • How to Set Configuration Parameters on the Queen (page 85) • How to Set Configuration Parameters on the Client (page 85) • Creating Certificates (page 90) SSL-Related Files and Settings The queen has the following files related to SSL communications and set-up: • Private key: Default one is /home/beehive/certs/server.key • Self-signed PEM certificate (that is, the base64-encoded ASCII representation of the certificate only; this is the version of the certificate that starts with "-----BEGIN CERTIFICATE-----" and ends with "-----END CERTIFICATE-----"). The default one is / home/beehive/certs/server.cert • Another copy of the self-signed certificate, also in PEM format, but this time formatted in X.509 structure. Note! This file may be missing from your Aster Database installation. To work around this problem, you must create the default PEM file from the server.cert file, saving it as /home/beehive/certs/server.pem. To create it, log in to the queen as root, change directories to /home/beehive/certs, and type: # openssl x509 -in server.cert -text >> server.pem • The queen’s SSL-related settings (see “SSL Settings Reference” on page 79) in the Aster Database queen configuration file, /home/beehive/config/procmgmtConfigs/ coordinator.cfg On the client side, the files related to SSL and its configuration are: 78 • Copy of the queen’s public key in PEM format. This is a copy of the queen’s server.pem. For example, you might save it as /home/mjones/certs/server.pem • The client’s SSL-related settings (see “Client-Side SSL Settings” on page 79), stored: • for Linux, in the client’s odbc.ini file; and • for Windows, in the ODBC parameter fields of the registry Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications SSL Settings Reference This section provides a reference to the available SSL settings on the Aster Database queen. Queen-Side SSL Parameters Below, we list the configuration flags that can be used on the queen to tune SSL behavior. Most queen-side flags have a corresponding client-side flag. When you change a flag on one side (client or server), you will typically have to make appropriate changes to the other side. • disallowPeerWithoutCertificates: If this flag is set, the client cannot communicate with its peer (server) without a valid certificate. This flag is defaulted to FALSE. • allowSelfSignedPeer: If this flag is set, Aster Database allows connections from clients with self-signed certificates. Defaults to TRUE. • Set either trustedCAFileName or trustedCAPath, depending on whether you have one or many CA certificates: • trustedCAFileName: The pathname of the single PEM-formatted CA certificate that Aster Database trusts. (You also have the option of trusting multiple CA certificates; see trustedCAPath, below.) Whenever the queen gets a certificate from the peer, the queen traverses the certificate chain to verify that the certificate specified by trustedCAFileName is part of the chain. If so, the peer is allowed to connect. • trustedCAPath: Directory containing PEM-formatted CA certificates that Aster Database trusts. The files inside this directory are looked up based on the CA subject name hash value. • sslCertificatePath: SSL certificate location. • sslPrivateKeyPath: SSL private key location. • sslFileType: The formatting type of the certificate. Set this to a string value of 1 for PEM-encoded certs (called “SSL_FILESYSTEM_PEM” on the client side) or 2 for ASN1encoded certs (called “SSL_FILETYPE_ASN1” on the client side). Default is 1. • secureMuleServer: If set, Aster Database will be configured to use a secure channel for its communication. If secureMuleServer is enabled, the configuration flags sslCertificatePath and sslPrivateKeyPath should be appropriately set. Client-Side SSL Settings Below, we list the configuration flags that can be used on the client side to tune SSL behavior. These settings work with the Aster Database ODBC client. Most client-side flags have a corresponding queen-side flag. When you change a flag on one side (client or server), you will typically have to make appropriate changes to the other side. • • Aster Client Guide EnableSSL: Enables/disables the use of SSL (string value of 0 for false, or 1 for true). • 0 = Disable SSL (default) • 1 = Enable SSL SSLEncryptReads: Determines whether the client expects data returned from database to be encrypted (string value of 0 for false, or 1 for true). • 0 = query results are unencrypted (the default) • 1 = query results are encrypted 79 Connect Using Database Drivers SSL Security for Aster Database-Client Communications • SSLAllowSelfSignedPeer: Determines whether the client allows peers with self signed certificates to communicate (string value of 0 for false, or 1 for true; default is 1). • SSLFileType: The certificate file type. A string value; one of: • SSL_FILETYPE_PEM (the default) • SSL_FILETYPE_ASN1 • SSLPrivateKeyPath: Path to the private key to be used. Optional. (A string value.) • SSLCertificatePath: Path to the SSL certificate to be used. (A string value.) • Set either SSLTrustedCADir or SSLTrustedCAFilename, depending on whether you have one or many CA certificates: • SSLTrustedCADir: Path to the directory containing CA certificates in PEM format. (A string value.) • SSLTrustedCAFilename: Filename of CA certificate in PEM format. (A string value.) Common SSL Configuration Scenarios This section presents common SSL configuration scenarios and shows how to set up each one. The scenarios are: • Scenario 1: Allowing connections from clients without certificates (page 80) • Scenario 2: Client has a copy of the queen’s public key (page 81) • Scenario 3: Client has a copy of a certificate you provide (page 82) • Scenario 4: Client must present a valid, CA-signed certificate (page 82) In all of the scenarios, we instruct you to set parameters on the queen and on the client. For instructions, see: • For the queen: “How to Set Configuration Parameters on the Queen” on page 85; and • For clients: “How to Set Configuration Parameters on the Client” on page 85 Scenario 1: Allowing connections from clients without certificates In this scenario, the Aster Database SSL configuration is edited to allow connections from clients that have no certificate. In such a configuration, any Aster Database ODBC client can establish an SSL-secured connection, provided the user authenticates successfully with Aster Database. Communications from the client to Aster Database (for example, queries submitted to the database) are SSL encrypted, and query results travel in the clear. Queen-Side Settings Make the following settings on the queen: 80 • disallowPeerWithoutCertificates=false • allowSelfSignedPeer=true • trustedCAFileName=/home/beehive/certs/server.pem • sslCertificatePath=/home/beehive/certs/server.cert • sslPrivateKeyPath=/home/beehive/certs/server.key • sslFileType=1 (A value of “1” means SSL_FILETYPE_PEM.) Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications • Ensure that secureWrites is set to false • Ensure that secureMuleServer is set to true • There is no need to set the trustedCAPath parameter. Client-Side Settings Make the following settings on each ODBC client: • EnableSSL=1 • SSLEncryptReads=0 • SSLAllowSelfSignedPeer=1 • SSLFileType=SSL_FILETYPE_PEM • There is no need to set the other SSL settings such as SSLPrivateKeyPath. Scenario 2: Client has a copy of the queen’s public key In this scenario, we edit Aster Database’s SSL configuration to allow connections only from clients that have a copy of queen’s public key. This scenario uses the default public key (/home/beehive/certs/server.pem) that is part of the standard queen installation. Queen-Side Settings Make the following settings on the queen: • disallowPeerWithoutCertificates=true • allowSelfSignedPeer=false • trustedCAFileName=/home/beehive/certs/server.pem • sslCertificatePath=/home/beehive/certs/server.cert • sslPrivateKeyPath=/home/beehive/certs/server.key • sslFileType=1 (A value of “1” means SSL_FILETYPE_PEM.) • Ensure that secureWrites is set to false. • Ensure that secureMuleServer is set to true. • There is no need to set the trustedCAPath parameter. Client-Side Settings Do the following: Aster Client Guide 1 Copy the queen’s public key (self-signed certificate), /home/beehive/certs/ server.pem, to the client. For this example, we will assume the client will store the public key as /home/jbloggs/certs/server.pem. 2 Make the following settings on each ODBC client: • EnableSSL=1 • SSLEncryptReads=0 • SSLAllowSelfSignedPeer=1 • SSLFileType=SSL_FILETYPE_PEM • SSLTrustedCADir=/home/jbloggs/certs/server.pem 81 Connect Using Database Drivers SSL Security for Aster Database-Client Communications Scenario 3: Client has a copy of a certificate you provide This scenario is the same as the preceding one, except that now we replace the default ASter Database certificate with your site’s own certificate. As before, this configuration allows connections only from clients that have a copy of queen’s public key. This scenario uses the public key, /home/beehive/certs/sampleco.pem. Queen-Side Settings Do the following: 1 Get your public key file and save it on the queen. For this example, we will save it as /home/beehive/certs/sampleco.pem on the queen. 2 Save the corresponding private key file on the queen. For this example, we will save it as /home/beehive/certs/sampleco.key on the queen. 3 Make the following settings on the queen: • disallowPeerWithoutCertificates=true • allowSelfSignedPeer=false • trustedCAFileName=/home/beehive/certs/sampleco.pem • sslCertificatePath=/home/beehive/certs/sampleco.pem • sslPrivateKeyPath=/home/beehive/certs/sampleco.key • sslFileType=1 (A value of “1” means SSL_FILETYPE_PEM.) • There is no need to set the trustedCAPath parameter. Client-Side Settings Do the following: 1 Copy the queen’s public key, /home/beehive/certs/sampleco.pem to the client. For this example, we will assume the client will store the public key as /home/jbloggs/ certs/sampleco.pem. 2 Make the following settings on each ODBC client: • EnableSSL=1 • SSLEncryptReads=0 • SSLAllowSelfSignedPeer=1 • SSLFileType=SSL_FILETYPE_PEM • SSLTrustedCADir=/home/jbloggs/certs/sampleco.pem Scenario 4: Client must present a valid, CA-signed certificate This scenario presents a more strict regime that requires all clients to present CA-signed certificates. In other words, clients cannot connect using a self-signed certificate, nor can they connect using a copy of the queen’s public key. Queen-Side Settings Do the following: 82 Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications 1 Get the root certificate of the CA (certificate authority) that signed your client’s certificate. Save the root certificate it on the queen. For this example, we will save it as /home/ beehive/certs/ca-cert.pem on the queen. 2 Make the following settings on the queen: • disallowPeerWithoutCertificates=true • allowSelfSignedPeer=false • trustedCAFileName=/home/beehive/certs/ca-cert.pem • sslFileType=1 (A value of “1” means SSL_FILETYPE_PEM.) • There is no need to set the trustedCAPath parameter if you use a single root certificate for all clients. Variation: If your client’s certificates were not all signed by the same CA, then you must set nCluster to recognize all the CA root certificates used to sign you clients’ certificates, like so: 1 Save the root certificates of all the signing CAs on the queen. 2 Set trustedCAPath to point to the directory that contains the root certificates. For example: • 3 trustedCAPath=/home/beehive/certs Un-set the queen configuration parameter, trustedCAFileName, by setting it to no value at all. For example: • trustedCAFileName= Client-Side Settings Do the following: 1 Save the client’s public key on the client. For this example, we will assume the client will store its public key as /home/jbloggs/certs/my-client-cert.pem. 2 Make the following ODBC settings on the client: • EnableSSL=1 • SSLEncryptReads=0 • SSLAllowSelfSignedPeer=0 • SSLFileType=SSL_FILETYPE_PEM • SSLTrustedCADir=/home/jbloggs/certs/my-client-cert.pem Repeat the above steps for all ODBC client machines in your environment. Encrypting Data Traffic By default, Aster Database encrypts only control-path communications, such as the login credentials an ODBC user submits, and the queries he or she submits. For implementations that demand greater security, Aster Database also gives you the option of SSL-encrypting the data traffic as the queen returns it to the client. Aster Client Guide 83 Connect Using Database Drivers SSL Security for Aster Database-Client Communications All other things being equal, switching any network connection from unencrypted to SSLencrypted has the effect of reducing the maximum available rate of data transmission on that connection. To set this up: Queen-Side Settings Make the following settings on the queen: • secureWrites=true Client-Side Settings Make the following settings on each ODBC client: • SSLEncryptReads=1 Adding AD-Based SSO Authentication to SSL-Secured Aster Database If your Aster Database is configured to authenticate users against Active Directory (“AD”), you can configure your Aster Database ODBC clients to authenticate against AD, too. With this configuration in place, each ODBC client will be required to authenticate against AD when it tries to connect to Aster Database. If the ODBC client authenticates successfully, an SSL channel is established automatically for communication between Aster Database and the client. To set this up: Queen-Side Settings Make the following settings on the queen: 1 Set up Aster Database user authentication as explained in the Aster Database User Guide. 2 Configure the queen SSL configuration flags as described in one of the scenarios above which you are planning on implementing. (For example, see “Scenario 1: Allowing connections from clients without certificates” on page 80 Client-Side Settings Do the following: 1 Set the flags in the client’s ODBC configuration file or registry as described in the scenario. 2 Set the EnableSSO flag in the client’s ODBC configuration file: • 3 84 EnableSSO=1 If EnableSSO is set to 1, you must also: • Ensure that ServerIP is set to the fully qualified domain name of the Aster Database queen and not to an IP address. • For 64-bit Linux machines: The ODBC driver assumes that libvas-gssapi.so is present at /opt/quest/lib64/. If /opt/quest/lib64/libvas-gssapi.so does not exist, locate libvas-gssapi.so by referring to the VAS documentation and set the GSSPath parameter to point to the installed location of libvas-gssapi.so. For example, if libvas-gssapi.so is deployed at /usr/lib64, then the GSSPath parameter needs to be set to /usr/lib64 in the ODBC.ini config file as shown below: Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications GSSPath=/usr/lib64 • For 32-bit Linux machines: The ODBC driver assumes that libvas-gssapi.so is present at /opt/quest/lib/. If /opt/quest/lib/libvas-gssapi.so does not exist, locate libvas-gssapi.so by referring to the VAS documentation and set the GSSPath parameter to point to the installed location of libvas-gssapi.so. For example, if libvas-gssapi.so is deployed at /usr/lib, then the GSSPath parameter needs to be set to /usr/lib in the ODBC.ini config file as shown below: GSSPath=/usr/lib How to Set Configuration Parameters on the Queen Many procedures in this document ask you to set configuration parameters on the Aster Database queen. Below, we explain how to do this. Persistent Setting of Parameters To edit a setting (and have your edit survive reboots), make the setting as shown below. Settings you change in this manner will survive reboots and soft restarts but they will not survive upgrades. 1 Login as root into queen and use a text editor to open the file, /home/beehive/config/ procmgmtConfigs/coordinator.cfg 2 Find the “queenExec” section which looks like: "taskName": "queenExec", "nodeIps": "REPLACE_NODE_IP", "executableLocation": "/home/beehive/bin/exec/queenExec", "maxTaskRestarts": -1 3 Add the executableArgs section to the above section as shown below: "executableArgs": "<flags in CSV format>" For example, executableArgs for Scenario 1 (described earlier in this chapter) will look like: "taskName": "queenExec", "nodeIps": "REPLACE_NODE_IP", "executableLocation": "/home/beehive/bin/exec/queenExec", "maxTaskRestarts": -1, "executableArgs": "--disallowPeerWithoutCertificates=false, --allowSelfSignedPeer=true,--trustedCAFileName=/home/beehive/certs/ server.pem,--sslCertificatePath=/home/beehive/certs/server.cert, --sslPrivateKeyPath=/home/beehive/certs/server.key,--sslFileType=1" 4 Soft restart and activate the cluster. How to Set Configuration Parameters on the Client This section explains how to set up the 2nd-generation Aster Database ODBC client for SSL communications with Aster Database. Consult the section that applies to your operating system: Aster Client Guide • “Linux ODBC DSN Set-Up”, below; or • “Windows ODBC DSN Set-Up” on page 86 85 Connect Using Database Drivers SSL Security for Aster Database-Client Communications Linux ODBC DSN Set-Up On a UNIX-like systems, DSNs are added by setting parameters in the odbc.ini file Sample ODBC.INI: This sample assumes your queen machine is called cqueen.asterengqa.com and that you are following Scenario 1 (outlined earlier in this chapter): [ODBC Data Sources] AsterTest=AsterDriverTest [AsterTest] Driver=AsterDriverTest SERVER=cqueen.asterengqa.com DATABASE=beehive PORT=2406 UID=testuser13 PWD=testuser133 SQLSupportedConversions=3 NumericAndDecimalAsDouble=1 EnableSSO=0 GSSPath= EnableSSL=1 SSLEncryptReads=0 SSLAllowSelfSignedPeer=1 SSLFileType=SSL_FILETYPE_PEM SSLPrivateKeyPath= SSLCertificatePath= SSLTrustedCADir= SSLTrustedCAFilename= Sample ODBCINST.INI: This sample assumes you have installed the driver in /Drivers/AsterDriver/ODBCDriver: [AsterDriverTest] Driver=/Drivers/AsterDriver/ODBCDriver/libAsterDriver_unixODBC.so IconvEncoding=UCS-4LE Sample aster.ini: This sample assumes you want to log error messages in /Drivers/AsterDriver: [driver] DriverManagerEncoding=UTF-32 DSILogging=1 ErrorMessagesPath=/Drivers/AsterDriver/ErrorMessages Windows ODBC DSN Set-Up On Windows, DSNs are added by modifying the Windows Registry using regedit.exe or using a .reg file. The standards surrounding key names (such as whether to use “Server” or “servername”) used in a connection string for ODBC driver are loose, so please take care to follow the examples we provide. Details for setting the values in the registry are given below. Choose the section that fits your needs and client type: 86 Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications • “Adding a driver to a Windows 32-bit operating system”, below. • “Adding a 64-bit driver to a Windows 64-bit operating system” on page 88 • “Adding a 32-bit driver to a Windows 64 bit operating system” on page 89 Adding a driver to a Windows 32-bit operating system Make the following registry settings: Driver for 32-bit Windows Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\AsterDriver32] "Driver"="C:\\AsterDriver-Win32\\ODBCDriver\\AsterDataODBCDSII.dll" [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\ODBC Drivers] "AsterDriver32"="Installed" [HKEY_LOCAL_MACHINE\SOFTWARE\Aster] [HKEY_LOCAL_MACHINE\SOFTWARE\Aster\Driver] "DSILogging"="0" "ErrorMessagesPath"="C:\\AsterDriver-Win32\\ErrorMessages" "DriverManagerEncoding"="UTF-16" The values in the keys above can be modified depending on where the driver is located on the local machine and what the name of the driver should be. The values above are based on the assumption that the driver folder is at "C:\AsterDriver-Win32" and the name of the driver is "AsterDriver32". For an example .reg file that makes these settings, contact Teradata support. DSN for 32-bit Windows To add a DSN for the above driver setup in the registry, make these entries in the registry: [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\AsterDSN32] "driver"="AsterDriver32" "SERVER"="10.51.12.100" "DATABASE"="beehive" "PORT"="2406" "UID"="beehive" "PWD"="beehive" "SQLSupportedConversions"="3" "NumericAndDecimalAsDouble"="1" "EnableSSO"="0" "SSLKeyFile"="\"\"" "GSSPath"="" "EnableSSL"="1" "SSLEncryptReads"="1" "SSLAllowSelfSignedPeer"="0" "SSLFileType"="SSL_FILETYPE_PEM" "SSLPrivateKeyPath"="\"\"" "SSLCertificatePath"="\"\"" "SSLTrustedCADir"="\"\"" "SSLTrustedCAFilename"="\"\"" "EnableSSO"="0" [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\ODBC Data Sources] "AsterDSN32"="AsterDriver32" Aster Client Guide 87 Connect Using Database Drivers SSL Security for Aster Database-Client Communications Adding a 64-bit driver to a Windows 64-bit operating system Below we list the registry entries for setting up a 64-bit driver on Windows. Make the registry settings shown below: • 64-bit driver for 64-bit Windows • 64-bit DSN for 64-bit Windows 64-bit driver for 64-bit Windows Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\AsterDriver64] "Driver"="C:\\AsterDriver-Win64\\ODBCDriver\\AsterDataODBCDSII.dll" [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\ODBC Drivers] "AsterDriver64"="Installed" [HKEY_LOCAL_MACHINE\SOFTWARE\Aster] [HKEY_LOCAL_MACHINE\SOFTWARE\Aster\Driver] "DSILogging"="0" "ErrorMessagesPath"="C:\\AsterDriver-Win64\\ErrorMessages" "DriverManagerEncoding"="UTF-16" 64-bit DSN for 64-bit Windows To add a DSN for the above driver setup in the registry, make these entries in the registry: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\AsterDSN64] "driver"="AsterDriver64" "SERVER"="10.51.12.100" "DATABASE"="beehive" "PORT"="2406" "UID"="beehive" "PWD"="beehive" "SQLSupportedConversions"="3" "NumericAndDecimalAsDouble"="1" "EnableSSO"="0" "SSLKeyFile"="\"\"" "GSSPath"="" "EnableSSL"="1" "SSLEncryptReads"="1" "SSLAllowSelfSignedPeer"="0" "SSLFileType"="SSL_FILETYPE_PEM" "SSLPrivateKeyPath"="\"\"" "SSLCertificatePath"="\"\"" "SSLTrustedCADir"="\"\"" "SSLTrustedCAFilename"="\"\"" "EnableSSO"="0" [HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\ODBC Data Sources] "AsterDSN64"="AsterDriver64" Above, the name of the DSN for the 64-bit driver is AsterDSN64. The server being connected to is 10.51.12.100. 88 Aster Client Guide Connect Using Database Drivers SSL Security for Aster Database-Client Communications Adding a 32-bit driver to a Windows 64 bit operating system Some applications running on a Windows 64-bit machine require a 32-bit driver. The 32-bit operations work on Windows 64 bit machines under a mechanism called “Wow6432Node.” Below we list the registry entries for setting up 32-bit drivers on Windows. Make the registry settings shown below: • 32-bit driver on 64-bit Windows • 32-bit DSN on 64-bit Windows 32-bit driver on 64-bit Windows Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBCINST.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBCINST.INI\AsterDriver32 ] "Driver"="C:\\AsterDriver-Win32\\ODBCDriver\\AsterDataODBCDSII.dll" [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBCINST.INI\ODBC Drivers] "AsterDriver32"="Installed" [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Aster] [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Aster\Driver] "DSILogging"="0" "ErrorMessagesPath"="C:\\AsterDriver-Win32\\ErrorMessages" "DriverManagerEncoding"="UTF-16" The values in the keys above can be modified depending on where the driver is located on the local machine and what the name of the driver should be. The values above are based on the assumption that the driver folder is at "C:\AsterDriver-Win32" for 32 bit drivers and the name of the driver is "AsterDriver32". For an example .reg file that makes these settings, contact Teradata support. 32-bit DSN on 64-bit Windows To add a DSN for the above driver setup in the registry, make these entries in the registry: [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC] [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI] [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI\AsterDSN32] "driver"="AsterDriver32" "SERVER"="10.51.12.100" "DATABASE"="beehive" "PORT"="2406" "UID"="beehive" "PWD"="beehive" "SQLSupportedConversions"="3" "NumericAndDecimalAsDouble"="1" "EnableSSO"="0" "SSLKeyFile"="\"\"" "GSSPath"="" "EnableSSL"="1" "SSLEncryptReads"="1" "SSLAllowSelfSignedPeer"="0" "SSLFileType"="SSL_FILETYPE_PEM" "SSLPrivateKeyPath"="\"\"" "SSLCertificatePath"="\"\"" "SSLTrustedCADir"="\"\"" Aster Client Guide 89 Connect Using Database Drivers Process SQL Statements in JDBC "SSLTrustedCAFilename"="\"\"" [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI\ODBC Data Sources] "AsterDSN32"="AsterDriver32" Above, the name of the DSN for the 32-bit driver is AsterDSN32. The server being connected to is 10.51.12.100. Creating Certificates Creating a Self-Signed Certificate You may to create your own self-signed certificate. The high-level steps for doing this using the OpenSSL tool are shown here, assuming you create a private key file called “host.key”: openssl genrsa 1024 > host.key chmod 400 host.key openssl req \-new \-x509 \-nodes \-sha1 \-days 365 \-key host.key > host.cert Regenerating SSL Certificates/Private Key During Aster Database Installation You may manually regenerate the default private key and certificate using the following command on the queen: 1 Working as root user, perform a soft shutdown on Aster Database: # ncli system softshutdown 2 Wait for the system to shut down, and then run the resetCert command: # /home/beehive/bin/lib/configure/ConfigureNCluster.py --resetCert This will back up any existing server.key/server.cert files present in /home/beehive/certs and generate a new private key and certificate file. Process SQL Statements in JDBC You can execute SQL statements by using a Statement or a PreparedStatement instance. Sometimes it is more convenient to use a PreparedStatement object for sending SQL statements to the database. This special type of statement is derived from the more general class, Statement, that you already know. If you want to execute a Statement object many times, it normally reduces execution time to use a PreparedStatement object instead. The main feature of a PreparedStatement object is that, unlike a Statement object, it is given an SQL statement when it is created. The advantage to this is that in most cases, this SQL statement is sent to the DBMS right away, where it is compiled. As a result, the PreparedStatement object contains not just an SQL statement, but an SQL statement that has been precompiled. This means that when the PreparedStatement is executed, the 90 Aster Client Guide Connect Using Database Drivers Process SQL Statements in JDBC DBMS can just run the PreparedStatement SQL statement without having to compile it first. Process a Simple Query in JDBC This example issueS a simple query and printS out the first column of each row using a Statement. Statement st = conn.createStatement(); ResultSet rs = st.executeQuery("SELECT * FROM mytable WHERE columnf = 500"); while (rs.next()) { System.out.print("Column 1 returned "); System.out.println(rs.getString(1)); } rs.close(); st.close(); This example issues the same query as before but uses a PreparedStatement and a bind value in the query. int fvalue = 500; PreparedStatement st = conn.prepareStatement("SELECT * FROM mytable WHERE columnf = ?"); st.setInt(1, fvalue); ResultSet rs = st.executeQuery(); while (rs.next()) { System.out.print("Column 1 returned "); System.out.println(rs.getString(1)); } rs.close(); st.close(); The instance returns a ResultSet containing the results. Aster Database does not support cursor-based server-side caching of results which would save a small amount of time in parsing. To retrieve data from the ResultSet instance, call next(), which returns true if there are results. The ResultSet instance can be closed by calling close(), or if another query is issued using the same Statement instance that was used to create this ResultSet. Statement stmt=con.createStatement(); ResultSet rs = stmt.executeQuery("select count(*) from page_views"); while (rs.next()) { System.out.println(rs.getInt(1)); } rs.close(); stmt.close(); You can also run Update/Insert/Delete statements and Create/Drop Table statements using the same method. Aster Client Guide 91 Connect Using Database Drivers JDBC Troubleshooting and Limitations JDBC Troubleshooting and Limitations Infinity and Negative Infinity Timestamp Conversion The JDBC driver converts 'infinity' and '-infinity' to specific TimeStamp values (a very large and a very tiny TimeStamp) in the ResultSet.getObject API, which can cause unexpected results on the client side. Aqua Data Studio and other client tools, like SQL Assistant JAVA edition also have this issue. These tools do not support 'infinity' and '-infinity' in the timestamp datatype. If they encounter these values, they will display specific TimeStamp values (a very large and a very tiny TimeStamp). If you want to retain the negative infinity and infinity values, you can use ResultSet.getString instead. Then you will be able to get the correct values for infinity and negative infinity, without this conversion to a very small or very large timestamp value. Thread Safety The Aster legacy JDBC driver (pre-5.0.3) supports thread safety for all JDBC objects. The new Aster JDBC Driver (5.0.3 and later), does not support JDBC object thread safety in all cases. If you have developed code based on the legacy JDBC driver which uses the same instance of a JDBC object across multiple threads simultaneously, you may have to modify your code to ensure that JDBC object thread safety is implemented by your application. Connect Reporting Tools to Aster Database This section shows how to connect various reporting and query tools to Aster Database: • Connect Aqua Data Studio to Aster Database (page 92) • Connect MicroStrategy to Aster Database (page 93) • See also: Connect Using Database Drivers (page 43) Connect Aqua Data Studio to Aster Database ADS support is being deprecated. AquaFold’s ADS lets you perform DDL operations and query data interactively, and it provides tools that help you write and manage queries efficiently. ADS is a third-party tool available for purchase from Aqua Fold directly. Aster Database is compatible with ADS version 10.0.2 with patch ads-10.0.7_03-patch.zip. 92 Aster Client Guide Connect Using Database Drivers Connect Reporting Tools to Aster Database Install ADS This section explains how to install ADS on your client workstation and connect to an Aster Database. 1 Download Aqua Data Studio version 10.0.2 or later from http://www.aquafold.com/ downloads.html 2 Install Aqua Data Studio on your client workstation as explained in your version of the Aqua Data Studio documentation at http://docs.aquafold.com/ aquadatastudio_11_documentation.html 3 Start Aqua Data Studio. 4 Select the command Server: Register Server. 5 In the list, select the “Aster nCluster Driver.” 6 Fill in the tabs as required. Apply the ADS Patch 1 Download the patch from AquaFold: http://dd1.aquafold.com/download/v10.0.0/ads-10.0.7_03-patch.zip 2 Unzip the patch files. 3 Replace the files under C:\Program Files\Aqua Data Studio 10.0 - 64bit\lib with the files from ads-10.0.7_03-patch.zip. 4 Restart ADS. Connect MicroStrategy to Aster Database Observe the following guidelines when connecting MicroStrategy to Aster Database: Versions MicroStrategy 9 or later is required, and Aster Database 5.0 or later is required. Platforms Supported Aster Database supports Intelligence Server clients running on Windows XP and Windows Vista with the Aster Database ODBC driver for Windows. Limitations Aster Database is only certified as a warehouse with MicroStrategy. Aster Database cannot be used as a repository. Set-up Instructions To connect MicroStrategy to Aster Database, follow the steps below. Prerequisites Aster Client Guide 1 Microstrategy supports the ODBC Driver Manager bundled with the Aster Database ODBC driver and the Windows Driver Manager only. 2 Make sure the following patches are applied to your MicroStrategy installation: 93 Connect Using Database Drivers Connect Reporting Tools to Aster Database MicroStrategy 8: Contact MicroStrategy Customer support for • DATABASE.PDS file that's certified with Aster Database • DTMAPPING.PDS file that's certified with Aster Database MicroStrategy 9: Contact MicroStrategy Customer support for • DTMAPPING.PDS file that works with Aster Database MicroStrategy 9.0.1 and above: No changes required Install Drivers On the client machine where MicroStrategy runs, install the database drivers: 1 Install the Aster Database ODBC driver. See “ODBC Driver” on page 44 for installation instructions. 2 Install the MicroStrategy VLDB driver, version 9 or later. Connecting to an Aster Database 1 Open your MicroStrategy project and choose Intelligence Server as your connection option. 2 Use the Configuration Wizard to create a New Project Source. 3 Create a New Database Instance and give it a name. 4 In the Connection Type field, choose Aster Database nCluster. 5 Create a New Database Connection and give it a name. 6 Create a new ODBC System DSN and give it a name. Once you have saved the System DSN, you can create reports in MicroStrategy that use the data from your Aster Database. In your Logical Table definitions in MicroStrategy, you can create queries that use special Aster Database tools such as nPath. Best Practices By following the guidelines below, you can avoid common errors in Aster DatabaseMicroStrategy integration. Schema changes • If your schema contains NUMERIC(X,0) type columns, you should replace these with INT or BIGINT type columns for a higher probability of success with existing MicroStrategy reports. • For your small- to medium-sized tables that have no BIGINT or INT columns, you should create the tables in Aster Database as replicated dimension tables. This takes more space in the cluster, but works better with MicroStrategy. Operational items When pointing an existing MicroStrategy report from another database to Aster Database, the warehouse schema should be “UPDATED” the first time when pointed to an Aster Database Instance. This updates the metadata inside MicroStrategy for correct MicroStrategy columns. This is particularly important when a schema is ported from some other database to Aster Database. 94 Aster Client Guide Connect Using Database Drivers Connect Reporting Tools to Aster Database SQL query changes • Aster Client Guide To use Aster Database’s nPath and other custom SQL-MapReduce functions: • Reports must be created as free-form SQL type reports in MicroStrategy. • Teradata Aster recommends that you embed the query in an MicroStrategy logical table and build the reports on top of that. • If there are existing reports that need to run on Aster Database, some tweaking may be needed in the VLDB Settings of MicroStrategy depending on the complexity of the dynamically generated queries. For example, you may need to turn off TEMP table creation, and so on. This is very rare, but editing VLDB settings can be useful in some cases. • If the following error appears in queries: “Unable to find function to convert from <x> to <y>”, and if x and y are datatypes like int, float, and so on, then you may need a workaround to explicitly cast the column types. Contact Teradata Support for help on this. 95 Connect Using Database Drivers Connect Reporting Tools to Aster Database 96 Aster Client Guide CHAPTER 4 Tools for .NET Environments Teradata Aster provides a suite of tools that enable you to use Aster Database as a data source in your .NET applications and reports. These tools include: • nClusterDNProvider—A managed .NET Data Provider (nClusterDNProviderInstaller_ng_i386.msi, nClusterDNProviderInstaller_ng_x64.msi, a database driver for ADO.NET) that allows Microsoft SQL Server Integration Services (SSIS) and .NET client applications to connect to an Aster Database server. • Tutorials, including the sample program program.cs that show how to use the Aster Database tools for .NET in a Microsoft reporting and BI environment. This section explains how to install these tools and how to start writing applications and reports that use databases in Aster Database. • Installing the Aster Database ADO.NET Driver (nClusterDNProvider) (page 97) • Performing SSIS Data Loading with nClusterDNProvider (page 99) • Sample Program for ADO.NET (page 113) Note: The nCluster OleDB Driver (nClusterOleDbInstaller_i386.msi) is no longer supported. Installing the Aster Database ADO.NET Driver (nClusterDNProvider) This section describes how to install nClusterDNProvider, the Windows Aster Database database driver for ADO.NET. This driver is a managed .NET Data Provider for Aster Database and allows .NET client applications (for example, Console, WinForms, ASP.NET, and Web Services) to send and retrieve data from Aster Database. Installation Prerequisites Make sure the .NET 2.0 framework is installed on your workstation before installing nClusterDNProvider. You can download it from: http://www.microsoft.com/downloads Installing nClusterDNProvider Follow the steps below to install nClusterDNProvider: Aster Client Guide 97 Tools for .NET Environments Installing the Aster Database ADO.NET Driver (nClusterDNProvider) 1 Get the Aster ADO.NET driver installer, whose name depends on your operating system: • For 32-bit, the installer is nClusterDNProviderInstaller_i386.msi • For 64-bit, the installer is nClusterDNProviderInstaller_x64.msi by doing one of the following: • Copy the client package for your operating system from your queen node. Clients are located in the directory /home/beehive/clients_all on the queen. • Download it from http://downloads.teradata.com/download/tools. 2 Place the installer on your client machine. 3 Run the installer. a In the Welcome screen, click Next. b In the Select Installation Folder screen: Choose the installation location. Specify who should be allowed to use the driver. Click Next. c In the Confirm window, click Next. d When the Installation Complete window appears, click Close. Note: Before proceeding, make sure that nClusterDNProvider is installed correctly. There should be several DLL files and the RegisterAsterProvider.exe file in the .NET provider install path (for example, C:\Program Files (x86)\Aster Data Systems\nCluster ADO.NET Provider (x86)). If not, nClusterDNProvider will not work. You can also check the installation from the Control Panel. If the size of nClusterADO.Net Provider (x86) is less than 1 MB and the version is 1.0.x, you need a new driver. Installing SQLServer and SSAS after installing the ADO.NET provider It is recommended to install SQLServer and SSAS before installing the Aster Database ADO.NET provider. When you install the ADO.NET provider, the installer checks the Windows registry to see if SSAS is installed. Then the correct files are installed and configured automatically. But in cases where the ADO.NET provider is already installed before SSAS/SQL Server, you will need to do the following steps before they can work together: 1 Find these four files: • Install.cmd • Install.vbs • Install.reg • nCluster.xsl These files can be found in the folder where the Aster Database ADO.NET Provider is installed, typically something like C:\Program Files (x86)\Aster Data Systems\nCluster ADO.NET Provider (x86). 2 98 Install the files manually. Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider When installed, these files enable the Aster Database ADO.NET provider to appear as an option within Microsoft Visual Studio and Microsoft Analysis Services. To install the files, run the Install.cmd file from the command line with the /codebase option. The /codebase option should have the full path to the Simba.Net.dll file. The syntax to run the command is: Install.cmd /codebase <full path>\Simba.Net.dll For example, the command will look similar to: Install.cmd /codebase "C:\Program Files (x86)\Aster Data Systems\nCluster ADO.NET Provider (x86)\Simba.Net.dll" If you are installing the ADO.NET provider to work with SQL Server 2005, you must use the /vs2005 option, as in: Install.cmd /vs2005 /codebase <full path>\Simba.Net.dll 3 Put the Aster Database cartridge (the file nCluster.xsl) into the corresponding directory for SSAS. This file should be placed into all the cartridge directories for SSAS, which may include all of the following directories, though the directories on your Windows machine may differ slightly: SQL Server 2008 C:\Program Files\Microsoft Analysis Services\AS OLEDB\10\Cartridges C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP \bin\Cartridges C:\Program Files\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\ IDE\DataWarehouseDesigner\UIRdmsCartridge C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE \PrivateAssemblies\DataWarehouseDesigner\UIRdmsCartridge SQL Server 2005 C:\Program Files\Microsoft SQL Server\MSSQL.3\OLAP\bin\Cartridges C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7 \IDE\DataWarehouseDesigner\UIRdmsCartridge C:\Program Files\Microsoft Visual Studio 8\Common7\IDE \PrivateAssemblies\DataWarehouseDesigner\UIRdmsCartridge Sample Program to Demonstrate the Driver The Program.cs program (“Sample Program for ADO.NET” on page 113) demonstrates the features supported by nClusterDNProvider. Performing SSIS Data Loading with nClusterDNProvider This section is a tutorial that shows you how to create a Data Flow task in SSIS for transferring data from a table to a flat file via the Aster Database ADO.NET driver, nClusterDNProvider. Overview SSIS is a tool for building extract, transform, and load (ETL) jobs. In .NET environments, SSIS provides a fast way to extract data from or load data into your Aster Database databases. Aster Client Guide 99 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider To connect to Aster Database, SSIS requires the Aster Database driver, nClusterDNProvider. This tutorial shows you how to set up SSIS to use the driver and provides an example for exporting data from Aster Database to a flat file. Procedure To set up SSIS to use nClusterDNProvider, follow these steps: 1 Run Microsoft SSIS 2 Choose File: New > Project to create a new integration project. 3 In the New Project dialog: 4 a Select Integration Services Project. b Enter your project a name c Click OK. In the Connection Manager tab at the bottom of the window, add a new Connection. To do this, right-click and select New ADO.NET Connection. 100 Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 5 In the Configure ADO.NET Connection Manager window, click New. In the next couple of steps we’ll create a widget (a database connection definition) that allows SSIS to connect to your Aster Database. Aster Client Guide 6 In the Connection Manager dialog, in the Provider drop-down box, choose Aster Data Provider, and click OK. 7 In the Connection Manager dialog, make the following settings: a Set the Database to the name of the database in Aster Database you want to connect to. b Set the Host to your Aster Database queen IP address. c Set Port to the queen port, which is usually 2406. d Set the User Id and Password to the credentials of a user with sufficient rights on your database in Aster Database. e Click OK. Note the name of the connection definition you are saving. Click OK again. 101 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 8 Choose View > Toolbox. 9 Drag and drop the Data Flow Task from the Control Flow Items list into the Control Flow designer panel. 10 Click the Data Flow tab. 102 Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 11 From the Data Flow Sources panel, drag an ADO NET Source object into the Data Flow panel. 12 To make this source a connection to your database in Aster Database, double-click ADO NET Source to configure it. The ADO.NET Source Editor dialog box appears. Aster Client Guide 103 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 13 In the ADO.NET Source Editor dialog box, configure the Connection Manager properties: a Click Connection Managers. b From the Data access mode drop-down menu, choose SQL command. c In the SQL command text field, enter a SQL query. 14 In the ADO.NET Source Editor dialog box, check the Columns property values: 104 a Click Columns. b Check the External Column and Output Column values. Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 15 In the ADO.NET Source Editor dialog box, configure the Error Output properties: a Click Error Output. b For each output column, set its Error property to Redirect row. 16 In the ADO.NET Source Editor dialog box, click OK. Aster Client Guide 105 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 17 From the Data Flow Destinations panel, drag a Flat File Destination object into the Data Flow panel. 18 Connect the green arrow of the ADO NET Source object to the Flat File Destination object. 106 Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 19 Configure the Flat File Destination object. Aster Client Guide a Double-click the Flat File Destination object. b In the Flat File Destination Editor, click Connection Manager. c Click New. d In the Flat File Format dialog box, click Delimited. e Click OK. 107 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 108 f In the Flat File Connection Manager Editor dialog box, in the Connection manager name field, enter the name of the output file of the connection manager. g Click Browse. h Select an output file and click Open. i Click OK. j In the Flat File Destination Editor, click Mappings. k Set the correct column mappings for the Flat File Destination object. l Click OK. Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider 20 From the Data Flow Destinations panel, drag a Flat File Destination object into the Data Flow panel. This object is the destination for all error records. 21 Connect the error output (red arrow) of the ADO NET Source object to Flat File Destination you just added. 22 Create another Flat File Connection Manager for the new Flat File Destination object that serves as the error destination. Aster Client Guide a Double-click the Flat File Destination object. b In the Flat File Destination Editor, click Connection Manager. c Click New. d In the Flat File Format dialog box, click Delimited. e Click OK. f In the Flat File Connection Manager Editor dialog box, in the Connection manager name field, enter the name of the output file to be used to store all error records. g Click Browse. h Select an output file and click Open. i Click OK. j In the Flat File Destination Editor, click Mappings. 109 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider k Set the correct column mappings for the error destination. l Click OK. 23 Save the project. 24 Run the project with debugging (Debug > Start Debugging). You can check the progress of the workflow in the Progress panel. 110 Aster Client Guide Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider When the export is successful, the flat file source and destination objects in the Data Flow pane turn green. If the export is not successful, the ADO NET Source object turns red. For example, you might get an exception like “Error: 0xC0047062 at Data Flow Task, ADO NET Source [16]: System.ArgumentException: Error loading assembly: C:\Program Files (x86)\Aster Data Systems\nCluster ADO.NET Provider (x86)\AsterDataC#DSII.dll.” In this case, configure the 32-bit Runtime in SSIS for the project or you install the 64-bit NClusterDNProvider. The Microsoft SQL Server Business Intelligence Studio (BIDS) is a Visual Studio plug in. For SQL Server 2008, this edition is 32-bit only. This requires using 32-bit drivers, including for Aster Database. However, in the 64-bit versions of Windows, there is a project flag that you must set to allow 32-bit drivers to operate. Select the project, right-click, and choose Properties. Then, set Run64BitRunTime to False. If not, you get an architecture mismatch and other connection errors. Aster Client Guide 111 Tools for .NET Environments Performing SSIS Data Loading with nClusterDNProvider Similarly, to run the package outside of BIDS—such as running a SQL Server Agent job or running the package by itself—click the tab Execution options and check the check box Use 32 bit run time. 112 Aster Client Guide Tools for .NET Environments Sample Program for ADO.NET Sample Program for ADO.NET This section provides a sample program, Program.cs that demonstrates some of the features supported by the Aster Database ADO.NET driver, nClusterDNProvider. Features Demonstrated in the Sample The Program.cs sample shows the following: • Insert into int column • DataReader • DataSet • Prepared query execution • Serializing to XML • CopyIn • CopyOut • Metadata operations • Indefinite timeout • Exception handling Steps to Build and Test the Sample Program 1 Create a new Visual C# Console Application project called “Program.cs”. 2 Replace the generated Program.cs with the Program.cs provided below. 3 Ensure that the reference to nClusterDNProvider is properly set. If not, then you can do a References: Add Reference and browse to nClusterDNProvider.dll. 4 In the main() method of Program.cs, edit the connectionString values to use your Aster Database user credentials. 5 Compile and run the application. Source Code of Program to Demonstrate nClusterDNProvider Below is the source code for the sample program, “program.cs” that demonstrates the use of the Aster Database ADO.NET driver. using using using using using using using System; System.Collections.Generic; System.Text; System.Data; System.Data.Common; System.IO; System.Threading; namespace ConsoleApplication2 { class Program { Aster Client Guide 113 Tools for .NET Environments Sample Program for ADO.NET // This method expects the following table in the backend: // // create table tableout(field_int int, field_numeric numeric) // distribute by hash(field_int); // // /* static void AddWithDataSet(DbConnection connection) { DataSet ds = new DataSet(); nClusterDataAdapter da = new nClusterDataAdapter("select * from tableout", conn); da.InsertCommand = new DbCommand("insert into tableout(field_int, field_numeric) " + " values (:a, :b)", conn); da.InsertCommand.Parameters.Add(new nClusterParameter("a",DbType.Int32)); da.InsertCommand.Parameters.Add(new nClusterParameter("b",DbType.Decimal)); da.InsertCommand.Parameters[0].Direction = ParameterDirection.Input; da.InsertCommand.Parameters[1].Direction = ParameterDirection.Input; da.InsertCommand.Parameters[0].SourceColumn = "field_int"; da.InsertCommand.Parameters[1].SourceColumn = "field_numeric"; da.Fill(ds); DataTable dt = ds.Tables[0]; DataRow dr = dt.NewRow(); dr["field_int"] = 4; dr["field_numeric"] = 7.3M; dt.Rows.Add(dr); DataSet ds2 = ds.GetChanges(); da.Update(ds2); ds.Merge(ds2); ds.AcceptChanges(); } static void GenerateXmlFromDataSet(DbConnection connection) { nClusterDataAdapter da = new nClusterDataAdapter("select * from tableb", connection); DataSet ds = new DataSet(); da.Fill(ds); ds.WriteXml("DataSetFeed.xml"); } */ // table schema: create table tableBytea (field_int int, field_bytea bytea) distribute by hash(field_int); static void ByteaDataType(DbConnection connection, String filename) { int res; FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read); BinaryReader br = new BinaryReader(new BufferedStream(fs)); Byte[] bytes = br.ReadBytes((Int32)fs.Length); Console.WriteLine(fs.Length); br.Close(); fs.Close(); DbCommand cmd = connection.CreateCommand(); cmd.CommandText = "insert into tableBytea values(1, ?)"; DbParameter param = cmd.CreateParameter(); param.ParameterName = "bytesData"; param.DbType = DbType.Binary; param.Value = bytes; cmd.Parameters.Add(param); res = cmd.ExecuteNonQuery(); 114 Aster Client Guide Tools for .NET Environments Sample Program for ADO.NET cmd.CommandText = "select field_bytea from tableBytea where field_int = 1;"; Byte[] result = (Byte[])cmd.ExecuteScalar(); fs = new FileStream(filename + ".suffix", FileMode.Create, FileAccess.Write); BinaryWriter bw = new BinaryWriter(new BufferedStream(fs)); bw.Write(result); bw.Flush(); fs.Close(); bw.Close(); } private static void CopyIn(DbConnection connection) { Console.WriteLine("Creating command..."); DbCommand cmd = connection.CreateCommand(); cmd.CommandText = "COPY t_test_adonet_in from 'foo.out'"; Console.WriteLine("Command created: " + cmd.CommandText); Console.WriteLine(); Console.WriteLine("ExecuteNonQuery: " + cmd.CommandText); cmd.ExecuteNonQuery(); } private static void CopyOut(DbConnection connection) { Console.WriteLine("Creating command..."); DbCommand cmd = connection.CreateCommand(); cmd.CommandText = "COPY t_test_adonet to 'foo.out'"; Console.WriteLine("Command created: " + cmd.CommandText); Console.WriteLine(); Console.WriteLine("ExecuteNonQuery: " + cmd.CommandText); cmd.ExecuteNonQuery(); } // table schema: create table t4 (i int, j char(10)) distribute by hash(i); public static void SimpleInsert(DbConnection connection) { DbCommand command = connection.CreateCommand(); command.CommandText = "insert into t_test_adonet values(11, 'eleven')"; Int32 rowsaffected; rowsaffected = command.ExecuteNonQuery(); Console.WriteLine("{0} rows added in table t_test_adonet", rowsaffected); } // table schema: create table t4 (i int, j char(10)) distribute by hash(i); public static void DataReader(DbConnection connection) { DbCommand command = connection.CreateCommand(); command.CommandText = "select * from t_test_adonet"; DbDataReader reader = command.ExecuteReader(); while (reader.Read()) { for (int i = 0; i < reader.FieldCount; i++) { Console.Write("{0} \t", reader[i]); } Console.WriteLine(); } } // table schema: create table t4 (i int, j char(10)) distribute by hash(i); public static void PreparedQuery(DbConnection connection) { // Declare the parameter in the query string DbCommand command = connection.CreateCommand(); Aster Client Guide 115 Tools for .NET Environments Sample Program for ADO.NET command.CommandText = "select * from t_test_adonet where i > ?"; // Now add the parameter to the parameter collection of //the command specifying its type. DbParameter param = command.CreateParameter(); param.ParameterName = "value1"; param.DbType = DbType.Int32; command.Parameters.Add(param); command.Parameters[0].Value = 2; using (DbDataReader reader = command.ExecuteReader()) { while (reader.Read()) { for (int i = 0; i < reader.FieldCount; i++) { Console.Write("{0} \t", reader[i]); } Console.WriteLine(); } } } // table schema: create table t_test_adonet (i int, j char(10)) distribute by hash(i); static void OutputParameter(DbConnection connection) { Console.WriteLine("OuputParameter start...."); // Send a query to backend. DbCommand command = connection.CreateCommand(); command.CommandText = "select * from t_test_adonet where i = 11"; // Now declare an output parameter to receive the first column of // the tablea. DbDataReader reader = command.ExecuteReader(); // Now, the firstcolumn parameter will have the value of // the first column of the resultset. Print(reader); Console.WriteLine("OuputParameter end."); } static void Print(DataTable tbl) { // For each row, print the values of each column. foreach (DataRow row in tbl.Rows) { for (int i = 0; i < tbl.Columns.Count; i++) { Console.Write("{0} \t", row[i]); } Console.WriteLine(); } } static void Print(DbDataReader reader) { // For each row, print the values of each column. for (int i = 0; i < reader.FieldCount; ++i) { Console.Write(String.Format("{0,12} |", reader.GetName(i))); 116 Aster Client Guide Tools for .NET Environments Sample Program for ADO.NET } // Write out the dividing line. Console.WriteLine(); for (int i = 0; i < reader.FieldCount; ++i) { Console.Write("=============="); } Console.WriteLine(); // Write out the actual data. while (reader.Read()) { for (int i = 0; i < reader.FieldCount; ++i) { Console.Write("{0,12} |", (reader.IsDBNull(i)) ? "<null>" : FormatData(reader[i])); } Console.WriteLine(); } } public static string FormatData(object obj) { if (obj is DBNull) { return "<null>"; } if (obj is Byte[]) { const string hexChars = "0123456789ABCDEF"; string binary foreach (byte { binary += binary += } = "0x"; b in (Byte[])obj) hexChars[(b >> 4) & 0x0F]; hexChars[b & 0x0F]; return binary; } else { return obj.ToString(); } } static void GetMetaDataCollections(DbConnection connection) { DataTable tbl = connection.GetSchema("MetaDataCollections"); Print(tbl); } static void GetRestrictions(DbConnection connection) { DataTable tbl = connection.GetSchema("Restrictions"); Print(tbl); } /* static void GetDatabases(DbConnection connection) { DataTable tbl = connection.GetSchema("Databases", new String[] { "beehive" }); Aster Client Guide 117 Tools for .NET Environments Sample Program for ADO.NET Print(tbl); } */ static void GetTables(DbConnection connection) { string[] restrictions = new string[4]; restrictions[2] = "tablea"; Print(connection.GetSchema("Tables", restrictions)); } static void GetColumns(DbConnection connection) { string[] restrictions = new string[4]; restrictions[2] = "tablea"; Print(connection.GetSchema("Columns", restrictions)); } static void GetViews(DbConnection connection) { string[] restrictions = new string[4]; restrictions[2] = "tablea_view"; Print(connection.GetSchema("Tables", restrictions)); } /* static void GetUsers(DbConnection connection) { DataTable tbl = connection.GetSchema("Users", new String[] { "beehive" }); Print(tbl); } * */ static void Setup(DbConnection connection) { Int32 rowsaffected; Console.WriteLine("Creating the tables ..."); DbCommand command1 = connection.CreateCommand(); command1.CommandText = "DROP TABLE IF EXISTS tableBytea, tableout"; rowsaffected = command1.ExecuteNonQuery(); DbCommand command2 = connection.CreateCommand(); command2.CommandText = "CREATE TABLE t_test_adonet (i int, j char(10)) distribute by hash(i)"; rowsaffected = command2.ExecuteNonQuery(); DbCommand command3 = connection.CreateCommand(); command3.CommandText = "CREATE TABLE tableBytea (field_int int, " + " field_bytea bytea) " + "distribute by hash(field_int)"; rowsaffected = command3.ExecuteNonQuery(); DbCommand command4 = connection.CreateCommand(); command4.CommandText = "create table tableout(field_int int, " + "field_numeric numeric) distribute by hash(field_int)"; rowsaffected = command4.ExecuteNonQuery(); } static void Main(string[] args) { DbConnection connection = null; //String connectionString = "Server=153.65.197.90;Port=2406;User Id=beehive;Password=beehive;Database=beehive;CommandTimeout=-1"; //DbConnection conn = new nClusterConnection(connectionString); try 118 Aster Client Guide Tools for .NET Environments Sample Program for ADO.NET { Console.WriteLine("Aster .Net Provider Test Program"); Console.WriteLine(); Console.WriteLine(); Console.WriteLine("Looking up provider factory..."); DbProviderFactory factory = DbProviderFactories.GetFactory("Aster.Net"); Console.WriteLine("Found provider factory."); connection = factory.CreateConnection(); string connectionString = "uid=beehive;pwd=beehive;dbname=beehive;ip=153.65.197.90;port=2406;NumericAndDecimalAsDo uble=1"; connection.ConnectionString = connectionString; // Connect to the database then retrieve the schema information. Console.WriteLine("Connecting..."); connection.Open(); Console.WriteLine("Connected."); Console.WriteLine(); // create the tables Setup(connection); // Metadata operations GetMetaDataCollections(connection); GetRestrictions(connection); //GetDatabases(connection); GetTables(connection); GetColumns(connection); GetViews(connection); //GetUsers(connection); DataReader(connection); SimpleInsert(connection); PreparedQuery(connection); OutputParameter(connection); //AddWithDataSet(connection); // GenerateXmlFromDataSet(connection); ByteaDataType(connection, "..\\..\\input\\out.txt"); CopyOut(connection); CopyIn(connection); } catch (Exception e) { Console.WriteLine(); Console.WriteLine("***********************************************************"); Console.WriteLine("Exception: " + e.Message); Console.WriteLine("Stack: "); Console.WriteLine(e.StackTrace); Console.WriteLine("***********************************************************"); if (connection != null) connection.Close(); } Console.WriteLine("Press any key to continue."); Console.ReadKey(); } } } Aster Client Guide 119 Tools for .NET Environments Sample Program for ADO.NET Possible Exceptions and Resolutions These are resolutions to possible exceptions: • • You might get an exception like “Exception: Unable to find the requested .Net Framework Data Provider. It may not be installed.” • Reason: Your 32-bit provider is not installed or is broken. • Resolution: Reinstall your 32-bit provider and make sure that the size and version are correct. Loading assembly error. You might get an exception like: Error -- 1056964601 : Internal error: The operation terminated unsuccessfully.Error -- 1055784933 : Errors in the high-level relational engine. The following exception occurred while the managed IDbConnection interface was being used: Error loading assembly: C:\Program Files (x86)\Aster Data Systems\nCluster ADO.NET Provider (x86)\AsterDataC#DSII.dll. If this exception occurred in SSAS: • 120 • Reason: Your 64-bit provider might not be installed correctly. • Resolution: Reinstall your 64-bit provider and make sure that the size and version are correct. If this exception occurred in SSIS, then: • Reason: The 32-bit Runtime not enabled. • Resolution: Right-click the project and choose Properties. Then, set Run64BitRunTime to False. • Note that in the SSAS view, no time zone information is shown for Aster Database columns of the date data type “time with time zone”. This is a known issue. Aster Client Guide CHAPTER 5 Data Loading To load data into Aster Database, you can use the Aster Database Loader Tool, the COPY command, the INSERT command, or a custom-defined SQL-MapReduce data loading function you have written. This section provides tips for efficient loading and shows how to load using the Aster Database Loader Tool. The following sections explain these utilities: • Best Practices for Data Loading (page 121) • Aster Database Loader Tool (page 125) • Troubleshoot Loading (page 143) Best Practices for Data Loading This section describes best practices for efficiently loading data into Aster Database. In particular, this section explains how to help optimize new loading projects using the tools provided by the Aster Database loader nodes and the Aster Database Loader Tool (ncluster_loader). The sections are: • Loading Terminology (page 121) • Scenario 1: Pre-Production Data Loading with Aster Database (page 122) • Scenario 2: Loading in a Production Environment with Aster Database (page 123) Loading Terminology We use the following terms in the text that follows: Aster Database loader node: In the cluster, a loader node is a node dedicated to data loading. Many loader nodes can operate in parallel. Aster Database Loader Tool (ncluster_loader) is the client application for initiating highspeed bulk loads. Aster Database Load Error Logging is a feature in ncluster_loader that allows you to perform loading that is more tolerant of poorly formatted input data. Load Error Logging sends malformed rows to an error logging table. Input data: Source input file(s) which are to be loaded into Aster Database. All source files are compatible with a format that Aster Database is able to load. Examples include the CSV Aster Client Guide 121 Data Loading Best Practices for Data Loading format (RFC 4180: http://www.rfc-editor.org/rfc/rfc4180.txt) or the tab-delimited format. Each file must use a consistent newline character throughout. Never mix UNIX and Windows-style newline characters in the same file; this will cause your load attempt to fail! Data staging node(s): Nodes where all the input data is present. In a typical setup the input data is stored on the local filesystem. However, other use cases where all the data is stored on a network-mounted storage device are possible. Row(s): In any given input data file individual rows are present. The used format for the input data describes where row boundaries are. Usually a “row” refers to a logical unit of information that needs to be stored as a unit in the data warehouse. One example for rows include call records which consist of source and destination phone numbers, call duration time, and so on. The following section highlights three different data loading scenarios and the respective best practices that should be performed to make the process as a whole as seamless as possible. Scenario 1: Pre-Production Data Loading with Aster Database To validate your loading process and data before deploying it to your production system, it is an accepted industry best practice to load to a pre-production or testing system. You can perform test data loading using the standard Aster Database command-line, bulk loading tool, ncluster_loader. For a complete overview of the tool’s command-line options refer to “Argument Flags” on page 127. The tool’s error logging features are useful for debugging your loading process. If there is a possibility that your input data contains malformed rows, you should consider using error logging for bulk load operations to accomplish the following goals. Below, we also show the command-line options for each task: 1 Make sure that even in the presence of malformed rows a given load operation succeeds; This can be accomplished by enabling error logging but not setting an error logging limit. (Set --el-enabled but do not set an --el-limit.) If you are not interested in what errors are present, malformed input rows can be discarded so that they are not stored in the cluster (--el-enabled --el-discard-errors). Tip: A UNIQUE or PRIMARY KEY violation in the data being loaded will always cause the load to abort. So if the table you are loading into contains these constraints and your load is failing, check the data you are loading to ensure it complies. 2 Store malformed rows into a separate table for later inspection; You should specify a custom error logging table to store malformed rows. This is done via the appropriate option (--el-enabled --el-table = 'my_errorlogging_table'). If the error logging feature of Aster Database is turned on without using --el-table to specify an error logging table, malformed rows are stored in the default system error logging tables (nc_errortable_part or nc_errortable_replicated). This is undesirable because the rows from different loads will be mixed together in the default 122 Aster Client Guide Data Loading Best Practices for Data Loading tables. Note that any custom error logging table has to inherit from the default system error logging table. To create such a table, see . 3 Abort data load operation in the presence of too many malformed rows; This is in particularly useful if you want a given load operation to abort if too many malformed rows are present in the input data (--el-enabled --el-limit = 100). In order to preserve atomicity for bulk load operations, the load operation fails as a transaction when the error limit is reached. When the operation fails, any rows already written by the transaction to the target table and error logging table are deleted. 4 Label malformed rows in the error logging table. If multiple operations are loading data into Aster Database at the same time (e.g., the preproduction system is testing integration of two separate data sources), you can label each load operation to identify which rows belong to which data source (--el-enabled -el-label = 'my_data_source'). The following types of errors in input data files can be detected: 1 Malformed datatypes (e.g., text value for an integer column); 2 Missing column values (e.g., the input data file provides data values for the first two columns, but not for the third one); 3 Additional column values (e.g., the input data file contains a row with more values than there are columns in the destination table); 4 Check constraint violations (e.g., the integer value of an imported data field exceeds the range allowed by the check constraint on the target table); 5 Character set conversion errors (e.g., input data file contains invalid UTF-8 characters); 6 Missing or invalid distribution key (e.g., for your distribution key, you have specified a column that has a distribution-key-disallowed datatype). Summary In an environment where not all operations are run in an automated fashion, where new data sources are to be integrated, or where existing ETL processes are to be changed, we recommend that you set an error logging limit to prevent accumulation of too many malformed rows in the error logging tables. If malformed data rows need to be inspected, we recommend that you use a custom error logging table or create a separate error logging table for that load job. Scenario 2: Loading in a Production Environment with Aster Database Teradata Aster recommends that, before you bulk load data into your production system, you first verify your loading procedure on a test system. In this section, we’ll focus on the differences between a data loading job in a testing environment and one in a production system. In a production deployment, we recommend that you follow the steps below to ensure you take advantage of the bulk loading and error logging tools in Aster Database: Aster Client Guide 123 Data Loading Best Practices for Data Loading 1 Performance when loading autopartitioned tables Beginning in Aster Database version 5.0, loading of logically partitioned tables has much lower overhead and performs better than previously. Specifically, performance is improved when loading data into a table with many child partitions. The improvement applies to tables that use either row or columnar storage, but is most pronounced for tables with row storage. 2 Run ANALYZE after you load each reasonably sized batch. When bulk loading data, it’s important to run ANALYZE regularly. In ncluster_loader, use the --auto-analyze (or -z) flag to do this. With this flag present, ncluster_loader will run ANALYZE on the target table or tables after it loads the data. While running, ANALYZE requires an amount of disk space on the cluster proportional to the amount of new data in the table being analyzed. For this reason, Teradata Aster recommends that your run ANALYZE after every load, rather than waiting for multiple loads to finish. Waiting too long can result in a large amount of the cluster’s disk space being consumed during the running of ANALYZE. Daily child tables offer a good example: When you load to daily child tables, run ANALYZE on each child table after you load the day’s data into that table. 3 Stay informed about cluster activity. As with any other production system component, it is important to stay informed about any irregular activity that is going on in the cluster, in this case in particular what is happening in the data loading path. If you want to make sure that large loads with too many malformed rows do not complete successfully, you can set an error logging limit for your load operation (--el-enabled -el-limit = '100000'). You can specify a limit clause with a value that is small enough to allow most loads to complete successfully while forcing the failure of loads with unexpected percentages of malformed rows. Exact values will depend on the size of the files being loaded and the expected malformed row ratios. You should use an alert system to monitor the size of your error logging tables. See for information. 4 Keep an eye on the volume of data in the load error logging tables. The system does not perform automatic truncation of the error logging tables! The error logging tables are regular tables in Aster Database. The same operations that an administrator would apply to regular tables thus need to be applied to error logging tables. Tasks include the appropriate setup of child table hierarchies for efficient data processing, regular VACUUM operations to free up disk space occupied by dead rows, and the monitoring of table growth over time. Custom error logging tables can be used by passing the appropriate arguments when you run the Aster Database Loader Tool (for example, -el-enabled --el-table = 'el_table_April_2010'). 5 Use an error logging file for faster re-try of rows that failed. Have ncluster_loader create a file containing the row data that failed to load. To do this, pass the argument, --el-errfile <my_error_file.txt> to the tool. Upon completion of the load, you can inspect the contents of the error logging file, fix issues you find there, and reload from that file. 124 Aster Client Guide Data Loading Aster Database Loader Tool 6 Watch your load error statistics. To monitor error rates, check the statistics tables, nc_all_errorlogging_stats and nc_user_errorlogging_stats, as shown in . These tables show you the malformed and well-formed row count for each load operation. Records from the statistics tables can be correlated with the malformed rows in the load error logging tables. Both the error logging and statistics tables contain an error logging label which can be used to identify a particular loading operation. 7 Trace malformed input data to the data source. An important aspect of a production system is that it must provide the user with the means to find the problem source when an operation fails. When loading data, the user can use the error information about malformed rows (SQL error code and error message, sqlerrcode and errmessage, respectively, in the load error logging table) to diagnose the problem in the input data. To do this, you must invoke your data loading operation with the error logging feature enabled (--el-enabled). As described above, this information can then be correlated with the error logging statistics table to find the source of the malformed data. To facilitate this process, you may wish to apply custom error logging labels to each load operation (--el-enabled --el-label = 'my_data_source'). Loading Best Practices Summary For any production system, the close monitoring of system components is absolutely mission critical. Correlating error information present in the error logging tables can reveal possible problems early in the process. Aster Database Loader Tool The Aster Database Loader Tool, ncluster_loader, is a command-line application for bulkloading data from files into Aster Database. It provides an alternative to the SQL COPY statement. Each invocation of the Aster Database Loader Tool is equivalent to a single invocation of the COPY statement. Each invocation is a single database transaction that loads the contents of the specified data file or files into the specified table or tables. This section includes: Aster Client Guide • Syntax (page 126) • Argument Flags (page 127) • Exit Status (page 132) • Install the Loader Tool (page 133) • Load Data with the Loader Tool (page 133) • Load from Multiple Files Using a Map File (page 135) • Examples (page 138) • Hints for Successful Loading (page 139) • Error Logging (page 142) 125 Data Loading Aster Database Loader Tool Syntax To run the Aster Database Loader Tool, you type: $ ncluster_loader [arguments] [schemaname.]tablename [ filename | dirname ] where • arguments are the command-line flags that control how the loader runs. The flags are explained in Argument Flags, below, or you can display the help by typing: $ ncluster_loader -? • schemaname is the optional name of the destination schema. If no schema name is provided, Aster Database will search the schemas listed in the schema search path. • tablename is the name of the destination table (See “Case-Sensitive Handling for Table Names” on page 126 if you wish to have Aster Database evaluate table names in a casesensitive manner.); • filename or dirname indicates the file or directory of files to be loaded. • Qualified path of the file containing the data to be loaded. The contents of the file must be in either CSV or text format, as described for the COPY statement. Details of the encoding used (such as non-default values for null or delimiter) are specified using the appropriate options, as described below; or • dirname Qualified path of the directory containing one or more data files to be loaded. All data files found within this directory are expected to be in the same format and will be loaded as a single transaction. Subdirectories will not be processed. filename If you don’t supply a file or directory name argument, Aster Database Loader assumes you want to load from STDIN. See “Load from STDIN Example” on page 138. Tip for Windows users: When using Aster Database Loader on Microsoft Windows, bear in mind: • If the filename or dirname contains spaces, make sure you enclose it in double quotes. • When specifying a dirname, you must use a double backslash at the end of the path. For example, use “c:\temp\loadFiles\\” to specify a directory called “loadFiles”. • Never mix UNIX and Windows-style newline characters in the same data file. Doing so will cause your load attempt to fail. Case-Sensitive Handling for Table Names To force Aster Database Loader to treat your table and schema names in a case-sensitive manner, you must surround the schema and table names in double quote marks when you invoke ncluster_loader. To pass a double quote mark to Aster Database Loader, you must prefix it with the escape character, which is a backslash. That means you type a double quote mark like so: \". If either the schema name, the table name or both names include capital letters, you must surround each name in escaped quotation marks, individually. For example, assuming you want to load to table "Bar" in schema "Foo", then you would invoke Aster Database Loader using this form: ncluster_loader.exe -h 10.51.3.100 -U mjones -w st4g0l33 \"Foo\".\"Bar\" mydata.csv -c 126 Aster Client Guide Data Loading Aster Database Loader Tool and in the map file, you would reference the table as: "table" : "\"Foo\".\"Bar\"", If you want to load to table "bar" in schema "Foo", you would still need to escape quote the schema and the table separately as follows: ncluster_loader.exe -h 10.51.3.100 -U mjones -w st4g0l33 \"Foo\".\"bar\" mydata.csv -c and in the map file, as: "table" : "\"Foo\".\"bar\"", Argument Flags In addition to the schemaname.tablename and filename/dirname arguments explained above, the Aster Database Loader Tool takes the following argument flags at the command line or in the map file. (Map files let you load from many input files in a single running of Aster Database Loader. See “Troubleshoot Loading” on page 143.) In the table that follows, the argument flags are sorted based on the long-form, command-line flag: • The left column lists the flag you use at the command line. • The middle column lists the flag you can use in a map file (See “Rules for Passing Arguments in a Map File” on page 136.). If no value appears in the middle column, then the argument is one that can only be passed at the command line. Table 5 - 6: Argument Flags for Aster Database Loader Tool Flag at Command Line Flag in Map File Description -a [ --auto-partition] auto-partition Specifies that ncluster_loader will run in autopatitioning mode, which automatically routes each row to the appropriate child table in a parent/child table created through inheritance. -B [ --begin-script ] arg begin-script Specifies the qualified path of the file containing SQL commands that should be executed when the transaction starts, i.e. immediately after the BEGIN command is issued to Aster Database. Note: Datareturning statements such as SELECT are not allowed in scripts executed by the ncluster_loader. -c [ --csv ] Aster Client Guide Pass this argument if you want ncluster_loader to interpret the input file as being in CSV format. By default ncluster_loader uses a text file with a tab-delimited format, often called “TSV”. 127 Data Loading Aster Database Loader Tool Table 5 - 6: Argument Flags for Aster Database Loader Tool (continued) Flag at Command Line Flag in Map File Description -C [ --columns ] arg columns An optional comma-separated list of the columns in the target table that will receive the data being loaded (the default is to assume that each input row contains exactly one data value for each column in the target table, in the order in which the table columns were defined). By using -C, you can specify that the ordering of the columns in the input file is different from the ordering in the table. You can also use -C to specify that this input file contains values for only some of the columns in the table. The input data is assumed to contain values for the columns in the order specified here. For example, to load data into columns 'col1' and 'col2', one could specify "col1, col2" as the value for this option. Column names not specified here are expected to get NULL values. See “Using the -C option with uppercase or special characters” on page 144. -d [ --dbname ] arg dbname The column delimiter character to use when interpreting the input file (must be a string that represents a valid single character, such as 'd' or '\n'). The default is a tab character ('\t') in text mode, a comma in CSV mode. -D [ --delimiter ] arg --date-style arg The name of the database to connect to (default is "beehive"). date-style Specifies the date format. If you are using a map file, you must specify a single date-style per map file. The following formats are supported: • ISO - This is the default. Uses ISO 8601-style dates and times (YYYY-MM-DD HH:MM:SS) like, for example, 2010-12-17 07:37:16-08 • SQL - Uses Oracle/Ingres-style dates and times. For example: 12/17/2010 07:37:16.00 PST • POSTGRES - Uses traditional PostgreSQL format. For example: Wed Dec 17 07:37:16 2010 PST • German - Uses dd.mm.yyyy for numeric date representations. For example: 17.12.2010 07:37:16.00 PST • SQL,DMY - For example: 17/12/2010 15:37:16.00 CET • SQL,MDY - For example: 12/17/2010 07:37:16.00 PST • Postgres,DMY - For example: Wed 17 Dec 07:37:16 2010 PST -e [ --escape ] arg Specifies the escape character for CSV input (must be a string that represents a valid single character, such as d or \n). The default is the quote value - double-quote by default. You may only pass this option when loading a csv-formatted file (--csv). Warning: ncluster_loader will fail if you pass this argument when loading a tab-delimited text file. 128 Aster Client Guide Data Loading Aster Database Loader Tool Table 5 - 6: Argument Flags for Aster Database Loader Tool (continued) Flag at Command Line Flag in Map File -E [ --end-script ] arg end-script Description Qualified path of the file containing SQL commands that should be executed when the transaction is about to commit, i.e., immediately before the END/COMMIT command is issued to Aster Database. Note: Data-returning statements such as SELECT are not allowed in scripts executed by the Aster Database Loader Tool. --el-discard-errors discard-errors or skiperrors Sub-flag that can accompany --el-enabled. If present, specifies that all errors (malformed rows) be discarded by Aster Database. By default, errors are not discarded. --el-enabled errorlogging If present, turns on error logging for this invocation of the ncluster_loader. This needs to be enabled for any other error logging option to be accepted. The default is disabled. See “Error Logging” on page 142. --el-label arg label Sub-flag that can accompany --el-enabled. The --el-label flag specifies the label to be used to tag the malformed rows. By default, Aster Database will use a statement identifier as the label value. (There’s one statement identifier per COPY command; if there is one map entry for many input files, then you’ll have a unique statement identifier per input file.) The label is useful for finding your failed rows in the error logging table and for finding statistics about the load attempt in the nc_all_errorlogging_stats table. -F --el-errfile arg errfile Sub-flag that can accompany --el-enabled. The --elerrfile flag introduces the pathname of the optional error logging file. If you use error logging, you must have an error logging table, and you can have an error logging file. Upon completion of the load, ncluster_loader writes the contents of the error logging table’s rawdata column (and no other columns) to the error logging file. Only the contents of this column are written to the file. The filename will have a numeral appended to it in the form, “_0”. For this option to work, you must have also specified an el-table. Regardless of whether or not you specify that an error logging file should be used, the error logging table will still contain all error rows upon completion of each load. -L [ --el-limit ] arg limit Sub-flag that can accompany --el-enabled. The --el-limit flag sets the maximum number of malformed rows allowed for each file being loaded. By default the limit is unbounded. If the error count during your load attempt exceeds the specified --el-limit, the transaction aborts and no rows are inserted into the destination table, and no rows are inserted into the error logging table. -S [ --el-schema ] arg schema Sub-flag that can accompany --el-enabled. The --el-schema flag specifies the name of the schema to which the error logging table belongs. If not specified, the public schema is used. Aster Client Guide 129 Data Loading Aster Database Loader Tool Table 5 - 6: Argument Flags for Aster Database Loader Tool (continued) Flag at Command Line Flag in Map File Description -T [ --el-table ] arg table Sub-flag that can accompany --el-enabled. The --el-table flag specifies the name of the error logging table that will capture malformed rows. If you are loading to a distributed table, the error logging table must be a distributed table. Likewise, if you are loading to a replicated table, the error logging table must be a replicated table. If this flag is not provided, Aster Database uses a default table: nc_errortable_repl is used if the load-destination table is a replicated table, or nc_errortable_part is used if the loaddestination table is a distributed table. Important: The system does not perform automatic truncation of the error logging tables! You must VACUUM them regularly. -f [ --force-loader ] force-loader filename (Not a flag; you pass file the file or directory name itself!) Instructs the Aster Database Loader Tool to use the loader node specified with the -l or --loader option even if the IP address provided is not known to Aster Database. Note that if this option is specified, the Aster Database Loader Tool will only try that single IP address and return an error status if the connection fails for any reason. Qualified pathname of the file or directory containing data to be loaded. The contents of the file must be in either CSV or text format, as described for the COPY statement. If you pass a directory name, then all data files found in the directory are loaded in a single transaction. Subdirectories are ignored. All files must be in the same format. If you pass no file or directory name, Aster Database Loader loads from STDIN. In a map file, use the file flag. At the command line, there is no flag for this argument; instead, ncluster_loader assumes that you pass the named-flag arguments first, then the table name, and finally the source file name. -? [ --help ] Shows online help for Aster Database Loader Tool. -h [ --hostname ] arg Host name or IP address of the Aster Database queen. Default is 'localhost'. This always points to the queen, even if you’re loading using loader nodes. See “Multiple Loader Nodes” on page 140. --header-included Indicates that the first line of the data file to be input is a comma delimited list of the column names. The number of skip-rows will be counted beginning after the header, if this option is specified. Cannot be combined with --columns. -l [ --loader ] arg 130 loader Preferred loader IP address. If a value is provided, the Aster Database Loader Tool tries to load to this IP address before trying any other loader node. Note that the Aster Database Loader Tool expects this IP address to be known by Aster Database and will ignore the address provided if this is not the case. To change this behavior one can use the -f or --force-loader flag. Aster Client Guide Data Loading Aster Database Loader Tool Table 5 - 6: Argument Flags for Aster Database Loader Tool (continued) Flag at Command Line Flag in Map File Description -m [ --map-file ] arg Name of the file containing mappings of data files or directories to target tables. This option allows you to load multiple directories into a table and also to load data into multiple tables within the same transaction. For details about the format used for the map file, see “Troubleshoot Loading” on page 143. -n [ --null ] arg String that is used to represent a null value. The default is \N in text (non-CSV) mode, and an empty value with no quotes in CSV mode. When loading any non-CSV delimited format (e.g. TSV), you can easily load files that contain empty strings (that is, files that don’t use the typical \N to represent nulls). To do this, run ncluster_loader with the -n flag, followed by two doublequote characters. That is, you type: ncluster_loader -n "" See also --null-backslash-escapes. Indicates that backslashes in the '-n/--null' flag should be treated as special characters, which will be processed according to the default rules for escaped strings. --null-backslashescapes You should use this flag whenever the null string contains a backslash, to ensure compatibility with future versions of Aster Database. TCP port on which Aster Database listens for connections. Default is 2406. -p [ --port ] arg -P [ --data-prefix ] arg data-prefix String that should be used as a prefix for each row (each line in a data file) being loaded into Aster Database. Important: The Aster Database Loader Tool will simply concatenate this value to each row without any parsing. You must insert the proper delimiter, quote, and escape characters in the prefix string. Character to be used as the quoting character for CSV input (must be a string that represents a valid single character, such as d or \n). The default is the double-quote. This option is only valid when CSV input has been specified (-c or --csv) and will cause the load to fail otherwise. -q [ --quote ] arg Note that if using a single quote (') as the quoting character, you should escape it when specifying it with -q as -q"\'". Otherwise any input data that includes a comma will cause the load to fail. Specifies how many rows of the file to skip before starting to load data. If combined with --header-included, --skip-rows will not start counting until after the first line in the file. The default is 0. --skip-rows arg -t [ --timeout ] arg timeout Timeout value in seconds for Aster Database connection attempts. Default is 30 seconds. tablename (Not a flag; you pass the table name itself!) table Name of the destination table. In a map file, use the table flag. At the command line, there is no flag; instead, ncluster_loader assumes that you pass the named-flag arguments first, then the table name, and finally the source file name or directory name. Aster Client Guide 131 Data Loading Aster Database Loader Tool Table 5 - 6: Argument Flags for Aster Database Loader Tool (continued) Flag at Command Line Flag in Map File Description --truncate-table truncate-table Performs a TRUNCATE operation to empty the table before beginning the load. If child tables are present, they will be truncated, too. By default, this feature is disabled. -U [ --username ] arg username If present, the Aster Database Loader Tool connects to the database with this username instead of the default ("beehive"). (The user account must have permission to do so, of course.) -v [ --vacuum-table ] Runs VACUUM on the target table if the load fails. If target child tables or partitions are present, they are VACUUMed, too. By default, this feature is disabled. -V [ --version ] Print the Aster Database Loader Tool version and exit. --verbose Run in verbose mode. -w [ --password ] arg password Connect to the database using the specified password (as opposed to providing a password at the prompt). -W [--password-prompt] Forces the Aster Database Loader Tool to prompt for a password before connecting to a database. It should automatically prompt for a password whenever the server requests password authentication. If no password prompt is issued and the server requires password authentication, the connection attempt will fail. -z [ --auto-analyze ] Runs ANALYZE on the table or tables after loading the data. By default, this is disabled. If data was loaded to child tables via autopartitioning, they will be analyzed, as well. Warning: Analyzing a columnar table may be slow if there are many columns. To improve the speed of statistics collection, execute a separate ANALYZE command after the load that only processes the columns involved in query row filters or grouping. Exit Status Table 5 - 7: The Aster Database Loader Tool exit values 132 Exit Value Description 0 The Aster Database Loader Tool terminated successfully. 2 An error internal to the Aster Database Loader Tool was detected. 3 Another error was detected. 4 Failed to establish a connection to Aster Database. 5 An error was detected while communicating with Aster Database. 6 Malformed input data detected. Aster Client Guide Data Loading Aster Database Loader Tool Install the Loader Tool Install the Aster Database Loader Tool on a data staging machine that has fast network connectivity to your Aster Database loader nodes. The data staging machine is where you will place the data files to be loaded. For a list of supported operating systems for the Aster Database drivers and utilities, see the Aster Database Drivers and Utilities Guide. Note that if you are upgrading, you can simply replace ncluster_loader with the newer version. Obtain the Loader Tool Installer Package To obtain the Aster Database Loader Tool package for your client operating system, download it from one of these places onto your client machine in the directory where you will install it: • To get the newest package, download it from http://downloads.teradata.com/download/tools • On your queen node, you can find the installers in the directory /home/beehive/ clients_all/<your_client_OS>. Install the Loader Tool on Linux 1 Obtain the Loader Tool Installer Package. 2 Unzip the file and expand the archive. 3 Set permissions to make the ncluster_loader file executable. 4 On Linux, ncluster_loader requires glibc 2.7 or later. Update glibc if needed. Install the Loader Tool on Windows 1 Obtain the Loader Tool Installer Package. 2 Unzip the file and expand the archive. 3 Run ncluster_loader.exe from the command line. Install the Loader Tool on AIX 1 Obtain the Loader Tool Installer Package. 2 Unzip the file and expand the archive. 3 Set permissions to make the ncluster_loader file executable. See also “AIX Client Dependent Libaries” on page 44. Load Data with the Loader Tool Connecting The Aster Database Loader Tool is a regular Aster Database client application. In order to connect to an Aster Database, you need to specify the host name or IP address of the queen node via the command line option -h. If the connection could not be made for any reason (e.g., insufficient privileges, server is not running on the targeted host, etc.), the Aster Database Loader Tool will return an error and terminate. Procedure: To load data with the Aster Database Loader Tool, do this: Aster Client Guide 133 Data Loading Aster Database Loader Tool 1 Install the Aster Database Loader Tool on your data staging machine. (See “Install the Loader Tool” on page 133.) 2 If you have not already created one or more loader nodes in Aster Database, create them now. 3 Prepare the file or files that contain the data you wish to load: a For hints on file formatting, see the descriptions of the --csv, --delimiter, and -null arguments, below. Use a consistent newline character in your input file(s)! b Determine your mapping of the input file’s field values to the columns of your target table. See the description of the --columns argument, below. c Determine any special parsing hints you need to provide to the loader routing. See the descriptions of the --escape, --quote, and --data-prefix arguments, below. 4 Place your data input file(s) on the data staging machine. 5 Figure out how you want to handle records that fail to load. See the section “Error Logging” on page 142. Teradata Aster recommends that you create an error logging table that will receive rows that fail to load. Be prepared to query the nc_all_errorlogging_stats table for statistics about your load attempt. 6 Figure out which advanced options of the Aster Database Loader Tool you will use: 7 a Do you need to load from multiple files or insert into multiple tables? If so, see the description of the --map-file argument and read the section, “Troubleshoot Loading” on page 143. b Do you need to automatically partition data when loading parent child tables with inheritance? Use the --auto-partition argument and read the section, “Loading Parent Child Tables with Inheritance” on page 140 c If you need to run an SQL script before and/or after the data is loaded, read the descriptions of the --begin-script and --end-script arguments, below. Load your data by running the Aster Database Loader Tool on your input file(s). See “Argument Flags” on page 127 for a list of the command-line options. The command you type will be similar to: $ ./ncluster_loader -h 10.50.25.100 –w beehive -D "~" customers input_data.txt 8 Check the results of your load attempt by querying the statistics tables, nc_all_errorlogging_stats and nc_user_errorlogging_stats. 9 Check your error logging table(s) for rows that failed to load. Be aware that if the load failed entirely (for example, if the number of errors exceeded the --el-limit), then no new rows will appear in your target table, and no error rows will appear in the error logging table. 10 If you wish to import the failed rows, find each row of input data that failed, edit it to fix it, and combine these fixed rows into a new input file. Re-run the Aster Database Loader Tool on the new input file. 134 Aster Client Guide Data Loading Aster Database Loader Tool Load from Multiple Files Using a Map File With the Aster Database Loader Tool you can load from multiple data files and insert into multiple tables in a single invocation by passing the --map-file or -m option. We refer to this as loading with a map file. All tables are loaded in a single database transaction. When loading a very large amount of data, you may choose to created multiple map files that each load their data files using a different loader. This can help speed up the process of loading a large amount of data. Procedure: 1 Prepare your data files for import. Each file should be formatted as usual for the Aster Database Loader Tool. All the files you submit in a single running of the Aster Database Loader Tool must be in the same format. 2 You can optionally pass connection parameters in the map file. Any connection parameters specified on the command line supercede those in the map file. The connection parameters you can specify are: • dbname - the name of the database • username - the name of the Aster Database user • password - the password of the Aster Database user • loader - the hostname or IP address of the Aster Database loader node • force-loader - same as the -f command line option. Instructs the Aster Database Loader Tool to use the loader node specified with the loader parameter even if the IP address provided is not known to Aster Database. Note that if this option is specified, the Aster Database Loader Tool will only try that single IP address and return an error status if the connection fails for any reason. • timeout - same as the -t command line option. Specifies the timeout value in seconds for Aster Database connection attempts. Default is 30 seconds. 3 Prepare your map file. The map file is a text file containing a set of logical text blocks, each surrounded by curly braces. Each block represents a file or directory to be loaded. The format is like this: { "loadconfig" : [ { "table" : "schema1.targettable1", "file" : "data/insert1.txt", "errorlogging" : { "enabled" : false } }, { "table" : "schema1.targettable1", "file" : "data/insert2.txt", "begin-script" : "input/mapfile/begin-script.sql", "end-script" : "input/mapfile/end-script.sql", "errorlogging" : { "enabled" : false } } ] } Aster Client Guide 135 Data Loading Aster Database Loader Tool In the above example, we assume the current directory (from which we invoke ncluster_loader) contains a subdirectory, data, which has two files, insert1.txt and insert2.txt, and we load these into table targettable1 in schema1. Error logging is turned of. Each block in the map file must contain the following required parameter flags and their values: • table specifies the name of the target table. Typically you will schema-qualify the table name as shown in the example above. You may omit the schema, in which case the user’s schema search path determines the schema. If your table name is case sensitive, you must surround the table name with backslash-escaped double quote marks (\"), like this: "table" : "schema1.\"TargetTable\"", • file specifies the name of the file or directory to be loaded. See “filename” in “Syntax” on page 126. • begin-script and end-script specify scripts to run either before or after the loading of the file. Both are optional. Each map file entry can have a separate beginscript and end-script. For each map file entry, ncluster_loader will run the begin-script, load the data from the file, then run the end-script. The begin-script and end-script require each statement to be on a single line. Do not include commands that begin or end the transaction in the script files. Here is an example of a valid script file: DROP table IF EXISTS foo; CREATE table foo (id int, sometext varchar(40)) DISTRIBUTE BY HASH (id); • errorlogging introduces a block that specifies how to handle malformed rows in the input file. Inside the errorlogging block (enclosed in curly braces) you pass the enabled parameter and, optionally, the parameters discard-errors, label, limit, schema, and table. These parameters correspond, respectively, to the command-line flags, --el-enabled, --el-discard-errors, --el-label, --el-limit, -schema, --el-table. Each block in the map file can contain the optional parameter flags listed in the middle column of the table, “Argument Flags” on page 127. See “Rules for Passing Arguments in a Map File”, below. 4 Run the Aster Database Loader Tool, passing the --map-file or -m option and the name of your map file. Pass additional command line flags as needed, observing the rules set forth in the preceding paragraph. Rules for Passing Arguments in a Map File Observe the following rules when you pass loader arguments in a map file: 136 1 Each block in a map file can contain the optional parameter flags listed in the middle column of the table, “Argument Flags” on page 127. Observe all rules spelled out there. 2 If you wish to use flags listed in the middle column of that table, you must pass them inside the map file, and you cannot use their command-line equivalents. Aster Client Guide Data Loading Aster Database Loader Tool 3 Those flags that have no value listed in the middle column can also be used when you’re loading with a map file, but you must pass them at the command line, instead. Example map file The example file reproduced here shows various valid combinations of loading parameters. Note that the third block in the example (the one inserting into “test3”) instructs the Aster Database Loader Tool to load all the files it finds in directory “my_data_dir”. { "dbname" : "beehive", "username" : "beeuser", "password" : "jw8s0F4", "loader" : "10.51.6.240", "force-loader" : true, "timeout" : 5, "loadconfig" : [ { "table" : "test1", "file" : "data/insert.text", "errorlogging" : { "enabled" : false } }, { "table" : "test2", "file" : "data/insert.text.n", "errorlogging" : { "enabled" : false } }, { "table" : "test3", "file" : "my_data_dir/", "errorlogging" : { "enabled" : false } }, { "table" : "test6", "file" : "data/insert.text.incorrect", "errorlogging" : { "enabled" : true, "label" : "vm_test_12-test6" } }, { "table" : "test12", "file" : "data/insert.text.incorrect", "errorlogging" : { "enabled" : true, "label" : "vm_test_12-test12", "discard-errors" : true } }, { "table" : "test13", "file" : "data/insert.text.incorrect.pk", "prefix" : "0", "columns" : "a,b", "errorlogging" : Aster Client Guide 137 Data Loading Aster Database Loader Tool { "enabled" : true, "label" : "vm_test_12-test13", "limit" : 100000, "schema" : "public", "table" : "nc_errortable_part" } } ] } Examples Simple Load Example Assume the following: • File to be loaded: 2010MarchSales_data.txt (assume current working directory) • Number of rows: 1000 rows encoded in text format • Delimiter: delimited by character "~" • Password: beehive • Destination table: sales_fact • Destination Aster Database: IP address 10.50.25.100 • Username: assume default • Database: assume default This is what you would type: $ ./ncluster_loader -h 10.50.25.100 –w beehive -D "~" sales_fact 2010MarchSales_data.txt After successful completion, you will see the following output indicating 1000 rows loaded: 1000 tuples were successfully loaded into table customers. Load from STDIN Example To load from STDIN, you simply omit the file name/directory name argument. (When loading from a data file, the last argument to ncluster_loader is the data file name.) For example, to import our UK sales data and replace the product name “NappyPlus” with “DiaperPlus” we could use sed and ncluster_loader together like so: $ sed s/NappyPlus/DiaperPlus/ 2010_UK_sales.csv | ./ncluster_loader -U mjones -w st4g0l33 sales_fact -c -l 10.51.50.240 Loading from STDIN lets you pipe your data through useful tools like sed, the text stream editor. Here’s an example of sed at work, ensuring that backslash escape sequences are properly formatted in the input data before Aster Database Loader sees the data. Here’s some sample data: # cat sampleData-3.tsv 5 How often do back-slash characters ('\') appear in your data? 6 And how often do you think they actually disappear: 1 \ ? 2 \ ? 3 \ ? 7 \W\a\y \t\o\o \o\f\t\e\n\! \! \! 138 Aster Client Guide Data Loading Aster Database Loader Tool Here’s how we run Aster Database Loader, piping its input data through sed: $ cat sampleData-3.tsv \ | sed -e 's_\\_\\\\_g' \ | ncluster_loader -h $QUEEN_IP -d my_db -U beehive -w beehive testo / dev/stdin Loading tuples using node '192.168.28.100'. 3 tuples were successfully loaded into table 'test'. Here are the result rows: $ act -h $SYSMAN_IP -d my_db -U beehive -w beehive -c 'SELECT * FROM testo ORDER BY id;' id | string ----+--------------------------------------------------------------------1 | This is just a line. 5 | How often do back-slash characters ('\') appear in your data? 6 | And how often do you think they actually disappear: 1 \? 2 \? 3 \? 7 | \W\a\y \t\o\o \o\f\t\e\n\! \! \! (4 rows) Example with Error Logging Use the same assumptions as in the previous example, and assume we will log malformed rows (that is, rows that the loader cannot interpret and therefore cannot load) to a table called “2010MarchSales_error_table,” tagging each error row with the label “2010MarchSalesErr”. At the end of the load attempt, the error data will also be copied to the file, /home/ccrisp/2010MarchSales_error.txt. We’ll set a limit of 100 error rows; if more than 100 errors are encountered, the load will be cancelled. To do this: 1 Create the custom error logging table: Run ACT as a user with table creation rights (for example, one with the catalog_admin role) and type: CREATE TABLE 2010MarchSales_error_table () INHERITS (nc_errortable_part); For more information on creating a custom error logging table, including information on permissions, . 2 Exit ACT, return to the command line, and type: $ ./ncluster_loader -h 10.50.25.100 –w beehive -D "~" --el-enabled --ellabel 2010MarchSalesErr --el-limit 100 --el-table 2010MarchSales_error_table --el-errfile /home/ccrisp/ 2010MarchSales_error.txt sales_fact 2010MarchSales_data.txt For more information on logging malformed rows, see “Error Logging” on page 142. Hints for Successful Loading Recommended Character Set Is UTF-8 The default character set for Aster databases is UTF-8, and the Aster team recommends that you load only UTF-8 formatted data when loading to char, varchar, and text columns. For the tools you use to prepare files and to connect to Aster, make sure you have set the default character set to UTF-8. This is particularly important if you are loading data from a Aster Client Guide 139 Data Loading Aster Database Loader Tool Windows-based machine. For example, if you will use an SSH client (e.g., putty) to run ncluster_loader, make sure you set the SSH client’s default character set to UTF-8. We recommend that, prior to loading, you convert your text files to UTF-8. For example, if you’re a Notepad++ user, you can use the command, “Convert to UTF8 without BOM.” Newline Character Make sure your data file uses a consistent character to represent newlines. If the file uses \r\n for newlines, then it should not also use \n for newlines, and vice versa. If your file contains both UNIX-style \n newlines and Windows-style \r\n newlines, then you must clean the file before you try to load it. The UNIX command, dos2unix, can be useful for doing this. Multiple Loader Nodes The Aster Database Loader Tool supports the use of many Aster Database loader nodes. For most loading tasks, the queen is sufficient to handle all loading, but for high volume loading, you can add dedicated loader nodes to your cluster. To use a loader node, you invoke one or more ncluster_loader instances that will load through that loader node. You may run many ncluster_loader sessions in parallel against one loader node, and you may use many loader nodes in parallel (with each node handling loads from a number of ncluster_loader instances). To do this, you invoke each ncluster_loader instance with the -l (and optionally -f ) argument to specify the loader node. The required flags are: • as always, the --hostname option (-h) provides the queen IP address; • the --loader flag (-l) provides the IP address of the desired loader node; and • Optionally, the --force-loader flag (-f ) forces the use of the desired loader node. Loading Parent Child Tables with Inheritance The --auto-partition option is retained in order to support parent/child tables created with inheritance. It is not used when working with parent/child tables created with autopartitioning. Using --auto-partition instructs the Aster Database Loader Tool to automatically send each row to the right child table during loading. Each row is directed to a table according to the check constraints you have set up on the child tables. For example, if you partition your data into daily child tables based on the contents of a timestamp column, each ultimate child table in your schema will have a CHECK constraint that specifies what value of timestamp may be loaded into that child table. When you load data, the autopartitioning feature will route each row to the appropriate child table, based on its timestamp value. Use autopartitioning like this: 140 Aster Client Guide Data Loading Aster Database Loader Tool 1 Set up the parent-child table schema in your database. On each ultimate child table, write a CHECK constraint that specifies what data may be loaded into that child table. Warning! Aster Database does not detect overlapping constraints on peer child tables. As a result, the correct placement of a row during loading can be indeterminate. Workaround: Take care that the constraints you define do not create overlapping logical partitions. A simple mistake would be to set up range constraints like this: CHECK ( ymdh BETWEEN '2005-07-01' AND '2005-08-01' ); CHECK ( ymdh BETWEEN '2005-08-01' AND '2005-09-01' ); In this example, it is not clear in which partition the ymdh value '2005-08-01' resides. 2 3 Prepare your data for loading: a Your data input file can contain data values that will end up in many different child tables. b To handle rows that do not fit your partitioning scheme, you can rely on the standard error logging feature of the Aster Database Loader Tool (see “Error Logging” on page 142) or create a check constraint that will catch rows that you do not want to include in your partitions. Run the Aster Database Loader Tool with the -a or --auto-partition flag. Using COPY with columnar tables A loading operation using the Aster Database Loader Tool, COPY, or INSERT can be expensive when the following conditions exist: • the target table uses columnar storage, AND • the target table has many logical partitions, AND • the loaded data matches many different logical partitions. In this case, the memory allocated to perform the load may be as large as the amount of source data. To avoid high memory requirements, it is best to divide a large load into batches. There are two alternative approaches: Aster Client Guide • Ensure that each batch only loads a small number of logical partitions in the columnar table. For example, when inserting data into a columnar table with weekly partitions, each batch may insert data for a single month. • Ensure that the size of each batch is a small fraction of system memory available at the worker nodes. This should only be done if the data being loaded into the columnar table has a mixture of records matching many different logical partitions. As an example, suppose that a year's worth of data is being loaded into a columnar fact table with weekly partitions, and the cluster has four physical worker nodes with 100GB of system memory in each node. Loading data in 40GB batches will use 10GB memory per physical worker, which is 10% of the overall available memory. 141 Data Loading Aster Database Loader Tool Error Logging Enabling Load Error Logging Use the --el-enabled flag (or the errorlogging flag inside a map file) to run the Aster Database Loader Tool in a mode in which it tolerates poorly formatted input rows and logs each bad row to a table. This differs from Aster Database Loader’s normal mode of running: • Running normally, Aster Database Loader aborts the load immediately if it encounters a bad input row, and it does not log the malformed input row to a table. • Running in --el-enabled mode, Aster Database Loader logs each malformed input row (that is, any row it cannot interpret for loading or cannot load due to datatype mismatch or check constraint violation) to an error table and continues to load the remaining rows in the load job. We refer to this as “error logging.” The --el-enabled flag is a master flag that operates with a set of sub-flags (--el-discarderrors, --el-errfile, --el-label, --el-limit, --el-schema, and --el-table) that fine-tune your handling of malformed rows. To use any of the sub-flags, you must first have specified the --el-enabled flag. (If you’re using a map file, the syntax is different. The master flag is errorlogging, and the sub-flags are discard-errors, errfile, label, limit, schema, and table.) The --el-discard-errors flag discards all malformed rows, the --el-label tags failed row data, the --el-limit flag sets a maximum number of allowed failed rows for the job, and the --el-table flag specifies a custom error logging table. See “Argument Flags” on page 127 for explanations. See “Example with Error Logging” on page 139 for example usage. To perform error logging, the Aster Database Loader Tool relies on the error handling features of the Aster Database COPY command in SQL. Load Error Logging Tables By default, Aster Database logs error rows to one of its default error logging tables, based on type of the target table: malformed rows for distributed tables go into table nc_errortable_part table, and malformed rows for replicated tables go into the nc_errortable_repl table. Alternatively, you can create your own error logging tables. You can perform SELECT, UPDATE, and DELETE operations on the error log tables, and you can inherit from the default error logging tables to create your own error logging tables. Statistics About Logged Errors To see the number of rows that loaded or failed to load, query the load error statistics tables, nc_all_errorlogging_stats and nc_user_errorlogging_stats. Each loading command generates a row in the tables. For a given transaction, the totalcount, goodcount, and malformedcount columns show the total number of rows you tried to load, the number of rows that successfully loaded, and the number of rows not loaded, respectively. 142 Aster Client Guide Data Loading Troubleshoot Loading Troubleshoot Loading Running Multiple Loaders Concurrently When running multiple ncluster_loader instances concurrently, eventually Aster Database will throw errors due to physical limitations imposed by system resources. Due to this, you should not run more than ten ncluster_loader instances concurrently. Load stalls upon cancellation or failure encountered during a load The Aster Database Loader Tool can take a lot of time to finish after a failure is detected. Likewise, if a Control+C is issued, the loader can take several minutes to cancel completely. Load fails on UNIQUE or PRIMARY KEY violation A UNIQUE or PRIMARY KEY violation in the data being loaded will always cause the load to abort. So if your load is failing, check your data and the schema of the table you are loading into for these violations. Invalid Input Syntax Error If you encounter an error of type: "Error: Invalid input syntax on partition key for..." then there may be a problem with the character encoding of your input data. Please check the encoding of your input data and the datatypes of your destination columns, fix any mismatches you find, and retry the load. Single quote character must be escaped when using the -q option If you encounter an error like: "ERROR: extra data after last expected column" you may have encountered a problem with command line parsing. If you used a single quote delimiter in your CSV input file, and the input data includes a comma within one of the text strings to be loaded, you will see this error. To correct this, you will need to escape the single quote character when specifying it using the -q option on the command line as -q"\'". Here is an example: Assume the file test.txt, which we are attempting to load into table table1 includes the following input data row: 1, 25, "Some text with a, comma in it." The following will not load, returning the "Error: extra data after last expected column" error: ncluster_loader -l 10.50.129.103 -U db_superuser -w db_superuser -d beehive -c -q"'" table1 test.txt Aster Client Guide 143 Data Loading Troubleshoot Loading The following will load: ncluster_loader -l 10.50.129.103 -U db_superuser -w db_superuser -d beehive -c -q"\'" table1 test.txt Using the -C option with uppercase or special characters When using the -C option where the column list has any uppercase or special characters, you must put the column list in double quotes. On Windows, this requires escaping the double quotes: Example on Linux: ./ncluster_loader -h 10.51.150.100 -l 10.51.150.240 -w db_superuser -U db_superuser -c -D , -C '"ID","NAME","$value"' test2 test2.csv On Windows, when using the -C option where the column list has any uppercase or special characters, you must put the column list in escaped double quotes. Example on Windows: ncluster_loader.exe -h 10.51.150.100 -w db_superuser -U db_superuser -c -C '\"ID\",\"NAME\",\"$value\"' test2 test2.csv Uppercase characters are passed as lowercase if not escape quoted If your data contains a table or schema with a capital letter in its name, you must escape quote the table or schema name. If you do not do this, all lower case letters are assumed. Because of this, the data will load into a schema or table having the same name all in lower case, if it exists. For example, if you specify the table Test.Try, Test.try, test.Try or test.try, the data will always be loaded into test.try unless you specify escape characters with enclosed quotes for the table name parameter to ncluster_loader as in this example: # /home/beehive/clients/ncluster_loader -U db_superuser -w db_superuser -c \"Test\".\"Try\" file001 Loading tuples using node 'localhost'. 4 tuples were successfully loaded into table '"Test"."Try"'. This example loads the data in the expected table (Test.Try). 144 Aster Client Guide CHAPTER 6 • Export and Load Tools In addition to Aster Database Loader (described in “Aster Database Loader Tool” on page 125) Aster Database provides the Aster Database Export Tool. Aster Database Export Tool The Aster Database Export Tool, ncluster_export, is a command-line application for bulkexporting data from Aster Database to a file or to STDOUT. It provides an alternative to the SQL COPY TO statement. Each invocation of the Aster Database Export Tool is equivalent to a single invocation of the COPY TO statement. Each invocation is a single database transaction that exports the contents of the specified table into the specified file or STDOUT. Synopsis To run the Aster Database Export Tool, you type: $ ./ncluster_export [arguments] [schemaname.]tablename [ filename ] You can also pipe the results through a standard UNIX command such as gzip. For example: $ ./ncluster_export -U mjones -w st4g0l33 -h 10.50.52.100 -d mydb mytable | gzip -c > mytable.gz In the synopsis above, the arguments are: • schemaname is the optional name of the source table’s schema. If no schema name is provided, Aster Database will search the schemas listed in your current schema search path. • tablename is the name of the source table. See “Case-Sensitive Handling for Table Names” on page 146 if you wish to have Aster Database evaluate table names in a casesensitive manner. To export from multiple tables, see “Exporting from Multiple Tables” on page 146. • filename indicates the name of the file that will receive the exported data. If you don’t supply a file or directory name argument, Aster Database Export assumes you want to export to STDOUT. If the filename contains spaces, make sure you enclose it in double quotes. Aster Client Guide 145 Export and Load Tools Aster Database Export Tool • arguments are the command-line flags that control how the exporter runs. The flags are explained in “Argument Flags for Exporter” on page 147, or you can display the help by typing: $ ncluster_export -? Case-Sensitive Handling for Table Names To force Aster Database Export to treat your table and schema names in a case-sensitive manner, you must surround the schema and table names in double quote marks when you invoke ncluster_export. To pass a double quote mark to Aster Database Export, you must prefix it with the escape character, which is a backslash. That means you type a double quote mark like so: \" To do this, surround the entire schemaname.tablename string in a single pair of double quotation marks. For example, assuming you want to export to table "t" in schema "S", then you would invoke Aster Database Export in this form: ncluster_export.exe -h 10.51.3.100 -U mjones -w st4g0l33 \"S\".\"t\" mydata.txt Exporting from Multiple Tables With ncluster_export you can export from multiple tables in a single invocation by creating a map file of tables to be exported and passing the --map-file or -m option with the filename. All tables are exported within a single database transaction. To export from multiple tables, perform the following steps: 1 Prepare your map file. The map file is a text file containing a set of logical text blocks, each surrounded by curly braces. Each block represents a table to be exported. The format is as follows: { "exportconfig" : [ { "table" : "exporttable1", "fileprefix" : "path/to/dir/file" }, { "table" : "exporttable2", "fileprefix" : "path/to/dir/prefix", "maxtuplesperfile" : 100, "columns" : "a,b" } ] } Each block must contain the following required parameters: • “table” specifies the name of the table to export. • “fileprefix” specifies the output file / file prefix to be used. Each block can contain the following optional parameters: 146 Aster Client Guide Export and Load Tools Aster Database Export Tool 2 • “maxtuplesperfile” can be used to specify how many records will be put in each output file. This allows outputting multiple files for a single table, with the specified number of records in each. • “columns” is a comma-separated list of the columns to be exported. Run ncluster_export, passing the --map-file or -m option and the name of your map file. $ ./ncluster_export -U mjones -w st4g0l33 -h 10.50.52.100 -d mydb -m "path/to/dir/mapfile" Argument Flags for Exporter Aster Database Export accepts the following flags. Table 6 - 1: Aster Database Export Flags Aster Client Guide Column Type -? [ --help ] Shows the online help. -B [ --begin-script ] arg Specifies the qualified path of the file containing SQL commands that should be executed when the transaction starts, i.e. immediately after the begin command is issued to Aster Database. NOTE: Datareturning statements such as SELECT are not allowed. Each command in the begin script should be on a separate line. Do not include BEGIN or END statements. -c [ --csv ] Exports the output files in CSV format (the default is to use Text format). -C [ --columns ] arg An optional comma-separated list of columns to be exported from the source table (the default is to export all columns). -d [ --dbname ] arg Specifies the name of the database to connect to (the default is 'beehive'). -D [ --delimiter ] arg Specifies the delimiter character to use when exporting data (must be a string that represents a valid single character, such as 'd' or '\n'). The default is a tab character ('\t') in text mode, a comma (',') in CSV mode. -E [ --end-script ] arg Specifies the qualified path of the file containing SQL commands that should be executed when the transaction finishes, i.e. immediately before the end command is issued to Aster Database. NOTE: Datareturning statements such as SELECT are not allowed. Each command in the end script should be on a separate line. Do not include BEGIN or END statements. 147 Export and Load Tools Aster Database Export Tool Table 6 - 1: Aster Database Export Flags (continued) Column Type -e [ --escape ] arg Specifies the escape character for CSV output (must be a string that represents a valid single character, such as 'd' or '\n'). The default is the quote value - double-quote by default. This option is only valid when CSV mode is specified. -f [ --force-loader ] Instructs ncluster_export to use the loader node specified with the '-l/--loader' option even if the IP address provided is not known to Aster Database. NOTE: ncluster_export will only try that single IP address and return an error status if the connection fails for any reason. filename (Not a flag; you pass the file or directory name itself!) filename indicates the name of the file that will receive -h [ --hostname ] arg Specifies the hostname or IP address of the machine on which Aster Database is running. Default is 'localhost'. -l [ --loader ] arg Preferred loader IP address. If a value is provided, ncluster_export will try to export through this IP address before trying any other loader node. NOTE: ncluster_export expects this IP address to be known by Aster Database and will simply ignore the address provided if this is not the case. To change this behavior one can use the '-f/--force-loader' option. -m [ --map-file ] arg Specifies the name of the file containing mappings of the tables to be exported. This option allows export of multiple tables within the same transaction. For details about the format used for the map file, see “Exporting from Multiple Tables” on page 146. -M [ --max-tuples-perfile ] arg Specifies how many records are exported to a single file. If the total number of records exceeds this number multiple output files with the suffix '_N' are produced, where N is an integer. -n [ --null ] arg Specifies a string that represents a NULL value. The default is '\N' in text mode and an empty value with no quotes in CSV mode. See also “--null-backslashescapes” on page 148. --null-backslash-escapes Indicates that backslashes in the '-n/--null' flag should be treated as special characters, which will be processed according to the default rules for escaped strings. the exported data. If you don’t supply a file or directory name argument, Aster Database Export assumes you want to export to STDOUT. If the filename contains spaces, make sure you enclose it in double quotes. You should use this flag whenever the null string contains a backslash, to ensure compatibility with future versions of Aster Database. 148 Aster Client Guide Export and Load Tools Aster Database Export Tool Table 6 - 1: Aster Database Export Flags (continued) Column Type -p [ --port ] arg Specifies the TCP port on which Aster Database is listening for connections. The default is port 2406. -q [ --quote ] arg Specifies the quote character for CSV output (must be a string that represents a valid single character, such as 'd' or '\n'). The default is the double-quote. This option is only valid if CSV mode is specified. schemaname (Not a flag; you pass schemaname is the optional name of the source table’s schema. If no schema name is provided, Aster Database will search the schemas listed in your current schema search path. the schema name itself!) -t [ --timeout ] arg Specifies the timeout value (in seconds) to use when connecting to Aster Database (default is 30). tablename (Not a flag; you pass the table name itself!) tablename is the name of the source table. See “CaseSensitive Handling for Table Names” on page 146 if you wish to have Aster Database evaluate table names in a case-sensitive manner. To export from multiple tables, see “Exporting from Multiple Tables” on page 146. -U [ --username ] arg Connects to the database with the specified username instead of the default ('beehive'). (You must have permission to do so, of course.) -V [ --version ] Prints the version of ncluster_export and exit. -w [ --password ] arg Connects to the database with the specified password (as opposed to providing a password at the prompt). -W [ --password-prompt ] Forces ncluster_export to prompt for a password before connection to the database. Installing Aster Database Export You can run ncluster_export directly on the queen or install it on your workstation. On the queen, it is located by default at /home/beehive/clients/ncluster_export. Supported Platforms For a list of supported operating systems for the Aster Database drivers and utilities, see the Aster Database Drivers and Utilities Guide for your version of Aster Database. On Linux, ncluster_export requires glibc 2.7 or later. If you are running on AIX, see “AIX Client Dependent Libaries” on page 44. Procedure To install the Aster Database Export Tool on a machine other than the queen: Aster Client Guide 149 Export and Load Tools Aster Database Export Tool 1 To obtain the Aster Database Export Tool package for your client operating system, download it from one of these places onto your client machine in the directory where you will install it: • To get the newest package, download it from http://downloads.teradata.com/download/ tools • 2 150 On your queen node, you can find the installers in the directory /home/beehive/ clients_all/<your_client_OS>. Set permissions to make the files executable. Aster Client Guide Index A about this book 10 ACT check version of ACT 19 command history 32 command reference 30 command-line flag reference 18 file management commands 32 installing 11 launching 16 launching on Linux or Solaris 17 launching on Mac 17 launching on Windows 17 list all databases 34 list all tables in Aster Database 33 parameter, setting with \set 31 parameters 18 parameters at SQL prompt 30 tab completion 30 transpose columns and rows in output 35 using ACT 24 ACT commands 30 command-line options 18 SQL-prompt options 30 Active Directory authentication SSL connections and 84 AIX configure ODBC Driver Manager 50 install ODBC driver 50 ANALYZE bulk loading and 124 Aster Data, about 10 Aster Database checking version of ACT 19 connecting, overview 43 Aster Database configuration settings SQL behavior 71 Aster Database Export: See ncluster_export. Aster support portal 10 auto-analyze flag in batch loading 132 auto-partition flag in batch loading 127 autopartitioning ncluster_loader and 140 B backslash Aster Client Guide adding to input stream with sed 138 allow backslash escapes 71 backslash escape sequences 138 begin-script argument during batch load 136 begin-script flag in batch loading 127 best practices loading 121 bulk load 121 autopartition 140 end-script 129 from many files 135 logging bad rows during import 142 map file 135 ncluster_loader 125 bulk loading arguments 127 flags 127 map file: supported flags 127 parameters 127 bytea handling bytea data in ODBC 57 C certificate 90 for ODBC connection 90 character set for loading 139 character set, default 44 characters, special 44 loading 139 child table automatically route data into 140 client 11 installing ACT client 11 recommended client settings 44 client settings, recommended 44 client software versions 19 client tools 11 client-side cursors in JDBC 67 column transpose columns and rows 35 column alignment of query output 34 columns argument during batch load 128 command buffer 32 command history 32 send to file 32 command reference 151 ACT command-line flags 18 ACT SQL prompt 30 command-line (in SQL) commands for ACT 30 command-line options for ACT 18 commands ACT command-line flags 18 ACT commands at SQL prompt 30 configuration queen system parameters 85 configuration settings SQL behavior 71 connect 43 JDBC driver 59 Microsoft .NET, drivers for 97 MicroStrategy 93 ODBC driver 44 connecting 43 overview 43 through JDBC 62 conventions 9 coordinator.cfg 85 COPY in ncluster_loader 125 copyright 10 csv flag in batch loading 127 cursor ACT, cursors in 26 distributed 67 JDBC, cursors in 67 customer support 10 customizing SQL behavior 71 D data autopartition during load 140 bulk load 121 encrypting data traffic 83 data export 145 data loading 121 database list all tables 33 list the databases in a cluster 34 database drivers 43 JDBC 59 Microsoft .NET, drivers for 97 ODBC 44 ODBC for Perl scripts 55 date date format in bulk loading 128 date of publication 10 date-style flag in batch loading 128 dbname flag in batch loading 128 delimiter flag in batch loading 128 152 delimiters setting in ACT 34 discard-errors flag in batch loading 129 distributed cursors 67 documentation conventions 9 documentation version and updates 10 documentation, about 9 dot-NET data provider 97 Driver Manager configuration 50 ODBC 50 drivers 43 JDBC 59 Microsoft .NET, drivers for 97 ODBC 44 ODBC for Perl scripts 55 E E prefix 71 edition 10 el-enabled flag in batch loading 129 ELT 121 enable_quoted_identifiers 71 encryption 77 of query results 83 SSL for ODBC 77 end-script argument during batch load 136 end-script flag in batch loading 129 error file flag in batch loading 129 error limit in bulk loading 129 error logging 142 loading, log of errors for 142 tables to capture load errors, default 142 error logging file 129 error logging table, naming 130 errorlogging argument during batch load 136 escape character 128 allow backslash escapes 71 bulk loading and 128 ncluster_loader and 128 escape flag in batch loading 128 escape sequence adding to input stream with sed 138 ETL 121 SSIS and 99 expanded output mode 35 export 145 Aster Database export tool 145 JDBC driver 59 ODBC driver 44 source table, case-sensitive name 146 tools for 145 export data 145 Aster Client Guide sample test program 69 export tool 145 F L FETCH_COUNT 26 FETCH_LIMIT 28 fetch-count 26 fetch-limit 28 field separator, setting in ACT 34 file redirect command history to file 32 redirect query history to file 32 redirect query results to file 32 file argument during batch load 136 file input to ACT 32 with \i 32 with -f 19 force-loader flag for bulk loading 130 format dates in bulk loads 128 label argument during batch load 136 label for data loading errors 129 LIMIT FETCH_LIMIT as alternative to 28 limit argument during batch load 136 limit rows returned at a time 26 limit total rows returned per query 28 line break character 140 list all tables 33 list databases command 34 load 121 autopartition 140 from many files 135 from SDTIN 138 logging bad rows during import 142 map file 135 load data 121 other tools 145 loader arguments 127 Aster Database loader tool 125 destination table 126 destination table, case-sensitive name 126 flags 127 map file: supported flags 127 parallel loading with 140 parameters 127 specifying a dedicated loader node 140 loader flag for bulk loading 130 loader node 121 parallel loading with 140 loader tool 125 loading 121 arguments 127 Aster Database Export tool 145 Aster Database Loader tool 125 best practices 121 character set for loading 139 flags 127 from SDTIN 138 handling nulls when loading data 131 JDBC driver 59 map file: supported flags 127 ODBC driver 44 parameters 127 SSIS and 99 troubleshooting 140 troubleshooting problems encountered when loading 143 loading errors 142 log errors 142 G get latest documentation 10 group list all groups in Aster Database 34 H help 10 help for ncluster_loader 130 history in ACT 32 hostname flag for bulk loading 130 I import from many files 135 logging bad rows during import 142 ncluster_loader 125 index list all indexes in Aster Database 33 install ACT client 11 Aster Database Loader Tool 133 J Java JDBC statements 90 JDBC 59 connecting through JDBC 62 cursors in 67 driver 59 how to write applications that use JDBC 90 query example 91 Aster Client Guide 153 custom error logging tables 142 loading errors, logging 142 SSL settings 79 output transpose columns and rows 35 M P malformed rows during load 142 map file 135 supported flags 127 map file for ncluster_loader 135 map-file flag for bulk loading 131 memory usage in ACT: limit with FETCH_COUNT paging of query results 26 Microsoft .NET, drivers for 97 Microsoft SSIS 99 MicroStrategy, connections for 93 parameter setting queen system parameters 85 settings for ACT 31 partitioning autopartition during load 140 performance tuning improve ACT performance with FETCH_LIMIT 28 limit ACT memory use with FETCH_COUNT 26 server-side cursors in ACT 26 Perl scripts, ODBC driver for 55 pipe to ncluster_loader 138 pivot column and row output 35 port flag for bulk loading 131 portal 10 preferences SQL settings 71 prefix argument during batch load 131 PreparedStatement in JDBC 90 pre-process data with sed 138 N nc_errortable_part 142 nc_errortable_repl 142 ncluster_export 145 source table, case-sensitive name 146 ncluster_loader 125 arguments 127 character set for loading 139 date format 128 destination table 126 destination table, case-sensitive name 126 flags 127 installing 133 loading from many files 135 loading to multiple tables 135 logging bad rows during import 142 map file: supported flags 127 parameters 127 .NET Data Provider 97 newline character 140 null handling nulls when loading data 131 null flag for bulk loading 131 O ODBC 44 bytea handling 57 driver 44 Driver Manager 50 Driver Manager configuration 50 ODBC driver for MicroStrategy 93 installing for use with Perl scripts 55 installing on AIX 50 installing on Linux 48 installing on Windows 46 154 Q queen system parameters, setting 85 query transpose results 35 typing in ACT 24 via Java JDBC calls 90 query buffer 32 send to file 32 query history 32 send to file 32 query results 32 send to file 32 query tool installing ACT client 11 quiet mode of ACT 21 quote character flag for bulk loading 131 quoted identifier enable in Aster Database 71 R recommended client settings 44 recommended loading settings 139 redirect command history to file 32 redirect query history to file 32 redirect query results to file 32 reporting tools Aster Client Guide MicroStrategy 93 results 32 send to file 32 row transpose columns and rows 35 running SQL scripts 32 with \i 32 with -f 19 S sample code Microsoft dot.NET example 99 schema list all schemas in Aster Database 34 schema argument during batch load 136 scripts 32 run SQL script with \i 32 run SQL script with -f 19 security SSL for ODBC 77 sed for backslash sequences 138 sed, using with loader 138 send command history to file 32 send query history to file 32 send query results to file 32 server-side cursors in ACT 26 set command in ACT 31 settings enable_backslash_escapes 71 enable_quoted_identifiers 71 SQL settings 71 SSL connection settings 79 SQL customizing SQL behavior 71 parameters that affect SQL command interaction 30 parameters that set ACT client behavior 18 parameters that set SQL behavior 71 redirect query results to file 32 scripts, running with \i 32 scripts, running with -f 19 SQL commands utility commands in ACT 30 SQL COPY 125 SQL prompt commands 30 SQL prompt, using in ACT 24 SQL tool, installing ACT client 11 SSH 17 SSH client 17 SSIS 99 SSL 77 with Active Directory authentication 84 Aster Client Guide SSL for ODBC 77 certificates for 90 client settings 79 common configuration scenarios 80 encryption of query results 83 how to set up on client 85 how to set up on queen 85 queen settings 79 reference to Aster Database SSL settings 79 SSO for ODBC connections 84 Statement in JDBC 90 STDIN loading data from 138 support 10 system tables nc_errortable_part 142 nc_errortable_repl 142 T tab completion 30 table list all tables in Aster Database 33 table argument during batch load 136 table argument inside errorlogging block 136 TD Wallet 73 commands 74 configure 74 install 74 technical support 10 Teradata Wallet 73 terminal installing ACT client 11 tools 11 MicroStrategy 93 SSH client 17 troubleshooting load attempt fails 140 loading, troubleshooting for 143 TRUNCATE before bulk loading 132 truncate-table flag for bulk loading 132 typeface conventions 9 U umlauts 44 updated documentation 10 URL 10 Aster Support URL 10 user list all users in Aster Database 34 UTF-8 loading 139 recommended client settings 44 155 utilities 11 JDBC driver 59 Microsoft .NET, drivers for 97 ncluster_export 145 ncluster_loader 125 ODBC driver 44 V VACUUM after bulk loading 132 vacuum-table flag for bulk loading 132 version check ACT version 19 documentation version 10 ncluster_loader version 132 W workaround certificate, missing self-signed cert 78 X x command 35 156 Aster Client Guide