Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download StudentGuide
Document related concepts
Open Database Connectivity wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Team Foundation Server wikipedia , lookup
Functional Database Model wikipedia , lookup
Relational model wikipedia , lookup
Clusterpoint wikipedia , lookup
Object-relational impedance mismatch wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Transcript
Student Guide Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Part #EDB315-90-1242-10 I Notice Copyright ©2000 Sybase, Inc. All rights reserved. No part of this publication may be reproduced, transmitted, or translated in any form or by any means, electronic, mechanical, manual, optical or otherwise, without prior written permission from Sybase, Inc. ® indicates registration in the United States of America. Sybase Trademarks Sybase, SYBASE (logo), ADA Workbench, Adaptable Windowing Environment, Adaptive Component Architecture, Adaptive Server, Adaptive Server Anywhere, Adaptive Server Enterprise, Adaptive Server Enterprise Monitor, Adaptive Server Enterprise Replication, Adaptive Server Everywhere, Adaptive Server IQ, Adaptive Warehouse, AnswerBase, Application Manager, AppModeler, APT Workbench, APT-Build, APT-Edit, APT-Execute, APT-FORMS, APT-Translator, APT-Library, ASEP, Backup Server, BayCam, Bit-Wise, Certified PowerBuilder Developer, Certified SYBASE Professional, Certified SYBASE Professional Logo, ClearConnect, Client-Library, Client Services, CodeBank, Column Design, ComponentPack, Connection Manager, CSP, Data Pipeline, Data Workbench, DataArchitect, Database Analyzer, DataExpress, DataServer, DataWindow, DB -Library, dbQueue, Developers Workbench, Direct Connect Anywhere, DirectConnect, Distribution Director, Electronic Case Management, Embedded SQL, EMS, Enterprise Application Server, Enterprise Application Studio, Enterprise Client/ Server, Enterprise Connect, Enterprise Data Studio, Enterprise Manager, Enterprise SQL Server Manager, Enterprise Work Architecture, Enterprise Work Designer, Enterprise Work Modeler, EWA, First Impression, Formula One, Gateway Manager, GeoPoint, ImpactNow, InfoMaker, Information Anywhere, Information Everywhere, InformationConnect, InstaHelp, InternetBuilder, iScript, Jaguar CTS, jConnect for JDBC, KnowledgeBase, Logical Memory Manager, MainframeConnect, Maintenance Express, MAP, MDI Access Server, MDI Database Gateway, media.splash, MetaWorks, MethodSet, MySupport, Net-Gateway, Net-Library, NetImpact, Next Generation Learning, O DEVICE, OASiS, OASiS logo, ObjectConnect, ObjectCycle, OmniConnect, OmniSQL Access Module, OmniSQL Toolkit, Open Client, Open ClientConnect, Open Client/ Server, Open Client/Server Interfaces, Open Gateway, Open Server, Open ServerConnect, Open Solutions, Optima++, Partnerships that Work, PB-Gen, PC APT Execute, PC DB-Net, PC Net Library, Power++, PowerJ, Power Through Knowledge, power.stop, PowerAMC, PowerBuilder, PowerBuilder Foundation Class Library, PowerDesigner, PowerDimensions, PowerDynamo, PowerScript, PowerSite, PowerSocket, Powersoft, PowerStage, PowerStudio, PowerTips, Powersoft Portfolio, Powersoft Professional, PowerWare Desktop, PowerWare Enterprise, ProcessAnalyst, Report Workbench, Report-Execute, Replication Agent, Replication Driver, Replication Server, Replication Server Manager, Replication Toolkit, Resource Manager, RW-DisplayLib, RW-Library, S Designor, S-Designor, SAFE, SAFE/PRO, SDF, Secure SQL Server, Secure SQL Toolset, Security Guardian, SKILS, smart.partners, smart.parts, smart.script, SQL Advantage, SQL Anywhere, SQL Anywhere Studio, SQL Code Checker, SQL Debug, SQL Edit, SQL Edit/TPU, SQL Everywhere, SQL Modeler, SQL Remote, SQL Server, SQL Server Manager, SQL SMART, SQL Toolset, SQL Server/CFT, SQL Server/DBM, SQL Server SNMP Sub Agent, SQL Station, SQLJ, STEP, SupportNow, Sybase Central, Sybase Client/Server Interfaces, Sybase Development Framework, Sybase Financial Server, Sybase Gateways, Sybase Learning Connection, Sybase MPP, Sybase SQL Desktop, Sybase SQL Lifecycle, Sybase SQL Workgroup, Sybase Synergy Program, Sybase Virtual Server Architecture, Sybase User Workbench, SybaseWare, SyberAssist, SyBooks, System 10, System 11, System XI (logo), SystemTools, Tabular Data Stream, The Enterprise Client/Server Company, The Extensible Software Platform, The Future Is Wide Open, The Learning Connection, The Model For Client/Server Solutions, The Online Information Center, Transact-SQL, Translation Toolkit, Turning Imagination Into Reality, UltraLite, UNIBOM, Unilib, Uninull, Unisep, Unistring, URK Runtime Kit for UniCode, Viewer, Visual Components, VisualSpeller, VisualWriter, VQL, WarehouseArchitect, Warehouse Studio, Warehouse WORKS, Watcom, Watcom SQL, Watcom SQL Server, Web.PB, Web.SQL, WebSights, Web Viewer, WorkGroup SQL Server, XA-Library, XA-Server and XP Server are trademarks of Sybase, Inc. or its subsidiaries. All other company and product names used herein may be the trademarks or registered trademarks of their respective companies. Restricted Rights Legend Use, duplication or disclosure by the Government is subject to restrictions set forth in subparagraph (c)(1)(ii) of DFARS 52.227-7013 for the DOD and as set forth in FAR 52.227-19(a)-(d) for civilian agencies. Sybase, Inc., 6475 Christie Avenue, Emeryville, CA 94608 How to Use This Guide Student Guide The pages of this Student Guide are divided into two sections: a slide and a set of student notes. Your instructor will project the slide during class. The student notes provide additional notes to accompany the slide. Putting It All Together Some pages include only a code fragment or a code summary. They are provided for your reference to help illustrate how the commands work in sequence. Use the code fragments to complete the labs. The instructor may or may not display these sections Icon Samples Caution—Possible damage to equipment, data, or software Critical note—Very important piece of information that should definitely be read Note—Information that expands upon or supplements the information in the main text NT—Information specific to the Windows NT platform Question Recovery Tip—A suggestion for improving backup and recovery. Tip—Best practice or "tips and tricks" information TryIt UNIX—Information specific to the UNIX platform We Want to Hear From You Feedback Sybase welcomes your feedback on our course materials. This includes corrections as well as suggestions for improvements. Please mark typographical errors, formatting errors, errors of fact, or areas that need clarification and send copies of those pages to the Sybase Education Products and Technology Group (EPTG). Be sure to include your name, phone number, address and/or e-mail address so that we can reach you should we need clarification about your concerns. Sybase EPTG address Sybase, Inc. Education Products and Technology Group 6475 Christie Avenue Emeryville, CA 94608 USA Sybase EPTG fax Sybase EPTG E-mail Fax (510) 922-3559 [email protected] Administering Adaptive Server IQ for the Data Warehouse Student Guide Module 1 Overview Module 2 Adaptive Server IQ Product When to Use Adaptive Server IQ (ASIQ)?. Module Overview .................................................................................................. 1-1 Using ASIQ to Split DSS and OLTP .......................................................................................... .............................................. 1-3 Distributed Data Mart ........................................................................................... 1-4 Data Warehouse .................................................................................................... 1-5 What Is ASIQ? ...................................................................................................... 1-6 What ASIQ Is Not ................................................................................................. 1-7 Advantages of ASIQ: Overview ............................................................................ 1-8 Advantages of ASIQ: Speed .................................................................................. 1-9 Advantages of ASIQ: Economy .......................................................................... 1-10 Advantages of ASIQ: Flexibility ......................................................................... 1-11 What Principles Are at Work? ............................................................................. 1-13 Column-Wise Data Structures ............................................................................. 1-14 Bit-Wise Indexing................................................................................................ 1-15 ASIQ Multiplexing .............................................................................................. 1-18 Benefits of ASIQ ................................................................................................. 1-19 Summary ............................................................................................................. 1-20 Lab 1: ASIQ Product Overview .......................................................................... 1-21 Architecture and Terminology Module Overview .................................................................................................. 2-3 ASIQ Platforms ..................................................................................................... 2-4 Module Map .......................................................................................................... 2-5 Adaptive Server Anywhere and Adaptive Server IQ ............................................. 2-6 Adaptive Server IQ Database Server .................................................................... 2-7 ASIQ Server and Database (Without Multiplex) .................................................. 2-8 IQ Main Store ....................................................................................................... 2-9 Catalog Store ....................................................................................................... 2-10 IQ Temporary Store............................................................................................. 2-13 IQ Message Log .................................................................................................. 2-14 Putting It All Together...ASIQ Database ............................................................. 2-15 Module Map ........................................................................................................ 2-16 ASIQ Tables ........................................................................................................ 2-17 Utility Database ................................................................................................... 2-18 ASIQ Indexes ...................................................................................................... 2-19 ASIQ Tables and ASIQ Indexes .......................................................................... 2-20 interfaces File and ASIQ Connectivity ................................................................ 2-21 ASIQ Architecture .............................................................................................. 2-22 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-1 Table of Contents Summary ................ Lab 2: Architecture Module 3 2-23 224 ASIQ Environment and Installation Module Overview .................................................................................................. 3-1 Module Map .......................................................................................................... 3-2 Disk Space ............................................................................................................ 3-3 Virtual Memory ..................................................................................................... 3-4 Swap ...................................................................................................................... 3-5 Basic Configuration Recommendations ................................................................ 3-6 Hardware Recommendations................................................................................. 3-7 Module Map .......................................................................................................... 3-8 Before Installing Software on UNIX .................................................................... 3-9 Preparing for ASIQ Installation on the UNIX Platform ....................................... 3-10 Installing Adaptive Server IQ .............................................................................. 3-11 Setting Environment Variables ............................................................................ 3-12 Starting the ASIQ Server - UNIX ...................................................................... 3-15 Starting the ASIQ Server -Simple Method ...................................................... 3-16 Starting the ASIQ Server - UNIX ...................................................................... 3-17 Server Configuration File.................................................................................... 3-18 Stopping the Server ............................................................................................. 3-19 Preparing for ASIQ Installation on the NT Platform ........................................... 3-20 Installing ASIQ Server Components .................................................................... 3-21 Starting the Server ............................................................................................... 3-22 Stopping the Server ............................................................................................. 3-23 Installing ASIQ Server Client Components ........................................................ 3-24 Module Map ........................................................................................................ 3-25 Validating the ASIQ Environment ...................................................................... 3-26 Lab 3-1: Installation ........................................................................................... 3-27 Module Map ........................................................................................................ 3-28 Setting Up ODBC ............................................................................................... 3-29 Connection to ASIQ using DBISQL ................................................................... 3-36 Starting DBISQL - GUI ...................................................................................... 3-37 Ready to Connect ................................................................................................ 3-38 Interactive SQL ................................................................................................... 3-39 Connect Using Command Line DBISQL ............................................................ 3-40 Connecting Through Sybase Central ................................................................... 3-41 Sybase Central..................................................................................................... 3-42 Sybase Central Features ...................................................................................... 3-43 Module Map ........................................................................................................ 3-44 Database Configuration File for Open Client 12 ................................................. 3-45 Setting Up JDBC and Open Client ...................................................................... 3-46 Setting Up JDBC and Open Client ...................................................................... 3-48 Setting Up JDBC ................................................................................................. 3-49 Connect Using ISQL ........................................................................................... 3-50 ISQL Versus DBISQL......................................................................................... 3-51 sp_iqstatus ........................................................................................................... 3-52 TOC-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Running sp_iqstatus Command to Check ASIQ .................................................. 3-53 Summary of Installation Steps ............................................................................ 3-54 Lab 3-2: Connectivity ...................................................................................... 3-55 Module 4 Creating Databases Module Overview .................................................................................................. 4-1 Database Review ................................................................................................... 4-2 Database Overview................................................................................................ 4-3 Creating a Database .............................................................................................. 4-4 Step 1: Select the Device Type .............................................................................. 4-5 Step 2: Estimate the Database Size ........................................................................ 4-6 Step 3: Create the Database .................................................................................. 4-7 Files Allocated When Creating Database .............................................................. 4-8 Full Syntax ............................................................................................................ 4-9 Command Options .............................................................................................. 4-10 Create Database: Example 1 ................................................................................ 4-14 Create Database: Example 2 ............................................................................... 4-15 Creating Databases: Sybase Central.................................................................... 4-16 Before Starting the Server with the New Database .............................................. 4-18 Command-Line Switches .................................................................................... 4-19 Other Recommended Switches............................................................................ 4-20 Command-Line or Configuration File Switches .................................................. 4-21 Communication Protocols.................................................................................... 4-22 Communication Protocol Switches...................................................................... 4-23 Starting the Server with the New Database ......................................................... 4-24 Creating a New DSN .......................................................................................... 4-25 Viewing the New Database: Sybase Central ........................................................ 4-26 sp_iqstatus ........................................................................................................... 4-27 Checking the Database with sp_iqstatus .............................................................. 4-28 Enlarging a Database .......................................................................................... 4-29 Enlarging a Database: Sybase Central ................................................................ 4-30 Decreasing a Database ........................................................................................ 4-31 Dropping a Database ........................................................................................... 4-32 Other Useful Stored Procedures .......................................................................... 4-33 sp_iqdbsize .......................................................................................................... 4-34 Summary ............................................................................................................. 4-35 Lab 4: Databases ............................................................................................... 4-36 Module 5 ASIQ Index Types and Datatypes Module Overview .................................................................................................. 5-1 Module Map .......................................................................................................... 5-2 Types of Indexes Used by Typical RDBMSs ........................................................ 5-3 ASIQ's Bitmap Indexes ......................................................................................... 5-4 Review: ASIQ Index Characteristics ..................................................................... 5-5 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-3 Table of Contents Review: Four Basic Index Types ........................................................................... 5-6 Fast Projection (FP) .............................................................................................. 5-7 Low Fast (LF) ..................................................................................................... 5-10 High Non Group (HNG)...................................................................................... 5-11 High Group (HG) ................................................................................................ 5-12 Generic Method for Indexing Columns ............................................................... 5-13 Combining Index Types....................................................................................... 5-14 Fast Indexes ......................................................................................................... 5-15 Step 1: By Default, an FP Index Is on Each Column ........................................... 5-16 Step 2: Create LF on Low-Cardinality Columns ................................................ 5-17 Step 3: Create HG on High-Cardinality Columns ............................................... 5-18 Step 4: Create HNG, If Needed .......................................................................... 5-19 Case Study........................................................................................................... 5-20 Case Study: Solutions ......................................................................................... 5-21 Module Map ........................................................................................................ 5-23 ASIQ Data Types ................................................................................................ 5-24 User-Defined Datatypes ...................................................................................... 5-27 Summary ............................................................................................................. 5-28 Lab 5: ASIQ Indexing and Datatypes ............................................................... 5-29 Module 6 Creating Tables and Indexes Module Overview .................................................................................................. 6-1 Steps for Deploying an ASIQ Installation ............................................................ 6-2 Module Map .......................................................................................................... 6-3 Create Tables in the ASIQ Database ...................................................................... 6-4 CREATE TABLE .................................................................................................. 6-5 Building Upon CREATE TABLE........................................................................... 6-6 CREATE TABLE - IQ UNIQUE .......................................................................... 6-7 CREATE TABLE - UNIQUE ................................................................................ 6-8 CREATE TABLE - PRIMARY KEY .................................................................... 6-9 Building Upon CREATE TABLE......................................................................... 6-10 FOREIGN KEY Constraint................................................................................. 6-11 Syntax: CREATE TABLE .................................................................................... 6-13 CREATE TABLE ................................................................................................ 6-14 Creating a Table with Sybase Central .................................................................. 6-16 Sybase Central: Steps 1-3 ................................................................................... 6-17 Sybase Central: Steps 4-6 ................................................................................... 6-18 Sybase Central..................................................................................................... 6-19 Viewing a Table................................................................................................... 6-2 1 Altering Tables .................................................................................................... 6-2 2 ALTER TABLE .................................................................................................. 6-2 3 TRUNCATE TABLE........................................................................................... 6-2 4 DROP TABLE..................................................................................................... 6-2 5 Module Map ........................................................................................................ 6-2 6 Local Temporary Tables: Method 1 .................................................................... 6-2 7 Local Temporary Tables: Method 2 .................................................................... 6-2 8 Module Map ........................................................................................................ 6-2 9 TOC-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Global Temporary Tables .................................................................................... 6-30 Module Map ........................................................................................................ 6-31 View: Defined ..................................................................................................... 6-32 Creating a View: Benefits ................................................................................... 6-33 View: Using Partitioned Tables ............................................................................ 6-34 CREATE VIEW .................................................................................................. 6-35 Views for Performance ........................................................................................ 6-36 Steps for Deploying an ASIQ Installation .......................................................... 6-37 Create the ASIQ Indexes .................................................................................... 6-38 Review: Index Types ........................................................................................... 6-39 Four Options for Creating Indexes ...................................................................... 6-40 CREATE INDEX: Syntax and Example .............................................................. 6-41 CREATE INDEX: Sybase Central ...................................................................... 6-42 CREATE INDEX: Restrictions ........................................................................... 6-46 Creating Indexes in Parallel ................................................................................. 6-47 Viewing ASIQ Indexes with sp_iqindex .............................................................. 6-49 Dropping Indexes ................................................................................................ 6-50 Create and Maintain Scripts ................................................................................ 6-51 Summary ............................................................................................................. 6-52 Lab 6: Creating Tables and Indexes..................................................................... 6-53 Module 7 ASIQ Memory Configuration Module Overview .................................................................................................. 7-1 Sybase ASIQ Memory Components ...................................................................... 7-2 Server Memory...................................................................................................... 7-3 Buffers, Page Size, and Blocks ............................................................................. 7-4 Buffer Caches ........................................................................................................ 7-5 Why Tune Shared Buffer Caches?......................................................................... 7-6 Main Buffer Cache ................................................................................................ 7-7 Temporary Buffer Cache ....................................................................................... 7-8 Viewing Buffer Cache Size.................................................................................... 7-9 "Active" Users ..................................................................................................... 7-10 Strategy for Sizing Caches .................................................................................. 7-11 Memory: The Big Picture ................................................................................... 7-12 Operating System Memory and Other Applications ............................................ 7-13 ASIQ Server ........................................................................................................ 7-14 ASIQ Memory Overhead .................................................................................... 7-15 Side Note on Setting Database Options .............................................................. 7-17 load_memory_MB option ................................................................................... 7-18 ASIQ Main and Temp Buffer Caches .................................................................. 7-19 Strategy for Sizing Buffer Caches ....................................................................... 7-20 Calculating Memory for Caches on UNIX .......................................................... 7-2 1 Calculating Memory for Caches on NT .............................................................. 7-2 2 Query Memory Requirements: Read-Only........................................................... 7-2 3 Strategy for Sizing Caches .................................................................................. 7-2 4 Setting Buffer Caches.......................................................................................... 7-2 5 Viewing the New Buffer Caches .......................................................................... 7-28 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-5 Table of Contents Memory and Swap Usage ................................................................................... 7-29 Memory Performance.......................................................................................... 7-30 Summary ............................................................................................................. 7-31 Lab 7: Memory Configuration ............................................................................ 7-32 Module 8 Loading Data from Files Module Overview .................................................................................................. 8-1 Data Sources.......................................................................................................... 8-2 Table Loading ....................................................................................................... 8-3 Data Loading ......................................................................................................... 8-4 Simple Example: Flat-File Load............................................................................ 8-5 Adding Rows with LOAD TABLE ........................................................................ 8-6 LOAD TABLE Command: Partial Syntax ............................................................ 8-7 Module Map .......................................................................................................... 8-8 load-specification / column-spec .......................................................................... 8-9 column-spec ........................................................................................................ 8-10 Inserting Using Fixed-Width Field Specification ................................................ 8-11 Inserting Variable-Length Fields ........................................................................ 8-12 Example: Loading from a Character File ............................................................. 8-13 Date and Datetime Considerations ...................................................................... 8-14 A Word About NULL ......................................................................................... 8-16 Module Map ........................................................................................................ 8-17 FROM Clause...................................................................................................... 8-18 Module Map ........................................................................................................ 8-19 load-options ........................................................................................................ 8-20 DELIMITED BY Option .................................................................................... 8-21 Inserting Variable-Length Fields ........................................................................ 8-22 QUOTES/ESCAPES Options ............................................................................. 8-23 WITH CHECKPOINT Option ............................................................................ 8-24 Putting It Together ............................................................................................... 8-25 PREVIEW load-option - Input Layout Description ............................................. 8-26 ROW DELIMITED BY load_option ................................................................... 8-27 Loading in Parallel .............................................................................................. 8-28 More Complex LOAD TABLE Example ............................................................ 8-29 Module Map ........................................................................................................ 8-30 What Happens During a load Session? ................................................................ 8-3 1 Notification Messages ......................................................................................... 8-3 2 Transaction Processing and LOAD TABLE ........................................................ 8-3 3 Deleting Data ...................................................................................................... 8-3 4 DELETE Statement............................................................................................. 8-3 5 DELETE, then LOAD TABLE ............................................................................ 8-3 6 UPDATE Command ............................................................................................ 8-3 7 Summary ............................................................................................................. 8-3 8 Lab 8: Loading Data from Files .......................................................................... 8-3 9 TOC-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Module 9 Advanced Data Loading from Files Module Overview .................................................................................................. 9-1 Review: Options for Table Loading ...................................................................... 9-2 Review: LOAD TABLE Command ...................................................................... 9-3 Advanced Insert Load Options .............................................................................. 9-4 Module Map .......................................................................................................... 9-5 Multifile Load ....................................................................................................... 9-6 Example: Multifile Load ....................................................................................... 9-8 Module Map .......................................................................................................... 9-9 Partial-Width Loads............................................................................................. 9-10 What Is a ROWID? ............................................................................................. 9-11 Partial-Width Loads ............................................................................................ 9-12 Example: Partial-Width Loads ............................................................................ 9-14 Example: Partial-Width Load - The Right Way ................................................... 9-15 Example: Partial-Width Load - The Wrong Way ................................................. 9-17 Module Map ........................................................................................................ 9-19 Adding dbspaces During a Load.......................................................................... 9-20 Good Loading Practices ...................................................................................... 9-21 Summary ............................................................................................................. 9-22 Lab 9: Advanced Loading Data from File .......................................................... 9-23 Module 10 Loading Data Using the INSERT Command Module Overview................................................................................................ 10-1 Table Loading ...................................................................................................... 10-2 INSERT from SELECT Command ...................................................................... 10-3 INSERT VALUES Command .............................................................................. 10-4 Module Map ........................................................................................................ 10-5 Partial-Width Inserts ............................................................................................ 10-6 Example: Partial-Width Insert - The Right Way ............................................... 10-7 Module Map ........................................................................................................ 10-9 Inserting from the IQ Main Store ................................................................... 10-10 Module Map ...................................................................................................... 10-13 Inserting Directly from a Foreign Database ....................................................... 10-14 Using CIS .......................................................................................................... 10-17 Summary ........................................................................................................... 10-18 Lab 10: Loading Data Using the insert Command .......................................... 10-19 Module 11 Using ASIQ Module Overview ................................................................................................ 11-1 ASIQ SQL ........................................................................................................... 11-2 Review: SELECT Statement ............................................................................... 11-3 SELECT Statement ............................................................................................. 11-4 Module Map ........................................................................................................ 11-6 ASIQ-Specific SQL Features .............................................................................. 11-7 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-7 Table of Contents KEY JOIN ........................................................................................................... 11-8 NATURAL JOIN................................................................................................. 11-9 OUTER JOIN .................................................................................................... 11-10 Expressions in Subqueries ................................................................................. 11-11 IF Condition ...................................................................................................... 11-12 CASE Statement ............................................................................................... 11-13 A Word on Quotes ............................................................................................ 11-14 Module Map ...................................................................................................... 11-15 ASIQ Numeric Functions .................................................................................. 11-16 ASIQ String Functions ...................................................................................... 11-17 ASIQ Date and Time Functions......................................................................... 11-18 ASIQ Datatype Conversion Functions............................................................... 11-19 Additional ASIQ Functions............................................................................... 11-20 Module Map ...................................................................................................... 11-21 Exporting Data from ASIQ................................................................................ 11-22 Exporting Data Using Redirection .................................................................... 11-23 Exporting Data Using Redirection .................................................................... 11-24 Exporting Data Using Data Extraction .............................................................. 11-25 Options for Data Extraction .............................................................................. 11-26 Data Extraction ................................................................................................. 11-28 Summary ........................................................................................................... 11-29 Lab 11: Using ASIQ ...................................................................................... 11-30 Module 12 ASIQ Operational Data Administration Module Overview ................................................................................................ 12-1 Module Map ........................................................................................................ 12-2 Transaction Processing ....................................................................................... 12-3 What Is a Transaction? ........................................................................................ 12-4 Transaction Commands ...................................................................................... 12-5 Chained and Unchained Transaction Modes ....................................................... 12-8 Completing Transactions ..................................................................................... 12-9 Module Map ...................................................................................................... 12-10 Locking ............................................................................................................. 12-11 Locks forDML ................................................................................................. 12-12 Locks for DDL .................................................................................................. 12-13 ANSI Isolation Levels........................................................................................ 12-14 Module Map ...................................................................................................... 12-15 Snapshot Versioning .......................................................................................... 12-16 Versioning Level ............................................................................................... 12-17 Table Versioning................................................................................................ 12-18 Versioning Example .......................................................................................... 12-19 Option to COMMIT After Every SELECT ...................................................... 12-24 ODBC AutoPreCommit .................................................................................... 12-25 Snapshot Versioning: Issues .............................................................................. 12-26 Module Map ...................................................................................................... 12-27 Data Security ..................................................................................................... 12-28 Users.................................................................................................................. 12-29 TOC-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Users and Groups .............................................................................................. 12-30 Special User ...................................................................................................... 12-31 Creating Users ................................................................................................... 12-32 Creating Users with Sybase Central .................................................................. 12-33 Special Groups .................................................................................................. 12-35 Creating Groups ................................................................................................ 12-36 Adding a User to a Group ................................................................................. 12-37 Table and View Permissions .............................................................................. 12-38 Stored Procedure Permissions .......................................................................... 12-39 REVOKE Statement ......................................................................................... 12-40 Permissions Hierarchy ...................................................................................... 12-41 Database Object Names and Prefixes ................................................................ 12-42 Summary ........................................................................................................... 12-43 Lab 12: ASIQ Operational Data Administration ............................................... 12-44 Module 13 Joins Module Overview ................................................................................................ 13-1 Types of Joins ...................................................................................................... 13-2 Ad-Hoc Joins ....................................................................................................... 13-3 When to Use Ad-Hoc Joins ................................................................................ 13-4 When to Create a Join Index................................................................................ 13-5 Ad-Hoc Joins Versus Join Indexes ...................................................................... 13-6 Creating a Join Index........................................................................................... 13-7 Step 1: Create Tables ........................................................................................... 13-8 Step 2: Define the Relationships.......................................................................... 13-9 Join Hierarchy ................................................................................................... 13-10 Types of Join Hierarchies .................................................................................. 13-11 Entity-Relationship Diagram ............................................................................ 13-12 Join Hierarchy with the Top Table .................................................................... 13-13 Creating a Join Index......................................................................................... 13-14 Step 3: Create the Indexes .................................................................................. 13-15 Step 4: Load Data.............................................................................................. 13-16 Creating a Join Index......................................................................................... 13-17 Step 5: Create the Join Index ............................................................................ 13-18 Creating a Join Index......................................................................................... 13-22 Step 6: Synchronize Join Indexes ...................................................................... 13-23 Multiple-Table Join Example ............................................................................. 13-25 To Create the Join Index.................................................................................... 13-26 Rules for Join Indexes ....................................................................................... 13-27 Estimating Join Index Size ................................................................................ 13-28 Actual Join Index Size ...................................................................................... 13-29 Dropping a Join Index ....................................................................................... 13-30 Data Modification in a Join Index ..................................................................... 13-31 How Join Indexes Are Used .............................................................................. 13-32 Summary ........................................................................................................... 13-33 Lab 13: Joins ................................................................................................... 13-34 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-9 Table of Contents Module 14 Monitoring and Troubleshooting Module Overview ................................................................................................ 13-1 Module Map ........................................................................................................ 13-2 Monitoring Users................................................................................................. 13-3 Monitoring Loads/Deletes/Inserts........................................................................ 13-4 Monitoring Loading Messages ............................................................................ 13-5 Monitoring Load Messages ................................................................................ 13-7 Monitoring the Database ..................................................................................... 13-8 dbcc_option ....................................................................................................... 13-12 Monitoring the Database ................................................................................... 13-14 Monitoring Buffer Caches ................................................................................. 13-17 Stopping the Buffer Cache Monitor .................................................................. 13-21 System Utilities .................................................................................................. 13-22 Module Map ...................................................................................................... 13-23 Troubleshooting Fields ...................................................................................... 13-24 ASIQ Server Won't Start .................................................................................. 13-25 User Cannot Access the Database or Database Objects ........................................................................................... 13-26 Database Fills Up During the Load ................................................................... 13-27 Processing Slows Down or Hangs ..................................................................... 13-28 Reporting Problems to Technical Support ........................................................ 13-29 Summary ........................................................................................................... 13-30 Lab 14 -Monitoring and Troubleshooting ........................................................ 13-31 Module 15 Performance and Tuning Module Overview .......................................................................................................... 15-1 Module Map.................................................................................................................... 15-2 IQ Query Engine ............................................................................................................ 15-3 IQ Query Engine Architecture ..................................................................................... 15-5 IQ Query Optimizer ....................................................................................................... 15-6 IQ Query Optimizer ....................................................................................................... 15-9 Module Map..................................................................................................................15-10 Choosing Correct Index Types .................................................................................. 15-11 Database Options for Performance ............................................................................15-14 IQ Query Plans .............................................................................................................15-15 IQ Query Tree ...............................................................................................................15-16 IQ Query Plans .............................................................................................................15-17 IQ Query Plan Detail ...................................................................................................15-23 QUERY_ROWS_RETURNED_LIMIT .................................................................. 15-24 NOEXEC and ROWCOUNT Options ......................................................................15-25 IQ Query Plan -Elements ............................................................................................15-26 More Query Options ....................................................................................................15-28 IQ Query Performance ................................................................................................15-29 Module Map..................................................................................................................15-30 Loading Data from a Fixed-Length ASCII Flat File ............................................ 15-31 TOC-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Using Disk Striping ........................................................................................... 15-32 Using Multiple dbspaces ................................................................................... 15-33 Spreading Access Across Separate Disks .......................................................... 15-34 Improving INSERT/UPDATE/DELETE Performance ....................................... 15-35 Memory Allocation ........................................................................................... 15-36 Limiting Number of Active Queries on the IQ Server ....................................... 15-38 Summary ........................................................................................................... 15-39 Lab 15: Performance and Tuning ...................................................................... 15-40 Module 16 Backup and Restore Module Overview ................................................................................................ 16-1 Backup Features .................................................................................................. 16-2 Archive Devices .................................................................................................. 16-3 Backup ................................................................................................................ 16-4 Types of Backups ................................................................................................ 16-5 Running a Backup ............................................................................................... 16-6 BACKUP DATABASE Syntax ........................................................................... 16-7 Example: Simple Backup .................................................................................... 16-8 SIZE and STACKER Options.............................................................................. 16-9 Example: Incremental Backup .......................................................................... 16-10 Backup: Notes and Restrictions ........................................................................ 16-11 Validating Your Database .................................................................................. 16-12 sp_iqcheckdb..................................................................................................... 16-13 Restore Features ................................................................................................ 16-14 Before You Restore ........................................................................................... 16-15 Restore Database ............................................................................................... 16-16 Restore Database: Full Restore .......................................................................... 16-17 Moving Database Files ...................................................................................... 16-18 Restore Database ............................................................................................... 16-20 Restore Database: Notes and Restrictions ......................................................... 16-21 ASIQ Backup and Restore Performance ........................................................... 16-22 Estimating Dump Size ...................................................................................... 16-23 DBLOG Utility .................................................................................................. 16-24 Option for Backing Up a Large Database .......................................................... 16-25 Summary ........................................................................................................... 16-26 Lab 16: Backup and Restore ............................................................................ 16-27 Module 17 Multiplexing Overview Module Overview ................................................................................................ 17-1 What Is IQ Multiplexing? ................................................................................... 17-2 Multiplex Architecture .........................................................................................17-3 Multiplex Features ...............................................................................................17-4 ASIQ Architecture (Simplex) ............................................................................. 17-6 ASIQ Architecture (Multiplex)............................................................................ 17-7 Write and Query Servers ...................................................................................... 17-8 Communication between Servers ........................................................................ 17-9 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-11 Table of Contents Multiplex Operating Modes .............................................................................. 17-10 IQM Advantages ............................................................................................... 17-11 Summary ........................................................................................................... 17-12 Lab 17: Multiplexing Overview ........................................................................ 17-13 Module 18 ASIQ Multiplex Hardware Module Overview ................................................................................................ 18-1 Module Map ........................................................................................................ 18-2 Storage Basics ..................................................................................................... 18-3 RAID Basics........................................................................................................ 18-4 RAID Diagram .................................................................................................... 18-5 RAID Levels........................................................................................................ 18-6 RAID Striping ..................................................................................................... 18-7 RAID Connectivity.............................................................................................. 18-8 Module Map ........................................................................................................ 18-9 Configuring Storage .......................................................................................... 18-10 IQM and Shared Disk Farm .............................................................................. 18-11 IQ Multiplex Hardware ... Putting it together .................................................... 18-12 Summary ........................................................................................................... 18-13 There is no lab for this module .......................................................................... 18-14 Module 19 Creating a Multiplex Database Module Overview .......................................................................................................... 19-1 Steps to Set up Multiplex .............................................................................................. 19-2 Step 1 - Create the Write Server .................................................................................. 19-3 Create the IQ Write Server ........................................................................................... 19-4 Steps t ........................................................................................... o Set up Multiplex19-11 Step 2 - Create Reader (Query) Servers ...................................................................19-12 Create Query Server ....................................................................................................19-13 Directory Structure ......................................................................................................19-18 Steps to Set up Multiplex ............................................................................................ 19-19 Step 3 - Synchronize ....................................................................................................19-20 Synchronize ..................................................................................................................19-21 What we have so far .....................................................................................................19-22 Steps to Set up Multiplex ............................................................................................ 19-23 Why Add A Temp Store?............................................................................................19-24 Step 4 - Add Temp Store for Writer ..........................................................................19-25 Add Temp Store for Writer .........................................................................................19-26 Steps to Set up Multiplex ............................................................................................ 19-30 Step 5 - Add Main IQ Store dbspaces ........................................................................19-31 Start Server in Simplex Mode ................................................................................... 19-32 Add Main IQ Store dbspaces ......................................................................................19-33 Add Main Store Dbspaces .......................................................................................... 19-38 Stop the Writer Server .................................................................................................19-39 TOC-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Table of Contents Steps to Set up Multiplex .................................................................................. 19-40 Step 6 - Add Temp Store for the Query Server................................................... 19-41 Summary ........................................................................................................... 19-42 There is no lab for this module .......................................................................... 19-43 Module 20 Multiplex Operations Module Overview ................................................................................................ 20-1 Working in the Multiplex Environment ............................................................... 20-2 Creating Database Objects .................................................................................. 20-3 Setting Memory .................................................................................................. 20-4 Loading Data ....................................................................................................... 20-5 Running Queries.................................................................................................. 20-6 Specifying Database Options .............................................................................. 20-7 Postsync.sql Script .............................................................................................. 20-8 Working in the Multiplex Environment ............................................................... 20-9 Setting Startup Parameters ................................................................................ 20-10 Starting and Stopping ........................................................................................ 20-11 Select the Start Mode ........................................................................................ 20-12 Stopping Multiplex Server ................................................................................ 20-13 Stopping Individual Servers .............................................................................. 20-14 Managing Servers .............................................................................................. 20-15 Restarting Servers.............................................................................................. 20-16 Working in the Multiplex Environment ............................................................. 20-17 Adding and Dropping Dbspaces ....................................................................... 20-18 To Add A Permanent (Main) Dbspace .............................................................. 20-19 To Create a Temporary Dbspace ........................................................................ 20-20 To Drop a Main or Temporary Dbspace............................................................. 20-21 Working in the Multiplex Environment ............................................................. 20-22 Backup the Multiplex ........................................................................................ 20-23 Restore the Multiplex ........................................................................................ 20-24 Summary ........................................................................................................... 20-25 There is no lab for this module .......................................................................... 20-26 Appendix A Migrating Data from Prior Versions Appendix Overview .............................................................................................. A-1 Why Is Data Migration Necessary? ...................................................................... A-2 Migration Steps ..................................................................................................... A-3 Step 1 - Install Version 12.0.................................................................................. A-4 Step 2 - Create 12.0 Database............................................................................... A-5 Method 1 ............................................................................................................... A-6 Method 2 ............................................................................................................... A-7 IQ CREATE ASIQ TEMPLATE ........................................................................... A-8 Edit Scripts ............................................................................................................ A-9 Database Creation Script ..................................................................................... A-10 ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 TOC-13 Table of Contents Module Map ........................................................................................................ A-15 Migration Scripts ................................................................................................. A-16 Data Extract Script .............................................................................................. A-17 Data Load Script ................................................................................................. A-19 Editing Scripts ..................................................................................................... A-21 Running Scripts ................................................................................................... A-22 Module Map ........................................................................................................ A-23 Load the Data from your Original Source ........................................................... A-24 Module Map ........................................................................................................ A-26 Insert the Data from an Indexspace ..................................................................... A-27 Module Map ........................................................................................................ A-28 Select Data into an ASCII file ............................................................................. A-29 bcp Data into an ASCII file ................................................................................. A-30 Migration Notes .................................................................................................. A-31 Summary ............................................................................................................. A-33 TOC-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 1 Adaptive Server IQ Product Overview Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Module Overview This module will provide a brief overview of Adaptive Server IQ (ASIQ) Objectives After completing this module, you should be able to: ■ List the key advantages of ASIQ ■ Describe the principles at work behind ASIQ ■ Explain what ASIQ is ■ Explain what ASIQ is not ■ This module answers the basic questions about how ASIQ's unique technology can be used for decision support systems (DSSs), data marts, or data warehousing. Note ■ ©2000 Sybase, Inc. ■ This is an introductory module. Later modules discuss the architecture as well as the indexes in greater detail. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 -1 Adaptive Server IQ Product Overview When to Use Adaptive Server IQ (ASIQ)? Decision support system (DSS) Distributed data mart Data warehouse DSS Decision support system. Although some queries will be requested regularly, management often asks for unexpected aggregates of data that traditionally require individualized tuning for each query. Data mart A special-purpose data warehouse focused on a single strategic or efficiency issue. A data mart often serves as a single department's analytical base. Data warehouse Extremely large database encompassing an enterprise-wide set of data. A data warehouse can be used as the source of data for a company's data marts or for direct DSS queries. 1-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Using ASIQ to Split DSS and OLTP Off-Loads DSS Query Burden from an OLTP Engine Splitting DSS and OLTP • Reduces contention between readers and writers Ad-hoc query applications, used in decision support systems • Queries are not predefined • Are often I/O intensive OLTP Online transaction processing involves heavy inserting, updating, and deleting. Typically, queries bring back one row or only a few rows. Indexes are used sparingly and a Database Administrator can tune for the limited types of queries performed. DSS Decision support systems are typically read-only data that has been filtered, preprocessed, scrubbed, or rolled up for analysis. The data may have originated in an OLTP database and may be as new as one day old. Ad-hoc query A query that the system has no prior knowledge of and for which no explicit tuning can therefore be done. Reduce contention Typically, readers block writers and writers block readers. Contention results. Separating scan-intensive DSS users and OLTP users is beneficial to both. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-3 Adaptive Server IQ Product Overview Distributed Data Mart Specialized Servers for Speciaized Needs Data mart example 1-4 This example shows a number of clients accessing a specialized data mart, depending on the client's needs. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Data Warehouse Centralized Enterprise-Wide Data Warehouse Example ©2000 Sybase, Inc. This example shows a number of clients accessing a central, enterprise-wide data warehouse ASIQ Server. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-5 Adaptive Server IQ Product Overview What Is ASIQ? A Powerful Ad-Hoc Query Server ■ Designed to dramatically increase the speed of DSS queries performed against large amounts of data ■ Short list of features: • Vertical data storage and compression • Optimized access methods for all types of data • Uses bit-wise storage technology • Ad-hoc query optimizer • Standard open interfaces • Supported by Sybase Central All of the features are discussed in detail in later modules. Access method An access method is really an index that is defined on each column. Sybase Central Database management tool in a graphical user interface. 1-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview What ASIQ s Not An online transaction-processing engine • Use Adaptive Server Enterprise instead A tool for tuning an existing database A server for applications that require real-time updates • Use Adaptive Server Enterprise or Adaptive Server Anywhere instead ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-7 Adaptive Server IQ Product Overview Advantages of ASIQ: Overview Speed • Designed for ad-hoc queries • Reduced I/O on retrieval • Fast to implement Economy • Set up once for all queries • Excellent data compression characteristics • Reduced disk storage Flexibility • Interoperates with multiple data sources • Works with off-the-shelf query tools Speed Consistently fast response time for complex ad-hoc queries. Accesses only the data relevant to the query, reducing I/O up to 98%. Accelerates iterative analysis and information. Economy Enables Database Administrators (DBAs) to tune data once rather than for each query. Supports a standard relational framework, Sybase's Open Client interfaces, and ODBC, allowing DBAs to leverage existing skills and infrastructure. Provides more answers to more questions for more users, enhancing your productivity. Improves performance without adding hardware. Compresses data, dramatically reducing disk requirements and cost. Flexibility Loading from various data sources, including SQL Server, Adaptive Server, and flat files. Can be directly accessed through OmniConnect, which includes a large number of gateways. Works with a wide range of popular, off-the-shelf query tools, giving users their choice. Handles any database schema, allowing business users to see data their way. 1-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Advantages of ASIQ: Speed Scalability • Handles large amounts of data • Takes advantage of multiple processors for loading and queries • Multiplexing allows a multiple server configuration Speed of loading • Fast loads are a critical success factor for a data warehouse • Indexes can be incrementally loaded or deleted without being dropped and recreated • Can load data while users are querying Scalability example ■ Multiplexing Will be discussed in further detail in later modules. ©2000 Sybase, Inc. When ASIQ query workload increases by 20 times, average ASIQ response time increases by only 5 times. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-9 Adaptive Server IQ Product Overview Advantages of ASIQ: Economy Saves Disk Space by Storing Only the Index Index sizes are kept small through a variety of data compression technologies The entire database can be fully indexed and made available for ad-hoc queries in less space than it takes to store the raw data Less space Typically 50-100% of the raw data size. The indexes are the data; there is no need for both structures. No summary tables needed Summary tables are typically not needed because ASIQ returns queries so quickly without them. Compact indexes More compact, bit-wise indexing requires less disk space than traditional b-tree indexes. 1-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Advantages of ASIQ: Flexibility Flexible (open systems support) • Integrated with Open Client and Open Server • Integrated with ODBC, JDBC • To the end user and DBA, ASIQ looks just like another RDBMS ( c o n t inued . . .) Note ■ ■ ODBC driver ©2000 Sybase, Inc. The most recent list of tested tools can be found in the Release Bulletin that came with your software. Some of the PC tools are BrioQuery, Business Objects, Forest & Trees, GQL, Holos, InfoMaker, Iridon Panorama, Microsoft Access, MicroStrategy and Cognos. Special ODBC driver ships with ASIQ. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-11 Adaptive Server IQ Product Overview Advantages of ASIQ: Flexibility Can load data from a variety of sources Client Example 1-12 The graphic above shows that data can be loaded directly from an ASE database. Data can also be loaded from flat files as well as other data sources. See Module 8, "Loading Data from Files," Module 9, "Advanced Data Loading from Files" and Module 10, "Loading Data Using the INSERT Command." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview What Principles Are at Work? ■ Column-wise data structures ■ Bit-wise indexing This is an overview foil. These principles are covered in the next few pages. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 -1 3 Adaptive Server IQ Product Overview Column-Wise Data Structures Stores Data Column-Wise Rather Than Row-Wise ■ Vertical partitioning of data ■ Returns those columns necessary for a query, reducing I/O SELECT count(*) FROM customer WHERE gender = "M" Column-wise Column-wise data structures are easily searched without having to scan the full rows. Vertical partitioning of the data means never performing a table scan. At most, ASIQ would perform a column scan. Row-wise Row-wise data structures, such as those used in traditional RDBMSs, can be searched only by reading an entire page and then locating the data on the page. Because the columns being searched for are only a fraction of the total data read, there is excessive I/O. 1-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Bit-WiseIndexing Benefits Both Low- and High-Cardinality Columns ■ Translates the distinct data elements of a column into a bit mask ■ Cardinality - The number of unique values in a column • Low-cardinality data: Fewer than 1500 unique values • High-cardinality data: 1500 unique values or more ■ Example: High-Cardinality Data ■ Primary keys ■ Telephone numbers ■ Social security number Low-Cardinality Data ■ Day of week ■ Gender ■ States (continued . . .) Bit-wise indexing Unlike other indexing methods, ASIQ indexes are based on the cardinality of the data and the way the data will be used. ASIQ has indexes for both low- and high-cardinality columns. ASIQ indexing types are discussed in depth in Module 5, "ASIQ Index Types and Data Types". ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-15 Adaptive Server IQ Product Overview Bit-WiseIndexing Low-Cardinality Example The table on the left shows the unindexed data The table on the right shows a bit-wise index (continued . . .) Note This is not an actual index. It is merely a representation of how bit-wise indexing works for low-cardinality data. This topic is discussed in detail in Module 5, "ASIQ Index Types and Data Types". 1-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Bit-Wise Indexing High-Cardinality Example Actual data in Binary Data is stored vertically High cardinality/ bit-wise indexing ©2000 Sybase, Inc. Sybase has patented technology for indexing high-cardinality data using bitmaps. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-17 Adaptive Server IQ Product Overview ASIQ Multiplexing Allows a multi-server configuration Designed for managing large query loads across multiple nodes Supports many users, each executing complex queries against a shared database IQ Temp Store Catalog Store IQ Temp Catalog IQ Temp Catalog Store Store Store Store ASIQ Multiplexing IQ Main IQ Temp Catalog Store Store Store Must have ASIQ 12.4.2 or higher. Multiplexing is implements and managed through Sybase Central. ■ This topic is discussed in detail in Module 17, Multiplexing". “ 1-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Benefits of ASIQ ■ Reduced ongoing tuning of databases ■ Reduced need for Database Administrator resources after initial install ■ Reduced disk requirements ■ Rapid access to information ■ Users can access data with a variety of query tools ■ Indexes can be efficiently incrementally added or dropped • Without reloading data ■ ASIQ Version 12.4.2 can support a multi-server configuration, ASIQ Multiplex For more information about the benefits of ASIQ, see www. sybase. com/products/dataware/iqindex. html. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 -1 9 Adaptive Server IQ Product Overview Summary ASIQ Is... ■ Leading technology that stores data vertically ■ Fast at ad-hoc query resolution • Bit-wise storage solves low- and high-cardinality queries ■ Economical on disk usage due to compressed storage formats ■ Flexible, interoperable, open system ASIQ Is Not... ■ An OLTP database ■ For applications requiring real-time updates ■ A tool to tune an existing database 1 - 20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Adaptive Server IQ Product Overview Lab 1: ASIQ Product Overview Exercise Overview Topics ■ DSS . OLTP ■ High- and low-cardinality data ■ Row-wise and column-wise data structures ■ Bit-wise indexing Goal After completing this lab, you should be able to: ■ Explain the basic principles of ASIQ General tasks You will take a written quiz. Lab setup No lab setup is required. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1-21 2 Architecture and Terminology Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Module Overview This module provides the necessary background information for performing installations and administration. Objectives After completing this module, you should be able to: ■ Describe the ASIQ architecture ■ Define ASIQ terminology and concepts ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2- Architecture and Terminology ASIQ Platforms ■ Sun Solaris (SPARC) ■ IBM RISC System/6000 AIX ■ HP-UX HP 9000/800 ■ Compaq UNIX ■ Windows NT ■ SGI IRIX ■ LINUX (coming soon) Refer to the Adaptive Server IQ Installation and Feature Guide for detailed information on each platform. Refer to the Adaptive Server IQ Release Bulletin for detailed information on product restrictions, operating system patches, new and changed features, and documentation errors. 2-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Module Map ■ ASIQ Database Server ■ ASIQ Database • IQ Store • Catalog Store & Transaction Log • IQ Temporary Store • IQ Message Log ■ ASIQ Tables ■ ASIQ Indexes ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-3 Architecture and Terminology Adaptive Server Anywhere and Adaptive Server IQ ■ Adaptive Server Anywhere (ASA) is a relational database system that can exist with or without IQ • Small memory footprint • ANSI SQL plus additional functionality • Proven database technology ■ Adaptive Server IQ (ASIQ) exists with ASA • Patented storage technology • Vertical data storage • High-performance database query engine Anywhere database An Adaptive Server Anywhere database. Anywhere tables Anywhere tables have a different format from IQ tables. Enterprise database An Adaptive Server Enterprise (ASE) database. Also known as a SQL Server store. T-SQL Transact-SQL is a superset of Structured Query Language (SQL). It includes all ANSI standard SQL plus additional commands. 2-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Adaptive Server IQ Database Server ■ Combination of Adaptive Server Anywhere network server and ASIQ Server ■ ASA contains the catalog, language parser, and connectivity ■ IQ pieces include data store and query engine ASIQ ©2000 Sybase, Inc. Allows multiple users to query a database at the same time, while another user inserts or deletes data. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-5 Architecture and Terminology ASIQ Server and Database (Wthout Multiplex) ■ Database specifically created to store ASIQ indexes ■ Includes the following components: • IQ Main Store (for data) • Catalog Store & Transaction Log (for metadata) • IQ Temporary Store (for temporary data) • IQ Message Log (for log file) ■ Has one database per ASIQ Server Components of an ASIQ database are discussed in detail in the following slides. ASIQ Multiplex has a similar component structure and is addressed later. 2-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology IQ Main Store ■ Contains the data (indexes) • Compressed in blocks on disk ■ First physical file name is dbname.iq • User specifies the name for additional files ■ One IQ Main Store per database • Can consist of many physical files • Can be raw or file system ■ Also contains an internally managed transaction log ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-7 Architecture and Terminology Catalog Store ■ Contains information called metadata needed to manage an IQ database ■ Contains the system tables and stored procedures • Traditional horizontal storage system ■ Physical file names are: • dbname.db • dbname.log (Transaction Log) ■ One Catalog Store per database • Can consist of multiple devices (unusual) • File System ONLY (continued . . .) Metadata 2-8 ■ Describes the data in your database—for example, the size and datatype for each column in a table. ■ An ASA type datastore with 4K data pages. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Catalog Store System Tables ■ In addition to the system tables, the Catalog Store contains ASIQ-specific system tables, for example: • SYSIQCOLUMN • SYSIQFILE • SYSIQINDEX • SYSIQINFO • SYSIQTABLE (continued . . .) SYSIQCOLUMN Lists information on columns in every table or view in the IQ store. SYSIQFILE Lists information on operating system files for the database. SYSIQINDEX Lists internal information on indexes in the database. SYSIQINFO Lists database characteristics in addition to those in the SYSINFO table. SYSIQTABLE Describes each table or view in the IQ Store. For a complete listing of system tables and views, see the Adaptive Server IQ Reference Guide. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 Architecture and Terminology Catalog Store Transaction Log ■ Only used to log changes to the Catalog Store (not the warehouse data) ■ Used to roll back (undo) or roll forward changes if necessary ■ Physical file name is dbname.log m Very small file (~ 15 to 20MB ) • Placement not an issue Default By default the dbname.log file is placed in the same directory as the .db file (but can be placed elsewhere). CREATE DATABASE command You can turn logging off using the CREATE DATABASE command, but this is not recommended. Pruning To prune this log, first stop the server and rename the dbname.log file, then restart the server. 2-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology IQ Temporary Store ■ Work area for the database • Sorting data during loads • Query processing work area ■ Physical file name is dbname. iqtmp ■ One IQ Temporary Store per IQ database • Can consist of several physical files • Can be raw or file system The IQ Temporary Store is used for both loads and queries. It is used to build HG indexes during loads. Queries use it to sort data during ORDER BY and join queries. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-11 Architecture and Terminology IQ Message Log ■ Readable log file for the ASIQ database that contains: • Error messages • Status messages • Insert notification messages ■ Physical file name is dbname. iqmsg ■ One IQ Message Log per IQ database • File system only 2-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Putting It All Together...ASIQ Database ASIQ Database Components (with No Multiplex) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-13 Architecture and Terminology Module Map • ASIQ Database Server • ASIQ Database • IQ Store • Catalog Store & Transaction Log • IQ Temporary Store • IQ Message Log . ASIQ Tables ■ ASIQIndexes 2-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology ASIQ Tables ASIQ tables are logical tables defined in the Catalog Store • A table is really a group of columns Three types of tables: • Base tables • Local temporary tables • Global temporary tables These tables are discussed in great detail in Module 6, "Creating Tables and Indexes." Base tables These tables: ■ Are sometimes called main or permanent tables. ■ Are stored in the main IQ Store and are a permanent part of the database. ■ Must be dropped explicitly. Use the CREATE TABLE statement to create a base table. Local temporary tables Use the DECLARE LOCAL TEMPORARY TABLE statement. The table will exist until the connection ends or within a compound statement in which it is declared. This table is completely inaccessible to other users. Global temporary tables Use the GLOBAL TEMPORARY option of the CREATE TABLE statement. The table will exist in the database until it is explicitly removed. Each user will have a separate instance of the data in a global temporary table. Those rows are visible only to the connection that inserts them. They are deleted when the connection ends. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 2-15 Architecture and Terminology Utility Database ■ Created automatically during ASIQ installation ■ Used for connection purposes when no "real" database exists ■ Never holds data ■ DO NOT DELETE this database When to use the utility database 2 -16 To start the database server when no database is specified during the START ENGINE command. To create, drop, or restore a database when you have no other database to which to connect. To start the database server or connect to a database when other databases are corrupt or unavailable due to media failure. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology ASIQ Indexes ■ Each data column is assigned one or more data access methods (indexes) • Usually determined before initial load ■ Index type selection determined by: • Cardinality of data • Column usage ■ Four different types of indexes (discussed in Module 5): • Default Index (FP) • LF • HNG • HG Cardinality Low - Fewer than 1500 unique values. High - 1500 or more unique values. Column usage Joins, GROUP BY, aggregation. Index types are discussed in further detail in Module 5, "ASIQ Index Types and Data Types." ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-17 Architecture and Terminology ASIQ Tables and ASIQ Indexes Tables Contain Indexes on Columns ■ An index is an access method ■ Each ASIQ index contains all the data for a column ■ A column has one or more indexes ■ A default index is placed on each column when the table is created 2-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology interfaces File and ASIQ Connectivity ■ TDS clients and servers find each other via lookups in an interfaces file ■ The interfaces file is required for: • Servers, to discover the net addresses they should listen to for client connections • Clients, to discover where to direct connection requests ■ The interfaces file has different names on different platforms: • UNIX: $SYBASEinterfaces • Windows: %SYBASE%\ini\sql.ini • Novell Netware (clients only): interfac ■ TDS clients and CIS servers use the interfaces file to find an ASIQ Server instance ASIQ ©2000 Sybase, Inc. ASIQ uses the command line during startup and not the interfaces file. TDS clients (for example, SQL Advantage) will need the interfaces file, but DBISQL and Sybase Central do not. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-19 Architecture and Terminology ASIQ Architecture Putting It All Together... 2-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Architecture and Terminology Summary ■ An ASIQ database contains: • IQ Main Store (contains the data) • Catalog Store & Transaction Log (contains metadata) • IQ Temporary Store (contains temporary data) • IQ Message Log (contains database log file) ■ The ASIQ table definition is stored in the Catalog Store ■ The data is stored in indexes in the IQ Main Store ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2-21 Architecture and Terminology Lab 2: Architecture Exercise Overview Topics ■ ■ ■ Goals After completing this lab, you should be able to: ■ Define ASIQ architecture and terminology General tasks Lab setup 2-22 Process architecture components and functions ASIQ architecture Terminology: • ASIQ database server • ASIQ database • IQ Store • Catalog Store & Transaction Log • IQ Temporary Store • IQ Message Log • ASIQ tables • ASIQ indexes ■ Complete a matching game. ■ Complete a multiple-choice quiz. ■ Complete fill-in-the-blank questions. ■ Participate in an instructor-led discussion. ■ No lab setup is needed. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 3 ASIQ Environment and Installation Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Module Overview This module provides the necessary background information to install ASIQ and to connect to ASIQ. Objectives After completing this module, you should be able to: ■ Prepare for an ASIQ installation ■ Install ASIQ ■ Verify the success of an ASIQ installation This module is divided into two sections: ■ Environment & Installation. ■ Connectivity. ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- ASIQ Environment and Installation Module Map ■ System Requirements • Disk space • Memory • Swap ■ Installing ASIQ ■ Validating ASIQ Installation ■ Setting Up ODBC ■ Setting Up JDBC and Open Client 3-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Disk Space ■ IQ binaries • UNIX: ~300MB (if all binaries are installed) • NT: ~120MB ■ Database size • IQ Main Store: 70 - 100% of raw data • IQ Temporary Store - Varies based on users and data ■ Data staging area • Space required to load raw data from file ■ Swap • 2-3 times the amount of physical memory (RAM) The IQ binary size will vary from platform to platform. Refer to the Adaptive Server IQ Installation and Feature Guide for a specific platform. Database size The size of the IQ Main Store depends on the types of IQ indexes used and the data itself. Staging area If you are loading from another server, you do not need this additional space. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-3 ASIQ Environment and Installation Virtual Memory Virtual = Physical Memory + Swap Disk Space ■ Recommended minimum of 512MB of RAM ■ When the system is out of physical memory, swapping occurs • Processes are copied (swapped) out of memory to disk • Inactive pages are written from memory (paged) to disk ■ ASIQ has its own buffering system • Decides how to page buffers in and out of memory • Uses the operating system's swap disk ■ ■ Sybase recommends 256MB of RAM dedicated to ASIQ. 512MB allows for the operating system and other necessary components. More is better As with any RDBMS software, ASIQ requires a lot of memory. The more memory you can allocate to ASIQ, the better your system performance will be. Paging and swapping There is always a fixed limit to the amount of memory in a system. As a result, operating systems sometimes can have only part of the data in memory and the rest on disk. Paging or swapping occurs when the operating system must go out to disk and retrieve any data before a memory request can be satisfied. The primary objective of good memory management is to avoid or minimize paging or swapping. VMSTATorSAR Use the UNIX VMSTAT command or the SAR command, if available, to get statistics on the number of running processes and the number of page-outs and swaps. Use this information to find out whether the system is paging excessively, then make any necessary adjustments. Try to put your swap files on special fast disks. For more information on VMSTAT, type MAN VMSTAT. 3-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Swap Also Known As Swap Disk or Swap Partition ■ Recommendation: Swap should be set between 2-3 times the physical memory ■ See specific operating system information about extending swap ■ ASIQ uses shared memory for reads and writes • ASIQ pages out buffers to the swap disk ■ Insufficient swap space may result in: • System supporting fewer users • A large process may be stopped for lack of virtual memory Swap files ■ ■ ■ Examining swap The most frequently used operating system files are swap files. When memory is exhausted, the operating system swaps pages of memory to disk to make room for new data. When the pages that were swapped are called again, other pages are swapped, and the required memory pages are brought back. This is very time-consuming for users with high disk-usage rates. In general, organize memory to avoid swapping and thus minimize use of operating system files. On HP-UX: as root, type /etc/swapinfo On Solaris:/usr/sbin/swap -s On Compaq: as root, type /sbin/swapon -s On NT: Control Panel->System->Performance ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-5 ASIQ Environment and Installation Basic Configuration Recommendations System Requirement Recommendation Disk Space IQ binaries (~120- ~300MB) + database size (70-100% raw data) + staging area + IQ temporary space + swap (2-3 times RAM) RAM Recommend 512MB (more is better) ■ ■ IQ temporary space 3-6 The IQ binary size number can vary from platform to platform. Check the Adaptive Server IQ Installation and Feature Guide for a particular platform. Default is set to 50% of database size. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Hardware Recommendations Number and Type of Processors ■ More is better • Faster loading with SMP • Better multiuser performance ■ Faster is better • 233 MHz (or more) • IQ is more CPU intensive than I/O intensive CPU intensive ©2000 Sybase, Inc. The reason ASIQ is more CPU intensive is because of compressing and decompressing data. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-7 ASIQ Environment and Installation Module Map • System Requirements . Installing ASIQ • UNIX install • NT install ■ Validating ASIQ Installation ■ Setting Up ODBC ■ Setting Up JDBC and Open Client 3-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Before Installing Software on UNIX ■ Decide what components to install • ASIQ Server will likely be installed ■ Optional Components • Sybase Open Client Libraries ▲ Only need if you will load data from another Sybase server or if you want to use the ISQL client A Version 11 or Version 12, not both A Should install Open Client 12 unless Sybase ASE 11.x server is running on the same UNIX server ■ Sybase Central on the UNIX server • You need Sybase Central only if you want to run it on the UNIX server • May be installed on a Windows 9598/Win NT client ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-9 ASIQ Environment and Installation Preparing for ASIQ Installation on the UNIX Platform Before You Begin 1. Check for proper version of the OS 2. Install kernel patches (if any) 3. Set kernel parameters to suggested values 4. Configure user resource limits with ULIMIT -A, then reboot 5. Verify network functionality with TELNET host For more information on the individual platforms, see Chapter 1, "Adjusting the Operating System Configuration," in the Adaptive Server IQ Installation and Configuration Guide. 3-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Installing Adaptive Server IQ UNIX 1. Set the environment variable $SYBASE to a file system for the ASIQ binaries 2. Create the user "sybase"; home for the user "sybase" is $S YBASE 3. Mount the CD-ROM drive 4. Start the install utility, SYBINSTALL 5. Follow options on SYBINSTALL screen • Accept the default username and password for the utility database • After installation, you need to set additional environment variables Mounting CD-ROM drive Check the Adaptive Server IQ Installation and Feature Guide for the exact commands for your platform. If you have an existing version of Open Client on your system, installing the Open Client libraries will add missing Open Client files and overwrite older files. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-11 ASIQ Environment and Installation Setting Environment Variables ■ $SYBASE • Set to the /sybase directory (should already be set) • Contains up to three subdirectories A asiq_12 A sybase-central_3.2 (optional) A ociq-11_0(optional) or ociq-12_0(optional) Open Client 3-12 ■ ■ Open Client 11 - ociq-11_0 Open Client 12 - ociq-12_0 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Environment Variables ■ $ASDIR • $ASDIR = $SYBASE/asiq_12 • This directory contains the ASIQ Server binaries bin/ charsets/ include java/ javaconnect/ lib res/ scripts tix/ m $LD_LIBRARY_PATH • $LD_LIBRARY_PATH = $ASDIR/lib • This directory contains critical ASIQ Server library files (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-13 ASIQ Environment and Installation Setting Environment Variables ■ $SYBASE_OCS • Required only if you install Open Client 12 • $SYBASE_OCS = $SYBASE/ociq-12_0 ■ $PATH • Modify your path to include the following: A $ASDIRbin ♦ This is for the ASIQ Server binaries A If Open Client 11 is installed: ♦ $SYBASEbin A If Open Client 12 is installed: ♦ $SYBASE_OCS/bin ■ $ASLOGDIR (optional) • Specify a directory for ASIQ Server log messages • A new log message file will be created each time the server is started 3-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Starting the ASIQ Server - UNIX ■ ASIQ Server must be started with a database ■ After installation, you must start the ASIQ Server with either the asiqdemo database or the utility database ■ asiqdemo database • Supplied with the software • Easiest way to start and test the server installation ■ utility database • Never holds data ■ A "database" is a database file with a .db extension • Example: asiqdemo.db The ASIQ Server must be started from a writable directory. The directory where the server is started becomes the default directory for all files created by the ASIQ Server. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-15 ASIQ Environment and Installation Starting the ASIQ Server - Simple Method ■ Syntax: start_asiq dbname ■ start_asiq is located in $ASDIR/bin ■ Starts the server as a background process ■ Example: % start_asiq asiqdemo.db ■ This example starts the server with the asiqdemo database, this example assumes you are in the $ASDIR/demo directory ■ The default start parameters will be used 3-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Starting the ASIQ Server - UNIX ■ Preferred method of starting the server in UNIX ■ Syntax: start_asiq @configuration_file.cfg dbname m Example: % start_asiq @$ASDIR/demo/asiqdemo.cfg $ASDIR/demo/ asiqdemo.db ■ This example starts the server with the asiqdemo database ■ The configuration file is used to specify options you want whenever you start your server ©2000 Sybase, Inc. Administering Adaptive Server IQforthe DataWarehouse, Version 1.0 3-17 ASIQ Environment and Installation Server Configuration File All Platforms ■ A file with a .cfg extension that contains server starting parameters ■ These parameters override default ASA engine starting parameters ■ Are critical parameters for ASIQ performance ■ Configuration file example: -n eptgdev2_asiqdemo -c 16m -gd all -gm 10 -gp 4 0 96 -x tcpip{port=2638} ■ Do not use commas to separate parameters in this file Additional parameters are required to use Sybase Open Client. Discussed in "Connectivity" module. Example: asiqdemo.cfg -n eptgdev2_asiqdemo Server name (machine name_dbname) -c 16m Catalog Store cache size -gd all Database starting permissions -gm 10 Number of connections (default is 10) -gp 4096 Catalog Store page size -x tcpip{port=2638} Network connection protocol; 2638 is the default port number 3 -18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Stopping the Server UNIX ■ Run the STOP utility • stop_asiq A This utility prompts the user to stop an ASIQ Server • dbstop A Requires connection parameters ■ Example: dbstop -c "uid=DBA;pwd=SQL;eng=eptgdev2;dbn=asiqdemo" ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-19 ASIQ Environment and Installation Preparing for ASIQ nstallation on the NT Platform Before You Begin 1. Check the OS • See the ASIQ Release Bulletin for latest OS release information • Special instructions apply to NT Enterprise servers using more than 2 GB of memory 2. Decide what components to install • ASIQ Server will probably be installed • Optional components A Sybase Open Client Libraries - Version 11 or Version 12 • Sybase Central on the NT server 3. Check for sufficient page file (swap) space • You will reboot ASIQ after software is installed 4. Decide on location for binaries • Approx. 110MB required for ASIQ Server binaries • Approx. 24MB for Open Client (optional) • Approx. 35MB for Sybase Central Java Edition (optional) 3-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Installing ASIQ Server Components Required for Operation As a Network Server 1. Log in using an account with Administrator privileges 2. Exit any Windows programs running on your machine 3. Insert CD-ROM into CD-ROM drive 4. Run setup.exe 5. Select products to be installed • ASIQ Server ▲ Click Change button to view subcomponents A Deselect unnecessary subcomponents (as desired) ♦ 16-bit ODBC drivers ♦ C++ development components • Sybase Central Java Edition • Open Client 6. Accept the default login (DBA) and password (SQL) for the utility database (can be changed later) 7. Click Finish to complete installation 8. Restart computer If you have an existing version of Open Client on your system, installing the Open Client libraries will add missing Open Client files and overwrite older files. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-21 ASIQ Environment and Installation Starting the Server Start ASIQ ■ Click Start on the taskbar, select Programs-> Sybase->Adaptive Server IQ 12->Start ASIQ Demo Database • Starts the asiqdemo database using the name of the server on which you are running ASIQ or ■ At the NT command prompt: asiqsrv12 @asiqdemo.cfg asiqdemo.db ■ This example assumes you are in the $ASDIR/demo directory See the UNIX example in the student notes for contents of asiqdemo.cfg file. To display the startup options available, enter the following at the NT command prompt: asiqsrv12 /? ASIQ Server must be started with a database, either the asiqdemo or utility database. After creating a database, you can start the server with that database. 3-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Stopping the Server Stop ASIQ - NT 1. Double-click the icon in the Windows system tray to display the server window 2. Click Shutdown ■ Command-line options during startup control whether a server is stopped even if there are active connections ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-23 ASIQ Environment and Installation Installing ASIQ Server Client Components Required for Connection to a Network Server 1. Log in using an account with Administrator privileges 2. Exit any Windows programs running on your machine 3. Insert Client Components CD-ROM into CD-ROM drive 4. Run setup.exe 5. Select products to be installed • ASIQ 12 client components A Decide which subcomponents to install (click Change) ♦ 16-bit ODBC drivers may not be needed ♦ C++ development components (if needed) • Sybase Central Java Edition (for DBA) • Open Client 12 (if needed) A Open Client used in this class A Not required at some sites 6. Click Finish to complete installation 7. Restart computer Sybase Central and ASIQ are installed in separate folders unless you specify otherwise. 3-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Module Map • System Requirements • Installing ASIQ ■ Validating ASIQ Installation ■ Setting Up ODBC ■ Setting Up JDBC and Open Client ©2000 Sybase, Inc. 25 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- ASIQ Environment and Installation Validating the ASIQ Environment Was the Installation Successful? ■ Examine the installation log files ■ Access the ASIQ Server using: • DBISQL • Sybase Central (for database administration) • ISQL • Third-party tools ■ First, set up ODBC and JDBC • Next section 3-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Lab 3-1: Installation Exercise Overview Topics ■ Goals After completing this lab, you should be able to: . Install ASIQ ■ Start the ASIQ Server using the sample database General tasks ■ Increase virtual memory on your machine ■ Install the ASIQ Server components ■ Start the ASIQ Server ■ Define and document the ASIQ environment Lab setup ■ ■ ©2000 Sybase, Inc. 27 Installing ASIQ Server components Sufficient disk space and shared memory must be available. Must have the following CD: • Sybase Adaptive Server IQ for Microsoft Windows NT 4.0, version 12.0 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- ASIQ Environment and Installation Module Map • System Requirements • Installing ASIQ • Validating ASIQ Installation . Setting Up ODBC • Connect using DBISQL (GUI) • Connect using DBISQL (Quite Mode) • Connect using Sybase Central ■ Setting Up JDBC and Open Client 3-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up ODBC Start the ODBC Administrator • Start->Programs->Sybase->Adaptive Server IQ 12->ODBC Administrator Create a DSN (Data Source Name) by clicking Add (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-29 ASIQ Environment and Installation Setting Up ODBC Add a New Source ■ Select the Adaptive Server IQ 12 Driver Click Finish (continued . . .) 3-30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up ODBC Name the Data Source (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-31 ASIQ Environment and Installation Setting Up ODBC Log In ■ Click the Login tab ■ Enter a user ID of DBA and a password of SQL (continued . . .) 3-32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up ODBC Database Tab ■ Click the Database tab ■ Enter the database name of asiqdemo ■ Enter the full path for the asiqdemo.db (Catalog Store) file • Click Browse, if desired Can specify server name and server start command file May want to uncheck the Auto start and Auto shut down (continued ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-33 ASIQ Environment and Installation Setting Up ODBC Network Tab ■ Click the Network tab ■ Click TCP/IP ■ For remote servers; identify the host and port • Examples: host=poobah:2345 host=157.133.75.36:2345 (continued . . .) 3-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up ODBC Test the Connection ■ Click the ODBC tab ■ Click the Test Connection button ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-35 ASIQ Environment and Installation Connection to ASIQ using DBISQL Interactive SQL ■ DBISQL is an ODBC Client for communicating with ASIQ • Comes with the software • Preferred method for communicating with ASIQ • Useful for database administration • Can be used as a simple query tool ■ DBISQL has two modes • GUI • Quiet Mode (no GUI) 3-36 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Starting DBISQL - GUI ■ Windows: • Start->Programs->Sybase->ASIQ 12->Interactive SQL ■ UNIX: % dbisql -c "uid=DBA;pwd=SQL;eng=eptgdev2_asiqdemo" Option Description "" Need to use double quotes -c Database connection parameters uid User id pwd Password eng Servername ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-37 ASIQ Environment and Installation Ready to Connect Interactive SQL ■ Start->Programs->Sybase->ASIQ 12->Interactive SQL ■ Specify the ODBC DSN you just created 3-38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Interactive SQL ■ Enter commands in the bottom window ■ Function keys • F1 = Help for DBISQL • F7 = List tables (select table to list the columns) • F9 = Execute command (or click the Execute button) • F10 = Activate menus at top of window ■ Stop = Cancel command ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-39 ASIQ Environment and Installation Connect Using Command Line DBISQL DBISQL (Quiet mode) ■ Syntax: dbisql ‘ -c keyword=value; . . . ' ] [-d] [-q] [-x] Option Description -d Command delimiter (defaut is ;) -q Quiet mode (no window or messages), used to submit scripts -x Syntax check only ■ Example: dbisql -c ‘uid=DBA;pwd=SQL;eng=eptgdev2_asiqdemo' -q load.sql ■ In this example load.sql is the name of the script file Parameters ■ ■ Always embedded in quotes. Separated by a semicolon. See Chapter 2 of the Adaptive Server IQ Administration Guide or Chapter 3 of the Adaptive Server IQ Reference Manual. 3-40 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Connecting Through Sybase Central ■ Graphical user interface to Sybase database products • Can use ODBC or JDBC to connect • This is a Java version of Sybase Central for use only with ASIQ To connect to ASIQ (NT): 1. Open the Adaptive Server IQ 12 folder • Click Start on the taskbar, select Programs->Sybase-> ASIQ12->Sybase Central 2. Enter a login and password in the connection dialog box To connect to ASIQ (UNIX): % cd $SYBASE/sybcentral % scjview ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-41 ASIQ Environment and Installation Sybase Central Connecting to a Database ■ On the Sybase Central menu bar, select Tools->Connect->ASIQ ■ In the connection box, click ODBC and enter the data source name 3-42 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Sybase Central Features ■ Create an IQ database ■ Add or remove database devices ■ Create tables, views, and indexes ■ Create stored procedures ■ Add users and groups and manage permissions ©2000 Sybase, Inc. 43 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- ASIQ Environment and Installation Module Map • System Requirements • Installing ASIQ • Validating ASIQ Installation • Setting Up ODBC ■ Setting Up JDBC and Open Client 3-44 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Database Configuration File for Open Client 12 ■ This step is required only if you access an ASIQ 12 with Open Client 12 ■ Add a parameter in the server configuration for the database name ■ Use a second -n parameter for the database name ■ This database name is used in the interfaces file as the server name ■ Example: -n eptgdev2_asiqdemo -c 16m - g d a l l -gm 10 -gp 4 0 96 -n asiqdemo ■ See the ASIQ Release Bulletin on Open Client Library changes ©2000 Sybase, Inc. 45 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- ASIQ Environment and Installation Setting Up JDBC and Open Client ■ Set up a server entry using DSEDIT • Start->Programs->Sybase->Open Client Directory Services Editor • Click OK (continued . . .) 3-46 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up JDBC and Open Client Under Server Object, click Add (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-47 ASIQ Environment and Installation Setting Up JDBC and Open Client ■ Enter the server name Enter the network address for the server (continued . . .) 3-48 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Setting Up JDBC ■ Dsedit should look similar to this: You can ping the server to test the connection you made ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-49 ASIQ Environment and Installation Connect Using ISQL ■ Start ISQL from command window ■ Example: isql -UDBA -PSQL -Sasiqdemo [-i<file>][-o<file>] ■ You are connected to your database • No concept of USE DATABASE • Command terminator default is still "go" 3-50 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation ISQL Versus DBISQL ■ You cannot direct output from DBISQL • No equivalent to ISQL -o parameter • IQ has TEMP_EXTRACT options for SELECT ■ DBISQL command terminator is a semicolon (;) ■ DBISQL can start the server, if desired ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-51 ASIQ Environment and Installation sp_iqstatus Shows information about the current database • Database name • Creation date • IQ Page size • Block size IQ-specific stored procedure, which is valid only on ASIQ servers If you are unable to connect and execute a command, check the "Troubleshooting" section of the Adaptive Server IQ Administration Guide. IQ Page size Adaptive Server IQ displays the IQ Page size in addition to the block size. For example: Page Size: 1024/512blksz/2bpc 3-52 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Running sp_iqstatus Command to Check ASIQ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3-53 ASIQ Environment and Installation Summary of Installation Steps ■ Prepare for the ASIQ install • Disk space • RAM ■ Perform the installation • Follow steps outlined and use installation guide ■ Verify the installation • DBISQL • Sybase Central (for database administration) • ISQL ■ Run sp_iqstatus to show database information For more information on the individual platform installations, see the appropriate Adaptive Server IQ Installation and Feature Guide. 3-54 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Environment and Installation Lab 3-2: Connectivity Exercise Overview Topics ■ ■ ■ Goals After completing this lab, you should be able to: ■ Set up a DSN using the ODBC Administrator ■ Use DSEDIT set up JDBC ■ Connect to the server using Interactive SQL General tasks ■ ■ ■ Access the asiqdemo database using Interactive SQL and ODBC. Run sp_iqstatus to list information about the asiqdemo database. Install student files for later use. Lab setup ■ Must have installed the ASIQ Server in Lab 3-1. ©2000 Sybase, Inc. 55 How to use the ODBC Administrator Setting up a server entry using DSEDIT Connecting to the server using Interactive SQL Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 3- 4 Creating Databases Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Creating Databases Module Overview This module describes how to create and administer an ASIQ database. Objectives After completing this module, you should be able to: ■ Determine space requirements for a database ■ Create/drop a database ■ Add space to a database ■ Drop space from a database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-1 Creating Databases Database Review ■ Created using CREATE DATABASE command IQ database Consists of the IQ Store, the Catalog Store, the Transaction Log, the IQ Temporary Store, and the IQ Message Log File containing related information. IQ Store Catalog Contains the IQ data as well as the database transaction log. Store Transaction Contains table definitions, system tables, and stored procedures. Log IQ Temporary Contains the updates to the metadata. Contains temporary tables Store IQ Message generated by certain queries. Log File Contains error, status, and insert notification messages for the IQ database. 4-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Database Overview dbspace ■ A dbspace is a logical name for a database device ■ Each database includes multiple dbspaces ■ The IQ Store, Catalog Store, Transaction Log, IQ Temporary Store, and IQ Message Log File all consist of dbspaces ■ Initially, each IQ Store has only one dbspace ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-3 Creating Databases Creating a Database 1. Select the device type 2. Estimate the database size 3. Create the database Step 1 Either raw or file. Step 2 Need to know the size of the raw data to be loaded. Step 3 This step is similar to using the DISK INIT and CREATE DATABASE commands on ASE. 4-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Step 1: Select the Device Type Raw Partitions Versus File Systems Advantages Raw Disadvantages - Managed by application, not the - Harder to manage file system - Performance can be higher - Avoid UNIX file system overhead - ASIQ has buffer control File System - Manageability - Additional overhead for reads - Double buffering - File system does not understand ASIQ pages Raw partitions Raw partitions are recommended for production environments because they do not use memory for file system buffering. File Systems ■ ■ set ■ ©2000 Sybase, Inc. If you are using file systems for your IQ database and running on Solaris or NT, you should consider turning off file system cache buffering. Some queries are competing within the OS for cache. To disable buffering: option public.OS_File_Cache_Buffering='Off You then need to cycle the ASIQ server. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-5 Creating Databases Step 2: Estimate the Database Size ■ Use 100% of the size of the raw data for the IQ Store • Additional space can always be added ■ The proper sizing of the IQ Temporary Store depends on: • Number of users • Loading of HG indexes • Types of queries ■ The IQ Temporary Store is 50% of the IQ Store by default ■ Question to ask • What is a good dbspace size? dbspace size 4-6 Recommendations are specific for each platform. Check the specific platform guide. dbspace sizing is also discussed in Module 15, "Performance & Tuning." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Step 3: Create the Database ■ Can either use CREATE DATABASE statement in DBISQL or use Sybase Central ■ Must have the server started with an existing database (asiqdemo or utility) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-7 Creating Databases Files Allocated When Creating Database ■ dbspaces created with CREATE DATABASE • dbname.iq - Main IQ Store • dbname.db - IQ Catalog Store • dbname.log - Transaction Log • dbname. iqtmp - IQ Temporary Store • dbname.iqmsg -Message Log dbname.iq You can specify a different name during database creation. You can add dbspaces to increase the size of the main IQ Store. dbname.db This is the initial dbspace for the Catalog Store; you can later add dbspaces to increase the size. dbname.log This is the dbspace for the Catalog Transaction Log; by default placed in the same directory as the .db file. dbname.iqtmp The required size can vary depending on query types and the amount of data. You can later add dbspaces to increase its size. dbname.iqmsg You can specify a different file name on database creation. In addition to these database files, the database server also uses a temporary file to hold information during a session. This file is not the same as the IQ Temporary Store. 4-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Full Syntax Required ■ The IQ PATH clause is required for creating an IQ database. Otherwise, you will create a standard Adaptive Server Anywhere database. Defaults: Relative Pathnames ■ The Catalog Store and the IQ Store are created relative to the working directory of the ASIQ Server. The Temporary Store and the Message Log are created in the same directory as the Catalog Store. ■ Creating a database with relative pathnames can cause unexpected results. It is better to create a database using fully qualified pathnames. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 Creating Databases Command Options ■ db-name • Usually databasename.db • For example: mydb.db m LOG OFF • Recommendation: do not use ■ TRANSACTION LOG clause • By default, log placed in the same directory as the .db device ■ MIRROR clause • By default, no transaction log mirror ■ CASE clause • By default, case sensitive (CASE RESPECT) (continued . . .) db-name ■ This is the full path for the Catalog Store file. TRANSACTION LOG ■ The database server tracks all changes made by all users in this log. It should be placed on a different physical device from the .db and .iq dbspaces. MIRROR ■ This is an identical copy of the transaction log. You must provide a pathname, usually on a separate device to mirror. CASE ■ Used during comparisons and string operations. 4 -10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Command Options ■ PAGE SIZE clause • By default, 4096 (PAGE SIZE 4K) • Recommend leaving at default ■ COLLATION clause • By default, collation of uppercase and lowercase letters is identical for a case-sensitive database ■ BLANK PADDING clause • By default, trailing blanks are ignored (ON) ■ JAVA clause • By default, installed (JAVA ON) (continued . . .) PAGE SIZE ■ This is the page size for the catalog tables. It can be 1024, 2048, or 4096 bytes. Using 4096 is recommended, because each server can support only one catalog page size. By setting this value high, you will always have an adequate page size for the Catalog Store. COLLATION ■ If data is all one case, use ISO_BINENG. If mixed-case data, use the default. BLANK PADDING ■ Should be set to ON to ignore trailing blanks in queries. JAVA ■ To use Java in your database, you must install entries for the Sybase runtime. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-11 Creating Databases Command Options ■ JCONNECT clause • By default, ON (JCONNECT ON) ■ IQ PATH • Required; should be an absolute file path ■ IQ SIZE • For raw, by default, use the entire raw device • For file, no default; you must specify a size in MB ■ IQ PAGE SIZE • By default, 65536 (64K) • For low-memory and high-user-count systems, use 32,768 (32K) • For Very Large Data Base (VLDB) systems (greater than 50GB), use 131,072 (128K) (continued . . .) JCONNECT ■ To use the jConnect JDBC driver, you need to install jConnect support. IQ PATH ■ The pathname where the IQ data will be stored. It can be a file or raw partition. IQ automatically detects which type is specified. There is a limit of 65536 devices. IQ SIZE ■ The size in MB of the file you specified with the IQ PATH clause. For raw partitions, you should always accept the default, which is not to specify size. For file, you can specify a value up to 128GB, depending on the operating system. IQ PAGE SIZE ■ The page size in bytes for the IQ Store. This value is a power of 2 between 1K and 128K. The IQ PAGE SIZE determines the default I/O transfer block size and maximum compression. Recommendations: • Low memory (<512MB) with more than 5 users and/or large database, use 32K. • Generally use the default. • For databases larger than 50GB, use 128K. ■ 4-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Command Options ■ BLOCK SIZE • Default is system generated by IQ Page Size • The default is virtually always optimal ■ MESSAGE PATH • By default, created relative to the directory of the .db file ■ TEMPORARY PATH • By default, created relative to the directory of the .db file • Recommend specifying a path ■ TEMPORARY SIZE • By default, one half the size of the value of IQ SIZE for files • Recommend specifying a size BLOCK SIZE ■ ■ ■ ■ The I/O transfer block size for the IQ Store. Valid values are between 512 and 32,768. For the default IQ Page Size of 64K, the Block Size default is 4096. See Module 7, "ASIQ Memory Configuration," for the table of default block sizes. Recommendation: Do not touch! MESSAGE PATH ■ ■ The pathname where the IQ message files will be stored. Can be a raw partition or file. IQ automatically detects which type. A file system is recommended. TEMPORARY PATH ■ The pathname of the temporary file containing temporary tables generated by certain queries. Can be raw partition or file. Raw device is recommended. ■ TEMPORARY SIZE ■ ■ ©2000 Sybase, Inc. The size in MB of either the raw partition or the operating system file you specified with the TEMPORARY PATH. For raw partitions, you should always accept the default, which is to use the entire raw partition. The default for file is one half the value of IQ SIZE. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-13 Creating Databases Create Database: Example 1 ■ NT example: CREATE DATABASE 'D:\\mydb\\mydb.db' BLANK PADDING OFF IQ PATH 'D:\\mydb\\mydb.iq' IQ SIZE 200 PAGE SIZE 4096 ■ Creates an IQ database called mydb with the following files all in the same directory: • mydb.db - mydb.iqtmp - mydb. log • mydb. iq - mydb. iqmsg m All files are created in d:\mydb ■ The size of the database is 200MB with a page size of 4K Example In NT, if you specify a path, any backslash characters must be doubled if they are followed by an "n" or an "x". This prevents them from being interpreted as a newline character or as a hexadecimal number. It is always safer to double the backslash. Specifying all the options and not relying on the defaults is a good idea. 4-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Create Database: Example 2 ■ UNIX raw partition example: CREATE DATABASE big.db IQ PATH '/dev/md/rdsk/big.iq' MESSAGE PATH '/sq/data/big.iqmsg' TEMPORARY PATH '/dev/md/rdsk/big.iqtmp' PAGE SIZE 4096 ■ Raw devices for: • IQ PATH • TEMPORARY PATH ■ How big will the IQ Main Store and IQ Temporary Store be? The IQ Main Store and IQ Temporary Store will be as big as the raw device has been defined. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-15 Creating Databases Creating Databases: Sybase Central (continued . . .) Sybase Central Open the Utilities folder and select Create Database. ■ Follow the create database wizard and select desired database options. • You need to select a file name for the new database, such as mydb.db. • Also, you can select the options listed in the create database command. Create Database wizard ■ 4-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Creating Databases: Sybase Central This is the last page of the wizard. The lab guides you through each step in the Create Database wizard. Caution: Clicking the Back button resets Sybase Central to the default values for creating a database. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-17 Creating Databases Before Starting the Server with the New Database ■ Create a configuration file for the new database • Specify database name • Specify port number • Specify any other startup options desired • Each option must be on its own line ■ Example: mydb.cfg -n eptgdev2_mydb -c 16m -gc 6000 gd all -gm 10 -gp 4 0 96 -gr 6000 -ti 4400 -tl 300 -x tcpip{port=2640 Definition of the command-line switches are coming. 4-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Command-Line Switches Server switches that are needed to run ASIQ effectively ■ -n = server name • Required if server is started without a database ■ -c = cache size for Catalog Store • Default = 2 MB • Recommend 16MB to keep entire catalog in cache ■ -gp = Catalog store page size • Default = 1024 • Recommend 4096 to allow more table columns ■ -gm = limits number of concurrent users • Default = 10 • Recommend to set as needed, up to maximum in license ©2000 Sybase, Inc. Administering Adaptive Server IQforthe Data Warehouse, Version 1.0 4-19 Creating Databases Other Recommended Switches ■ -gc = checkpoint interval (minutes) • Default = 60, recommend 6000 ■ -gr = max recovery time interval (minutes) • Default = 2, recommend 6000 ■ -ti = default client timeout (minutes) • Interval to disconnect inactive clients • Default = 240, recommend 4400 (~73 hours) ■ -tl = default network timeout (seconds) • Interval a "liveness" packet is sent to a client • Default = 120, recommend 300 or higher 4-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Command-Line or Configuration File Switches Server switches that override the SET OPTION command ■ These parameters can be on the command line or in the configuration file • -iqmc = main cache size A Specify main cache size in MB ▲ Overrides MB values of the database options • -iqtc = temporary cache size A Specify main cache size in MB ▲ Overrides MB values of the database options ■ Example (command line): %asiqsrv12 50 @asiqdemo.cfg asiqdemo.db -iqmc 50 -iqtc These switches are useful to start the database if memory has been overallocated and the database will no longer start. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-21 Creating Databases Communication Protocols ■ Server starts up with all available protocols • Default behavior ■ TCP/IP supported on all platforms ■ WinNT platforms • IPX - Client and server (and Win95 clients) • NetBIOS - Client and server (and Win95 clients) • Named pipes (for Win 3.x clients using ODBC) 4-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Communication Protocol Switches ■ Use the -x parameter to specify protocols • Default TCP/IP port will be 2638, if not specified ■ Examples: -x tcpip(MyIP=192.75.2 0 9 . 1 2 : 2 3 6 7 ) A Equates to an IP address with port 2367 -x tcpip A Use only TCP/IP with default port -x tcpip(port=2640) A Start server using port 2640 ©2000 Sybase, Inc. 23 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4- Creating Databases Starting the Server with the New Database ■ Start server from command line ■ Assume in the appropriate directory ■ NT example: asiqsrv12.exe @megaphone.cfg megaphone.db ■ UNIX example: asiqsrv12 @megaphone.cfg megaphone.db 4-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Creating a New DSN Set up ODBC See the section "Setting Up ODBC" in Module 3, "ASIQ Environment and Installation," for more information ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-25 Creating Databases Viewing the New Database: Sybase Central Sybase Central To connect: Tools->Connect->Adaptive Server IQ. Specify the user ID and password with the DSN name or use JDBC and specify the correct port number. Example The database created is called mydb. The server is eptgdev2. Double-click eptgdev2_mydb to see a statistics folder and mydb (DBA) database. 4-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases sp_iqstatus ■ Provides detailed information about the current database ■ Includes percentages of the IQ Store and IQ Temporary Store used ■ Can use Interactive SQL or DBISQL ■ Example on next page ©2000 Sybase, Inc. 27 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4- Creating Databases Checking the Database with sp_iqstatus ■ sp_iqstatus: 4-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Enlarging a Database ■ Syntax: CREATE DBSPACE dbspace -name AS filenam e ... [ { IQ STORE | IQ TEMPORARY STORE | CATALOG STORE } . . . [ [SIZE] file-size ] ■ Example: CREATE DBSPACE mydb_tmp_2 AS 'e:\\s2\data\\mydb_2.iqtmp' IQ TEMPORARY STORE SIZE 200 ■ Creates an additional dbspace for the IQ Temporary Store ■ When you create a dbspace, you use a logical name and a physical name (logical AS physical) CREATE DBSPACE SIZE ©2000 Sybase, Inc. 29 ■ The default is to add space to the IQ Store. ■ The size is in MB, from 1 to 128,000, and either raw partition or operating system file. You can only specify SIZE for the IQ Store and the IQ Temporary Store, not the Catalog Store. ■ A database can have up to 65,536 dbspaces, including the initial dbspaces created when you create the database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4- Creating Databases Enlarging a Database: Sybase Central 1. In the appropriate database, select the Dbspaces folder and double-click Add Dbspace 2. Follow the wizard to add a dbspace: • Enter the dbspace name • Click the type of data to be stored: IQ or IQ Temporary • Enter the filename and size (optional) • Click OK 4-30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Decreasing a Database Drops the dbspace ■ A dbspace containing data cannot be dropped ■ dbspaces must be dropped in the reverse order they were created ■ Cannot drop any of the four initial dbspaces ■ Syntax: DROP DBSPACE dbspace-name ■ Example: DROP DBSPACE mydb_tmp_2 Because of the way ASIQ fills dbspaces with data, it is unlikely that you will be able to drop the last dbspace, especially if disk striping is used. To find out if you can drop a dbspace, run sp_iqstatus. Look at the DB Blocks value and compare this value to the Main IQ Blocks used (or Temporary) to see whether the Max block number is in the dbspace. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4-31 Creating Databases Dropping a Database Removes a Database and All of Its Contents ■ Syntax: DROP DATABASE db-filename m Example: DROP DATABASE 'D:\mydb\mydb.db' ■ Drops database mydb m You must specify the database file name you defined during the CREATE DATABASE operation ■ The database must be stopped before you can drop it 4-32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Other Useful Stored Procedures ■ sp_iqcheckdb • Checks the validity of the current database ■ sp_iqdbsize • Lists the size of the current database ■ sp_iqdbstatistics • Contains statistics for the main IQ Store ©2000 Sybase, Inc. 33 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4- Creating Databases sp_iqdbsize For more information on sp_iqdbsize and other stored procedures, see Chapter 13 of the ASIQ Reference Manual. 4-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Databases Summary ■ Determine the device type, either file or raw. ■ Determine approximate database size. ■ To create a database, use CREATE DATABASE. ■ To view database particulars, use sp_iqstatus. ■ To drop the database, use DROP DATABASE. ■ To enlarge the database, use CREATE DBSPACE. ■ To decrease the database, use DROP DBSPACE. ©2000 Sybase, Inc. 35 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 4- Creating Databases Lab 4: Databases Exercise Overview Topics ■ ■ . . CREATE DATABASE The four files created after CREATE DATABASE and their locations CREATE DBSPACE, DROP DBSPACE DROP DATABASE Goals This module introduces the process of creating an IQ database. After completing this lab, you should be able to: ■ Create a database ■ Identify the files associated with a database General tasks ■ Using Sybase Central, create an ASIQ database. ■ Check the status of the database. ■ Add a dbspace to the database. Lab Setup ■ 4-36 The ASIQ Server should be installed with the asiqdemo database running. Connectivity to the database should have been verified. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 5 ASIQ Index Types and Datatypes Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Module Overview This module discusses ASIQ indexes, including how they work and guidelines for choosing appropriate types. It also discusses ASIQ datatypes, including range and storage information. Objectives After completing this module, you should be able to: ■ Define the different ASIQ indexes and choose from among them ■ Discuss ASIQ datatypes ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5- ASIQ Index Types and Datatypes Module Map ■ ASIQ Index Types ■ ASIQ Datatypes 5-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Types of Indexes Used by Typical RDBMSs ■ For most indexed columns, use a balanced-tree (B-tree) index B-tree index ■ ■ B-tree indexes are excellent at finding single-data values. Data that is not relevant to a query is quickly screened out. ■ B-tree indexes are not good at handling columns with few unique values (such as Y/N or M/F). These indexes are quite large and not well suited for large query results. ■ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-3 ASIQ Index Types and Datatypes ASIQ's Bitmap ndexes ■ Bitmaps represent the given value as a particular bit mask (for example, 0, 1) ■ Efficiently indexes data regardless of cardinality ■ Easier to maintain (no need to drop and build indexes) 5-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Review: ASIQ ndex Characteristics ■ Fast to load, fast to query ■ Compact • Requires less disk space than traditional B-tree indexes • More data can be kept in memory ■ Column-wise storage structures • Key to fast query resolution ■ Each column has one, two, or three indexes ■ Index type selected based on datatype, cardinality, and use of data ■ Each index (access method) is stored and manipulated differently ■ Can incrementally add or delete rows use of data Compact Type, cardinality, and All three aspects of the data are taken into consideration in assigning index types. Data and indexes usually take up less space in ASIQ than in other RDBMSs. Compression depends on the size and nature of the data itself. Incremental update ASIQ is designed for periodic loads (including incremental loads), not realtime updates. Access method ©2000 Sybase, Inc. ASIQ indexes are really data access methods, but they are called indexes. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-5 ASIQ Index Types and Datatypes Review: Four Basic Index Types Each index type is discussed in great detail later. This discussion is intended only as an overview. Fast projection 5-6 Also referred to as the default index. This index has two subtypes: Fast Fast Projection (FFP) and Fast Fast Fast Projection (FFFP). Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Fast Projection (FP) A Column's Raw Data Is Compressed ■ Default column storage, automatically created by ASIQ, cannot be removed ■ Placed on all columns: needed for SELECT list columns, string searches, ad-hoc table joins ■ Typically, you supplement this default index with one or more of the other ASIQ index types ■ Used with wildcard string searching—for example, LIKE '%sys%' ■ Good for calculation expressions—for example, SUM (A + B) ■ Only index that is valid for BIT datatype and VARCHAR >255 ■ Data column examples: • Address • Name • Text descriptions • Project descriptions (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-7 ASIQ Index Types and Datatypes Fast Projection (FP) Subtype: Fast Fast Projection (FFP) ■ Automatically created when you use IQ UNIQUE parameter on CREATE TABLE command • IQ UNIQUE defines the discrete values for the column • Discussed further in Module 6, "Creating Tables and Indexes" ■ If fewer than 256 values in a column, data is stored as Fast Fast Projection (FFP) instead of FP • One-byte look-up table • Will create a FFFP after 256 values are detected ■ The optimizer may use the FFP in terms of distinct values to determine query plans (continued . . .) 5-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Fast Projection (FP) Subtype: Fast Fast Fast Projection (FFFP) custjype ■ Automatically created when you use IQ UNIQUE parameter on CREATE TABLE command • IQ UNIQUE defines the discrete values for the column • Discussed further in Module 6, "Creating Tables and Indexes" ■ If more than 255 and fewer than 65,536 values in a column, data will be stored as FFFP instead of FP • Two-byte look-up table • Will revert to FP after 65,536 values are detected ■ The optimizer may use the FFFP in terms of distinct values to determine query plans ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-9 ASIQ Index Types and Datatypes Low Fast (LF) Bitmap Index Used with Low-Cardinality Data ■ A series of value-based bitmaps used for processing all queries ■ Ideal for columns with a low number of unique values (<1500) ■ Used with the following query operations: • Search arguments in WHERE clause • Joins • GROUP BY • ORDER BY ■ Data column examples: • Male/female • Yesno • Product name • State or province • Date (if fewer than 1500 distinct values) A LF index will allow a maximum of 10,000 distinct values, although 1,500 distinct values is the recommended cutoff. A load will fail if more than 10,000 unique values are inserted. 5-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes High Non Group (HNG) Bit-Wise Index Optimized for Range Searches and Aggregate Functions ■ Example: SELECT SUM(sales) FROM customer ■ A non-value based bitmap ideal for operations involving ranges or aggregations ■ For High and Low Cardinality data ■ Ideal for columns used in: • Ranges • BETWEEN • SUM() and AVG() functions ■ Examples: • Date • Amount • Quantity ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-11 ASIQ Index Types and Datatypes High Group (HG) ASIQ Index Used for High-Cardinality Data ■ Enhanced B-tree index to process equality and GROUP BY operations on high-cardinality data ■ Needed for columns with a high number of unique values (>1500) ■ Used with the following query operations: • Search arguments in WHERE clause • Joins • GROUP BY • ORDER BY ■ A HG index created as UNIQUE or as a PRIMARY KEY will not have a G-Array structure and will use less space ■ Data column examples: • Airport code • Product code • Employee ID 5-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Generic Method for Indexing Columns Four Basic Steps ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-13 ASIQ Index Types and Datatypes Combining Index Types ■ Reasonable combinations are marked with a Prior to ASIQ 12.0 ■ ■ 5-14 In previous versions, usually two indexes were placed on each column—an FP and another index. This is no longer true. Some columns will have up to three indexes: the FP and two other indexes. In previous versions of ASIQ (prior to 12), an HG contained an HNG index; therefore, you did not want to put both an HG and HNG on the same column. This is no longer true. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Fast Indexes ■ LF and HG indexes are considered "fast" indexes ■ Fastest of all the indexes ■ ASIQ internally knows the cardinality ■ Will have the best performance ■ Still important to use HNG and FP where appropriate ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-15 ASIQ Index Types and Datatypes Step 1: By Default, an FP Index Is on Each Column ■ Every column has the default index (FP) • CREATE TABLE command builds this index • Cannot be removed ■ This index is used during a query when no other index is defined ■ This is the fastest structure for producing the original column data ■ Performance is still faster than for most RDBMSs because only one column of data is fetched ■ No additional indexes are need for columns that will be used only for projection or columns with the BIT or large VARCHAR (>255) datatype In most cases, one or two additional indexes are assigned to each column to provide the flexibility to handle different types of queries. CREATE TABLE 5-16 The syntax for CREATE TABLE is defined in Module 6, "Creating Tables and Indexes." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Step 2: Create LF on Low-Cardinality Columns ■ Create an LF index on any column with <1500 distinct values that is used in: • A join (ad-hoc or join index) • SELECT statement with functions min, max, count, or count(distinct) • WHERE clause predicates: equalityinequality, in list, EXISTS • GROUP BY • ORDER BY ■ Examples: state gender department Join index ■ This is a index that physically stores the join information in the database. See Module 13, "Joins." ■ Cardinality can be determined by running GROUP BY queries on raw data loaded into FPs. If you have trouble determining the data cardinality and/or are unable to determine how the data will be used, the initial load can be done with only an FP index and additional indexes can be added later without reloading all the data. ■ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-17 ASIQ Index Types and Datatypes Step 3: Create HG on High-Cardinality Columns ■ Create an HG index on any column with >1500 distinct values that is used in: • A join (ad-hoc or join index) • SELECT statement with functions min, max, count, or count(distinct) • WHERE clause predicates: equalityinequality, in list, EXISTS • GROUP BY • ORDER BY 5-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Step 4: Create HNG, If Needed ■ Create an HNG index on all columns used: • In BETWEEN or RANGE comparisons • With AVG() or SUM() functions ■ Examples ... where sold_date between '1/5/99' and '2/5/99' ... where revenue > 10 0 0 SELECT sum(revenue) from ... ...where customer_name like 'Syb%' ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-19 ASIQ Index Types and Datatypes Case Study ■ How would you index the following table without knowing the query mix? Orders ■ Solutions on next page 5-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Case Study: Solutions ■ How would you index the following table without knowing the query mix? Orders 1. Every column gets the FP index, by default 2. Low-cardinality columns get an LF index that are used in: • Join (ad-hoc or oin index) • SELECT statement with min, max, count, count(distinct) • WHERE clause predicates: equalityinequality, in list, EXISTS • GROUP BY and ORDER BY 3. High-cardinality columns get an HG index in same situations as step 2 (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-21 ASIQ Index Types and Datatypes Case Study: Solutions 4. HNG index on columns that are used in: • BETWEEN or RANGE comparisons • With AVG() or SUM() functions Note: Because we do not know the query mix there are other possible index choices. How would the previous index choices change based on the following query? ■ Sample query select avg(quantity), order_date from Orders where order_date between '1/1/98' and '12/31/98' group by order_date • Frequency = 10,000/yr Solutions 5-22 Sample query: ■ Add an HG index on order_date to support the GROUP BY. HNG index will be useful for range search. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Module Map • ASIQ Index Types ■ ASIQ Datatypes ©2000 Sybase, Inc. 23 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5- ASIQ Index Types and Datatypes ASIQ Data Types (continued . . .) VARCHAR > 255 ■ ■ ■ ■ ■ 5-24 Only an FP index is allowed on any VARCHAR column longer than 255. Only LIKE query predicates and SUBSTR functions are supported. There is no storage advantage for choosing VARCHAR over CHAR. There is no performance advantage for choosing VARCHAR over CHAR. UNSIGNED INT is best for primary keys. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes ASIQ Datatypes (continued . . .) ■ ■ The display of timestamp data outside the range of 1600-02-28 23:59:59 to 7911-01-01 00.00:00 may be incomplete, but the complete datetime value is stored in the database. You can see the complete value by first converting it to a character string. FLOAT or REAL A HNG index in not allowed on any FLOAT or REAL. SMALLDATETIME DATETIME TIMESTAMP These are identical datatypes. DATETIME & SMALLDATETIME are user-defined datatypes with a datatype of TIMESTAMP. These were added for compatibility with ASE. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-25 ASIQ Index Types and Datatypes ASIQ Datatypes You should avoid defaulting precision and scale settings for NUMERIC and DECIMAL datatypes, because these are different values in ASIQ, ASE, and ASA. DOUBLE An HNG index is not allowed on any DOUBLE. MONEY SMALLMONEY MONEY and SMALLMONEY are user-defined datatypes with an underlying datatype of NUMERIC. These were added for compatibility with ASE MONEY = NUMERIC(19,4) SMALLMONEY = NUMERIC(10,4) 5-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes User-Defined Datatypes Aliases for Built-in Datatypes ■ Created using the CREATE DOMAIN statement • Syntax: CREATE DOMAIN domain_name system_datatype • Example 1: CREATE DOMAIN ename CHAR(35) ■ Can also be created using sp_addtype stored procedure • Syntax: sp_addtype type_name, system_datatype • Example 2: sp_addtype "ssn", "varchar(11)" Example 1 CREATE TABLE employee (employee_id int, employee_name ename) drop domain ■ ■ To drop a user-defined type, use DROP DOMAIN. Can be carried out only if the datatype is not in use. sp_droptype ■ ■ To drop a user-defined type, use sp_droptype. Can be carried out only if the datatype is not in use. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5-27 ASIQ Index Types and Datatypes Summary ■ Choosing ASIQ index types • General methodology based on cardinality, joins, and query use ■ ASIQ datatypes • Choose appropriate datatypes when creating ASIQ tables 5-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Index Types and Datatypes Lab 5: ASIQ Indexing and Datatypes Exercise Overview Topics ■ Read the case study information for the megaphone database . ASIQ index types: FP, LF, HNG, HG ■ Cardinality of columns ■ ASIQ datatypes ■ SQL queries Goals This module introduces the various types of ASIQ indexes. After completing this lab, you should be able to: ■ Correctly determine the index types General tasks Using a case study approach: ■ Read the case study information. ■ Examine the queries. ■ Identify and assign LF indexes. ■ Identify and assign HG indexes. ■ Identify and assign HNG indexes. ■ Perform a datatype matching game. Case study ■ ©2000 Sybase, Inc. 29 The megaphone database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 5- 6 Creating Tables and Indexes Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Module Overview This module describes how to create ASIQ tables and indexes using the correct syntax. It also covers how to create ASIQ views using the correct syntax. Objectives After completing this module, you should be able to: ■ Create, drop, and alter tables ■ Create/drop indexes ■ Create views ■ Make scripts ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes Steps for Deploying an ASIQ Installation • Properly plan and size the database • Create ASIQ database(s) • Choose ASIQ datatypes and index types for each column ■ Create ASIQ tables ■ Create ASIQ indexes >-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Module Map ■ Examining Three Types of Tables: • Base Tables • Local Temporary Tables • Global Temporary Tables ■ Creating Views ■ Creating Indexes All of these topics are discussed in this module. Base tables ■ ■ Local temporary tables . Global temporary tables ■ ■ ©2000 Sybase, Inc. Sometimes called main or permanent tables. Stored in the main IQ Store and a permanent part of the database. Must be dropped explicitly. Use the CREATE TABLE statement to create a base table. Use the DECLARE LOCAL TEMPORARY TABLE statement. Created in the IQ Temporary Store. The table will exist until the connection ends or within a compound statement in which it is declared. This table is completely inaccessible to other users. Use the GLOBAL TEMPORARY option of the CREATE TABLE statement. The table will exist in the database until it is explicitly removed. Created in the IQ Temporary Store. Each user will have a separate instance of the data in a global temporary table. Those rows are visible only to the connection that inserts them. They are deleted when the connection ends. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-3 Creating Tables and Indexes Create Tables in the ASIQ Database Clients ASIQ table 6-4 ■ ■ Remember that this is a logical table and does not contain data. The ASIQ table definition is stored in the IQ Catalog Store. The data is stored in indexes in the IQ Store. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE TABLE Simple Example ■ Translate entities and attributes to tables and columns ■ Determine column characteristics • Name • Datatype • Properties ■ Example: CREATE TABLE employee (emp_id int NOT NULL ,lname varchar(30) NULL ,fname varchar(30) NULL ,salary money NULL) ASIQ default The ASIQ default setting is to allow NULLs (ANSI standard). This can be changed using the ALLOW_NULLS_BY_DEFAULT option, and setting it to OFF. Column limit The number of columns allowed in a table is 10,000. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-5 Creating Tables and Indexes Building Upon CREATE TABLE... Column Constraints ■ IQ UNIQUE • Defines the approximate number of unique values for a column ■ UNIQUE • Allows a value to occur once in a column • Creates an HG index, by default ■ PRIMARY KEY • Same as UNIQUE constraint except that a table can have only one PRIMARY KEY constraint ■ Examples of each to follow. ■ ASIQ does not enforce multicolumn (table-level) UNIQUE and PRIMARY KEY constraints. Also, ASIQ does not enforce REFERENCES and CHECK constraints, column or table level. See Fast Track to Adaptive Server Enterprise documentation for more information on these constraints. . 6-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE TABLE - IQ UNIQUE CREATE TABLE customer (customer_id int NOT NULL, cust_type char(4) IQ UNIQUE (30), postalcode char(9) NULL) ■ IQ UNIQUE constraint defines the approximate cardinality of a column • Used for internal FP index storage optimization • <256 values - Single-byte lookup table (FFP index) • >255 and <65536 - Two-byte lookup table (FFFP index) • Default value is 65,537 ■ Use this constraint for optimal storage and performance • You will use less disk space • Optimizer will make better decisions ■ ■ ■ Example ©2000 Sybase, Inc. Sybase recommends that you specify IQ UNIQUE whenever you have a reasonable idea of how many unique values a column will hold. Do not specify IQ UNIQUE if you do not have a good estimate for the number of unique values. The optimizer will use this value (if a LF or HG is not defined on that column) in the query plan. Remember that a LF or HG store column statistics and those will be used first, if available. ■ If you don't set the IQ UNIQUE in CREATE TABLE, you will get the plain FP index. If the IQ UNIQUE indicates a FFP to be built, but the number of values during a load are exceeded, an FFFP will be created if the value is < 65,536. ■ In this example, the cust_type has 30 expected distinct values. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 Creating Tables and Indexes CREATE TABLE-UNIQUE CREATE TABLE stores (store_id char(4) NOT NULL ,store_name varchar(20) NOT NULL ,store_address varchar(40) NOT NULL ,unique (store_id )) ■ UNIQUE constraint creates a unique HG index on the column • No G-Array with the HG index • If a column is low cardinality, consider creating a unique LF index instead of a UNIQUE column constraint ■ Can be used only on a single column ■ The optimizer will use the UNIQUE constraint when developing a query plan, which is the same as creating a unique HG separately Example Can also be defined as follows: CREATE TABLE stores (store_id char (4) UNIQUE ,store_name varchar(20) NOT NULL ,store_address varchar(40) NOT NULL) This is the column-level syntax of the UNIQUE constraint. ■ ■ 6-8 ASIQ does not enforce multicolumn uniqueness. The column cannot contain any NULL values. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE TABLE - PRIMARY KEY CREATE TABLE product (product_code CHAR(5) NOT NULL ,product VARCHAR(25) NOT NULL ,price MONEY NULL ,primary key (product_code)) ■ PRIMARY KEY constraint creates a unique HG index on the column • No G-Array with the HG index • If a column is low cardinality, consider creating a unique LF index instead of a PRIMARY KEY column constraint ■ Can be used only on single-column primary keys ■ The optimizer will use the PRIMARY KEY constraint when developing a query plan, which is the same as creating a unique HG index separately Example Can also be defined as follows: CREATE TABLE product (product_code CHAR(5) PRIMARY KEY ,product VARCHAR(25) NOT NULL ,price MONEY NULL) This is the column-level syntax of the PRIMARY KEY constraint. ■ ■ ■ ©2000 Sybase, Inc. ASIQ does not enforce multicolumn primary keys. The column cannot contain any NULL values. You cannot specify the PRIMARY KEY and UNIQUE constraints for the same column. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 Creating Tables and Indexes Building Upon CREATE TABLE... FOREIGN KEY Constraint ■ Specifies which column(s) in the created table compare with the columns in the referenced list ■ This constraint is not enforced in ASIQ, but will provide improved query optimization ■ Should be used only if there are no orphans • Meaning that there are corresponding values in the referencing table • If there are orphans, ASIQ may provide incorrect query results ■ Can be a column-level constraint or a table-level constraint ■ When defined in tables, permits the use of the KEY JOIN syntax in queries Column-level and table-level constraints ■ ■ KEY JOIN 6-10 Column-level constraint: • If there is only one column in the table to be compared with a column in the referenced table, use a column-level constraint. Table-level constraint: • If there is more than one column in the table to be compared with the columns in the reference table, use a table-level constraint. • A composite key consists of more than one column. This is an optional method for creating table joins. Discussed further in Module 11, "Using ASIQ." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes FOREIGN KEY Constraint Column-Level Example CREATE TABLE titles (title_id tid not null ,title varchar(80) null ,pub_id char(4) null references publishers(pub_id) UNENFORCED ,notes varchar9200) null) ■ You must specify the UNENFORCED keyword • ASIQ does not check this constraint during insertion or deletion to this table; used for query optimization only ■ The publishers table must exist and have a unique index on pub_id (continued . . .) Column-level syntax REFERENCES primary-table-name [(primary-column-name)] Notes ■ The table name following references must already have a PRIMARY KEY or UNIQUE constraint defined on the columns referenced. Dropping constraints ■ To remove a constraint, use ALTER TABLE, which is discussed later in this module. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-11 Creating Tables and Indexes FOREIGN KEY Constraint Table-Level Example CREATE TABLE salesdetail (stor_id char(4) ,ord_num varchar(2 0) ,title_id tid ,qty smallint not not not not null null null null ,FOREIGN KEY(stor_id, ord_num) REFERENCES sales (stor_id, ord_num) UNENFORCED ,FOREIGN KEY(title_id) REFERENCES titles (title_id) UNENFORCED) The above example does not imply a composite index. Remember that a foreign key is not enforced. Therefore, by defining foreign keys in ASIQ is only useful for documentation purposes. 6-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Syntax: CREATE TABLE ■ Create a table in the database CREATE [ GLOBAL TEMPORARY ] TABLE [ owner.]table-name ... ( column-definition [ column-constraint]... [, column-definition [column-constraint] ...]) [, table-constraint ] ...) column-definition column-constraint column-name data-type [ [NOT] NULL] { UNIQUE | PRIMARY KEY | IQ UNIQUE (integer) REFERENCES table-name [(column-name)] UNENFORCED} The REFERENCES (FOREIGN KEY column-level) constraint is not enforced but will be used for query optimization. table-constraint ©2000 Sybase, Inc. Table constraints are not enforced. You can specify UNIQUE, PRIMARY KEY, and CHECK constraints at the table level, but these are not currently useful in an ASIQ implementation. The FOREIGN KEY constraint will be used for query optimization. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-13 Creating Tables and Indexes CREATE TABLE A More Complex Example CREATE TABLE books (isbn char(20) PRIMARY KEY IQ UNIQUE (150000) ,copyright_date date NOT NULL ,title char(100) REFERENCES titles (title) UNENFORCED ,author char (50) REFERENCES authors (author) UNENFORCED ,price money NULL) ■ You can combine IQ UNIQUE with the other column constraints (continued . . .) Custom data types 6-14 You can define customized data types for your database. See the CREATE DOMAIN statement in Module 5, "ASIQ Index Types and Datatypes." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE TABLE Automatic Indexing ■ ASIQ automatically creates an FP index on each column • If you specify an IQ UNIQUE count < = 255, loading will be optimized to an FFP • If you specify an IQ UNIQUE count >= 256 and <= 65,536, loading will be optimized to an FFFP • If you specify an IQ UNIQUE count >= 65,537 (or do not use IQ UNIQUE), you will create a regular FP index ■ An HG index will be created on a column with a PRIMARY KEY or UNIQUE single-column constraint • This will be an HG index without a G-Array • Works well for a single column with high cardinality • Must alter table and drop primary key constraint to drop the HG index Do not confuse UNIQUE & IQ UNIQUE ■ Auto Indexing ■ ©2000 Sybase, Inc. ■ UNIQUE allows a value to occur only once in a column and must be defined as NOT NULL. IQ UNIQUE is an approximation of the number of distinct values. There is a database option called AUTOINDEXING, this may be set to ON or OFF. By default, it is set ON. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 6-15 Creating Tables and Indexes Creating a Table with Sybase Central 1. Connect to the database 2. Click the Tables folder 3. Double-click Add Table 4. Enter a name for the table 5. Enter the columns, datatypes, and other attributes 6. Click Save ■ ■ ■ 6-16 Do not create multicolumn primary keys (very easy to do in Sybase Central). You will seemingly have a multicolumn HG index on those columns, but that index does not actually get created. If you do create a multicolumn primary key, you must alter the table and drop the primary key constraint for the table and then create the individual indexes. A CREATE TABLE example using Sybase Central follows on the next several foils. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Sybase Central: Steps 1-3 1. Connect to the database • Select Tools->Connect->Adaptive Server IQ • Enter user ID, password, port number 2. Click the Tables folder 3. Double-click Add Table ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-17 Creating Tables and Indexes Sybase Central: Steps 4-6 4. Enter a name for the table 5. Enter the column names, datatypes, and other attributes 6. To create the table in the database, click the Save icon in the toolbar 6-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Sybase Central Specifying IQ UNIQUE ■ Click the Advanced Column Properties icon in the toolbar (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-19 Creating Tables and Indexes Sybase Central Specifying a Primary Key ■ Click the Key icon in the Table window ■ The icon changes to a key symbol with a "P" ■ When you click the Save icon in the toolbar, the table is created and the selected column will have an HG index created on it • It will also have an FP index by default ■ To create a primary key on a column, the column must not allow NULLs. Click the nulls box to remove the checkmark, which indicates that NULLs are allowed. ■ Do not create multicolumn primary keys. The HG index will not be physically created and the table will have to be modified and the correct indexes will have to be added separately. 6-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Vileewing a Tab ■ Syntax: sp_iqtable [table-name] ■ Example: sp_iqtable customer ■ Results: ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-21 Creating Tables and Indexes Altering Tables ■ Partial syntax: ALTER TABLE [owner.] table-name { ADD column-definition [column-constraint]...| ADD table-constraint | MODIFY column-name [NOT] NULL | { DELETE | DROP } column-name | { DELETE | DROP } UNIQUE (column-name [, ...])| { DELETE | DROP } PRIMARY KEY | { DELETE | DROP } FOREIGN KEY | RENAME new-table-name | RENAME column-name TO new-column-name } See the ASIQ Reference Manual for full syntax. Sybase Central 6-22 You can alter tables in Sybase Central by displaying the property sheet. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes ALTER TABLE Examples ■ Add a column: ALTER TABLE customer ADD country char(20) ■ Add a column with a FOREIGN KEY constraint: ALTER TABLE customer ADD sales_contact INTEGER REFERENCES employee (emp_id) UNENFORCED ■ Drop a column • You can safely drop a column in an ASIQ table with indexes ALTER TABLE customer DROP cust title ©2000 Sybase, Inc. 23 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes TRUNCATE TABLE ■ Syntax: TRUNCATE TABLE [owner.]table-name ■ Example: TRUNCATE TABLE customer ■ Deletes all rows from the table ■ Column definitions and constraints remain intact, as well as permissions ■ Rows not entered into the transaction log • Very fast; size of data does not affect speed • Not recoverable This command is reserved for a table owner or you must have Database Administrator authority. ASIQ has a transaction log located in the IQ Main Store (not to be confused with the Catalog Store Transaction Log called dbname.log) which is managed entirely by ASIQ. Further discussion on this transaction log is in Module 12, "ASIQ Operation Data Administration ". Notes 6-24 Disk space for the data becomes available for other uses within the database when you truncate a table. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes DROP TABLE ■ Syntax: DROP TABLE tablename ■ Example: DROP TABLE customer ■ Deletes all records and removes table definition from database ■ Very fast; size of data does not affect speed Notes Disk space for the table and its indexes becomes available for other uses within the database when you drop a table. Sybase Central 1. Connect to the database. 2. Click the Tables folder. 3. Right-click the table and select Delete from the popup menu. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-25 Creating Tables and Indexes Module Map ■ Examining Three Types of Tables: • Base Tables • Local Temporary Tables • Global Temporary Tables ■ Creating Views ■ Creating Indexes ■ Base tables ■ Local temporary tables . Sometimes called main or permanent tables. Stored in the main IQ Store and a permanent part of the database. Must be dropped explicitly. Use the CREATE TABLE statement to create a base table. Use the DECLARE LOCAL TEMPORARY TABLE statement. Created in the IQ Temporary Store. The table will exist until the connection ends or within a compound statement in which it is declared. This table is completely inaccessible to other users. Global temporary tables . Use the GLOBAL TEMPORARY option of the CREATE TABLE statement. The table will exist in the database until it is explicitly removed. Created in the temporary IQ Store. ■ Each user will have a separate instance of the data in a global temporary table. Those rows are visible only to the connection that inserts them. They are deleted when the connection ends. 6-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Local Temporary Tables: Method 1 ■ Syntax: DECLARE LOCAL TEMPORARY TABLE table-name ...( column-definition [column-constraint] ... [, column-definition [column-constraint] ...]) ...[ ON COMMIT { DELETE | PRESERVE } ROWS ] ■ Example: DECLARE LOCAL TEMPORARY TABLE MyTable (colA int ,colB char(4)) ■ The table will exist until the connection ends or within a compound statement in which it is declared ■ This table is completely inaccessible to other users Notes ■ You cannot use ALTER TABLE on a local temporary table. ON COMMIT ■ Only allowed for temporary tables. By default, the rows of a temporary table are deleted ON COMMIT. ©2000 Sybase, Inc. 27 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes Local Temporary Tables: Method 2 ■ Syntax: CREATE TABLE #table-name ... or SELECT INTO #table-name ■ Example: CREATE TABLE #MyTable (colA int ,colB char (4)) ■ The table will exist until the connection ends or within a compound statement in which it is declared ■ This table is completely inaccessible to other users ■ 6-28 You will only have FP indexes unless you create additional indexes. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Module Map ■ Examining Three Types of Tables: • Base Tables • Local Temporary Tables • Global Temporary Tables ■ Creating Views ■ Creating Indexes Base tables ■ ■ Local temporary tables . Sometimes called main or permanent tables. Stored in the main IQ Store and a permanent part of the database. Must be dropped explicitly. Use the CREATE TABLE statement to create a base table. Use the DECLEARE LOCAL TEMPORARY TABLE statement. Created in the IQ Temporary Store. The table will exist until the connection ends or within a compound statement in which it is declared. This table is completely inaccessible to other users. Global temporary tables . Use the GLOBAL TEMPORARY option of the CREATE TABLE statement. The table will exist in the database until it is explicitly removed. Created in the temporary IQ Store. ■ Each user will have a separate instance of the data in a global temporary table. Those rows are visible only to the connection that inserts them. They are deleted when the connection ends. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-29 Creating Tables and Indexes Global Temporary Tables ■ Syntax: CREATE GLOBAL TEMPORARY TABLE table-name ...( column-definition [column-constraint] ... [, column-definition [column-constraint] ...]) ...[ ON COMMIT { DELETE | PRESERVE } ROWS ] ■ Example: CREATE GLOBAL TEMPORARY TABLE GlobTable (colA int ,colB char(4)) ■ The table will exist in the database until it is explicitly removed ■ Each user will have a separate instance of the data • Those rows are visible only to the connection that inserts them • Those rows are deleted when the connection ends, by default ON COMMIT ■ Allowed only for temporary tables. By default, the rows of a temporary table are deleted ON COMMIT. ■ The database contains one definition of a global temporary table, just like a base table. Each user has rows that are visible only to that connection. ■ You can drop a global temporary table using the DROP TABLE statement. You cannot drop or alter a global temporary table while other connections are using it. ■ 6-30 To create a global temporary table, double-click the Advanced Table property, and select Global Temporary Table. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Module Map • Examining Three Types of Tables: • Base Tables • Local Temporary Tables • Global Temporary Tables ■ Creating Views ■ Creating Indexes ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-31 Creating Tables and Indexes View: Defined A view: • Is a stored SELECT statement that behaves like a table • Does not store data • May SELECT from more than one table ■ ■ 6-32 A view is the derived results of the SELECT statement. You cannot create indexes on a view. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Creating a View: Benefits ■ Improved security • Limits certain columns and/or rows to certain users • Gives users access to views instead of to tables ■ Improved usability • Simplifies and customizes the user's perception • Provides logical data independence ■ Improved performance for views • Optimizer will select a better query plan when doing multiple table joins • Using views with UNION to partition large (deep) fact tables by some time period A Partitioning improves load/delete performance A Easy method for rolling data in and out ▲ Example coming up ... ©2000 Sybase, Inc. 33 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes View: Using Partitioned Tables A view with UNION • Keeps the table together logically • IQ will perform UNION in parallel (if CPU resources are available) 6-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE VIEW Syntax: CREATE VIEW ... [owner.] view-name [(column-name[,...])] ... AS select-without-order-by ... [WITH CHECK OPTION] Example: CREATE VIEW colorado_customer AS SELECT * FROM customer WHERE state = ‘CO' Quotes ■ ■ ■ WITH CHECK OPTION Rejects any updates or inserts to the view that do not meet the criteria of the SELECT. ASIQ currently ignores this option. Sybase Central 1. 2. 3. 4. 5. ©2000 Sybase, Inc. Single quotes are used to indicate strings. Double quotes are used to indicate table or column names. Two single quotes are used to include a single quote inside a string. Connect to the database. Click the Views folder. Double-click Add View. Enter the table and columns to be used. From the File menu, select Execute Script. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-35 Creating Tables and Indexes Views for Performance Join Views ■ Example: CREATE VIEW emp_dept AS SELECT emp_lname, emp_fname, dept_name FROM Employee, Department WHERE Employee.dept_id = Department.dept_id ■ A performance improvement may be seen by using join views instead of performing joins in SELECT statements • The more tables that are joined, the greater the performance gain DROPVIEW DROP VIEW [owner . ] view-name VIEW restrictions A view cannot include an ORDER BY clause. 6-36 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Steps for Deploying an ASIQ Installation • Properly plan and size the database • Create ASIQ database(s) • Map proper ASIQ datatypes and choose index types for columns • Create ASIQ tables ■ Create ASIQ indexes ©2000 Sybase, Inc. 37 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes Create the ASIQ Indexes ■ Indexes are defined on table columns ■ Contain all the data for a column ■ Each column has an FP index by default and usually one or two additional indexes are defined • See Module 5, "ASIQ Indexes and Datatypes," for information on choosing the correct indexes for a column 6-38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Review: Index Types ■ Four index types: • Fast Projection (FP) • Low Fast (LF) • High Group (HG) • High Non Group (HNG) ■ FP indexes are automatically created on all the columns during the CREATE TABLE operation ■ HG indexes are automatically created on PRIMARY KEY and UNIQUE columns during the CREATE TABLE • HG is created regardless of cardinality ■ All indexes that are automatically created by ASIQ have the prefix ASIQ_... FP The FP index has two subtypes: FFP and FFFP. ©2000 Sybase, Inc. 39 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes Four Options for Creating Indexes ■ CREATE INDEX command ■ Add Index wizard in Sybase Central ■ Create indexes in parallel (after data is loaded) ■ With the UNIQUE or PRIMARY KEY column constraint The first three approaches are discussed in the next several foils. The fourth approach was previously discussed in this module. It is generally more efficient to create all indexes on columns before loading data. 6-40 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE INDEX: Syntax and Example ■ Syntax: CREATE [UNIQUE] [index-type] INDEX index_name ON [owner] .table_name (column_name) [NOTIFY integer] ■ Example: CREATE HG INDEX cust_customer_id_hg ON customer(customer_id) ■ Creates a High Group index on the customer table for the customer_id column ■ Index names must be unique across the database UNIQUE ■ index-type { HG | HNG | LF } index-name ■ Must be unique for a database. Limited to 128 characters. NOTIFY ■ Gives notification messages after a specified number of records are read for the index. The default is 100,000 records. To turn off, set to0. ©2000 Sybase, Inc. Unique indexes can be created only on LF and HG index types. Recommend using this when possible Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-41 Creating Tables and Indexes CREATE INDEX: Sybase Central 1. Double-click Add Index 2. Follow wizard (continued on next few foils) (continued . . .) 6-42 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE INDEX: Sybase Central Add Index Wizard 1. Enter name of new index (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-43 Creating Tables and Indexes CREATE INDEX: Sybase Central Add Index Wizard 2. Choose the kind of index (HG, HNG, or LF) • If you choose Low disk, a High group will be created instead • The Low disk index will be removed in the future 3. Specify a notification record number ■ The notification message is associated with inserts, updates and deletes. This number indicates the number of records read before sending a message. (continued . . .) 6-44 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes CREATE INDEX: Sybase Central Add Index Wizard 4. Choose the column in the table the index is placed on 5. Click Finish ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-45 Creating Tables and Indexes CREATE INDEX: Restrictions ■ There is always an FP index on each column • Cannot drop the FP index ■ Can create additional indexes after the data is inserted • Data does not need to be reinserted ■ Index name must be unique in the database Adding additional indexes after the data is inserted is an option. This method of inserting data from an existing index can be slow. It is usually faster to create all the appropriate indexes before inserting data. It may be done when there is insufficient memory for an insert, update or delete. 6-46 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Creating Indexes in Parallel ■ To create new indexes on data loaded in the database in parallel • When additional indexes are needed • System resources (memory/cpu) dictate creating indexes after loading data ■ Syntax: Begin Parallel IQ Create LF index ... Create HG index... End Parallel IQ (continued . . .) ©2000 Sybase, Inc. 47 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- Creating Tables and Indexes Creating Indexes in Parallel ■ Restrictions • No other DDL statements by any user can be executed while this command is executing • Multiple indexes on the same column will be performed serially • Cannot create indexes on temp tables or global temp tables • Cannot use Sybase Central to create indexes in parallel ■ Performance recommendation • Limit number of indexes to create to the number of CPUs 6-48 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Viewing ASIQ Indexes with sp_iqindex ■ Syntax: sp_iqindex [table-name] ■ Example: sp_iqindex customer ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-49 Creating Tables and Indexes Dropping Indexes ■ Drops the data for that index column ■ Cannot drop the FP index ■ Cannot drop indexes created by the CREATE TABLE command, must alter table ■ Syntax: DROP INDEX [[owner].table-name.] index_name ■ Example: DROP INDEX cust_customer_id_hg Sybase Central 6-50 1. Click the table name, right-clicking to display options. 2. Select the Delete option. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Create and Maintain Scripts Be Able to Automate ASIQ Site Rebuilds ■ Keep all ASIQ DDL commands in a scripts ■ May be useful to create two scripts for each table • Script 1 contains table and index creation • Script 2 contains the load script for the table (coming soon...) ■ Have a CREATE DATABASE script Sybase Central ©2000 Sybase, Inc. Sybase Central does not have the ability to create scripts. Use this tool with caution. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6-51 Creating Tables and Indexes Summary ■ CREATE TABLE • Alter to add or drop columns. • View using sp_iqtable. ■ CREATE INDEX • Usually additional one or two indexes per column besides FP. • Parallel index creation. • View using sp_iqindex. ■ CREATE VIEW ■ Sybase Central • Create/alter/drop tables. • Create/drop indexes. ■ Always create scripts! 6-52 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating Tables and Indexes Lab 6: Creating Tables and Indexes Exercise Overview Topics ■ Use a script to create tables and indexes ■ CREATE TABLE command ■ CREATE INDEX command ■ Viewing tables and indexes using sp_iqtable and sp_iqindex commands Goals This lab introduces the creation of ASIQ tables and indexes. After completing this lab, you should be able to: ■ Create and use a script to create tables and indexes in an ASIQ database General tasks ■ View the creation script ■ Run the script to create the tables and indexes for the megaphone database. ■ View the tables and indexes Lab setup ■ ©2000 Sybase, Inc. 53 This lab assumes that you have created the megaphone ASIQ database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 6- 7 ASIQ Memory Configuration Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Module Overview This module explains how to properly configure ASIQ memory before loading data. Objectives After completing this module, you should be able to: ■ Identify ASIQ memory components ■ Set ASIQ memory buffer caches ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7- ASIQ Memory Configuration Sybase ASIQ Memory Components Overview ■ Server memory ■ Buffers ■ Buffer caches • Main buffer cache (IQ Store cache) • Temporary buffer cache (IQ Temporary Store cache) ■ "Active" users 7-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Server Memory ■ ASIQ allocates memory from a single memory pool ■ Used for: • Reading data from disk to resolve queries • Reading data from disk when loading data from flat files • Managing connections, transactions, buffers, and database objects ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-3 ASIQ Memory Configuration Buffers, Page Size, and Blocks Definitions ■ Buffer - A region in memory that holds uncompressed data read from and written to a database ■ Page size - Amount of data swapped in and out of memory • Determined by create database, IQ page size • Automatically determines the data compression • See Module 4, "Creating Databases" ■ Page = Buffer ■ Block - Minimum unit for performing disk I/O • By default, determined by IQ page size ASIQ Page Size (KB) Resulting Block Size (bytes) 1 512 2 512 4 512 8 512 16 1024 32 2048 64 4096 128 8192 256 16,384 512 32,768 7-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Buffer Caches Two Types ■ Main buffer cache • Area in memory for IQ Main Store buffers • Shared by all users ■ Temporary buffer cache • Area in memory used for IQ Temporary Store buffers • Shared by all users Each of these topics is discussed in detail later in the module. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-5 ASIQ Memory Configuration Why Tune Shared Buffer Caches? Reduce Physical I/O If Used Properly ■ Used for all reads and writes ■ Can reduce the amount of physical I/O if: • Multiple readers are accessing the same database • The buffer cache is large to prevent excess thrashing 7-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Main Buffer Cache ■ Initialized when the database is started ■ Default main buffer cache size is 8MB • Too small for most applications ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-7 ASIQ Memory Configuration Temporary Buffer Cache ■ Initialized when the database is started ■ This cache is used for ad-hoc joins, grouping, and sorting ■ Default cache size is 4MB • Too small for most applications 7-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Viewing Buffer Cache Size Use sp_iqstatus Example: Main IQ buffers = 8MB Temporary IQ buffers = 4MB ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-9 ASIQ Memory Configuration "Active" Users ■ Each ASIQ user has a process ■ This process needs about 5MB per "active" user in a database • An active user is one who is accessing or performing a query in the database • The documentation says 10MB, which is on the high side (an active user can range from 1MB to 10MB) 7-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Strategy for Sizing Caches Steps 1. Understand host platform's memory component requirements 2. Calculate ASIQ cache sizes 3. Set buffer caches ©2000 Sybase, Inc. 11 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7- ASIQ Memory Configuration Memory: The Big Picture Step 1 - Understand the Components ■ Operating system and other applications ■ ASIQ Server ■ ASIQ memory overhead ■ ASIQ main buffer cache ■ ASIQ temp buffer cache Detailed discussion of each component follows 7-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Operating System Memory and Other Applications ■ UNIX operating system (OS) • Varies by platform, but typically 60MB ■ NT • OS needs typically 30MB or more • 4MB + 25% of total memory ■ Other applications on the server • Query tool memory needs, if any • Middleware on the server • OLAP (OnLine Application Program) servers ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-13 ASIQ Memory Configuration ASIQ Server ■ Adaptive Server IQ • Varies by platform • Typically, 30MB 7-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration ASIQ Memory Overhead ■ Active users • 5MB each ■ ASIQ load operations • Default 0MB • Control heap memory allocation during a load ■ File system caches • Additional 30% of total memory to handle file buffering (continued . . .) Active user A user who is accessing or querying the database. Load operations The load_memory_MB option allows a user to specify an upper limit on the amount of heap memory subsequent loads can use. File system Thirty percent (30%) is in addition to the OS overheads for file systems. You can avoid this overhead by using raw partitions for your database. 30% may be a little on the high side. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-15 ASIQ Memory Configuration ASIQ Memory Overhead load_memory_MB option ■ Used to specify an upper bound on the amount of heap memory a load will use ■ Help ensure a load will complete even if it incurs degradation in load performance ■ Default is 0 • Meaning that there is no upper limit ■ Allowed values are 0-500MB ■ Recommend setting this value to non-zero to avoid exhausting virtual memory during a load 7-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Side Note on Setti ng Database Options ■ There are several database options which can be set ■ We will introduce them as needed ■ These options change how DBISQL interacts with the database ■ Syntax: SET [ TEMPORARY ] OPTION . . . [userid. | PUBLIC. ] option-name = [option-value] ■ Example on next slide ... ©2000 Sybase, Inc. Administering Adaptive Server IQforthe Data Warehouse, Version 1.0 7-17 ASIQ Memory Configuration load_memory_MB option ■ Every load uses some amount of heap memory ■ If the user-specified limit is lower than required for the load, the user will receive an error ■ The user must increase or turn off the load_memory_MB option ■ Example: SET OPTION LOAD MEMORY MB = 30 0 Turn off Set the value to 0. load_memory_MB option 7-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration ASIQ Main and Temp Buffer Caches Determined after calculating remaining memory Guideline is to split remaining memory between main and temp buffer caches: • 40% main buffer cache • 60% temp buffer cache 40/60 split This is a general rule of thumb. Some applications may be optimal with a split of 50/50 or 60/40. Start with the 40/60 split guideline and watch performance and adjust buffer caches. Refer to Module 15, "Monitoring and Troubleshooting." Maximum memory cache ■ settings ■ . . . . ■ Solaris 32 bit-3.3GB. Solaris 64 bit - Effectively unlimited. Compaq Tru64 (DEC-UNIX) - Effectively unlimited. SGI IRIX - Effectively unlimited. IBM RS6000 AIX - 2GB *. HP UNIX - 3GB *. NT-2.5GB**. * Must use the server startup option -iqsmem to get this much memory. ** WIN NT Enterprise only, 2GB for other WIN NT platforms. Server startup switches ©2000 Sybase, Inc. See Module 2 and 12, "Admin & Performance Guide," for more information. ■ -iqwmem creates a pool of memory that is locked down and cannot be paged. ■ -iqsmem is used to increase available memory. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-19 ASIQ Memory Configuration Strategy for Sizing Buffer Caches Steps 1. Understand host platform memory components - complete 2. Calculate ASIQ cache sizes 3. Set buffer caches 7-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Calculating Memory for Caches on UNIX Step2 Example: UNIX server with 1GB of RAM Memory Use Memory Available with... Amount Used (MB) 1.Total(nMBi) Raw Partitions UNIX File Systems 1000 1000 2. O/S 60 940 940 3. ASIQ Server 30 910 910 4. UFS overhead (30% of step 3) 273 5. ASQ actve users (10 usersIi) 50 860 587 6. load_memory_MB 100 760 487 7. Avaiable for caches 760 487 8. Main cache MB (40% of step 6) 304 195 9. Temp cache MB (60% of step 6) 465 292 Step 7 637 This is the amount of memory remang for the man and temp buffer caches. Steps 8 and 9 The rule of thumb is to allocate 40% of the remainng memory for the main buffer cache and 60% for the temporary buffer cache. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-21 ASIQ Memory Configuration Calculating Memory for Caches on NT Example: NT server with 512MB of RAM Memory Use Memory Available with... Amount Used (MB) 1.Total(nMBi) Raw Partitions NTFS File Systems 512 512 2. O/S 30 482 482 3. ASIQ Server 30 452 452 4. UFS overhead (30% of step 3) 136 5. ASIQ active users (5 users) 25 427 291 6. load_memory_MB 100 327 191 7. Avaiable for caches 327 191 8. Main cache MB (40% of step 6) 131 76 9. Temp cache MB (60% of step 6) 196 115 Step 7 316 This is the amount of memory remang for the man and temp buffer caches. Steps 8 and 9 The rule of thumb is to allocate 40% of the remainng memory for the main buffer cache and 60% for the temporary buffer cache. 7-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Query Memory Requirements: Read-Only ■ No insert overhead, so caches can be larger ■ Typically, you would set cache values to include insert operations and not adjust for different environments • Allows for mixed load of inserting and querying the database Changing cache values ©2000 Sybase, Inc. 23 In the previous version of IQ, you would adjust cache values for the l°ad environment and for the query environment. Now, that you can have a database being loaded at the same time it is being queried, the cache values cannot be adjusted as easily. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7- ASIQ Memory Configuration Strategy for Sizing Caches Steps 1. Understand host platform memory components - complete 2. Calculate ASIQ buffer cache sizes - complete 3. Set buffer caches 7-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Setting Buffer Caches 1. Ensure database is up 2. Set the buffer caches • Do not exceed physical memory • Syntax and examples to follow 3. Make sure all users are disconnected and stop the server 4. Restart the server • New cache sizes are now in effect (continued . . .) Step 1 Start the ASIQ server Step 3 Unix: stop_asiq NT: Open the ASIQ Server icon and click the shutdown button (or stop the service if started by NT Services) If you overallocate memory for caches such that the server will not start, use the following server starting parameters to override the cache sizes: ■ -iqmc x - Overrides the setting for the main cache at server startup. ■ -iqtcy - Overrides the setting for the temp cache at server startup. where x and y are in megabytes of memory. These options can also be put in the configuration file or on the startup command line. ©2000 Sybase, Inc. 25 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7- ASIQ Memory Configuration Setting Buffer Caches Setting the Buffer Cache for the IQ Store ■ Syntax: SET OPTION "PUBLIC".MAIN_CACHE_MEMORY_MB = #_of_MB Setting the Buffer Cache for the IQ Temporary Store ■ Syntax: SET OPTION "PUBLIC".TEMP_CACHE_MEMORY_MB = #_of_MB (continued . . .) Need to have PUBLIC user ID; otherwise, when the database is restarted, it will revert to the default settings. The PUBLIC user ID is a special user used to record PUBLIC permissions. 7-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Setting Buffer Caches Example: Remember to restart the database. If you set buffer cache sizes higher than your system will accommodate, ASIQ will not open the database. The cache options for start up ---- iqmc and -iqtc—should now be used to reset the defaults and open the database. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-27 ASIQ Memory Configuration Viewing the New Buffer Caches ■ Use sp_iqstatus ■ Main IQ buffers: 40MB ■ Temporary IQ buffers: 60MB 7-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Memory and Swap Usage Putting It All Together... ■ Pages are being swapped out to disk so that other virtual pages can be moved into physical memory ■ Swapping occurs when an entire process is moved to disk ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-29 ASIQ Memory Configuration Memory Performance ■ Keep I/O to a minimum • Use large buffer caches, if available ■ Avoid operating system paging • Don't overallocate physical memory ■ Monitor memory usage during loads and queries • Running out of memory results in excess swapping Monitoring swap 7-30 ■ UNIX vmstat command. ■ UNIX sar command. ■ Windows NT Task Manager. . IQ UTILITIES tool (Will be discussed in Module 14, "Monitoring and Troubleshooting") Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Memory Configuration Summary Memory Configuration ■ Control memory usage by setting IQ page size during create database. ■ Calculate the memory available for cache sizes: • Adust main and temp caches accordingly. • Do not overallocate physical memory. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 7-31 ASIQ Memory Configuration Lab 7: Memory Configuration Exercise Overview Topics How to view buffer cache sizes: sp_iqstatus SET OPTION "PUBLIC".MAIN_CACHE_MEMORY_MB SET OPTION "PUBLIC".TEMP_CACHE_MEMORY_MB Goals General tasks Lab setup 7-32 This lab explains how to configure ASIQ before loading. After completing this lab, you should be able to: ■ Properly resize main and temp buffer cache sizes ■ Restart the server with the new values ■ View the current cache sizes. ■ Reset the main and temporary caches to new values. ■ Restart the server. ■ View the new cache sizes. ■ This lab assumes that the ASIQ Server is up and running with the megaphone database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 8 Loading Data from Files Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Module Overview This module explains how to bulk load tables from flat files. It also introduces the LOAD TABLE, UPDATE and DELETE commands. Objectives After completing this module, you should be able to: ■ Load from flat files with fixed-length and variable-length fields ■ Describe incremental loads and deletes Basic loading techniques ■ This module covers fixed- and variable-length flat-file loads. Advanced loading techniques ■ Module 9, "Advanced Data Loading from Files," covers: • Loading from multiple files. • Partial-width loads. • Adding a dbspace during a load. Insert techniques ■ Module 10, "Loading Data Using the INSERT Command," covers inserting from foreign databases. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-1 Loading Data from Files Data Sources ASIQ Can Insert Data into Tables from. ■ Flat files • Fixed-width • Variable-width ■ ASE tables ■ Other ASIQ tables ■ Other databases on remote servers ■ Direct entry ■ Tools 8-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Table Loading Three Options ■ LOAD TABLE • Loads data from files ■ INSERT FROM SELECT • Loads information from databases on the same server or remote server ■ INSERT VALUES • Inserts data "manually" into specified table columns Table loading ©2000 Sybase, Inc. ■ The LOAD TABLE command is covered in this module. ■ Module 10, "Loading Data Using the INSERT Command," covers: • INSERT FROM SELECT command. • INSERT VALUES command. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-3 Loading Data from Files Data Loading ■ Multiple users can query a database table while one user loads or inserts data into that table ■ Multiple users can update the database concurrently • Must be inserting or deleting from different tables • Performance needs to be considered during large updates ■ Must have permissions to modify the table or have Database Administrator authority 8-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Simple Example: Flat-File Load Example: ■ Assumes the width of input columns matches the width of columns in the employee table ■ No explicit data conversion is needed employee table CREATE TABLE employee (employee_id char (4) ,employee_lname char (30) ,employee_fname char (30) ESCAPES OFF QUOTES OFF Required commands for ASIQ. Futher detail on page 8-23. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-5 Loading Data from Files Adding Rows with LOAD TABLE ■ LOAD TABLE appends the contents of the file to the existing rows ■ It does not replace any existing rows ■ To empty out an existing table and reload it, use TRUNCATE TABLE TRUNCATE TABLE 8-6 For syntax, see Module 6, "Creating Tables and Indexes." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files LOAD TABLE Command: Partial Syntax LOAD TABLE [owner].table-name [ ( load-specification, ...)] FROM 'filename-string', ... [ FORMAT { 'ascii' | 'binary' } ] ... [ DELIMITED BY string ] ... [ STRIP { ON | OFF } ] ... [ QUOTES { ON | OFF } ] ... [ ESCAPES { ON | OFF } ] [ ESCAPE CHARACTER character ] [ WITH CHECKPOINT ON | OFF ] ... [ loadoptions ] load-specification {column-name [ column - spec ] | FILLER (filler-type) } column-spec { ASCII (input-width) | BINARY [ WITH NULL BYTE ] | PREFIX { 1 | 2 | 4 } | 'delimiter-string' | DATE (input-dateformat) | TIME (input-time-format) | DATETIME (input-datetime-format)} [ NULL ( { BLANKS | ZEROS | 'literal',...} ) ] ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-7 Loading Datafrom Files Module Map LOAD TABLE Syntax ■ load-specification / column-spec ■ FROM clause ■ load-options Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files load-specification / column-spec Specify the Column Name with a column-spec or FILLER ■ Syntax: {column-name [column-spec]|FILLER(filler-type)} m FILLER (filler-type) • Indicates specified fields to skip in the source input file ■ Example: LOAD TABLE books (FILLER(50) , .... • Ignore the first 50 characters of the data file when loading the books table column-spec ©2000 Sybase, Inc. See the next foil. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-9 Loading Data from Files column-spec Specifies the Column Format of the Input ■ ASCII(number_bytes_wide) or quoted field delimiter • Specifies width of ASCII input columns ■ DATE, TIME, DATETIME, TIMESTAMP (format) • Converts ASCII datetime input ■ NULLS ({ BLANKS | ZEROS I literal' [, 'literal']...}) • Specifies which input values to convert to NULL on insert These topics are covered in detail on the following pages. 8-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Inserting Using Fixed-Width Field Specification ASCII column-spec with an Integer-Width Value ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-11 Loading Data from Files Inserting Variable-Length Fields Specify ASCII column-spec with a delimiter_string 8-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Example: Loading from a Character File Variable-Length Fields, Pipe Delimited \x0a ©2000 Sybase, Inc. This is the delimiter for a new line, used with DBISQL. If you are using Open Client (ISQL), you use '\n'. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-13 Loading Data from Files Date and Datetime Considerations ■ For date and datetime columns coming from character fields, use the DATE and DATETIME functions ■ DATE column-spec • DATE uses 4 bytes • January 1, 0001 to December 31, 9999 ■ DATETIME column-spec • DATETIME uses 8 bytes of storage • Default format: YYYY-MM-DD hh:mm:sssssss ■ BCP binary date formats are not supported • Use character (-c) formats when BCP-ing out • Use a format file when dealing with dates (continued . . .) Examples follow on the next foil. Notes 8-14 BCP a few test rows and do a test load into ASIQ before doing all rows. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Date and Datetime Considerations ■ Examples of supported input formats Notes ■ ■ ©2000 Sybase, Inc. For AM/PM in the DATETIME format, you can specify either AM or PM as a placeholder and the correct AM/PM will be substituted. For example, you might have another input data row of 0115199409:23:00PM and use the same INSERT_CONVERSION_OPTION as above: DATETIME(MMDDY Y Y Yhh: mm: ss AM). Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-15 Loading Data from Files A Word About NULL ■ Which values to convert to NULL at time of insert ■ Destination column must allow NULLs ■ Example: Syntax NULL ({ BLANKS | ZEROS | 'literal'...}) BLANKS ■ Indicates that blanks convert to NULLs. ZEROS ■ Indicates that binary zeros convert to NULLs. literal ■ Indicates that literals convert to NULLs. You can list up to 20 literal values. 8-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Module Map LOAD TABLE Syntax • load-specification / column-spec ■ FROM clause ■ load-options ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-17 Loading Data from Files FROM Clause ■ Syntax: FROM 'filename-string', . . . ■ The filename-string is passed to the server as a string • Subject to formatting requirements • To indicate directory paths in NT, the backslash character (\) must be represented by two backslashes ■ Example: FROM ' C : \\asiqdata\\input.dat' ■ This example shows the file to be on the same server as the ASIQ server \\ ■ ■ 8-18 ASIQ treats the leading backslash in a string as an escape character when the backslash precedes an "n," an "x," or another backslash. You must double each backslash required by the NT naming convention. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Module Map LOAD TABLE Syntax • load-specification / column-spec • FROM clause ■ load-options ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-19 Loading Data from Files load-options The complete listing of load-options: ■ Detail to follow 8-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files DELIMITED BY Option ■ Used to specify a global column delimiter • Default delimiter is a comma ■ Must be a string consisting of one ASCII character or its hexadecimal representation ■ Example: LOAD TABLE books ( name, ... ) FROM 'C:\\iq\\books.txt' DELIMITED BY '| ' QUOTES OFF ESCAPES OFF; ■ What about a newline in the file? • This option should be used in conunction with the ROW DELIMITED BY option, coming up ... ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-21 Loading Datafrom Files Inserting Variable-Length Fields Special Characters for Delimiting Variable-Length Fields with DBISQL 8-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files QUOTES/ESCAPES Options QUOTES option ■ QUOTES is ON by default ■ If you omit a column-spec definition for an input field, the load looks for a quote character ■ You must set this option to OFF for ASIQ ESCAPES option ■ ESCAPES is ON by default ■ If you omit a column-spec definition for an input field, characters following the backslash character are interpreted as special characters ■ You must set this option to OFF for ASIQ ©2000 Sybase, Inc. 23 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Data from Files WITH CHECKPOINT Option ■ A Checkpoint flushes the caches out to disk before doing a commit ■ Default setting is OFF ■ If set to ON, a checkpoint is issued after successful completion of the load ■ If not specified and recovery is required, the data file used to load the table is needed ■ Recommend setting this option to ON 8-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Putting It Together... File Using Special Character-Delimited Fields ■ od -c is a UNIX octal dump in character mode ©2000 Sybase, Inc. 25 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Data from Files PREVIEW load-option - Input Layout Description ■ Example: ■ Performs the load and also provides extra information ■ Describes the LOAD TABLE command, not the table definition • Indicates what the load is expecting to read In table 'month', the full width insert of 7 columns will begin at record 1. 8-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files ROW DELIMITED BY load_option ■ Specifies the end of each row ■ Example: LOAD TABLE lineitem (l_partkey ASCII(4), l_suppkey ASCII(4), l_quantity ASCII(6), l_orderkey ASCII(6)) FROM 'C:\\iq\\archive\\lineitem.txt' ESCAPES OFF QUOTES OFF ROW DELIMITED BY '\x0a' WITH CHECKPOINT ON; ■ You can either specify '\n' or its ASCII value '\x0a' Row delimiter Can be any string from 1 to 4 characters. Caveats for using ROW Can use this option only if all fields within the row: DELIMITED BY Contain NO binary data. ■ ■ Have column delimiters or are ASCII fixed length. ■ Data defined by DATE or DATETIME column spec. ■ Row delimiter must be distinct from column delimiter. What not to do ©2000 Sybase, Inc. LOAD TABLE license (license_id '|', license_description ' | ' , expiration_date ' | ' , renewal_date ' | ' , amt_due '|', amt_paid '|', price_paid '|', discount '\x0a' ) FROM 'C:\\iq\\archive\\lineitem.txt' ROW DELIMITED BY '\x0a' Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-27 Loading Datafrom Files Loading in Parallel ■ Must have ASIQ Version 12.4.2 or higher ■ Much faster than a regular load ■ Must specify the ROW DELIMITED BY in the LOAD TABLE command ■ AND, data must be delimited and have a newline ■ Example: col 1 | col2 | col3 |\n ■ Note: You must have a delimiter before the newline ■ Can be any delimiter 8-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files More Complex LOAD TABLE Example LOAD TABLE sales (l_orderkey '\x09', l_quantity ASCII(4) NULL("9999"), FILLER(1), l_shipdate DATE('YYYY/MM/DD')) FROM ‘/work/data/sales.t' QUOTES OFF ESCAPES OFF NOTIFY 1000 SKIP 4 ROW DELIMITED BY '\x0a' WITH CHECKPOINT ON; ■ Inserts into sales table from the flat file sales. t m Each row is delimited, load will not be parallel ■ Used with variable- and fixed-width input fields ■ Skips the first four rows ■ Notification message displayed every 1000 rows ■ QUOTES OFF and ESCAPES OFF (required for ASIQ) ■ Load is followed by a checkpoint ©2000 Sybase, Inc. 29 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Datafrom Files Module Map LOAD TABLE Syntax • load-specification / column-spec • FROM clause • load-options 8-30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files What Happens During a load Session? ■ Data inserted into all the indexes for the specified columns ■ If there is an IQ UNIQUE constraint, the appropriate type of FP (FFP or FFFP) will be created ■ Data converted as specified in the LOAD TABLE statement ■ Notification messages displayed to standard output unless redirected ■ Notifications are made in the IQ Message Log Notification messages ■ ■ FP Note ©2000 Sybase, Inc. ■ ASIQ displays notification messages about your database during insert and load. You can adjust the frequency of these messages, or omit them, with the NOTIFY load-option. The statistics in these messages indicate when you need to perform maintenance and optimization tasks, such as adding more dbspaces. If you exceed the number of values for an FFP specified in the IQ UNIQUE clause, a FP (not an FFFP) is created. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8-31 Loading Data from Files Notification Messages ■ This output file is explained in detail in Module 15, "Performance and Tuning." 8-32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Transaction Processing and LOAD TABLE ■ A savepoint occurs automatically before the data is loaded ■ If the load is successful, ASIQ releases the savepoint ■ If the load is unsuccessful, the transaction is rolled back to the savepoint For more information on transaction processing, see Module 12, "ASIQ Operational Data Administration." ©2000 Sybase, Inc. 33 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Data from Files Deleting Data Three Options ■ DELETE statement • Remove rows that meet specified criteria ■ DROP TABLE • Remove entire table, including data ■ TRUNCATE TABLE • Delete all rows from a table ■ ■ 8-34 For DROP TABLE syntax, see Module 6, "Creating Tables and Indexes." For TRUNCATE TABLE syntax, see Module 6, "Creating Tables and Indexes." Remember that TRUNCATE TABLE is not a logged operation and is very fast. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files DELETE Statement ■ Syntax: DELETE [FROM] [owner.]table-name ... [ FROM table-list ] ... [ WHERE search-condition] ■ Examples: DELETE FROM customer WHERE customer_id IN (SELECT customer_id FROM customer WHERE postalcode = '17349') DELETE FROM books WHERE book id = 105 delete ©2000 Sybase, Inc. 35 ■ ■ Deletes all rows that satisfy the search condition from the table. If no WHERE clause is specified, all rows are deleted. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Data from Files DELETE, then LOAD TABLE It may be faster to truncate and load all data • Overhead is involved in performing a delete Can avoid backfilling empty space by using: SET OPTION "PUBLIC."APPEND_LOAD"='ON' This option is OFF by default. 8-36 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files UPDATE Command ■ New with Version 12.4.2 ■ Used to modify rows of a table ■ Table may be a base table or temporary table ■ Base table cannot be part of any join index ■ Syntax: UPDATE table ... SET column-name = expression, ...[ FROM table ] ...[ WHERE search-condition ] . . . [ ORDER BY expression [ASC | DESC ] , ... ] ■ Example: UPDATE employee SET dept_id = 4 0 0 where emp_id = 111; Normally, the order in which rows are updated does not matter. With the NUMBER(*) function, an ordering can be useful to get increasing numbers added to the rows in some specified order. For better performance, do not use the ORDER BY clause in the UPDATE command. ©2000 Sybase, Inc. 37 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- Loading Data from Files Summary ■ Loading tables from flat files: • Can load flat files with fixed-length fields. • Can load flat files with variable-length fields. ■ Use rule of thumb to estimate load times. ■ Understand delete syntax to delete data from tables. ■ Understand incremental loads. 8-38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data from Files Lab 8: Loading Data from Files Exercise Overview Topics ■ LOAD TABLE statement ■ DELETE statement ■ TRUNCATE TABLE statement ■ How to insert from a flat file ■ How to delete rows from a table ■ ASCII command ■ Input field terminators ■ Data from ASCII files Goals This lab explains how to load and delete data from ASIQ. After completing this lab, you should be able to: ■ Load data using a variable-length field ASCII file ■ Delete data from a table ■ Truncate a table General tasks Lab setup ©2000 Sybase, Inc. 39 ■ Load the residential_customer table from a variable-width flat file. ■ Interpret the insert notification messages. ■ Delete data from the residential_customer table. ■ Truncate the residential_customer table. ■ Reload the residential_customer table. ■ This lab assumes that you have created the megaphone database and have created all the tables and indexes in that database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 8- 9 Advanced Data Loading from Files Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Module Overview This module outlines how to build on the LOAD TABLE command. Objectives After completing this module, you should be able to: ■ Load from multiple files ■ Perform partial-width loads ■ Add dbspaces during a load What is covered in this module? Covered in this module: More on the LOAD TABLE command: ■ • Partial-width inserts • Multifile loads Covered in Module 10, "Loading Data Using the INSERT Command" • INSERT FROM SELECT command • INSERT VALUES command ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9- Advanced Data Loading from Files Review: Options for Tabe Loading ■ LOAD TABLE • Loads data from files ■ INSERT FROM SELECT • Loads information from databases on the same server or remote server ■ INSERT VALUES • Inserts data "manually" into specified table columns 9-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Review: LOAD TABLE Command ■ Partial syntax: LOAD TABLE [owner].table-name [ ( load-specification, ...)] FROM 'filename-string', ... [ FORMAT { 'ascii' | 'binary' } ] ... [ DELIMITED BY string ] ... [ STRIP { ON | OFF } ] ... [ QUOTES { ON | OFF } ] ... [ ESCAPES { ON | OFF } ] [ ESCAPE CHARACTER character ] [ WITH CHECKPOINT ON | OFF ] ... [ loadoptions ] load-spec {column-name [ column - spec ] | FILLER (filler-type) column-spec { ASCII (input-width) | BINARY [ WITH NULL BYTE ] | PREFIX { 1 | 2 | 4 } | 'delimiter-string' | DATE (input-date-format) | TIME (input-time-format) | DATETIME (inputdatetime-format)} [ NULL ( { BLANKS | ZEROS | 'literal',...} ) ] ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-3 Advanced Data Loading from Files Advanced Insert Load Options load options and When to Use Them ■ ON FILE ERROR {ROLLBACK|FINISH|CONTINUE} • Only applies when doing multifile loads • Determines what action to take during a load if a file error occurs ■ START ROW ID row_record_id_# • Sets the starting record to the record number specified • Necessary for partial-width inserts (covered later) 9-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Module Map ■ Load from Multiple Files ■ Partial-Width Loads ■ Adding dbspaces During a Load ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-5 Advanced Data Loading from Files Multifile Load Allows large amounts of data to be loaded in a single pass (continued . . .) Notes ■ ■ 9-6 Specify one or more files for loading. There is no limit to the number of files that you can specify. Files processed sequentially from left to right. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Multifile Load ■ Faster than using multiple inserts ■ Used in LOAD TABLE to specify more than one file in the FROM clause • Can insert multiple files in same insert ■ If ASIQ cannot find or access files, the ON FILE ERROR load option provides the following options: • ROLLBACK - Roll back the entire insert transaction • FINISH - Finish inserting the records and commit • CONTINUE - Go to the new file ON FILE ERROR Specify one of these actions for ASIQ in the event one of the files specified suboptions does not exist or there are permission problems accessing the file. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-7 Advanced Data Loading from Files Example: Multifie Load LOAD TABLE lineitem ( l_shipmode ASCII(15), l_quantity ASCII(8), FILLER(1)) FROM '/d1/MILL1/lineitem.f1', '/d1/MILL1/lineitem.f2' ON FILE ERROR FINISH QUOTES OFF ESCAPES OFF ■ Insert into the lineitem table from two files ■ If there is an error accessing the second file, commit the records from the first file 9-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Module Map • Load from Multiple Files ■ Partial-Width Loads ■ Adding dbspaces During a Load ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-9 Advanced Data Loading from Files Partial-Width Loads Why Perform a Partial-Width Load? Use in these cases: ■ Multiple files, vertically partitioned by a column ■ May need to add a column later ■ Memory requirements do not allow a single-width insert Use either LOAD TABLE or INSERT statement ■ With START ROW ID load option Multiple files ■ Use if you have several files that are horizontally partitioned. Memory requirement ■ Use if you do not have enough memory to perform a full-width insert. Memory management is discussed in Module 7, "ASIQ Memory Configuration." ■ INSERT command 9-10 ■ Discussed in Module 10, "Loading Data Using the INSERT Command." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files What Is a ROWID? ■ One up number assigned to each row ■ Column is part of the table but not displayed, by default ■ To display this column, use the ROWID function ■ Example: select ROWID(table_name), * FROM table name ROWID ©2000 Sybase, Inc. 11 ■ The datatype is a BIGINT. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9- Advanced Data Loading from Files Partial-Width Loads colA colB colC colD In this example, there is a table with data in colA and colB. Now you want to load data into colC and colD. (continued . . .) 9-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Partial-Width Loads ■ Use START ROW ID to specify the starting record ID • Example on left without START ROW ID ■ Steps must be performed in the correct order; otherwise, the results may be incorrect Partial-width load ■ Also known as a vertical load or insert. ■ ■ This is an advanced operation. If you do not perform all the steps correctly in a partial-width insert, you may insert data incorrectly. • Never use this type of insert unless you are an experienced ASIQ user and are familiar with your data. Full-width inserts, which insert into all the column indexes on a table at the same time, ensure row-level integrity and are less error prone. ■ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-13 Advanced Data Loading from Files Example: Partial-Width Loads This is the table into which the data is inserted CREATE TABLE ( l_partkey l_suppkey l_quantity l_orderkey smallint smallint int lineitem null, null, null, int null) (continued . . .) 9-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Example: Partial-Width Load - The Right Way First load LOAD TABLE lineitem (l_partkey ASCII(4), l_suppkey ASCII(4), FILLER(13)) FROM ‘/d1/MILL1/tt.t' QUOTES OFF ESCAPES OFF WITH CHECKPOINT ON; ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-15 Advanced Data Loading from Files Example: Partial-Width Load - The Right Way Second load LOAD TABLE lineitem (FILLER(8), l_quantity ASCII(6), l_orderkey ASCII(6), FILLER(1)) FROM ‘/d1/MILL1/tt.t' START ROW ID 1 QUOTES OFF ESCAPES OFF WITH CHECKPOINT ON; ■ Note the START ROW ID option used in the second LOAD TABLE statement 9-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Example: Partial-Width Load - The Wrong Way Incorrect partial-width load LOAD TABLE lineitem (l_partkey ASCII(4), l_suppkey ASCII(4), FILLER(13)) FROM ‘/d1/MILL1/tt.t' QUOTES OFF ESCAPES OFF WITH CHECKPOINT ON; LOAD TABLE lineitem (FILLER(8), l_quantity ASCII(6), l_orderkey ASCII(6), FILLER(1)) FROM ‘/d1/MILL1/tt.t' QUOTES OFF ESCAPES OFF WITH CHECKPOINT ON; ■ The result of the SELECT statement below shows that 10 rows are stored instead of the correct number of 5 (10 rows affected) (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-17 Advanced Data Loading from Files Example: Partial-Width Load - The Wrong Way ■ Here is the data from the previous example: What went wrong? How do you fix it? What went wrong? ■ START ROW ID was omitted in the second LOAD TABLE. How do you fix it? ■ Delete records where rowid > 5 and issue second LOAD TABLE statement with START ROW ID 1. Truncate the table and issue both LOAD TABLE statements again. This time, do not forget the START ROW ID 1. ■ 9-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Module Map • Load from Multiple Files • Partial-Width Loads ■ Adding dbspaces During a Load ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-19 Advanced Data Loading from Files Adding dbspaces During a Load What to Do If You Run Out of Space in the Database ■ ASIQ stops loading and suspends the load ■ Add a dbspace; ASIQ will continue the load ■ Example: CREATE DBSPACE mydb_2 AS 'e:\\s2\data\mydb_2.iq' SIZE 200 ■ In Windows NT, create a dbspace called mydb_2 to add 200MB to the IQ Store of the current IQ database (mydb) CREATE DBSPACE dbspace-name AS filename ... [ {IQ STORE | IQ TEMPORARY STORE | CATALOG STORE }] ...[ [SIZE] file-size ] Default Create a dbspace for the IQ Store. You can monitor the amount of space used in a database by looking at the load notification message. 9-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Good Loading Practices ■ Use LOAD TABLE to load from files • Try to load in parallel by using the Row Delimited By option with a delimited flat file, when possible • Load from as many files as possible as one transaction ■ Place data files on separate physical drive from database files ■ Create all IQ indexes before loading data ■ Set up memory for loading • Use Load_Memory_MB option to limit heap size • Provide adequate temp cache for HG indexes ■ Schedule loading during low-usage intervals • Consider the IQ Multiplex option for high-volume systems ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9-21 Advanced Data Loading from Files Summary ■ LOAD TABLE • Fastest method ■ Load flat files together, if available, using multifile load method. • Specify ON FILE ERROR load option. ■ Do partial-width loads with caution. • Specify START ROW ID load option. ■ Remember to check the notification messages during a load. • You may need to add dbspace if the database runs out of room. 9-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Advanced Data Loading from Files Lab 9: Advanced Loading Data from File Exercise Overview Topics ■ Load data from multiple files ■ Add a dbspace during a load ■ IQ message file and load notification messages ■ sp_iqstatus Goals This lab develops your data-loading skills. After completing this lab, you should be able to: ■ Load from multiple files into an ASIQ database ■ Add a dbspace during a load General tasks Lab setup ©2000 Sybase, Inc. 23 ■ Load the telco_facts table from multiple flat files. ■ Monitor the IQ MSG file during a load. ■ Add a IQ Store dbspace when the database becomes full during a load. ■ Load the month, status, and service tables from a script. ■ Check the space utilization of the database using sp_iqstatus. ■ This lab assumes that you have created the megaphone database and have created tables and indexes in that database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 9- 10 Loading Data Using the INSERT Command Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Module Overview This module explores more loading techniques using the INSERT statement. Objectives After completing this module, you should be able to: ■ Perform partial-width inserts ■ Insert from the IQ Main Store ■ Insert directly from a foreign database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-1 Loading Data Using the INSERT Command Table Loading Three Options ■ LOAD TABLE • Loads data from files ■ INSERT from SELECT • Loads information from databases on the same server or remote server ■ INSERT VALUES • Inserts data "manually" into specified table columns LOAD TABLE 1 0 -2 For more information, see Module 8, " Loading Data from Files," and Module 9, "Advanced Data Loading from Files." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command INSERT from SELECT Command ■ Syntax: INSERT [INTO ] [owner.]table-name [ (column-name [, ...]) ] [ LOCATION servername.dbname] ... insert-load-options ... SELECT statement ■ Allows mass insertion into a table with the results of a SELECT statement ■ Example: INSERT INTO dept_head (name, dept) NOTIFY 20 SELECT emp_fname ||'' || emp_lname AS name, dept_name FROM employee JOIN department ON emp_id = dept_head_id [ LIMIT number-of-rows ] [ NOTIFY number-of-rows ] [ SKIP number-of-rows ] [ START ROW ID number ] ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-3 Loading Data Using the INSERT Command INSERT VALUES Command ■ Syntax: INSERT [INTO ] [owner.]table-name [ (column-name [, ...]) ] ... VALUES ( expression ... ) ■ Inserts a single row with specified values ■ Example: INSERT INTO department (dept_id, dept_name) VALUES (230, 'Eastern Sales') 10-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Module Map ■ Partial-Width Inserts ■ Inserting from the IQ Store ■ Inserting Directly from a Foreign Database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 0 -5 Loading Data Using the INSERT Command Partial-Width Inserts ■ Use either LOAD TABLE or INSERT statement (previous module includes an example of a partial-width insert using the LOAD TABLE statement) • Use START ROW ID insert option • Do not use INSERT VALUES command INSERT VALUES 1 0 -6 Do not use this command for partial-width inserts because you cannot specify START ROW ID using this format. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Example: Partial-Width Insert - The Right Way The first INSERT INSERT INTO lineitem (l_partkey, l_suppkey) LOCATION ase_srv1.part_db ( SELECT l_partkey, l_suppkey FROM lineitem ORDER BY l_partkey) i Must have Open Client Libraries installed (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-7 Loading Data Using the INSERT Command Example: Partial-Width Insert - The Right Way The second INSERT INSERT INTO lineitem (l_quantity, l_orderkey) START ROW ID 1 LOCATION ase_srv1.part_db ( SELECT l_quantity, l_orderkey FROM lineitem ORDER BY l_partkey) ■ Note the START ROW ID option used in the second INSERT statement (5 rows affected) 1 0 -8 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Loading Data Using the INSERT Command Module Map • Partial-Width Inserts ■ Inserting from the IQ Main Store ■ Inserting Directly from a Foreign Database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-9 Loading Data Using the INSERT Command Inserting from the IQ Main Store (continued . . .) Catalog Store 10-10 You can also insert data from the IQ Catalog Store. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Inserting from the IQ Main Store ■ Syntax: INSERT INTO asiq_table1 (col1, col2, col3,...) [ insert-load-options ] ... select col1, col2, col3 ... from asiq_table2 [WHERE ... ] ■ The table you are inserting into must exist in the database you are connected to (continued . . .) For maximum efficiency, insert as many rows as possible in one INSERT statement. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-11 Loading Data Using the INSERT Command Inserting from the IQ Main Store ■ Example: INSERT INTO lineitem (l_partkey, l_suppkey, SELECT l_quantity, l_order_key) l_partkey, l_suppkey, l_quantity, l_order_key FROM partitem ■ This example inserts data into the lineitem table from the already-populated partitem table 10-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Module Map • Partial-Width Inserts • Inserting from the IQ Main Store ■ Inserting Directly from a Foreign Database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-13 Loading Data Using the INSERT Command Inserting Directly from a Foreign Database The Database Can Be on a Different Computer Client (continued . . .) 10-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Inserting Directly from a Foreign Database 1. Connect to both databases using the same user ID and password for Unix • Must have an entry for the foreign server in the directory services file 2. On the ASIQ database, issue a statement using this syntax: INSERT INTO asiq_table (col1, col2, col3, ...) LOCATION 'servername.dbname' ( SELECT col1, col2, col3, . . . FROM table ) 3. Issue an explicit COMMIT to commit the insert COMMIT; (continued . . .) Step 1 In NT we have CIS, which permits you to set up an external login to remote servers, so you do not need to have the same user IDs and passwords on both servers. LOCATION ■ ■ ©2000 Sybase, Inc. ASE - The Adaptive Server Enterprise server you are connecting to must exist in the interfaces file on the local machine. Other - Need to have OMNI and DirectConnect access a foreign server, prior to ASIQ 12.0.4. Use CIS with IQ 12.0.4 and above. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-15 Loading Data Using the INSERT Command Inserting Directly from a Foreign Database ■ Example: INSERT INTO customer (customer_id, cust_type, ...) LOCATION prod.asedb SELECT customer_id, cust_type, ... FROM customer; COMMIT; ■ Insert data from the customer_id, cust_type, ... columns in the customer table in the database asedb.dba on the server prod, into the corresponding columns of the customer table in the current database This example inserts the same columns as the above example, but only for the rows in which the value of customer_id = 1234: INSERT INTO customer (customer_id, cust_type, ...) LOCATION prod.asedb SELECT customer_id, cust_type, ... FROM customer WHERE customer_id = 1234; COMMIT; Sybase connectivity libraries must be installed on your system, and the load library path environment variable must point to them. ■ 64-bit platforms must use Open Client 12 libraries to load from foreign data servers. ■ 32-bit platforms may use either Open Client 11 or Open Client 12. 10-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Using CIS ■ CIS (Component Integration Services) allows you to access data on a foreign server ■ You construct local tables in a proxy database • The data resides on the foreign server ■ Currently, CIS is only available on NT for ASIQ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 10-17 Loading Data Using the INSERT Command Summary ■ When using the INSERT statement, run DBISQL or the client application on the same machine. ■ There are two ways to use the INSERT command: INSERT from SELECT and INSERT VALUES. ■ Use INSERT from SELECT command when: • Loading data from the IQ Store or IQ Catalog Store. • Loading directly from ASE or other foreign data source using the LOCATION clause. ■ Use INSERT VALUES command when: • Loading data row-by-row. 10-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Loading Data Using the INSERT Command Lab 10: Loading Data Using the insert Command Exercise Overview Topics ■ ■ CREATE TABLE and CREATE INDEX commands (review from Module 6, "Creating Tables and Indexes") INSERT command Goals This lab introduces how to load data using the INSERT command. After completing this lab, you should be able to: ■ Use a script or Sybase Central to create a new table with indexes ■ Write a script to load the new table using the INSERT command from an existing table in the database General tasks ■ ■ ■ Lab setup ©2000 Sybase, Inc. ■ Use a script or Sybase Central to create a table called co_residential_customer. Use a script or Sybase Central to create indexes on the co_residential_customer table. Create a script to load co_residential_customer from the existing residential_customer table. This lab assumes that you have created the megaphone ASIQ database. You must have also created the residential_customer table and populated it with data. Administering Adaptive Server IQforthe Data Warehouse, Version 1.0 10-19 11 Using ASIQ Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Using ASIQ Module Overview This module introduces additional SQL statements and functions specific to ASIQ. Objectives After completing this module, you should be able to: ■ Write SQL statements using ASIQ-specific features ■ Use ASIQ functions in SQL statements ■ Export data to files ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-1 Using ASIQ ASIQ SQL ■ Conforms to the ANSI SQL 89 standard ■ Has many additional features not commonly found in other SQL implementations ■ Supports several functions not in the ANSI SQL standard Details on these topics to follow. 1 1 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Review: SELECT Statement Syntax: Examples to follow. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 1 -3 Using ASIQ SELECT Statement ■ Example 1: select type, avg(price) from titles group by type ■ Example 2: select stores.stor_id, qty, title_id, stor_name from salesdetail, stores where salesdetail.stor_id = stores.stor_id ■ Example 3: select distinct au_lname, state from authors where state = 'UT' (continued . . .) ■ ■ 1 1 -4 Refer to Introduction to SQL class materials. The above examples use the pubs2 database. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ SELECT Statement ■ Example 4 (SELECT INTO): declare @var1 char(40) select fname into @var1 from customer where cust_no = 1234 ■ Results: into@var1 Haskell ■ Example 5 (ROLLUP) select year, model, color, sum(sales) from sales_tab group by ROLLUP year, model, color ROLLUP ©2000 Sybase, Inc. The ROLLUP operator in the GROUP BY clause allows you to analyze subtotals using different levels of detail. It creates subtotals that roll up from a very detailed level to a Grand Total. Rollup requires an ordered list of grouping expressions with all the aggregate functions available to the GROUP BY clause (sum, avg, min, max). DISTINCT cannot be used with ROLLUP and ROLLUP cannot be used in a SELECT statement that is nested with INSERT, UPDATE, or DELETE. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 1 -5 Using ASIQ Module Map ■ ASIQ-Specific SQL Features • Joins • Expressions in Subqueries • IF Condition • CASE statement ■ ASIQ-Specific SQL Functions • Numeric • String • Date and Time • Datatype Conversion ■ Exporting Data Each of these topics is covered in the next few slides. 1 1 -6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ ASIQ-Specific SQL Features Joins ■ Syntax: ■ The FROM table list creates a result set consisting of all the columns from the tables specified Examples in the next few slides. JOIN Joins compare two or more tables by specifying a column from each and comparing the values in those columns row by row. INNER JOIN All matching rows in the component tables are in the result set. OUTER JOIN Joins that include all rows regardless of whether there is a matching row. LEFT OUTER JOIN Selects all rows from the first table that meet the statement's restrictions. The second table generates values if there is a match on the join condition. Otherwise, the second table generates NULL values. RIGHT OUTER JOIN Selects all rows from the second table that meet the statement's restrictions. The first table generates values if there is a match on the join condition. Otherwise, the first table generates NULL values. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-7 Using ASIQ KEY JOIN ■ Allows key joins between tables based on foreign key relationships ■ Example: SELECT emp_lname, id, order_date FROM sales_order KEY JOIN employee is the same as: SELECT emp_lname, id, order_date FROM sales_order, employee WHERE sales_order.sales_rep = employee.emp_id ■ Must have FK and PK defined to use KEY JOIN Case sensitivity 11-8 By default, ASIQ databases are case-insensitive in their string comparison. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ NATURAL JOIN ■ Allows natural j oins between tables based on common column names ■ Example: SELECT emp_lname, dept_name FROM employee NATURAL JOIN department is the same as: SELECT emp_lname, dept_name FROM employee, department WHERE employee.dept_id = department.dept_id You should be careful when using join operators. The server will join columns that are not intended to match, thus producing unwanted results. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11 - 9 Using ASIQ OUTER JOIN ■ Example: SELECT emp_lname, dept_name FROM employee LEFT OUTER JOIN department is the same as: SELECT emp_lname, dept_name FROM employee, department WHERE employee.dept_id *= department.dept_id List all the employees and find the department names, if one exists. 11-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Expressions in Subqueries ■ ASIQ allows expressions in subqueries • Example: SELECT customer_lname, customer_fname, ( SELECT product_name FROM product WHERE customer_id = customer.customer_id AND product_id = 123) FROM customer ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-11 Using ASIQ IF Condition Syntax: IF condition . . . THEN expression ... [ ELSE expression END IF Example: IF SomeValue > AnotherValue THEN SET AnotherValue = SomeValue; END IF; Transt-SQL IF statement BEGIN...END 11-12 Here is an alternative to the ASIQ IF statement. IF expression . . . statement ... [ELSE [ IF expression ] statement ] ... Use BEGIN and END to group a set of statements into a single compound statement. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ CASE Statement ■ Syntax: CASE expression WHEN expression THEN expression [, [ELSE expression ] END ■ Example: SELECT (CASE gender WHEN 'F' THEN 'Ms. ' WHEN 'M' THEN 'Mr. ' END) as Title, lname FROM customer Results Titl Mr. e Mr. Ms. Ms. Mr. Alternative syntax CASE WHEN search-condition THEN expression [,...] [ELSE expression ] END lname Mollan Sharpe d Cates Jones Miller Example: SELECT (CASE WHEN gender='F' THEN 'Female Customer' WHEN customer_id > 10000 THEN 'New Customer' ELSE 'Regular Customer' END) as Type FROM customer ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-13 Using ASIQ A Word on Quotes ■ Single quotes are used to indicate strings ■ Double quotes are used to indicate table or column names ■ Two single quotes are used to include a single quote inside a string 11-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Module Map • ASIQ-Specific SQL Features • Joins • Expressions in Subqueries • IF Condition • CASE statement ■ ASIQ-Specific SQL Functions • Numeric • String • Date and Time • Data Type Conversion ■ Exporting Data ■ ■ ©2000 Sybase, Inc. Details on these topics to follow. The list of functions on the next several pages are not supported by ASE. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-15 Using ASIQ ASIQ Numeric Functions "TRUNCATE" The double quotes are required because of the key word conflict with the TRUNCATE TABLE statement. ASE ASE has equivalent numeric functions. 11-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ ASIQ String Functions ASE ©2000 Sybase, Inc. ASE has equivalent string functions except for the ASIQ LOCATE function. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-17 Using ASIQ ASIQ Date and Tme Functions ■ ASIQ provides two classes of date and time functions • One set is T-SQL compatible; use DATEADD, DATEDIFF, DATEPART, and GETDATE • The other is the ANSI standard set of functions ■ Example: SELECT days( date('1997-09-17'),2) ■ Result: 1997-09-19 00:00:00.000 See the ASIQ Reference Manual for a complete list of date and time functions. Examples SELECT today(*) = 1999-03-14 SELECT YMD(1992,15,1) = 'Mar 1 1993' SELECT DATEFORMAT('1999-01-01','Mmm Dd, yyyy' = Jan 1, 199 9 SELECT DATENAME(month, '1999/3/17') = March SELECT DAYNAME('1987/05/02') = Saturday 11-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ ■ ^ ASIQ Datatype Conversion Functions ■ CAST • Returns the value of the expression converted to the specified datatype ■ Syntax: CAST(expression AS data type) ■ Example: CAST('This is a long string' AS CHAR(10)) ■ Result: This is a Conversion functions The following functions are supported by both ASIQ and ASE: CONVERT (datatype, expression [ ,format-style] ) HEXTOINT (hexadecimal-string) INTTOHEX (integerexp) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-19 Using ASIQ Additional ASIQ Functions 1 1 -2 0 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Module Map • ASIQ-Specific SQL Features • Joins • Expressions in Subqueries • IF Condition • CASE statement • ASIQ-Specific SQL Functions • Numeric • String • Date and Time • Data Type Conversion ■ Exporting Data ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11-21 Using ASIQ Exporting Data from ASIQ ■ Two methods available • Redirection using ># with a SELECT statement A File size limit is 2GB • Data extraction using SET options with a SELECT statement A Fastest method of exporting data A File size limit is 128GB ■ Data extraction using SET options is new in v12.4.2 • Replaces OUTPUT TO command in earlier versions A OUTPUT TO is not supported in v 12.4.2 • Syntax to support this feature is temporary • Information on this feature is only contained in the Release Bulletin 11 - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Exporting Data Using Redirection ■ Use the redirect symbol ># anywhere in a command to direct output to a file ■ Example: select * from employee ># empfile.txt ■ The file is placed relative to the directory where the session (DBISQL or ISQL) was started ■ Option to control the output on next page ... (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11 - 23 Using ASIQ Exporting Data Using Redirection Options Controlling the Output ■ Set Option OUTPUT_FORMAT = 'text' • Many file options available - See ASIQ Reference Manual, Chapter 5 ■ Set Option OUTPUT_LENGTH = 0 • Controls the length of fields • 0 is the default - No truncation of data ■ Set Option NULLS • Controls how NULLS are displayed/written ■ To redirect output and error messages, use >& filename ■ Append output to an existing file using >># or >>& • Will overwrite an existing file by default ■ For more information on exporting data using redirection, see the ASIQ Administration and Performance Guide. ■ Also, the ASIQ Reference Manual, Chapter 5, has the DBISQL options. 11 - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Exporting Data Using Data Extraction ■ New feature to support high-volume data extraction • Currently limited to disk file or a named pipe ■ SET OPTION commands control extract behavior • Name of the file • Format: ASCII, binary, or binary swap • Column and row delimiters • NULL as zero (default is NULL) ■ Option names all begin with "Temp_" • To reflect this is a temporary implementation • Option names will be changed in a future release • Functionality will continue to be supported ■ String fields will be unquoted • No support currently for change to quoted strings ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11 - 25 Using ASIQ Options for Data Extraction ■ Set the name of the file • Set Option Temp_Extract_Name = '[path/]filename' A Path and file are relative to the ASIQ Server A Default is ’’, meaning standard output • Existing file will be overwritten ■ File Format - ASCII by default • Set Option Temp_Extract_Binary = 'ON' for binary • Set Option Temp_Extract_Swap = 'ON' for Binary Swap ■ Column Delimiter - Comma by default • Set Option Temp_Extract_Column_Delimiter = '|' (for example) (continued . . .) 11 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Options for Data Extraction ■ Row delimiter - newline by default • Set Option Temp_Extract_Row_Delimiter = return (for example) • Use appropriate delimiter from your database collation order ■ Nulls as Zero Option - OFF by default • Set Option Temp_Extract_Null_As_Zero = 'On' • If set will represent values as follows: ▲Arithmetic - 0 ▲" for character data A 1900-01-01 for dates A 00:00:00.000000 for times A 1900-01-01 00:00:00.000000 for timestamp datetime If you set an extract option, execute a SELECT statement, and then execute a second SELECT statement without changing the extract filename, the second file will overwrite the first file. There is no method for appending data to a file. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11 - 27 Using ASIQ Data Extraction ■ Example: Set Temporary Option Temp_Extract_Name = 'C:\data\customer.dat'; Select * from residential_customer; Set Temporary Option Temp_Extract_Name = ''; 11 - 28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Using ASIQ Summary ■ Review of SELECT statement ■ You can use ASIQ-specific SQL: • Key joins, natural joins, expressions in subqueries • IF condition, CASE statement ■ There are ASIQ-specific SQL functions for: • Numeric, string, date/time, datatype conversion ■ There are two methods to export data from ASIQ: • Redirection using ># • Data extraction using SET options ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 11 - 29 Using ASIQ Lab 11: Using ASIQ Exercise Overview Topics ■ ■ Goals This lab develops your SQL skills. After completing this lab, you should be able to: ■ Write SQL statements using ASIQ-specific features ■ Use ASIQ functions against the IQ database ■ Export data to a file General tasks Lab setup ■ Write a query using the NATURAL JOIN syntax. ■ Write a query using the CASE statement. ■ Write a query using the rowid function. ■ Export data using a SELECT statement ■ ■ 11 - 30 Writing SQL statements ASIQ functions This lab assumes that you have created the megaphone database and have created tables and indexes in that database. The data should be loaded in all the tables from Labs 8, 9, and 10. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 12 ASIQ Operational Data Administration Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Module Overview This module provides data administration topics relevant to the ASIQ environment. Objectives After completing this module, you should be able to: ■ Define a transaction and how it is implemented in ASIQ ■ Describe ASIQ's locking ■ Describe how data security is implemented through the use of users, groups, and permissions ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-1 ASIQ Operational Data Administration Module Map ■ Transactions ■ Locking ■ Snapshot Versioning ■ Security 1 2 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Transaction Processing ■ Allows many users to read from the database while it is being updated ■ ASIQ's approach to transaction processing is called snapshot versioning • Allows multiple readers and one writer per table • Further details to follow ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 2 -3 ASIQ Operational Data Administration What Is a Transaction? ■ A logical unit of work ■ Takes the database from one consistent state to another ■ All-or-nothing processing • Everything is committed or • Everything is rolled back ■ Most IQ transactions begin and end automatically based on the command issued • You can also issue explicit commands to begin or end a transaction See Chapter 8, "Transactions and Versioning," in Adaptive Server IQ Administration and Performance Guide for more information. 1 2 -4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Transaction Commands ■ BEGIN TRANSACTION • To begin a user-defined transaction BEGIN TRAN[SACTION] [transaction-name] ■ COMMIT • To make any changes to the database permanent COMMIT [work] ■ ROLLBACK • To undo any changes made since the last COMMIT or ROLLBACK ROLLBACK [work] (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 2 -5 ASIQ Operational Data Administration Transaction Commands ■ SAVEPOINT • To establish a savepoint within a current transaction SAVEPOINT [savepoint-name] ■ ROLLBACK TO SAVEPOINT • To cancel any changes made since a savepoint ROLLBACK TO SAVEPOINT [savepoint-name] ■ CHECKPOINT • Writes changes to disk CHECKPOINT (continued . . .) 12-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Transaction Commands WITH CHECKPOINT ON ■ During a delete or a load to a table, you should specify WITH CHECKPOINT ON ■ A checkpoint is issued after successful completion of the statement ■ Example: LOAD TABLE mytable (...) FROM ... QUOTES OFF ESCAPES OFF WITH CHECKPOINT ON; See Module 8, "Loading Data from Files," for more information on the LOAD TABLE command and WITH CHECKPOINT load-option. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 2 -7 ASIQ Operational Data Administration Chained and Unchained Transaction Modes ■ Chained • Do not need to specify an explicit BEGIN to start a transaction • Commits a transaction only when an explicit COMMIT is executed or the statement executed has an autocommit • This is the default mode for ASIQ • ANSI standard ■ Unchained • Must specify an explicit BEGIN to start a transaction • This is the default mode for ASE ■ Can control the mode by setting the CHAINED database option autocommit 12-8 The following is a partial list of commands that have an automatic commit. . ALTER . CREATE . DROP . GRANT . REVOKE Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Completing Transactions ■ DML is not automatically committed • DML operations include inserts, updates, deletes, and queries ■ During inserts, updates or deletes • If it does not finish, the entire transaction is rolled back • If the connection is lost, the transaction is rolled back • Need to specify a COMMIT ■ DDL is automatically committed • DDL operations include CREATE, DROP, and ALTER ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 2 -9 ASIQ Operational Data Administration Module Map • Transactions ■ Locking ■ Snapshot Versioning ■ Security 12-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Locking ■ Locking is handled automatically by ASIQ ■ When a table is locked, no other transaction can have write access • Can only have read access ■ Locking is based on: • Data manipulation language (DML) operations • Data definition language (DDL) operations ■ DDL with IQ Multiplex • You cannot perform DDL in Multiplex mode • You can make changes only in Simplex mode ■ ■ ©2000 Sybase, Inc. DML and DDL commands to follow. See Chapter 8 of Adaptive Server IQ Administration and Performance Guide for more information on locking. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-11 ASIQ Operational Data Administration Locks for DML ■ ASIQ permits one writer and multiple readers per table ■ Rules: • Reads do not block writes • Writes do not block reads ■ The first transaction to open a table in write mode gains access • The second write transaction receives an error DML examples 12-12 ■ INSERT... . . . . LOAD TABLE. DELETE... SELECT... UPDATE... Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Locks for DDL ■ DDL operations include CREATE, DROP, and ALTER • Lock out all other readers and writers ■ Rules: • Cannot start any DDL while any other transaction is using the table • Cannot start any DDL while other DDL commands are running • DDL commands are automatically committed DDL examples ■ . . . . . ©2000 Sybase, Inc. CREATE TABLE... ALTER TABLE... DROP TABLE... CREATE DB SPACE. DROPDBSPACE... CREATE INDEX... Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-13 ASIQ Operational Data Administration ANSI Isolation Levels ■ ANSI defines four isolation levels • Level 0 allows dirty reads • Level 1 prevents dirty reads • Level 2 ensures repeatable reads • Level 3 prevents phantom reads ■ When updating a table, ASIQ uses level 3 ■ ASIQ protects you from data inconsistencies by: • Ensuring only one user can modify a table at a given time • Keeping changes invisible to other users until the changes are complete • Maintaining time-stamped snapshots of data objects in use at any time Phantom reads 12-14 When a transaction begins, updates made by other transactions are invisible to it. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Module Map • Transactions • Locking ■ Snapshot Versioning ■ Security ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-15 ASIQ Operational Data Administration Snapshot Versioning ■ ASIQ's method of managing changes to a table ■ Also referred to as table-level versioning ■ Allows updates to tables while users are reading the data ■ Supports ANSI isolation level 3 12-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Versioning Level ■ To the user, the unit of versioning is at the table level • Internally, data is versioned at the page level ■ ASIQ assigns a version ID to each table • Each active version of a table resides on disk • The version ID is not accessible to users ■ For performance reasons, you may not want to write to multiple tables at the same time ■ ■ ©2000 Sybase, Inc. A complete copy of each table version would be very large. ASIQ conserves resources by allowing table versions to share pages that are not being updated. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-17 ASIQ Operational Data Administration Tabe Versioning How Does It Work? 1. Upon connection, users see committed tables • These are called table blockmaps 2. When a table is modified, new blockmaps are created for changed pages • The "old" blockmaps are retained by current users • When changes are committed to a table, the new blockmap becomes available 3. A user must issue an explicit COMMIT statement or exit the database and log back in to use the new blockmap • The old blockmaps are retained until all users are no longer using that blockmap commit Most query tools cannot issue a COMMIT statement. There is an ODBC option AUTOPRECOMMIT = ON that issues a COMMIT before every SELECT. Example to follow. 12-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Versioning Example ■ At t1, two users are in the database reading data from the customer, product, and order tables (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-19 ASIQ Operational Data Administration Versioning Example ■ At t2, a write user logs into the database and begins inserting into the customer table. The read users still see the original versions of the customer, order, and product tables. (continued . . .) 12-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Versioning Example At t3, the changes to the customer table are committed. The read users still see the original versions of the customer, ,order and product tables. (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-21 ASIQ Operational Data Administration Versioning Example At t4, a new read user logs in and sees the new version of the customer table as well as the order and product tables. Read users (continued . . .) 1 and 2 still do not see the new version of the customer table. 12-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Versioning Example ■ At t5, read users 1 and 2 have issued a COMMIT or they have exited from the database and logged back on. All users see the new version of the customer table and the old version is gone. In IQ Multiplex, the new table version is replicated from the Catalog Store on the write server to the Catalog Stores on all reader servers. After the writer commits changes, the new version will be replicated and available to users on the reader servers in about 1 minute. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-23 ASIQ Operational Data Administration Option to COMMIT After Every SELECT ■ In Interactive SQL, select Command->Options, and then select After Every Command checkbox 12-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration ODBC AutoPreCommit ■ Turn to ON to COMMIT before execution of a statement ■ Default for AutoPreCommit is OFF ■ Set in either the Windows registry or the .ini file ■ Example: AutoPreCommit=ON ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-25 ASIQ Operational Data Administration Snapshot Versioning: Issues ■ Version management does require some overhead, but the effect on performance is minimal in the Catalog Store • ASIQ Server performance may suffer from multiple table inserts/deletes ■ Disk space can become an issue • May need to increase your disk space in order to store overlapping versions ■ Data consistency between tables must be understood by all users 12 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Module Map • Transactions • Locking • Snapshot Versioning ■ Security ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-27 ASIQ Operational Data Administration Data Security ■ ASIQ allows privacy of appropriate information within the database ■ Security is implemented at various levels: • Users • Groups • Object/command-level permissions 12 - 28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Users ■ You should set up individual user IDs for each user • Lets you tell which connects are which users ■ DBA (Database Administrator or database owner) has authority to add new users ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-29 ASIQ Operational Data Administration Users and Groups ■ User • A login can access a database ■ Groups • Enables a collective name for a set of users • Simplifies administration ■ Users in groups • SYS group • PUBLIC group 12 - 30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Specal User DBA User ■ A user with Database Administrator (DBA) authority is the database administrator or database owner ■ May be granted or transferred to other users ■ Responsible for the overall management of the database • Adds new users and permissions • Authority to create database objects and assign ownership ■ Default password "SQL" ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-31 ASIQ Operational Data Administration Creating Users DBISQL ■ Syntax: GRANT CONNECT TO userid [,...] IDENTIFIED BY password [,...] ■ Example: GRANT CONNECT TO laura_smith IDENTIFIED BY secret; ■ To create a user with no password, type: GRANT CONNECT TO userid ■ 12 - 32 A user with no password cannot connect to the database. This user is useful when you are creating groups. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Creating Users with Sybase Central 1. Connect to the database 2. Click Users and Groups folder 3. Double-click Add User 4. Follow wizard (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-33 ASIQ Operational Data Administration Creating Users with Sybase Central ■ Click Finish 12-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Special Groups ■ SYS group • Owner of all system tables and views for the database • Automatically created, no password • Cannot connect to the database as SYS ■ PUBLIC group • Member of the SYS group with read access for some of the system tables and views • Any new user is automatically a member of the PUBLIC group • Automatically created with connect permissions to the database ■ Can use REVOKE to remove membership in these groups REVOKE ©2000 Sybase, Inc. Syntax to follow. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-35 ASIQ Operational Data Administration Creating Groups DBISQL ■ Syntax: GRANT GROUP TO userid ■ Allows the user to have members ■ Example: GRANT CONNECT TO marketing; GRANT GROUP TO marketing; This example creates a user (marketing) with no password. You can not connect to the database with the marketing user. Sybase Central 12-36 1. 2. 3. 4. Connect to the database. Click the Users and Groups folder. Double-click Add Group. Follow wizard. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Adding a User to a Group DBISQL ■ Syntax: GRANT MEMBERSHIP IN GROUP group-name TO userid ■ Example: GRANT MEMBERSHIP IN GROUP marketing TO laura_smith ■ Must have already granted connect permis sion to laura_smith Steps to create a user in 1. a group 2. 3. 4. ©2000 Sybase, Inc. GRANT CONNECT TO laura_smith; GRANT CONNECT TO marketing; GRANT GROUP TO marketing; GRANT MEMBERSHIP IN GROUP marketing TO laura_smith; Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-37 ASIQ Operational Data Administration Table and Vew Permissions Syntax: Example: There are several ways to grant user permissions in Sybase Central • See Chapter 10, "Managing User IDs and Permissions," in the Adaptive Server IQ Administration and Performance Guide. 12-38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Stored Procedure Permissions ■ Must have EXECUTE permission to execute a stored procedure ■ Similar to granting permissions on tables and views ■ Example: GRANT EXECUTE ON my_procedure TO l aura_smith ■ Procedures execute with the permissions of their owners Inserts/deletes Any procedure that inserts or deletes from a table will execute successfully only if the owner of the procedures has the correct level of permissions. Therefore, the owner must have INSERT or DELETE permission on that table. Sybase Central You can grant user permissions on procedures in Sybase Central. See Chapter 10, "Managing User IDs and Permissions," in the Adaptive Server IQ Administration and Performance Guide. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-39 ASIQ Operational Data Administration REVOKE Statement ■ The REVOKE statement is the exact converse of the GRANT statement ■ Example: REVOKE 12 - 40 SELECT ON customer FROM laura smith Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Permissions Hierarchy ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-41 ASIQ Operational Data Administration Database Object Names and Prefixes ■ Every database object is an identifier; up until now, all database objects have been referred to using their simple names ■ Example: SELECT * from employee ■ Tables, procedures, and views all have an owner ■ Example: SELECT * from "DBA".employee ■ When referring to a database obect, you need a prefix unless: • You are the owner of the object • The database object is owned by a group ID of which you are a member To get around typing in 1. GRANT CONNECT TO sales; "DBA" 2. GRANT GROUP TO sales; 3. GRANT RESOURCE TO sales; 4. GRANT SELECT ON tablename to sales; 12 - 42 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Operational Data Administration Summary ■ Snapshot versioning is ASIQ's approach to transaction management. ■ Locking is handled automatically by ASIQ. • One writer and multiple readers at the table level ■ Keep data secure by controlling a user's access to the database and the objects within the database. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 12-43 ASIQ Operational Data Administration Lab 12: ASIQ Operational Data Administration Exercise Overview Topics ■ ■ ■ Goals This lab develops your DBA skills. After completing this lab, you should be able to: ■ Set up read-only users ■ See what happens when the database is being updated General tasks ■ Create a user using Sybase Central. ■ GRANT the appropriate permissions to the new user. ■ Update the residential_customer table in one window as the DBA user. ■ Query the residential_customer table in a separate window as the read-only user. Lab setup ■ ■ 12 - 44 Creating a user Granting permissions Snapshot versioning This lab assumes that you have created the megaphone database and have created tables and indexes in that database. The data should be loaded in all the tables from Labs 8, 9, and 10. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 13 Joins Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Joins Module Overview This module discusses ad-hoc oins and predefined join indexes. It also explores when and how to create an ASIQ join index. Objectives After completing this module, you should be able to: • Compare ad-hoc joins to join indexes • Create a join index • Synchronize a join index See Chapter 4, "Adaptive Server IQ Indexes," in the Adaptive Server IQ Administration and Performance Guide for additional information on join indexes. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 13-1 Joins Types of Joins ■ Ad-hoc joins • Joined at query time ■ Join index • An internal structure that relates table columns • Represents two or more tables, including inner, left outer, and right outer rows • May provide better performance than ad-hoc joins • Requires additional disk space and time to create 1 3 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Ad-Hoc Joins ■ A join merges rows from two tables into a result table ■ Six types of ad-hoc join algorithms • Nested loop • Hash • Sortmerge • Sort/merge push down • Nested loop push down • Hash push down ■ Additional join type called Cartesian nested loop • Joins without a join condition in the WHERE clause Nested loop ■ Takes a row from the larger table of the join and compares it to each row of the smaller table. ■ You want the rows from the smaller side of the join to stay in memory. Hash ■ Similar to nested loop, except the smaller side is "hashed" into an in-memory table with a hashing function applied to the join keys. ■ Requires fewer comparisons than a nested loop, but adds the cost of computing the hash key for each row on both sides of the join. Sort/merge Sort/merge ■ Sorts the rows on both sides and then compares the rows. push down ■ Bloom-filter based semijoin that significantly reduces the number of rows projected and the number of rows to be sorted. Nested loop push down ■ IQ turns the keys of the smaller side of the join into an IN list on the large table. Hash push down ■ Combination of a hash join and ASIQ's special index processing. Cartesian nested loop ■ A variation of the nested loop join. A many-to-many or Cartesian join will produce a row for every combination of rows from both sides. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-3 Joins When to Use Ad-Hoc Joins Instead of Join Indexes ■ Always try ad-hoc joins first before creating join indexes ■ If the ratio of data in oin tables is higher than 10 to 1 • For example, lookup tables • Ad-hoc joins recommended due to data explosion ■ For self-joins ■ For many-to-many relationships 1 3 -4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins When to Create a Join Index ■ After trying ad-hoc queries ■ If certain tables are typically joined together in queries ■ There is adequate disk space ■ Fairly static tables ■ Ratio of data in join index is less than 10 to 1 • This limit is due to high disk overload • Recommend a ratio of less than 5 to 1 for performance reasons ■ ■ ©2000 Sybase, Inc. The Adaptive Server IQ Administration and Performance Guide recommends the ratio of data to create a join index to be less than 10to1. For performance reasons, a ratio of less than 5 to 1 is preferred. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 3 -5 Joins Ad-Hoc Joins Versus Join Indexes ■ Join indexes can involve multiple tables and are subject to certain rules Even if extra build time is available and additional disk space is not an issue, join indexes cannot be created in all cases. Synchronizing 13-6 This topic is discussed in detail later in the module. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Creating a Join Index Step Overview ■ Step 1: Create the tables involved in the join index CREATE TABLE... ■ Step 2: Define the relationships ■ Step 3: Create the indexes for the tables being joined CREATE INDEX... ■ Step 4: Load the data into the tables LOAD TABLE... or INSERT... ■ Step 5: Create the join index CREATE JOIN INDEX... ■ Step 6: Synchronize join index, if required SYNCHRONIZE JOIN INDEX... ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 13-7 Joins Step 1: Create Tables ■ Create the tables involved in the join index • CREATE TABLE command • Sybase Central ■ See Module 6, "Creating Tables and Indexes" ■ Example: The example in this module joins customer and product tables. You would also need to create the product table as part of step 1. 1 3 -8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Step 2: Define the Relationships ■ Must determine the relationships between each pair of tables ■ Types • One-to-many (ONE>>MANY) • Many-to-one (MANY>>ONE) • One-to-one (ONE>>ONE) One-to-many . ONE >>MANY ■ One row in one table potentially matches with one or more rows in another table. Many-to-one . MANY>>ONE ■ One or many rows in one table potentially match with one row in another table. One-to-one . ONE >>ONE ■ One row in the first table potentially matches with one row in the second table. ■ ©2000 Sybase, Inc. If the join relationship specified is incorrect, the extra rows are discarded and not included in the join index. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 3 -9 Joins Join Hierarchy ■ All ASIQ join relationships must have a join hierarchy • A tree that illustrates how all the tables in a join are connected • Determine the top table, which is always on the MANY side ■ Rules • A table can occur only once • A table can have only one arrow leaving it • All tables must be connected • An arrowhead represents the MANY side of a relationship Notes 13-10 In this example, Table C is the "top table." Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Types of Join Hierarchies ■ Linear joins • Each table is related to the table above it, until you reach the top table • Each pair of tables represents a one-to-many relationship ■ Star joins • Each table is related to one table in the middle • The table in the middle represents the MANY side, sometimes called the fact table • The ONE tables are called dimension tables ■ You can create join indexes or have ad-hoc joins for both linear and star joins ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-11 Joins Entity-Relationship Diagram ■ This example is used throughout this module In this entity-relationship diagram, a customer can have one or more orders. 13-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Join Hierarchy with the Top Table ■ order table is the "Top Table" order becomes the top table because it is on the MANY side of the relationship. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-13 Joins Creating a Join Index Step Overview • Step 1: Create the tables involved in the join index CREATE TABLE... • Step 2: Define the relationships ■ Step 3: Create the indexes for the tables being joined CREATE INDEX... ■ Step 4: Load the data into the tables LOAD TABLE... or INSERT... ■ Step 5: Create the join index CREATE JOIN INDEX... ■ Step 6: Synchronize the join index, if required SYNCHRONIZE JOIN INDEX ... 13-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Step 3: Create the Indexes ■ Create the column indexes for the tables being joined ■ Columns involved in oins must have an HG or LF index ■ Example: CREATE HG index customer_id_HG on customer(customer_id); CREATE LF index cust_type_LF on customer(cust_type); ■ To determine the index types for each column, see Module 5, "ASIQ Index Types and Data Types." ■ To create indexes on each column, see Module 6, "Creating Tables and Indexes." Remember to create all the indexes for the customer and order tables. Sybase Central ©2000 Sybase, Inc. ■ Indexes can be created either using the CREATE INDEX command or using Sybase Central. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-15 Joins Step 4: Load Data ■ Load the data into the tables • Can use LOAD TABLE or INSERT command • See Modules 8, 9, and 10 on data loading ■ Example: LOAD TABLE customer (customer_id '|' ) FROM ... You need to load or insert data into the order table also. 13-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Creating a Join Index Step Overview • Step 1: Create the tables involved in the join index CREATE TABLE... • Step 2: Define the relationships • Step 3: Create the indexes for the tables being joined CREATE INDEX... • Step 4: Load the data into the tables LOAD TABLE... or INSERT... ■ Step 5: Create the join index CREATE JOIN INDEX... ■ Step 6: Synchronize join index, if required SYNCHRONIZE JOIN INDEX... ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 13-17 Joins Step 5: Create the Join Index ■ Must be DBA or the table owner and have RESOURCE authority to CREATE JOIN INDEX ■ Syntax: CREATE JOIN INDEX join-index-name FOR join-clause ■ Example: CREATE JOIN INDEX customer_order_join FOR customer FULL OUTER JOIN order ON customer.customer id = order.customer id (continued . . .) join-clause [ ( ] join-expression join-type join-expression [ on search-condition ] [ ) ] join-expression { table-name | join-clause } join-type [ NATURAL ] FULL [ OUTER ] JOIN Another example and further explanation follow later in this module. 13-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Step 5: Create the Join Index Specifying the Join Type ■ Join type is always FULL OUTER; OUTER keyword is optional ■ If you are joining equivalent columns with the same names, you specify NATURAL JOIN ■ If you are joining columns based on keys, you must have specified FOREIGN and PRIMARY keys ■ Use the ON clause to join equivalent values in two tables (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQforthe DataWarehouse, Version 1.0 13-19 Joins Step 5: Create the Join Index ■ Same example (natural oin): CREATE JOIN INDEX nat_customer_order_join FOR customer NATURAL FULL OUTER JOIN order ■ Join column names must be the same in the tables being joined (continued . . .) Keyjoin CREATE JOIN INDEX key_customer_order_join FOR customer FULL OUTER JOIN order ■ 13 - 20 Foreign and primary keys must be defined during the CREATE TABLE operation for customer and order tables in the above Key join example. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Step 5: Create the Join Index ■ Results: • When you create a join index, a message in the log appears: Sybase Central ©2000 Sybase, Inc. ■ ■ You can create a join index in Sybase Central. See Chapter 4 of the Adaptive Server IQ Administration and Performance Guide. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-21 Joins Creating a Join Index Step Overview • Step 1: Create the tables involved in the join index CREATE TABLE... • Step 2: Define the relationships • Step 3: Create the indexes for the tables being joined CREATE INDEX... • Step 4: Load the data into the tables LOAD TABLE... or INSERT... • Step 5: Create the join index CREATE JOIN INDEX... ■ Step 6: Synchronize join index, if required SYNCHRONIZE JOIN INDEX... 13 - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Step 6: Synchronize Join Indexes ■ This step ensures that the data is loaded in the correct order for the join ■ It automatically occurs when: • You create the join index • Changes are made to the top table of the join hierarchy ■ Otherwise, must be manually performed, which can be time-consuming • Schedule during off-peak hours • Synchronize after entire set of inserts and deletes (do not do after every insert or delete) (continued . . .) If you complete step 5 (CREATE JOIN INDEX) before step 4 (LOAD TABLE or INSERT), you do not have to synchronize the data for the initial setup. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-23 Joins Step 6: Synchronize Join Indexes ■ Syntax: SYNCHRONIZE JOIN INDEX [ join-index-name [, join-index-name] ■ If you omit the index name, ASIQ synchronizes all oin indexes • Because this process is so time consuming, it is best to specify joins and schedule them individually ■ Example: SYNCHRONIZE JOIN INDEX customer_order_join 13 - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Multiple-Table Join Example ■ Join hierarchy: ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-25 Joins To Create the Join Index CREATE JOIN INDEX cust_prod_order_join FOR customer FULL OUTER JOIN order ON customer.customer_id = order.customer_id FULL OUTER JOIN product ON order.product_id = product.product_id ■ If no key definitions exist, a ONE>>MANY is assumed left to right 13 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Rules for Join Indexes Conditions and Restrictions ■ You must be Database Administrator or table owner ■ You can combine steps 1 and 2 by defining relationships when you create the table (define keys) ■ You need to synchronize after additional inserts/deletes unless you have updated the top table ■ This approach is not recommended for data ratios greater than 10 to 1 (less than 5 to 1 may provide more optimal performance) ■ Join indexes cannot be created on many-to-many relationships ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-27 Joins Estimating Join Index Size ■ Stored procedure syntax: sp_iqestjoin table1, #ofrows, table2, #ofrows [, relation] [, iq_page_size] ■ Estimates the space needed to create oin indexes ■ Example: sp_iqestjoin customer, 10000, order, 30000 ■ Run this procedure for each pair of tables being joined Default values 13 - 28 ■ ■ Relationship: one-to-many. IQ page size: 64KB. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Actual Join Index Size ■ Stored procedure syntax: sp_iqjoinindexsize (join-index-name) ■ Returns the total size of the index in bytes ■ Example: sp_iqjoinindexsize ©2000 Sybase, Inc. customer_order_join Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-29 Joins Dropping a Join Index ■ Syntax: DROP JOIN INDEX join-index-name ■ Example: DROP JOIN INDEX customer_order_join ■ You must first drop the join index before dropping an underlying table 13 - 30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Data Modification in a Join Index ■ Always insert, update, or delete from the underlying tables, not from the join index ■ You cannot do partial-width inserts to tables that participate in a join index • Drop the join index, do the partial-width insert, and then recreate the join index • Do not forget to synchronize ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-31 Joins How Join Indexes Are Used ■ The SELECT statement determines whether a join index will be used • Determined by the FROM and WHERE clauses • Do not need to specify the join index name in a query ■ ASIQ can use the same join index to resolve a query that involves the full join relationship or a query that involves a contiguous subset ■ Queries that can be resolved • Entire relationship • AtoD • BtoD • DtoF • CtoE • EtoF • C to E to F ■ Example of a query that cannot be resolved using the join index: • EtoD 13-32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Joins Summary ■ Try using ad-hoc joins first. ■ Create a join index if: • Ratio of data in join tables is less than 10 to 1 (recommend less than 5 to 1). • There are no many-to-many relationships. • Same tables are frequently joined together in queries. • There is adequate disk space. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 13-33 Joins Lab 13: Joins Exercise Overview Topics ■ Goals This lab develops your data-loading skills. At the end of this lab, you should be able to: ■ Estimate the size of a join index before loading ■ Create a join index ■ Synchronize a join index General tasks ■ Participate in an instructor-led question-and-answer session. ■ Use the sp_iqestjoin procedure to estimate the size required for a join index. ■ Create a join index. ■ Synchronize a join index. Lab setup ■ 13 - 34 Join indexes This lab assumes that you have created the megaphone database on an ASIQ Server. Tables need to be created in that database and should be fully populated. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 14 Monitoring and Troubleshooting Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Module Overview This module identifies techniques to monitor ASIQ databases. In addition, it describes how to use ASIQ utilities to monitor and troubleshoot. Objectives After completing this module, you should be able to: ■ Monitor an ASIQ database ■ Monitor performance using IQ UTILITIES ■ Troubleshoot problems For further information, see Chapter 13, "Monitoring and Tuning Performance," in Sybase Adaptive Server IQ Administration and Performance Guide. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-1 Monitoring and Troubleshooting Module Map ■ Monitoring • Monitoring Files • Monitoring Users • Monitoring Deletes/Loads/Inserts • Using System Stored Procedures • IQ UTILITIES • System Utilities ■ Troubleshooting • What do I do if...? 1 4 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring Users ■ To see the connected users, use Sybase Central ■ Double-click the Connected Users folder for the database • More information can be obtained by double-clicking the connection ID ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-3 Monitoring and Troubleshooting Monitoring Loads/Deletes/Inserts Look at insert, update and delete notification messages... 90000 Records, 8 Seconds Mem: 3 0mb/M31 Main Blks: U5872/4%, Buffers: U6392/L9 Temporary Blks: U11721/28%, Buffers: U3083/L169 Main I: L173970/P2725 O: D27387/P21574 C:D21400 Temporary I: L341295/P7229 O: D50851/P13638 C:D32200 100000 Records, 8 Seconds Mem: 3 0mb/M31 Main Blks: U7784/6%, Buffers: U6392/L9 Temporary Blks: U13677/33%, Buffers: U3121/L116 Main I: L179988/P2725 O: D28345/P22521 C:D20500 Temporary I: L351507/P7230 O: D53062/P14611 C:D30100 ■ Final notification message at the end of the output: 100000 records were inserted into 'customer'. Insert for 'customer', completed in 86 seconds. 100000 records inserted. Mem: 14mb/M31 Main Blks: U23932/19%, Buffers: U6393/L0 Temporary Blks: U41/0%, Buffers: U9/L1 Main I: L249754/P2778 O: D36672/P30625 C:D14900 Temporary I: L421871/P14378 O: D63112/P15701 C:D28000 ■ This information is in the .iqmsg file ■ See "Interpreting Notification Messages" in Chapter 5 of the SybaseAdaptiveServerIQAdministrationandPerformanceGuide 14-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring Loading Messages ■ Three phases of the loading process • Phase 1 - Read data from all files and write LF, HNG, and FP indexes • Phase 2 - Sort data for HG indexes; continue writing other indexes to disk • Phase 3 - Write HG indexes to disk ■ Notification messages are issued based on the NOTIFY frequency • Default 100,000 rows • NOTIFY can be changed using LOAD TABLE or INSERT command option • Can also be set using Set Option Notify_Modulus (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 4 -5 Monitoring and Troubleshooting Monitoring Loading Messages ■ During Phase 1, NOTIFY reports time required to READ data from the source • Reports current memory (IQ caches + Heap)/ Max memory used so far Mem: 3 0mb/M31 • Shows blocks in use and percentage of total, buffers in use and buffers locked Main: Blks: U5872/28% Buffers: U6392/L9 Temporary Blks: U13677/33% Buffers: U3121/ L116 • Reports logical and physical reads, buffer destruction and compression information Main I:L179988/P2725 O:D28345/P22521 C:20500 Temporary I:L351507/P7230 O:53061/P14611 C:30100 (continued . . .) 1 4 -6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring Load Messages ■ After Phase 1, you will see similar notification messages as load phases complete, including a final report at the end of the load • If NOTIFY is set high, you may not see a report until the load completes • If NOTIFY is set too low, you may be inundated with messages ■ See Chapter 5, ASIQ Administration and Performance Guide, for more information on the load notification messages ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-7 Monitoring and Troubleshooting Monitoring the Database Stored Procedures ■ Examples of each to follow (continued . . .) 14-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring the Database sp iqcheckdb ■ Checks database validity • Reads every database page from disk into memory ▲ This may take a long time to run!! • Verifies and repairs • Saves information until next sp_iqdbstatistics is issued ■ Upon successful completion, updates the free list to reflect true storage allocation • Does not update free list if errors are found ■ Example on next page The free list is a list of blocks on disk that are available. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 4 -9 Monitoring and Troubleshooting Monitoring the Database sp iqcheckdb ■ This is a partial list ■ The statistics marked with an asterisk are the key statistics to look for orphaned blocks • If you see a non-zero value for any of these statistics, your database probably has a serious consistency problem ■ Need to issue a checkpoint before running this stored procedure to get accurate results (continued . . .) 14-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting sp_iqcheckdb and dbcc_option ■ dbcc_option determines processing done by sp_iqcheckdb • Works in conjunction with server start option -iqdroplks A -iqdroplks is used to drop leaked blocks with certain dbcc_options ■ dbcc_option default = 0 • Checks every page and object in the database (may run long) • Gathers statistics and reports results • With -iqdroplks, updates the free list if no errors are detected by sp_iqcheckdb ■ Other dbcc_options on next page (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-11 Monitoring and Troubleshooting dbcc_opton ■ Other dbcc_option values • 1 - Checks all blockmaps (runs fast) A Checks for leaked blocks and reports findings only • 2 - Not used • 3 - Checks blockmaps (runs fast) A With -iqdroplks and no errors detected, will update the freelist • 4 - Checks all indexes in database and reports errors (continued . . .) 14-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting dbcc_opton • 5 - Extensive consistency check on all indexes and reports errors ▲ Runs long with many LF and HG indexes in database • 6 - Checks and repairs all indexes, report repairs performed • 7 - Extensive check and repair (if possible) on indexes A Reports corrections ▲ Runs long with many LF and HG indexes in database ■ See Chapter 2, ASIQ Troubleshooting and Error Messages Guide ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-13 Monitoring and Troubleshooting Monitoring the Database sp iqdbstatistics sp iqdbsize (continued . . .) 14-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring the Database sp iqstatus ■ Things to look for: see next page (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-15 Monitoring and Troubleshooting Monitoring the Database sp iqstatus ■ Main and Temporary IQ Blocks Used • If percentage is high, add space to either store ■ Old Versions • Shows number of old table versions in the database and space used • These versions will be dropped and space recovered when no longer referenced by active transactions ■ Active Txn Versions • Number of active write transactions and amount of data created and destroyed ■ See Chapter 1, ASIQ Troubleshooting and Error Messages Guide, for more information on sp_iqstatus 14-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring Buffer Caches IQ UTILITIES ■ Monitors the performance of the buffer caches ■ Collects statistics on the buffer cache's memory and I/O on ASIQ ■ Statistics are stored in a log file Syntax: IQ UTILITIES { MAIN | PRIVATE } INTO dummy_table_name START MONITOR 'monitor_options [...]' (continued . . .) MAIN Starts monitoring the main buffer cache for the IQ Store of the database to which you are connected. PRIVATE Starts monitoring the temp buffer cache for the Temporary Store of the database you are connected to. monitor_options -summary Displays summary information for both the main and temp buffer caches (default). -cache Displays activity in detail for the main or temp buffer cache. -cache_by_type Same results as -cache, but broken down by IQ page type. -io Displays detailed main or temp buffer cache I/O rates and compression ratios. -interval Specifies the reporting interval in seconds (default = 2 seconds). More monitor options continued on next page. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-17 Monitoring and Troubleshooting Monitoring Buffer Caches Examining and Saving Monitor Results ■ Results are stored in a text file called: • dbname.connection#-main-iqmon for main buffer cache results • dbname.connection#-temp-iqmon for temp buffer cache results ■ The text file may be named with a suffix specified in the monitor options (see below) ■ To see the results, use a text editor and display or print the file • File(s) will be created in same directory as .db file (continued . . .) dbname.connection# Represents your database name and the connection number. You can check the monitor output results using the UNIX command tail -f <name_of_file>. On NT, the file cannot be monitored while the process is running. You can check the monitor output results using the UNIX command tail -f <name_of_file>. On NT, the file cannot be monitored while the process is running. monitor_options (continued) -buffalloc - Displays information on the main or temp allocator. contention Displays many key buffer cache and memory manager locks. Displays information about processing threads. Specifies a -threads file suffix for the monitor output file. file suffix 14-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Monitoring Buffer Caches ■ Example: create table monitor_table (a varchar(1)); IQ UTILITIES PRIVATE INTO monitor_table START MONITOR '-summary -interval 20'; ■ First need to create a dummy table • IQ table only used for monitoring • The datatype for column a in the monitor _table can be a char(1) ■ Start the buffer cache monitor and record activity for the main and temp buffer caches (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-19 Monitoring and Troubleshooting Monitoring Buffer Caches ■ Example of the -summary option Sybase Adaptive Server IQ Performance Monitor ■ Users C = connected, A = Active ■ See Chapter 13, "Monitoring and Tuning Performance," in the Adaptive Server IQ Administration and Performance Guide for more information. 14-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Stopping the Buffer Cache Monitor ■ Syntax: IQ UTILITIES { MAIN | PRIVATE } INTO dummy_table_name STOP MONITOR ■ Example: IQ UTILITIES PRIVATE INTO monitor_table STOP MONITOR; ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-21 Monitoring and Troubleshooting System Utilities Monitor CPU Use and Paging ■ UNIX: • ps • vmstat • sar ■ NT: • Performance Monitor • Task Manager 14-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Module Map • Monitoring • Monitoring Files • Monitoring Users • Monitoring Loads • Using System Stored Procedures • IQ UTILITIES • System Utilities ■ Troubleshooting • What do I do if...? See the Adaptive Server IQ Troubleshooting and Error Guide for a number of troubleshooting problems/scenarios and methodology to handle or correct problems. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-23 Monitoring and Troubleshooting Troubleshooting Fields What Do I Do If... ■ The ASIQ Server won't start? ■ A user cannot access a database? ■ The database fills during a load? ■ Processing slows down or hangs? 14-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting ASIQ Server Won’t Start Possible Cause ■ May already be up! ■ Transaction log file does not match the database ■ Not using valid user ■ Environment variables are not set correctly ■ Network connections are not working ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-25 Monitoring and Troubleshooting User Cannot Access the Database or Database Objects ■ User may not have a valid user ID and/or password ■ The server was started with DBA use only ■ Database files may be missing ■ Data source is not correctly defined, ODBC ■ The database is in the process of being restored ■ User does not have correct permissions on database objects 14-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Database Fills Up During the Load ■ ASIQ: • Suspends the load • Keeps checking to see whether another dbspace has been added ■ Allocate more space to the database • Use CREATE DBSPACE (syntax in notes) ■ Example of what to do: CREATE DBSPACE mydb_2 AS 'e:\\s2\data\mydb_2.iq' SIZE 200 CREATE DBSPACE dbspace-name AS filename ... [ {IQ STORE | IQ TEMPORARY STORE | CATALOG STORE }] . . . [ [SIZE] file-size ] Default Create a dbspace for the IQ Store. You can monitor the amount of space used in a database by looking at the load notification message. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-27 Monitoring and Troubleshooting Processing Slows Down or Hangs ■ Just one user, or several, or everyone? • If one user, what is the process? • If everyone, check the database or server ■ Look at paging to determine whether thrashing is a problem ■ Check buffer caches; they may be set too small • Be sure not to set IQ buffer cache sizes so that total memory requirements are exceeded ■ Look at query, check for correct indexes 14-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Reporting Problems to Technical Support ■ Have ready: • Site ID • Environment information (platform and versions) A sp_iqstatus • Exact problem or query with error message ■ May need: • IQ Message Log, OS Error Log, Server Config File • stktrc.iq (stack trace file in same directory as .db file) ■ Determine: • Severity of problem • If problem is reproducible (DBISQL or ISQL or both) ■ See Chapter 2, ASIQ Troubleshooting and Error Guide, for the extensive checklist of things to do before contacting Technical Support ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-29 Monitoring and Troubleshooting Summary Good System Administrator Practices ■ Ensue tables have been created correctly, use IQ UNIQUE ■ Ensure index types have been chosen correctly ■ Set the correct values for memory ■ Perform the following on a regular basis: • Backups • Monitoring the logs • Monitor the space usage • Check performance Regular backups 14-30 See Module 16, "Backup and Restore," for more information. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Monitoring and Troubleshooting Lab 14 - Monitoring and Troubleshooting Exercise Overview Topics ■ Goals This lab explains how to use IQ UTILITIES monitoring tool. After completing this lab, you should be able to: . Start and stop IQ UTILITIES . View the log file created by IQ UTILITIES General tasks Lab setup ©2000 Sybase, Inc. IQ UTILITIES ■ Create a dummy table. ■ Start IQ UTILITIES. ■ Run queries in the megaphone database. ■ Look at the results. . Stop IQ UTILITIES. ■ This lab assumes that the ASIQ Server is up and running with the megaphone database loaded with data. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 14-31 15 Performance and Tuning Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Module Overview This module describes the IQ Query Engine and identifies performance tips for queries and updates. It also including information on how to read an IQ Query Plan. Objectives After completing this module, you should be able to: ■ Describe the IQ Query Engine ■ Define techniques to improve queries ■ Be able to read an IQ Query Plan ■ Define techniques to improve updates For more information on performance and tuning see Chapter 12, "Managing System Resources," in the Adaptive Server IQ Administration and Performance Guide. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 15-1 Performance and Tuning Module Map ■ Define the IQ Query Engine ■ Improving Query Performance • IQ Query Plan ■ Improving Update Performance 1 5 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Engine ■ Similarities with Traditional Query Engines • Once ASIQ has completed all the work it can in the indexes for each table, rows are projected and then joined or grouped in the same ways as in other query engines • Whenever multiple access paths (or operator algoritms) are available, then a cost-based decision must be made, often based on predicated selectivity estimates ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 5 -3 Performance and Tuning IQ Query Engine ■ Differences from Traditional Query Engines • Effects of most predicates local to a table can be combined with each other before any rows are actually projected from the table • Numerous query operators are available directly within the indexes, including: A Grouping and distincting ▲ Inter-column comparisons ■ Hash containment 1 5 -4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Engine Architecture Three Parts ■ IQ Server Front End • Includes parsing incoming statements, stored procedures, security checking ■ IQ Optimizer (more detail to follow) • Includes join optimization, index selection, subquery optimization and grouping algorithm selection ■ IQ Run-Time Engine • Includes the execution of predicates, joins, groupings and subqueries, as well as sorting ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-5 Performance and Tuning IQ Query Optimizer ■ Join Optimization • All possible join plans that avoid cartesian product joins are considered • Applicable oin indexes are considered • Join algorithms selection is done from: A Nested loop A Hash A Sortmerge A Sort/merge push down A Nested loop push down A Hash push down (continued . . .) ■ Nested loop ■ ■ Hash ■ Takes a row from the larger table of the join and compares it to each row of the smaller table. You want the rows from the smaller side of the join to stay in memory. Similar to nested loop, except the smaller side is "hashed" into an in-memory table with a hashing function applied to the join keys. Requires fewer comparisons than a nested loop, but adds the cost of computing the hash key for each row on both sides of the join. Sort/merge Sort/merge ■ Sorts the rows on both sides and then compares the rows. push down ■ Bloom-filter based semijoin that significantly reduces the number of rows projected and the number of rows to be sorted. Nested loop push down ■ IQ turns the keys of the smaller side of the join into an IN list on the large table. ■ Combination of a hash join and ASIQ's special index processing. Hash push down 15-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Optimizer ■ Index Selection • The optimizer does a cost-based selection of what order to execute predicates and which index to use for each predicate • The optimizer considers: ▲Predicate type ▲Indexes available on that column (or columns) ▲Estimated selectivity of this predicate ▲Estimated number of rows (continued . . .) Predicate ©2000 Sybase, Inc. A condition in the where clause which accesses only one table. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 5 -7 Performance and Tuning IQ Query Optimizer ■ Grouping Algorithm Selection • Three different algorithms are compared for each GROUP BY or DISTINCT operator A Sort-based A Hash-based ▲ Index-based (continued . . .) 1 5 -8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Optimizer ■ Subquery Optimization • Some EXISTS subqueries are flattened into distinct joins • Results of uncorrelated subqueries are used directly in the indexes of tables in the parent query blocks • Correlated subqueries ▲ Are migrated to minimize subquery re-executions A Have a 1-value result cache, and outer references are ordered to further minimize re-executions A Are optimized to maximize invariant operators ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 5 -9 Performance and Tuning Module Map • Define the IQ Query Engine ■ Improving Query Performance • IQ Query Plan ■ Improving Update Performance 15-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Choosing Correct Index Types ■ To achieve the best performance, you must choose the correct indexes for your data ■ Review of index types Name Abbreviation Fast Projection FP Low Fast LF High Non Group HNG High Group HG ■ The optimized FP stores (FFP and FFFP) can reduce storage and improve some queries (continued . . .) Consider creating new tables with IQ UNIQUE and loading data from existing tables to take advantage of the optimized FP stores (FFP and FFFP). ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-11 Performance and Tuning Choosing Correct Index Types ■ ASIQ indexes are optimized according to the number of distinct values Unique Values Recommended Index Type < 1500 LF/HNG > = 1500 HG/HNG ■ LF and HG are the "fast" indexes that you should always use on any column used in a join, GROUP BY, or WHERE clause • They provide column cardinality to the optimizer ■ Use the UNIQUE or PRIMARY KEY constraint on any applicable column (continued . . .) Follow the algorithm outlined in Module 5, "ASIQ Index Types and Datatypes," to ensure you have chosen the correct index(es) for each column. UNIQUE or PRIMARY KEY 15-12 Use UNIQUE or PRIMARY KEY only on single columns. For HG indexes, no G-Array is constructed, thus reducing space and loading time. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Choosing Correct Index Types ■ Recommended index types based on the query ■ These recommendations should not be your only criteria for picking an index • See Module 5, "ASIQ Index Types and Datatypes" ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-13 Performance and Tuning Database Options for Performance ■ Set option public.Force_No_Scroll_Cursors = 'On' • Off by default - buffer results set to permit scrolling of data • Better performance by setting to ON to turn off buffering ■ Set option public.OS_File_Cache_Buffering = 'Off • For NT and Solaris only • Turns off file system caching for these platforms (ON by default) • Has no effect on database stores built with raw devices 15-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plans ■ IQ uses a Volcano style data-flow query engine • Queries are transformed into an upside-down tree of operator nodes, each of which can accept rows from any nodes immediately below it, and can pass rows up to the node above it • Each node transforms the row stream in some way, such as filtering some rows out, unioning or joining multiple row streams together, etc. • A 'Query Plan' is the textual display of a query tree ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-15 Performance and Tuning IQ Query Tree ■ A Query Tree consists of "nodes" • Each node represents a stage of work • Each node has a name and a number ■ Can be visualized as an upside down tree • Top of the tree is the root node • Lowest node is the leaf 15-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plans ■ The Query: select colA, colB, colC from tableA, tableB where tableA.joincol = tableB.joincol group by groupcol ■ The Query Tree: ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-17 Performance and Tuning IQ Query Plans ■ Messages about queries, including join order and join algorithm • Output to dbname. iqmsg file • Default is OFF ■ Example: SET OPTION QUERY_PLAN = 'ON'; /* A look at customers who have the following */ /* service call waiting, caller id, and voice */ /* mail by fiscal period */ /* i.e. Q1,Q2,Q3,Q4 for year= 1998 */ select service_key,fiscal_period,count(*) from telco_facts T,month M where T.month_key=M.month_key and service_key = 4 group by fiscal_period,service_key order by fiscal_period,service_key ■ Results in IQ Message file: : Query Plan: :1 #07: Scrolling Cursor Store : Child Node 1: #05 : Output Vector: 3 entries (14 data bytes) :2 . #05: Order By : Child Node 1: #04 : Output Vector: 3 entries (14 data bytes) :3 . . #04: Group By (Hash) : Child Node 1: #03 : Output Vector: 3 entries (14 data bytes) :4 . . . #03: Join (Hash) : Left Child Node: #01 (continued . . .) 15-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plans ■ Results (continued): : Right Child Node: #02: Valid Join Algorithms: NLJ, SMJ, SMPDJ, HJ, HPDJ : Left Input Table 1: telco_facts AS T : Right Input Table 1: month AS M : Join Result Constraint: Many to 1 : Condition 1: (T.month_key = M.month_key) : Output Vector: 2 entries (6 data bytes) :5 . . . . #01: Vertical Cursor : Table Name: T : Table Row Count: 1200000 : Condition 1 (Invariant): (T.service_key = 4) : Condition 1 Selectivity: 0.10000000 : Output Vector: 2 entries (8 data bytes) :5 . . . . #02: Vertical Cursor : Table Name: M : Table Row Count: 24 : Output Vector: 2 entries (6 data bytes) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-19 Performance and Tuning IQ Query Plans ■ Here is what the IQ Query Tree would look like for: select service_key,fiscal_period,count(*) from telco_facts T,month M where T.month_key=M.month_key and service_key = 4 group by fiscal_period,service_key order by fiscal_period,service_key 15-20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plans ■ Query plans appear in the server log (.iqmsg) file • All user's query plans plus all other server messages are interleaved in the server log file • All plan lines in the server log are prefixed with the date, time connection id, and message number for query plan lines ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-21 Performance and Tuning IQ Query Plans ■ Other useful set options with query plan: • QUERY_DETAIL A Additional information, part of QUERY_PLAN • QUERY_NAME A Adds one line to the query plan with the name string supplied; makes it much easier to find your query plan in the server log file • QUERY_PLAN_AFTER_RUN A Delays query plan printing so that actual rows projected (and other information) can be included in the plan • QUERY_ROWS_RETURNED_LIMIT ▲ Limits the number of output rows from a query • NOEXEC A Generates the query plan but does not execute the plan 15 - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plan Detail ■ QUERY_DETAIL • When turned ON, additional information is displayed as part of QUERY_PLAN • Output to dbname. iqmsg file • Default is OFF ■ Example: SET OPTION QUERY_PLAN = 'ON'; SET OPTION QUERY_DETAIL = 'ON'; select service_key,fiscal_period,count(*) from telco_facts T,month M where T.month_key=M.month_key and service_key = 4 group by fiscal_period,service_key order by fiscal_period,service_key; ■ Results in IQ Message file: ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-23 Performance and Tuning QUERY_ROWS_RETURNED_LIMIT ■ A query will be rejected if the estimated number of rows is greater than the value of this option ■ Default is 0, no row limit ■ Example: SET OPTION QUERY_ROWS_RETURNED_LIMIT = 10; Result set from query was > 10 15-24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning NOEXEC and ROWCOUNT Options NOEXEC ■ Does not execute the query plan ■ Example: SET OPTION NOEXEC = 'ON'; ROWCOUNT ■ Sets the number of rows to return to the users ■ Example: SET ROWCOUNT 10; ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-25 Performance and Tuning IQ Query Plan - Elements ■ All of the Nodes in the Query Plan have a Node Type • Leaf Node A One table or one join index • Join Node A One of the join types • Group By A One of the grouping operator's • Order By A Sort operator used by sort-merge joins, subquery outer reference ordering and sort-based grouping, as well as the ORDER BY clause • Vertical Grouping Cursor ▲ Leaf node where grouping is being done in the indexes (continued . . .) 15 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Plan - Elements ■ Scrolling Cursor Store • Results cache that appear at the top of a query tree to support scrolling cursor access ■ Store • Results cache for invariant portions of correlated subqueries ■ Subquery • Executes correlated subqueries, or uncorrelated subqueries that could not be used directly with the parent query block indexes ■ Filter • Site of non-pushable non-join predicates ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-27 Performance and Tuning More Query Options Optimization Options ■ INDEX_PREFERENCE • Sets the index to use for query processing • This option will override the optimizer's index choice for testing purposes • Under most circumstances, it should not be changed ■ JOIN_OPTIMIZATION (default ON) • ASIQ optimizes the join order • If OFF, join order is determined by the order of tables ■ JOIN_MAX_HASH_ROWS (default 125,000 rows) • Sets the maximum estimated number of rows that the query optimizer will consider for a hash algorithm ■ MAX_CARTESIAN_RESULT (default 100,000,000 rows) • Limits the number of result rows from a Cartesian join ■ ROW_COUNTS (default OFF) • Specifies whether to count the number of rows in a query when it is opened ■ QUERY_TEMP_SPACE_LIMIT (default 2000MB) • Sets the maximum temp cache memory per user • Some queries may require more memory, set as needed Additional query optimization options 15 - 28 ■ There are additional options that can be adjusted. They are not listed because these options are designed primarily for internal Sybase use. ■ To see the complete list, see Chapter 12, "Managing System Resources," in the Adaptive Server IQ Administration and Performance Guide. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning IQ Query Performance Experimentation ■ The IQ Optimizer usually does a good ob of creating the query plan, but there will be times when you have a specific query that just runs too slow. Things you can try: • You can often influence the choice of algorithms by experimenting with the options: Join_Preference, Aggregation_Preference, and Index Preference • You can force the optimizer to accept your specified join order exactly by setting Join_Optimization off ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 15-29 Performance and Tuning Module Map • Define the IQ Query Engine • Improving Query Performance • IQ Query Plan ■ Improving Update Performance 15 - 30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Loading Data from a Fixed-Length ASCII Flat File Fastest Way to Load Data ■ Try to achieve a parallel load by: • Specifying the ROW DELIMITED BY in the LOAD TABLE command • AND, data must be delimited and have a newline col 1 | col2 | col3 |\n For more information on this topic, see Module 8, "Loading Data from Files." ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-31 Performance and Tuning Using Disk Striping ■ You should use disk striping whenever possible ■ Allows you to spread data from a single file across multiple disk drives ■ Significant performance gains over traditional file management ■ Disk striping is on by default • Will place data in round-robin fashion on all disks • Do not use if you intend to drop any dbspaces Internal striping 15 - 32 Provided by ASIQ via SET OPTIONS; refer to the Adaptive Server IQ Administration and Performance Guide for more information. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Using Multiple dbspaces ■ Create multiple dbspaces for your IQMain Store and IQ Temporary Store data ■ These files can be spread across multiple disks ■ Use CREATE DBSPACE command CREATE DBSPACE dbspace-name AS filename ... [ { IQ STORE | IQ TEMPORARY STORE | CATALOG STORE } ... [ [SIZE] file-size ] See Module 6, "Creating Tables and Indexes," for more information on the CREATE DBSPACE command. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-33 Performance and Tuning Spreading Access Across Separate Disks Place the data staging area on separate disks from the IQ Store, Catalog Store, and IQ Temporary Store 15-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Improving INSERT/UPDATE/DELETE Performance ■ Ensure you have some working space in your IQ Main Store ■ Also, ensure you have adequate room in the IQ Temporary Store ■ INSERT, UPDATE, DELETE, and SYNCHRONIZE commands need extra space ■ Certain deletions require a large amount of work space • HG indexes require temp cache for sorting • Make temp cache as large as possible to avoid paging to disk ■ After deleting data for faster incremental loading, you can avoid backfilling empty rows using APPEND_LOAD database option • Applies to LOAD TABLE or INSERT commands • This leaves unused row IDs in tables ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-35 Performance and Tuning Memory Allocation Queries and Updates ■ Calculate total memory available for cache sizes • Adjust main and temp caches accordingly; usually set once • If the database is not being used, you can adjust caches for different operations • Do not overallocate physical memory ■ Keep I/O to a minimum • Use large buffer caches, if available ■ Monitor memory usage during loads and queries ■ Running out of memory results in excess swapping (continued . . .) Monitoring swap ■ ■ ■ UNIX VMSTAT command. UNIX SAR command. Windows NT Task Manager. For more information on configuring memory, see Module 7, "ASIQ Memory Configuration." 15 - 36 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Memory Allocation Semaphores ■ In UNIX, if an ASIQ Server has an abnormal exit or a user has an ungraceful termination from an IQ session, a semaphore may be left behind in shared memory ■ To eliminate unneeded semaphores, you should run the ipcs command ■ Identify shared memory segments and semaphores that have a very old create date but have not been recently used with the ipcs -a command ■ Use the ipcrm with -m parameter to specify the memory segment ID and the -s parameter to specify the semaphore ID number ■ Example: % ipcrm -m mid1 -m mid2 ... -s sid1 -s sid2 ... Abnormal exit ©2000 Sybase, Inc. Killing processes may also result in semaphores or shared memory being left behind. Under normal circumstances, semaphores and shared memory segments are cleaned up automatically. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-37 Performance and Tuning Limiting Number of Active Queries on the IQ Server ■ The server startup parameter -iqgovern limits the number of queries executing concurrently • Default = (2 x number of CPUs) + 10 • Use this option to optimize memory usage, especially on servers with many CPUs and limited memory 15 - 38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Performance and Tuning Summary ■ To improve query performance: • Use a sensible schema design. • Follow the recommendations for where to create indexes and what types to create. • Define the IQ UNIQUE attribues on all columns < 10000. ■ To improve load performance: • Load from a flat file, in parallel, if possible. • Use disk striping. • Use multiple dbspaces. • Spread access across disks. ■ Ensure that memory is allocated correctly. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 15-39 Performance and Tuning Lab 15: Performance and Tuning Exercise Overview Topics ■ ■ Goals General tasks Lab setup 15 - 40 SET OPTION QUERY_PLAN Query Trees This lab explains how to use the QUERY_PLAN option. After completing this lab, you should be able to: ■ View a query plan ■ Draw the Query Tree ■ Set the QUERY PLAN option to ON. ■ Run a query. ■ Look at the results in the IQ Message Log. ■ Draw the query tree. ■ This lab assumes that the ASIQ Server is up and running with the megaphone database loaded with data. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 16 Backup and Restore Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Backup and Restore Module Overview This module discusses backup and restore procedures for ASIQ databases. Objectives After completing this module, you should be able to: ■ Back up ASIQ databases ■ Restore ASIQ databases ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-1 Backup and Restore Backup Features ■ Three types of online database backup: • Full • Incremental • Incremental-since-full ■ Use multiple drives for backups, allowing parallel backups ■ Can run concurrently with other database operations, except metadata changes ■ Process backs up committed data Full backup Copies a complete copy of the database. Incremental backup Copies all transactions since the last backup of any type. Incremental-sincefull backup Copies all transactions since the last full backup. Committed data A backup begins with an automatic checkpoint. All data that is not committed when this checkpoint occurs is not included in the backup. 16-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Archive Devices ■ The archive device can be either disk or a magnetic tape drive ■ Disk backups must go to file systems, not raw devices ■ Supported tape devices • Digital linear tape (DLT) on UNIX • 4mm DDS • 8mm ■ Stacker drives with multiple tapes ■ ©2000 Sybase, Inc. BACKUP does not support: • Fixed-length tape drives on UNIX systems, such as quarter inch cartridge (QIC) drives. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 6 -3 Backup and Restore Backup ■ First the Catalog Store is backed up for a database and then the IQ Store Temporary data 16-4 The IQ Temporary Store is not backed up. However, the metadata and any other information needed to recreate the Temporary IQ Store is backed up. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Types of Backups ■ Full • Full backup of Catalog Store • Full backup of IQ Store • Default action ■ Incremental • Full backup of Catalog Store • Backs up changes to IQ Store since last IQ backup of any type ■ Incremental-since-full • Full backup of Catalog Store • Backs up changes to IQ Store since last full IQ backup Transaction Log ©2000 Sybase, Inc. The Catalog Server Transaction Log file is used for recovery purposes. This log is not used to restore the database, recover committed IQ transactions, or restore the Catalog Store. The restore program does check for existence of the log: ■ Full restore - The log must not exist. It must be deleted (or renamed or moved) before starting. ■ Incremental restore - The log must exist. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-5 Backup and Restore Running a Backup ■ Attended • Operator must be present • Prompts to mount archive media • Default action ■ Unattended • No prompts are issued • You must make appropriate size estimates and set up devices in advance 16-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore BACKUP DATABASE Syntax BACKUP DATABASE ... [CRC ON | OFF] ... [ATTENDED ON | OFF] ... [BLOCK FACTOR integer] ... [{FULL|INCREMENTAL|INCREMENTAL SINCE FULL}] ...TO 'archive_device' [SIZE #_of_KB] [ STACKER #_of_tapes_in_stack] ... [WITH COMMENT 'string'] ■ CRC - Sets 32-bit cyclical redundancy checking to ON ■ BLOCK FACTOR - Number of blocks to write at a time CRC ■ ■ BLOCK FACTOR The default is 25. SIZE & STACKER These options are discussed in the following slides. ©2000 Sybase, Inc. Activates 32-bit cyclical redundancy checking on a per-block basis. Used during a backup and automatically verified during a restore, affecting performance on both commands. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-7 Backup and Restore Example: Simple Backup ■ UNIX example: BACKUP DATABASE TO '/dev/rmt/0n' ■ Makes a full backup of the database to one tape device ■ The Catalog Store is backed up first, then the IQ Main Store 1 6 -8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore SIZE and STACKER Options ■ SIZE (KB) • Maximum size of the backed up data on that stripe • Should specify for unattended tape backups on platforms that do not reliably detect the end-of-tape marker • Should also specify whether backup from disk is not the default of 2GB ■ STACKER • Indicates that you are backing up to an automatically loaded multitape stacker device • Specify the number of tapes in that device Do not specify SIZE or STACKER if you are using a third-party backup product, because size information is conveyed in the vendor_specific_information string. To specify this string, see the section "Performance Backups with Non-Sybase Products," in the Adaptive Server IQ Administration and Performance Guide. The value of SIZE must be a multiple of 64. If you specify a SIZE that is not a multiple of 64, it will automatically be rounded up to a multiple of 64. If SIZE is not specified it is automatically set to 1.5GB. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 6 -9 Backup and Restore Example: Incremental Backup ■ UNIX example: BACKUP DATABASE INCREMENTAL TO '/dev/rmt/0n' SIZE 150 WITH COMMENT 'March 17 incremental backup of mydb database' 16-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Backup: Notes and Restrictions ■ You need Database Administrator privileges to run a backup ■ ASIQ does not rewind tapes before using them ■ After a backup, label each archive device ■ System-level backups are not adequate • A restore from a system-level backup could cause data loss or corruption ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-11 Backup and Restore Validating Your Database ■ Should run sp_iqcheckdb to make sure the database is in a usable state • Example on next page ■ Reads every database page from disk into memory and does various consistency checks ■ Statistics displayed on the IQ Store followed by the Temporary Store ■ Can take a long time on large databases This stored procedure consumes most of the database server's time. Run during nonpeak hours. Do not run sp_iqcheckdb while another user is doing inserts and deletes. Results may be misleading. sp_iqcheckdb can falsely report leaked blocks if there are multiple versions of any given table. 16-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore sp_iqcheckdb ■ Key statistics to look for are the orphaned blocks • These statistics have an asterisk next to them • If there is a non-zero value for any of these statistics, your database probably has a serious consistency problem • To get accurate results, run a checkpoint first ■ Partial results: ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-13 Backup and Restore Restore Features ■ Allows parallel restore using multiple tape drives ■ Can restore full database ■ Can restore from incremental database backup ■ Can restore to a new location • Useful for a lost device 16-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Before You Restore... ■ You must have Database Administrator privileges ■ You must be connected to the utility_db database ■ No user can be connected to the database being restored ■ For a full restore, the Catalog Store and the Catalog Transaction Log (.db and .log files) must not exist ■ For an incremental restore, the Catalog Store and the Transaction Log must exist To ensure exclusive access, start the database server with the -gd Database Administrator option set, but do not start the database you are restoring. utility_db To connect to the utility database: ■ From the command prompt, use the following command: asiqsrv12 -n mydb ■ ■ This command starts the database server but does not load a database— in this case, mydb. In Interactive SQL, select utility_db as the database name when you connect. The user ID and password are DBA and SQL, respectively. Catalog Store This is the DB file. This Catalog Transaction Log is the LOG file. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-15 Backup and Restore Restore Database ■ Syntax: RESTORE DATABASE 'db_file' FROM 'archive_device'[FROM archive_device] ... [RENAME dbspace_name TO 'new_dbspace_path']... ... [CATALOG ONLY] ■ db_file • Relative or absolute path ■ CATALOG ONLY option • Displays the header information only for the database dump ■ RENAME option • Specifies which files to restore and to which dbspaces • Cannot rename the transaction log using this option When you rename or move all other files in the database, it is preferable to rename the log file. To move or rename the log file, use the DBLOG utility. 16-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Restore Database: Full Restore ■ NT example: RESTORE DATABASE 'mydb.db' FROM 'C:\\asiq\\backup1' FROM 'C:\\asiq\\backup2' ■ The mydb database is restored from two disk files ■ All database files will be restored to their original locations ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-17 Backup and Restore Moving Database Files Two Methods ■ To move the database file that holds the Catalog Store (by default, the DB file), specify the new name as db_file m Example: RESTORE DATABASE 'c:\\newdir\\mydbnew.db' FROM 'C:\\asiq\\backup1' FROM 'C:\\asiq\\backup2' ■ For any other database file, use the RENAME option • See example on next foil (continued . . .) 16-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Moving Database Files ■ Example: RESTORE DATABASE 'salesdb.db' FROM '/dev/rmt/0n' RENAME '/s1/salesdb.1' TO '/s2/salesdb.1' ■ Useful if one or more dbspaces are lost due to disk failure Example ©2000 Sybase, Inc. This example restores the file /s1/salesdb.1 to a new location /s2/salesdb.1 and the file /s1/salesdb.2 to the same location. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-19 Backup and Restore Restore Database Displaying Header Information ■ Example: RESTORE DATABASE 'salesdb.db' FROM '/dev/rmt/0n' CATALOG ONLY ■ This option displays only the header information for the database and does not restore any data 16 - 20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Restore Database: Notes and Restrictions ■ You must restore full and incremental backups in the correct order ■ If the database is corrupt or if you are moving files, you must restore a full backup ■ To ensure tapes have been restored correctly, run sp_iqcheckdb ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-21 Backup and Restore ASIQ Backup and Restore Performance ■ Multiple tape drives provide the best performance ■ Set CRC OFF in the BACKUP command to improve the speed of backup and restore ■ Increase memory during backup • Adjust BLOCK FACTOR ■ Keep the size of the Catalog Store small • Creating non-IQ tables in it will increase backup and restore time ASIQ data is already in a compressed format. Hardware compression adds little in the way of performance or additional capacity. 16 - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Estimating Dump Size ■ ASIQ backs up compressed data (no zeros) ■ sp_iqstatus • Main IQ Blocks Used output gives size of compressed data ■ Use 11880 (Main IQ Blocks Used) * 4096 (Block Size in bytes) = Total Bytes ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-23 Backup and Restore DBLOG Utility ■ Should run this utility after: • Using RESTORE with a new database name • Using RESTORE with RENAME ■ Allows you to display or change the name of the Transaction Log or mirror associated with the database ■ Syntax: dblog [switches] database-file ■ Switches • -m mirror-name - Set transaction log mirror name • -l log-name - Set the transaction log name 16 - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Option for Backing Up a Large Database ■ Some databases may be too large to ever back up ■ Consider alternatives • Archive the source data for reloading • Use IQ Data Extraction utilities to selectively extract data ▲ Limit output to WHERE clauses A Take periodic "pictures" of tables and archive ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 16-25 Backup and Restore Summary ■ Determine a data backup and recovery strategy. ■ Schedule routine backups. ■ Mix full and incremental backups. ■ Test restores on a periodic basis. ■ Use sp_iqcheckdb to validate the database before a backup and after a restore. 16 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Backup and Restore Lab 16: Backup and Restore Exercise Overview Topics ■ BACKUP DATABASE statement ■ Connecting to the utility_db ■ RESTORE DATABASE statement ■ sp_iqcheckdb stored procedure Goals This lab explains how to back up and restore data from ASIQ. After completing this lab, you should be able to: ■ Back up an ASIQ database to file ■ Connect to the utility database ■ Restore an ASIQ database from file General tasks Lab setup ©2000 Sybase, Inc. ■ Back up the megaphone database to file. ■ Disconnect from the megaphone database. ■ Rename the Catalog Store and Transaction Log files for a full restore. ■ Start the server using the utility_db database. ■ Reload the megaphone database from file. ■ Run sp_iqcheckdb to ensure the database is in a usable state. ■ This lab assumes that you have created the megaphone database and have created all the tables and indexes in that database. Data should be loaded into the database. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 16-27 17 Multiplexing Overview Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview Module Overview This is an introduction module for ASIQ multiplexing Objectives After completing this module, you should be able to: ■ Explain the features of ASIQ multiplexing ■ Understand the architecture and terminology for ASIQ multiplexing ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 17-1 Multiplexing Overview What Is IQ Multiplexing? ■ An IQ multiplex consists of: • A number of nodes, each running ASIQ • A shared disk subsystem connected to all nodes • One node designated as the writer server • Other nodes designated as query server ■ Extends the speed, flexibility, and low cost of ownership of ASIQ to a multinode configuration ■ Extends support to multiterabyte databases, many billions of rows, and hundreds of simultaneous queries with high availability 1 7 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview Multiplex Architecture Nodes ■ ■ ■ ■ ©2000 Sybase, Inc. Individual nodes can be different configurations (CPUs, memory, disk). Must be all UNIX or NT, can not mix. Can have up to 120 nodes. One write server that performs updates. The IQ Store must be on a raw device. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17-3 Multiplexing Overview Multiplex Features ■ Data is shared among nodes ■ Each IQ node has its own local Temporary and Catalog Store ■ All data is stored in the shared IQ database, which is typically on a fiber channel ■ Could have multiple instances of IQ on one node without a fiber channel ■ 95% performance scaling ■ SMP-like management and tuning (continued . . .) 17-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview Multiplex Features ■ No interruption in database if node crashes ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17-5 Multiplexing Overview ASIQ Architecture (Simplex) Review ■ An ASIQ database consists of three main components: IQ Store Contains the IQ data as well as the database transaction log. Catalog Store Contains table definitions, system tables, and stored procedures. IQ Temporary Store 17-6 Contains temporary tables generated by certain queries. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview ASIQ Architecture (Multiplex) ■ Each server has its own IQ Temp Store and Catalog Store and shares the IQ Main Store ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17-7 Multiplexing Overview Write and Query Servers ■ Write Server • There is only one in an ASIQ Multiplex configuration • Handles all loading or updates to the database while queries are submitted • Can be used as Reader ■ Query Server • Read-only servers submitting queries • Also know as Read or Reader Server One Write Server 17-8 By allowing a single write server, the overhead and scalability limits of a distributed lock manager is eliminated. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview Communication between Servers ■ The IQ Catalog Store data is copied from the Write Server to the Reader Servers ■ This is done though a process called Synchronization, further discussion to follow ■ SQL Remote is the data-replication technology shipped as part of ASIQ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17 -9 Multiplexing Overview Multplex Operating Modes Two Modes ■ Normal • Readers may acces new versions of tables as soon as they are available • Use this for best performance and availability ■ Strict • Maintains a "single version of the truth" • Readers will not have access to the new versions until all readers have access to a new version 17-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview IQM Advantages ■ Scalability is unlimited ■ Start small, easy to grow ■ Can handle hundreds to thousands of concurrent queries ■ High availability ■ No need for extensive training ■ Low cost of ownership • Data compression • Low maintenance • No ongoing tuning of databases ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17-11 Multiplexing Overview Summary ■ Use an IQ multiplex if you want to: • Scale query performance past limits of one node. • Match I/O to CPU capabilities. • Have high availability. ■ You can start small and add more nodes. 17-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplexing Overview Lab 17: Multiplexing Overview Exercise Overview Topics ■ Multiplex Architecture ■ Write Server vs Query Server ■ Strict vs Normal Mode ■ Features ■ Advantages Goal After completing this lab, you should be able to: ■ Explain the basic principles of ASIQ Multiplexing General tasks You will take a written quiz. Lab setup No lab setup is required. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 17-13 18 ASIQ Multiplex Hardware Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware Module Overview This module describes hardware storage for ASIQ multiplex. Objectives After completing this module, you should be able to: ■ Understand storage basics ■ Identify the correct hardware needs for ASIQ multiplex ©2000 Sybase, Inc. Administering Adaptive Server IQ for the DataWarehouse, Version 1.0 18-1 ASIQ Multiplex Hardware Module Map ■ Storage Overview • Storage Basics • RAID Basics ■ Configuring Storage 18-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware Storage Basics ■ Disks are getting bigger and faster ■ Common disk sizes are 9GB, 18GB, 36GB and 72GB ■ Common rotation speeds are 7200 rpm and 18,000 rpm ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 8 -3 ASIQ Multiplex Hardware RAID Basics ■ Redundant Array of Inter-connected Disks (RAID) ■ RAID Controller performs many functions: • Connects and manages many disks • Presents many disks as ONE or MORE larger "virtual" devices • Performs performance optimizations • Improves reliability • Other functions: Remote mirroring, snapshoting, mirror management, etc ■ All those functions are invisible to host I/O system ■ ■ 1 8 -4 RAID "mimics" and behaves like a single device. Hosts can NOT tell a difference between a RAID and "genuine" disk. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware RAID Diagram ■ Host computer only communicates with RAID Controller RAID Controller handles individual disk I/O and is transparent to host. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 18-5 ASIQ Multiplex Hardware RAID Levels ■ RAID Controller improves reliability by adding parity or even mirroring data inside a given "virtual" device ■ There are variations called RAID-level: • RAID0: Striping • RAID1: Mirroring • RAID0+1: Striping + Mirroring • RAID5: Parity • RAID-S: EMC variant of RAID5 1 8 -6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware RAID Striping ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 18-7 ASIQ Multiplex Hardware RAID Connectivity ■ Raid Controller connect to host system using various connections • Most common are SCSI, Fibre Channel (FC), ESCON and SSA 1 8 -8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware Module Map • Storage Overview A Storage Basics A RAID Basics ■ Configuring Storage ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 8 -9 ASIQ Multiplex Hardware Configuring Storage ■ Shared disk storage is the foundation for IQ Multiplex, along with: • Top Level Shared Directory • Shared IQ Main Store ■ IQM is based on multiple nodes being able to physically access shared storage devices ■ Recommend shared storage configuration with one or more Fiber Channel connected RAID Controllers connected as RAW devices to all nodes (hosts) in IQ Multiplex IQM would not perform as well in an NFS, NAS, or network drive environment. Even a cluster file system (available on some platforms) would perform poorly in an IQM environment. 18-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware IQM and Shared Disk Farm ■ Raw devices on for IQ Store ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 18-11 ASIQ Multiplex Hardware IQ Multiplex Hardware... Putting it together 18-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. ASIQ Multiplex Hardware Summary ■ Multiplexing takes advantage of a shared disk subsystem connected with fiber channel ■ Additional CPU power and memory space for processing queries is gained when attaching more systems to the shared disk array ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 18-13 ASIQ Multiplex Hardware There is no lab for this module. 18-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. 19 Creating a Multiplex Database Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Module Overview This module outlines the steps to set up a multiplex environment using Sybase Central from scratch Objectives After completing this module, you should be able to: ■ Create a Write Server ■ Create a Query Server ■ Create Temp Stores ■ Synchronize ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-1 Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Query Server The final steps are to create DDL and load data. 1 9 -2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Step 1 - Create the Write Server ■ Using Sybase Central, open up Utilities Folder ■ Run Create Muliplex Wizard ■ Enter appropriate information for: • Host Name • Shared Top Level File System Directory • Device for the IQ Main Store • Name for the IQ Write Server • Port number • Database name ■ Screen shots on next few pages (continued . . .) Before you start the wizard you need: • A shared directory with space for multiplex control information. Approx. 30MB per node is required. • Shared raw device(s) for the IQ Main Store. Typically a fiber channel disk array. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 9 -3 Creating a Multiplex Database Create the IQ Write Server Open the Utilities Folder 1 9 -4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create the IQ Write Server Run the Create Multiplex Wizard • Enter the Host Name for the Write Server ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 9 -5 Creating a Multiplex Database Create the IQ Writer Server ■ Specify the shared top level directory and the initial raw device for the IQ Main Store 1 9 -6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create the IQ Writer Server ■ Name the Write Server (MPXServer01 is the default) ■ Specify a name for the IQ Catalog Store ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-7 Creating a Multiplex Database Create the IQ Writer Server ■ Do not specify size since the device is raw (not accessible) ■ Use the default IQ Page Size, in this example 1 9 -8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create the IQ Writer Server ■ In the example, we use the default userid, password and port number ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 1 9 -9 Creating a Multiplex Database Create the IQ Writer Server ■ Summary of options for multiplex write server ■ Click finish to create write server ■ The write server will be running in multiplex mode Screens for specifying Case Sensitivity, Ignore Trailing Blanks, Install Java Classes and Sort Order do exist, but were not shown. 19-10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Reader ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-11 Creating a Multiplex Database Step 2 - Create Reader (Query) Servers ■ Write server must be in Multiplex mode ■ Open Sybase Central - Utilities folder ■ Run Create Reader Wizard ■ Enter appropriate information for: • Host Name for the Query Server • Name for Query Server • Share Directory name • Raw Store Device • Port number In order to create a query server you must have created a multiplex server 19-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create Query Server ■ Open the Utilities folder ■ Select Create Query Server ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-13 Creating a Multiplex Database Create Query Server ■ Specify the Host Name for the Query Server ■ Give a name for the Query Server 19-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create Query Server ■ Specify the shared top level directory for the Multiplex ■ Specify the IQ Store raw device ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-15 Creating a Multiplex Database Create Query Server ■ Specify the port number (2638 is the default) Be sure to change the port number if building the Write and Query Servers on the same node. 19-16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Create Query Server ■ Summary of options for the Query Server ■ Click Finish to create a IQ Multiplex Query Server This step can be repeated for the number of Query Servers that will be in the IQ Multiplex configuration ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-17 Creating a Multiplex Database Directory Structure ■ The shared directory contains: • A directory for each server • A subdirectory for the database After this wizard completes you must synchronize in order to use the Query Server. Synchronization is next... 19-18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Query Server ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-19 Creating a Multiplex Database Step 3 - Synchronize ■ Synchronization Process does the following: • Overwrites all Query Server Catalog Stores with Writer Catalog Store • Resets all SL Remote Control • Reboots all Servers ■ Needs to be done: • Before you start the multiplex for the first time • After the schema has changed (Any DDL changes) • When new query servers are added • After a restore 19 - 20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Synchronize ■ Open the Multiplex Utilities folder in Sybase Central ■ Select Synchronize the Multiplex ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-21 Creating a Multiplex Database What we have so far... ■ Using Sybase Central, we have a write server called MPXServer01 and one query server called Query01 19-22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Query Server ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-23 Creating a Multiplex Database Why Add A Temp Store? ■ After creating a Write Server there is only a 10MB Temp Store • Much too small to perform any database task ■ Must add Temp Store to Writer before: • Creating any DDL • Loading any data 19 - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Step 4 - Add Temp Store for Writer ■ In Sybase Central, access the Multiplex folder ■ Highlight the Writer Server (MPXServer01) ■ Right click and access Properties The temporary store can be raw or file, in this example we show file. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-25 Creating a Multiplex Database Add Temp Store for Writer ■ Click Add DB Space 19-26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Add Temp Store for Writer ■ Follow Wizard ■ Enter name of the IQ Temporary db space ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-27 Creating a Multiplex Database Add Temp Store for Writer ■ Choose the type of dbspace, IQ Temporary data The Main store choice will be locked when the Writer is in Mulitplex mode. 19-28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Add Temp Store for Writer ■ Create the Temp Space ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-29 Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Query Server 19 - 30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Step 5 - Add Main IQ Store dbspaces ■ First need to start the Server in Simplex mode ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-31 Creating a Multiplex Database Start Server in Simplex Mode 19-32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Add Main IQ Store dbspaces ■ Access the multiplex folder • Highlight the Writer server (MPXServer01) • Right click and access Properties • Run the Add dbspace Wizard ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-33 Creating a Multiplex Database Add Main IQ Store Dbspaces ■ Click Add DB Space 19-34 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Add Main IQ Store Dbspaces ■ Create a raw dbspace for the IQ Main Store ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-35 Creating a Multiplex Database Add Main IQ Store Dbspaces ■ Select type of dbspace - Store IQ data In Simplex mode, the Temp Store can not be selected. 19-36 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Add Main IQ Store Dbspaces ■ Leave device size blank as these are all raw devices ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-37 Creating a Multiplex Database Add Main Store Dbspaces ■ We now have ... 19-38 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Stop the Writer Server ■ We are in Simplex Mode, need to stop the write server and Synchronize ■ Access the multiplex folder • Highlight the Writer server (MPXServer01) • Right click and select Stop Be sure to synchronize to ensure the Query Servers know about the new Main Store dbspace. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-39 Creating a Multiplex Database Steps to Set up Multiplex 1. Create the IQ Write Server 2. Create Query Servers 3. Synchronize 4. Add Temp Store for Writer 5. Add Main IQ Store dbspaces 6. Add Temp Store for Query Servers 19 - 40 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database Step 6 - Add Temp Store for the Query Server ■ Similar Steps as for adding temp store for Writer ■ Start the Write Server and all the Query Servers in multiplex mode ■ Using Sybase Central, access the Multiplex folder • Highlight the Reader Server • Right click and access Properties • Run the Add dbspace wizard • Create the Temp Space The Temp space may be raw or file system. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-41 Creating a Multiplex Database Summary ■ Follow the steps using Sybase Central • Create the IQ Write Server • Create Reader Servers • Synchronize • Add Temp Store for Writer • Add Main IQ Store dbspaces • Add Temp Store for Reader • Ready to create DDL and load data! 19 - 42 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Creating a Multiplex Database There is no lab for this module. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 19-43 20 Multiplex Operations Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Module Overview This module provides administration topics relevant to the multiplex environment Objectives After completing this module, you should be able to: ■ Create Database Object, Set Options, Load Data, Run Queries ■ Start and Stop Servers ■ Add/Drop Dbspaces ■ Backup and Restore ©2000 Sybase, Inc. 1 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20- Multiplex Operations Working in the Multiplex Environment ■ Getting Started • Creating Database Objects • Setting Memory • Loading Data • Running Queries ■ Start and Stop Servers ■ Add/Drop Dbspaces ■ Backup and Restore 20-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Creating Database Objects ■ All schema changes must be done on the write server in simplex mode ■ Steps: 1. Backup the database, if needed 2. Start the write server in simplex mode 3. Create objects interactively or with scripts (same as any ASIQ database) 4. Synchronize the multiplex 5. Backup the database (multiplex mode) Creating Objects ©2000 Sybase, Inc. 3 See Module 6, "Creating Tables and Indexes" and Module 13, "Joins" Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20- Multiplex Operations Setting Memory ■ In the top level shared directory modify the params.sql file for all servers to set memory ■ Add the -iqmc and -iqtc switches • -iqmc = main cache size A Specify main cache size in MB • -iqtc = temporary cache size A Specify temp cache size in MB ■ To take effect you must restart all servers params.sql More information on this file is coming in this module. Memory settings See Module 7, "ASIQ Memory Configuration" 20-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Loading Data ■ You can use any of the following methods • Loading from flat files • Inserting specified values • Inserting selected rows from a database ■ You can be in either simplex or multiplex mode • After running in simplex mode you must synchronize Basic loading techniques ■ Module 8, "Loading Data from Files", covers: • fixed-and variable-length flat-file loads. Advanced loading techniques ■ Module 9, "Advanced Data Loading from Files", covers: • Loading from multiple files. • Partial-width loads. • Adding a dbspace during a load. Insert techniques ■ Module 10, "Loading Data Using the INSERT Command," covers inserting from foreign databases. ■ If Loading in Simplex mode, you must synchronize in order to have new versions available on the Query Servers. If Loading in Multiplex mode, synchronization is done automatically by SQL Remote. ■ ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20-5 Multiplex Operations Running Queries ■ Transparent to users that this is a multiplex environment ■ Should set the 'Commit After Select' in the dbisql GUI under Command on menu ■ Or issue a Commit on the Reader ■ You many notice a delay of up to a minute to see changes that were made on the write server • Depends on Write Server's mode (strict vs normal) Using ASIQ 20-6 ■ Module 11, "Using ASIQ," covers SQL statements and functions specific to ASIQ. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Specifying Database Options ■ If you want to specify different database options on one particular server you can edit a script called postsync.sql m Allows you to specify user logins, permissions, etc. which are unique to a particular query server ■ This file runs automatically after synchronization ■ Any option set on the Write Server will be also set on all Query Servers • After Synchronization, the Writer's catalog is copied to all Query Servers ■ Add node specific options in the postsync.sql script to have local options for each server • Cannot set cache sizes using this script (use params.sql) ©2000 Sybase, Inc. 7 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20- Multiplex Operations Postsync.sql Script ■ All servers have this script in their directory ■ This script is executed during synchronization ■ This script contains one entry which should not be removed ■ Add other SQL commands as needed ■ Restrictions: • No DDL or DML • All commands need to be valid SQL commands fro ASIQ 20-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Working in the Multiplex Environment • Getting Started • Creating Database Objects • Setting Memory • Loading Data • Running Queries ■ Start and Stop Servers ■ Add/Drop Dbspaces ■ Backup and Restore ©2000 Sybase, Inc. 9 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20- Multiplex Operations Setting Startup Parameters ■ There is aparams.cfg file that sets options to control server startup ■ This script is located in the database directory for the server ■ It is automatically run when you start a server ■ Example: -n MPXServer01 -c 16m -gd all -gm 10 -gp 4 0 96 -ti 4 4 0 0 -tl 3 0 0 -iqmc 50 0 -iqtc 5 0 0 - x tc pip{ por t=2638} Server switches ■ ■ ■ ■ ■ ■ ■ 20 -10 -n = server name • Required if server is started without a database -c = cache size for Catalog Store • Default = 2 MB • Recommend 16MB to keep entire catalog in cache -gm = limits number of concurrent users • Default = 10 Recommend to set as needed, up to maximum in license -gp = Catalog store page size • Default = 1024 • Recommend 4096 to allow more table columns -ti = default client timeout (minutes) • Interval to disconnect inactive clients • Default = 240, recommend 4400 (~73 hours) -tl = default network timeout (seconds) • Interval a "liveness" packet is sent to a client • Default = 120, recommend 300 or higher Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Starting and Stopping ■ Sybase Central is the easiest way ■ Open the utilities folder and run Start Multiplex ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 2 0 -11 Multiplex Operations Select the Start Mode ■ Choose how to start the servers 20-12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Stopping Multiplex Server ■ To stop the entire multiplex: ■ Choose Stop Multiplex (wrong option is highlighted here) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20-13 Multiplex Operations Stopping Individual Servers ■ Open the database folder in Sybase Central and select the desired server ■ Right click on the server and choose stop from the menu ■ You can also stop individual servers on each node using dbstop (NT) or stop_asiq (UNIX) 20-14 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Managing Servers ■ To view the current state of your server open the Multiplex folder and right-click on the desired server ■ From the popup menu, you can: • Start • Stop • Delete • Properties Delete ©2000 Sybase, Inc. ■ ■ ■ You can only delete query servers You must be running in simplex mode to delete a query server The delete will fail if another user is logged in as DBA Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20-15 Multiplex Operations Restarting Servers ■ If one of the servers in your multiplex becomes unavailable you will need to shut down your multiplex and restart it ■ If the write server fails and can not be restarted you can specify one of the query servers to be the new write server • See Chapter 3 "Managing the Multiplex Environment", in the Adaptive Server IQ Multiplex User's Guide Must Synchronize anytime DDL is performed on the Write Server. 20 -16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Working in the Multiplex Environment • Getting Started • Creating Database Objects • Setting Memory • Loading Data • Running Queries • Start and Stop Servers ■ Add/Drop Dbspaces ■ Backup and Restore ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20-17 Multiplex Operations Adding and Dropping Dbspaces ■ Review: There are two types of dbspaces • permanent (IQ Main Store) • temporary (IQ Temporary Store) ■ Rules • You can not add or remove permanent dbspaces from the database while a multiplex is operating • For a permanent dbspace, the write server must be in simplex mode • For a temporary dbspace, all servers must be started in multiplex mode • To drop temp space on any server contact tech support 20 -18 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations To Add A Permanent (Main) Dbspace Steps: 1. Start the write server in simplex mode 2. Connect to the database and open the Multiplex folder 3. Click on the server name and choose File Properties 4. Click Add Dbspace 5. Follow the wizard 6. Synchronize Add Dbspace ©2000 Sybase, Inc. See Module 19, "Creating a Multiplex Database", page 35-39 for detailed screen shots of the Add Dbspace wizard to create a permanent IQ dbspace. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20-19 Multiplex Operations To Create a Temporary Dbspace Steps 1. Start all the servers in multiplex mode 2. Follow steps 3-5 on the previous page • Note: You do not have to synchronize Add Dbspace 20 - 20 See Module 19, "Creating a Multiplex Database", page 26-30 for detailed screen shots of the Add Dbspace wizard to create a temporary IQ dbspace. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations To Drop a Main or Temporary Dbspace Steps 1. Run Multiplex --> Utilities --> Stop Multiplex 2. Start the write server in simplex mode 3. Start dbisql and run DROP DBSPACE 4. Run Multiplex —-> Utilities —-> Synchronize Multiplex You cannot drop a main dbspace if it is striped and has been loaded. DROP DBSPACE ©2000 Sybase, Inc. See Module 4, "Creating Databases", page 31 for syntax. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20 - 21 Multiplex Operations Working in the Multiplex Environment • Getting Started • Creating Database Objects • Setting Memory • Loading Data • Running Queries • Start and Stop Servers • Add/Drop Dbspaces ■ Backup and Restore 20 - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Backup the Multiplex Steps 1. Log onto the Write Server 2. Run the BACKUP command 3. Do a regular file system backup of the partition where the top level share directory is or 1. Stop Write Server 2. Copy all files and dbspaces using OS Level Backup 3. Start Write Server BACKUP command ©2000 Sybase, Inc. See Module 16, "Backup and Restore", for syntax. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20 - 23 Multiplex Operations Restore the Multiplex ■ Perform a restore only when instructed by Technical Support ■ Three possible areas of data failure: • The Main IQ Store, on a shared disk array • The top level shared directory, on an NFS partition or shared disk • One or more .iqtmp files, on local disks ■ Try doing a synchronize first 20 - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Multiplex Operations Summary ■ Everything is driven through Sybase Central ■ Working in the Multiplex environment is very similar to working with one node ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 20 - 25 Multiplex Operations There is no lab for this module. 20 - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. A Migrating Data from Prior Versions Administering Adaptive Server IQ for the Data Warehouse Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Appendix Overview This appendix provides the necessary background information for migrating data from Adaptive Server IQ version 11.x into Adaptive Server IQ 12.0. Objectives After completing this appendix, you should be able to: ■ Understand the different methods to migrate data from prior versions ■ Choose a method and move data into ASIQ 12.0 from earlier versions ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A -1 Migrating Data from Prior Versions Why Is Data Migration Necessary? ■ The disk format of Adaptive Server IQ 12.0 is different than that of version 11 .x • Need to move data into 12.0 database A-2 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Migration Steps 1. Install ASIQ 12.0 2. Create the 12.0 database • Create dbspaces • Create tables 3. Load the data into the new 12.0 database ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-3 Migrating Data from Prior Versions Step 1 - Install Version 12.0 ■ See Module 3, "ASIQ Environment and Installation," on how to install ASIQ Version 12.0 ■ May want to upgrade your current Sybase IQ Server to Version 11.5.1EBF 0007 or higher • This provides a parallel unload utility and a command that generates migration scripts A-4 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Step 2 - Create 12.0 Database Two Methods ■ Method 1 - Copy and modify old scripts to build new database ■ Method 2 - Use the database creation script created by the IQ CREATE ASIQ TEMPLATE utility ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-5 Migrating Data from Prior Versions Method 1 Copy and Modify Old Scripts ■ Copy old scripts from 11 .x IQ Server and modify ■ Need to create a 12.0 database • See Module 4, "Creating Databases" ■ Need to create dbspaces • See Module 4, "Creating Databases" ■ Need to create tables and indexes • See Module 6, "Creating Tables and Indexes" • Review Module 5, "ASIQ Index Types and Data Types," for 12.0 modifications A-6 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Method 2 Database Creation Script ■ To create this script and other migration scripts, the existing ASIQ Server must be 11.5.1 EBF 0007 or higher ■ Must have Open Client 11.1 .x loaded ■ Easiest method ■ The IQ CREATE ASIQ TEMPLATE command creates three scripts for you to use in migration • Database creation script • Data extraction script • Data load script ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-7 Migrating Data from Prior Versions IQ CREATE ASIQ TEMPLATE ■ Syntax: IQ CREATE ASIQ TEMPLATE PATHNAME = name [APPEND][CREATE] [USEGO] [USESEMICOLON] ■ Example: IQ CREATE ASIQ TEMPLATE PATHNAME = "/home/usr/u/den101/migrate.sql" CREATE USESEMICOLON PATHNAME Required. Specifies the directory and filename for the generated script. Use single or double quotes to enclose name. APPEND| CREATE Specifies whether to append the newly generated script to the existing pathname or create a new file. USEGO| USESEMICOLON Specifies the SQL statement terminator. Either "go" or ";". A-8 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Edit Scripts General Guidelines ■ Divide the output into three files (Search for the word "edit"): • Database creation script • Data extraction script • Data load script ■ Replace all instances of "???" with the appropriate text • In the script there are comments with examples of what text should go there ■ Make any additional edits, as required ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-9 Migrating Data from Prior Versions Database Creation Script ■ Creates a database with the same name ■ Creates dbspaces ■ Creates tables • Automatically creates the columns in each table alphabetically, regardless of the order in which they were created ■ Edit the database creation script carefully! (continued . . .) If you have permanent private indexspaces in Sybase IQ 11.x, remove those from the script. A -10 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Database Creation Script ■ In ASIQ 12.0 database defaults have changed • You need to specify CASE RESPECT to get a case-sensitive database • You need to specify COLLATION ISO_BINENG to get the ASCII-style collation • The new default behavior is that trailing blanks are significant ■ See Module 4, "Creating Databases," for syntax and examples (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A -11 Migrating Data from Prior Versions Database Creation Script ■ The first part of the script creates the database and connects to the database ■ Example: CREATE DATABASE ‘my_db' CASE IGNORE BLANK PADDING ON IQ PATH ‘my_db' IQ SIZE 1172 BLOCK SIZE 4096 IQ PAGE SIZE 6553 6; ■ There are no .TI files in ASIQ 12.0; therefore, space used for .TI files in 11.5.x should be allocated in the 12.0 main dbspace (continued . . .) A -12 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Database Creation Script ■ The next part of the script creates additional dbspaces ■ Example: CREATE DBSPACE my_db_1 AS '/users/my_db_1' IQ STORE 1172; (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-13 Migrating Data from Prior Versions Database Creation Script ■ The final part of the script creates tables and indexes ■ Columns in tables are in alphabetic order, regardless of the order they were on your 11.5.x database • May want to change the order and make it match the 11.5.x order ■ Check all indexes to make sure they are correct • Do not use LD indexes; future releases will not support this index ■ Example: CREATE TABLE customer (name varchar(40) NOT NULL ,address varchar(40) NOT NULL ,phone char(10) NOT NULL...) LD index A -14 When a LD index is specified, it is currently replaced with an HG index. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Module Map Step 3 - Loading Data ■ Use migration scripts ■ Load the data from your original source ■ Insert the data from an indexspace ■ Select the data into a file and load Loading data ©2000 Sybase, Inc. Some of these options are not available depending on various factors. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-15 Migrating Data from Prior Versions Migration Scripts ■ Use the data extraction script • Created using the IQ CREATE ASIQ TEMPLATE command • Run this script on Version 11.5.1 EBF 0007 or higher ■ Use the data load script • Created using the IQ CREATE ASIQ TEMPLATE command • Run this script on Version 12.0 ■ These scripts unload the 11.x data and reload it into a 12.0 database in binary format ■ Scripts need to be edited before running on the appropriate server A -16 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Data Extract Script ■ To be run on Sybase IQ 11.5 Server ■ The unloaded format is in binary ■ You need to specify the tape devices to which you will be writing • You must choose a local tape device ■ You need at least one tape for each indexset (table) in a version 11.x indexspace (database) • You can unload one indexset and specify multiple devices and they will be written to in parallel • You cannot write more than one indexset per tape (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-17 Migrating Data from Prior Versions Data Extract Script ■ Example: IQ UNLOAD BINARY ASIQ PREVIEW BLOCKFACTOR = 400 TO ‘/dev/rmt/1' SELECT customer_id ,cust_type ,organization r • • • FROM customer go BLOCKFACTOR A -18 The BLOCKFACTOR specified in the IQ UNLOAD must be the same as the BLOCKFACTOR in the LOAD command in the data load script. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Data Load Script ■ To be run on the new 12.0 ASIQ database ■ Must specify the device name for the data ■ Must use QUOTES OFF and ESCAPES OFF ■ Specify the same BLOCKFACTOR that was used in the IQ UNLOAD command (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A-19 Migrating Data from Prior Versions Data Load Script ■ Example: LOAD INTO TABLE customer (customer_id BINARY ,cust_type BINARY ,organization BINARY ...) FROM ‘/dev/rmt/1' FORMAT BINARY STRIP OFF QUOTES OFF ESCAPES OFF PREVIEW ON BLOCK FACTOR 400 UNLOAD FORMAT; A - 20 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Editing Scripts General Notes ■ Careful editing of scripts prevents many load problems ■ Things to look for: • Do not use quotation marks as delimiters for column values • IQ LOAD requires both QUOTES OFF and ESCAPES OFF • Do not use the LIMIT option • Start the database using the start_asiq parameter -gd all ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 21 Migrating Data from Prior Versions Running Scripts General Notes ■ Use DBISQL ■ Example: DBISQL -c "uid=dba; pwd=sql; dbf=asiqdemo.db" -q myload.sql A - 22 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Module Map Step 3 - Loading Data • Run migration scripts ■ Load the data from your original source ■ Insert the data from an indexspace ■ Select the data into a file and load ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 23 Migrating Data from Prior Versions Load the Data from your Original Source Insert Data Directly from ASE ■ Steps: 1. Connect to both ASE and ASIQ 12 using the same user ID and password 2. On ASIQ, issue a statement using this syntax: INSERT INTO asiq_table LOCATION 'ase_servername.ase_dbname' {select col1, col2, col3, ... from ase_table} 3. Issue a COMMIT to commit the insert You can also use this method to move selected columns from a pre-version 12 ASIQ database into a version 12 database. A - 24 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Load the Data from Your Original Source Load from Original Flat Files ■ See Module 8, "Loading Data from Files" ■ Example: LOAD TABLE customer (customer_id ‘| ' , cust_type ‘| ' , organization ‘| ' , country ‘\x0a' FROM ‘/work/data/customer1. dat' ESCAPES OFF QUOTES OFF WITH CHECKPOINT ON; Loading from flat files requires a large amount of disk space. ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 25 Migrating Data from Prior Versions Module Map Step 3 - Loading Data • Run migration scripts • Load the data from your original source ■ Insert the data from an indexspace ■ Select the data into a file and load A - 26 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Insert the Data from an Indexspace Insert Data Directly from 11.5.x Sybase IQ ■ Example: INSERT INTO asiq12_table LOCATION 'asiq11_servername.asiq11_indexspace' {select col1, col2, col3, ... from asiq11_indexset} ■ Issue a COMMIT to commit the insert ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 27 Migrating Data from Prior Versions Module Map Step 3 - Loading Data • Run migration scripts • Load the data from your original source • Insert the data from an indexspace ■ Select the data into a file and load A - 28 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Select Data into an ASCII file ■ This method is not recommended due to the following drawbacks: • It is much slower • You cannot use parallel dumping to multiple devices • Requires a large amount of disk space • Must do datatype conversions manually ■ Example: select * from employee ># empfile.dat ■ Then use LOAD TABLE command: LOAD TABLE employee FROM 'c:\\temp\\empfile.dat' ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 29 Migrating Data from Prior Versions bcp Data into an ASCII file ■ Must have CIS with IQ Version 11.5.x to bcp out to a file ■ bcp data out to a flat file from IQ 11.5.x, then use LOAD TABLE command on ASIQ version 12 Server ■ Requires a large amount of disk space A - 30 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Migration Notes Data Conversion Issues ■ If you loaded source data in binary format for 11 .x, check to see whether there are date, datetime, or scaled integer columns • You should not load into v12 from the original sources • These datatypes need to be converted into the new format before loading (continued . . .) ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 31 Migrating Data from Prior Versions Migration Notes Memory Settings ■ You need to change the defaults for main and temporary cache before loading into the Version 12.0 database ■ Example: SET OPTION "PUBLIC".MAIN_CACHE_MEMORY_MB = #_of_MB SET OPTION "PUBLIC".TEMP_CACHE_MEMORY_MB = #_of_MB ■ Can add these SET OPTION commands to the scripts, but remember to disconnect from the database and reconnect for the settings to take effect A - 32 Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 ©2000 Sybase, Inc. Migrating Data from Prior Versions Summary ■ Choose the best load method for your site. ■ Use migration scripts with caution. • Edit these scripts very carefully! ©2000 Sybase, Inc. Administering Adaptive Server IQ for the Data Warehouse, Version 1.0 A - 33
