* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download What`s new in OE DB - FTP Directory Listing
Survey
Document related concepts
Transcript
What’s New in the OpenEdge Database Norwegian PUG Event Richard Banville, Fellow, OpenEdge Development April 8, 2014 Introduction Parallel development between OpenEdge 10.2B and OpenEdge 11 • Including rollout of some enhancements in the older releases • Some OE 10.2B releases occurred after OE 11.0 shipped – This confuses where some features exist Release information provided with each topic • Indicates there and above Not a deep dive into each feature • Just going to scratch the surface 2 © 2014 Progress Software Corporation. All rights reserved. Agenda Major Features Not So Major Features Relatively Unknown or Forgotten Features 3 © 2014 Progress Software Corporation. All rights reserved. Major Features 4 © 2014 Progress Software Corporation. All rights reserved. Transparent Data Encryption – TDE (OE 10.2B) Very low overhead (0 – 4%) Transparent Flexible Secure 5 Application transparent data encryption Full index query support No need to move data Encrypt individual objects or areas Can change encryption policies online Storage engine encrypts blocks on disk Provides secure encryption key storage Support industry standard ciphers Limits access to physical data © 2014 Progress Software Corporation. All rights reserved. OpenEdge Transparent Data Encryption (TDE) Database Storage Engine How Does It Work? Keys Key Store Write I/O Encrypted Data Encrypt Shared Memory plain text Buffer Pool (plain text block) & Database Decrypt Policies Policy Area Read I/O Product Install 6 Key store • Database Master Key (DMK) • Admin/User Passphrase • Manual/Automatic © 2014 Progress Software Corporation. All rights reserved. Encryption Policy Area • Encryption Policies • What to encrypt • How to encrypt (cipher) Other Encryption Features Encrypted backup/restore Optionally encrypt binary data dump Migration • Encrypt data over time or immediately • No application changes – Full index capabilities on encrypted data Supporting utilities • Enable/disable • Epolicy manage <area | table> rekey | update | encrypt • Epolicy scan • Table/index move w/possible re-encryption • Can change encryption policies online 7 © 2014 Progress Software Corporation. All rights reserved. Multi-tenant Tables (OE 11.0) Simplifies Development of Multi-tenant Applications Car Rental Ease maintenance & app. upgrades Application Basic multi-tenant paradigm Multi-tenancy • Multi-tenant tables built into the database Tenants share same schema definition Data physically partitioned by tenant identity Customer Tenant 1 (Hertz-Boston) Tenant 2 *Fictitious example 8 © 2014 Progress Software Corporation. All rights reserved. 2 Mark Wahlberg 3 Steve Carrell 4 Robin Williams 7 Harrison Ford (Hertz-Chicago) Minimal application changes • Just establish tenant identity (per database) 1 Ben Affleck 8 Berrnie Mac 2 Dennis Rodman Tenant 3 (Rent-a-wreck) 7 Nick Nolte 9 Lindsay Lohan 10 Charlie Sheen Multi-tenant Tables (OE 11.0) Simplifies Development of Multi-tenant Applications Car Rental Application Index keys unique per tenant partition Multi-tenancy Shared & tenant sequences/tables Query is tenant specific • Authenticate as tenant – _User – Client Principal – SQL only _users Customer Tenant 1: Hertz-Boston FIND CUSTOMER WHERE cust-num=2. Tenant 2 • Same application & schema *Fictitious example 9 © 2014 Progress Software Corporation. All rights reserved. 2 Mark Wahlberg 3 Steve Carrell 4 Robin Williams 7 Harrison Ford (Hertz-Chicago) • Assert tenant identity Useful for database consolidation 1 Ben Affleck 8 Berrnie Mac 2 Dennis Rodman Tenant 3 (Rent-a-wreck) 7 Nick Nolte 9 Lindsay Lohan 10 Charlie Sheen Multi-tenant Tables (OE 11.0) Simplifies Development of Multi-tenant Applications Car Rental Application Keys unique per tenant partition Multi-tenancy Shared & tenant sequences/tables Query is tenant specific • Authenticate as tenant – _User Tenant 1: Hertz-Boston FIND CUSTOMER WHERE cust-num=2. 1 Ben Affleck 2 Mark Wahlberg 3 Steve Carrell – Client Principal 4 Robin Williams – SQL only _users 7 Harrison Ford • Assert tenant identity Useful for database consolidation • Same application & schema *Fictitious example 10 Customer © 2014 Progress Software Corporation. All rights reserved. Tenant 3: Rent-a-wreck FIND CUSTOMER WHERE cust-num=2. 8 Berrnie Mac 2 Dennis Rodman 7 Nick Nolte 9 Lindsay Lohan 10 Charlie Sheen Multi-Tenancy Enhancements Resource Governors (OE 11.1) • -LGovernor - % of lock table any one tenant (user group) is allowed to occupy • -nGovernor – Number of database connections any one tenant is allowed to consume Tenant Index Rebuild online (OE 11.3) • proutil proutil <db> -C-C enablemtidxbld <db> enablemtidxbld (online & offline) • proutil <db> -C mtidxbuild table <name> index <name> proutil <db> -C mtidxbuild table <db> index <name> tenant | group <name> tenant | group <name> • Acquires EXCLUSIVE table partition lock • Tenant data for associated table is restricted to no-lock • Index being rebuilt is offline • Can run multiple concurrently • Data access to table available for all other tenants and other tables of this tenant 11 © 2014 Progress Software Corporation. All rights reserved. Multi-Tenancy Enhancements Tenant Data Recovery (OE 11.3) • Basically a restore, roll forward, binary transfer between databases <db>-C-Cdatamove datamove source <db> • proutil Proutil <db> source <sourcedb> tenant <name> table <name> sequence [recs<n> <n>] [ tenant <name> [ table <name> ] | [ sequence < <name> name> ] [ recs ] • Schema validation performed automatically Progress OpenEdge Tenant/Group Data Move Source Database: <source_db> Destination Database: <dest_db> Table Instance Control Txn Size 1. Move Table Data by Tenant 2. Move Sequence Data by Tenant 3. Move Table Data by Group 4. Set Records per Transaction ( Current: 100 ) 5. Quit Enter your selection: 12 © 2014 Progress Software Corporation. All rights reserved. Not So Major Features 13 © 2014 Progress Software Corporation. All rights reserved. Utility Enhancements Roll forward enhancements (OE 11.3) • rfutil rfutil <db> forward –ailist <ailistname> <db>-C-Crollrollforward –ailist <ailist-name> • Can now specify a list of files to roll forward rather than individual ones – Avoids redo on each restart • Also per file completion status information provided 10:01:10: 50% of aimage file processed (100 notes processed)... Dbrestrict (OE 11.3) • Restricts access to the database for processes other than the specified operation proutil <db> -C dbrestrict [ datamove | rollforward ] [ enable | disable | status] • Unlike “oplock”, allows access by certain read-only activities and prostrct add • For “datamove” allows DB Broker, promon & proshut • Prorest can enable –dbrestrict too! 14 © 2014 Progress Software Corporation. All rights reserved. Utility Performance Online Backup Pause (OE 11.3) • Requires new backup header version for bi cluster information – Applies to online only – Not backward compatible – On by default – Use –bibackup all option for backward compatibility • Backup active clusters only • Skips unused clusters regardless of where they are in the cluster ring or bi file • Restore will reorder cluster to compact state probkup online <db> [ incremental ] <device-name> [ bibackup {active | all } ] [enableai ] [ enableaiarchiver – aiarchdir <dir-list > ] 15 © 2014 Progress Software Corporation. All rights reserved. Index Rebuild Performance (OE 10.2B06, OE 11.2) -TB 64 -datascanthreads # threads for data scan phase 1.5 X #CPUs -TMB merge block size ( default -TB) 64 -TF merge pool fraction of system memory (in %) 80% -mergethreads # threads per concurrent sort group merging X -threadnum = 1.5 X #CPUs -threadnum -TM -rusage -silent 16 sort block size (8K – 64K, note new limit) # concurrent sort group merging # merge buffers to merge each merge pass report system usage statistics a bit quieter than before © 2014 Progress Software Corporation. All rights reserved. 2 or 4 32 -rusage -silent Index Rebuild Performance (OE 10.2B06, OE 11.2) -TB 64 -datascanthreads # threads for data scan phase 1.5 X #CPUs -TMB merge block size ( default -TB) 64 -TF merge pool fraction of system memory (in %) 80% -mergethreads # threads per concurrent sort group merging X -threadnum = 1.5 X #CPUs -threadnum -TM -rusage -silent 17 sort block size (8K – 64K, note new limit) # concurrent sort group merging # merge buffers to merge each merge pass report system usage statistics a bit quieter than before © 2014 Progress Software Corporation. All rights reserved. 2 or 4 32 -rusage -silent Networking Control Philosophy: Throughput by keeping server busy without remote client waits! Process-based control • -Ma, -Mn, -Mi – Controls the order users are assigned to servers Resource-based control • -Mm <n> – Maximum size of network message – Client & server startup New tuning knobs – resource-based control • Alleviate excessive system CPU usage by network layer • Control record data stuffed in a network message – Applicable for “prefetch” queries 18 © 2014 Progress Software Corporation. All rights reserved. Networking – Prefetch Query No-lock query with guaranteed forward motion or scrolling • Multiple records stuffed into single network message • Browsed static and preselected queries scrolling by default FOR EACH customer NO-LOCK: …. end. DO PRESELECT EACH customer NO-LOCK: …. end. define query cust-q for customer SCROLLING. open query cust-q FOR EACH customer NO-LOCK. repeat: get next cust-q. end. 19 © 2014 Progress Software Corporation. All rights reserved. -NmsgWait (OE 10.2B06 & OE 11.1) TCP I/O Server -Mm 8192 Remote Client Determines how long a server waits for remote request Benefit: Could reduce polling thereby reducing system CPU Detriment: Could make remote client wait longer for a lock Tuning suggestions: 20 There are better ways to reduce polling. Leave it alone Set it to 1 (default is 2) © 2014 Progress Software Corporation. All rights reserved. FOR EACH customer NO-LOCK: …. end. -prefetchPriority (OE 10.2B06 & OE 11.1) TCP I/O Server -Mm 8192 Remote Client Defers check (poll) for requests from other remote clients while network buffer is populated with multiple records Reduces System CPU costs Tuning suggestions: 21 Set to 100 (worked best in my tests) Some field testing suggests 50 is best Watch out for slower response to non-prefetch queries © 2014 Progress Software Corporation. All rights reserved. FOR EACH customer NO-LOCK: …. end. -prefetchDelay (OE 10.2B06 & OE 11.1) TCP I/O Server -Mm 8192 Remote Client Puts multiple records in first message when set. If not set, first response message only contains 1 record Benefits: Can improve browse widget window population Can reduce TCP I/O for a single query by 1 Tuning suggestions: 22 In theory, enabling this should improve performance I have seen no difference in my testing © 2014 Progress Software Corporation. All rights reserved. FOR EACH customer NO-LOCK: …. end. Avg record size = 200 bytes ~40 records per msg maximum (-Mm 8192) -prefetchNumRecs and –prefetchFactor (OE 10.2B06 & OE 11.1) TCP I/O Server -Mm 8192 Remote Client FOR EACH customer NO-LOCK: …. end. Record size and –Mm matter! Determines how many records are “stuffed” in a message Number of records vs percentage full Reduces network traffic and improves TCP/IP I/O efficiency Tuning suggestions: 23 -prefetchNumRecs 100 records Reduce if remote clients complain of “choppy” behavior Default is 16 records © 2014 Progress Software Corporation. All rights reserved. • Avg record size = 200 bytes • ~40 records per msg maximum Too small –Mm negates affect Enhancements to Database Buffering System Memory Memory access much faster than disk access Reduce read and write I/O via caching Process Shared Memory Database Buffer Pool Bi Buffers Ai Buffers File system cache I/O Multi level caches .d.d .d.d .d .d 24 .d.b © 2014 Progress Software Corporation. All rights reserved. Multi level caches .d.a • Memory is finite • Working set concept Managing the buffer pool • Primary buffer pool • Alternate buffer pool • Private Buffers • Replacement mechanisms The Database Buffer Pool LRU MRU Database Buffer Lookup 25 Buffer pool hash table 64 128 96 1024 Primary Buffer Pool -B buffers Database block lookup via single hash table Replacement policy via LRU mechanism • LRU vs MRU end of replacement chain • Manages “working set” well • Easily “fouled” • High housekeeping cost © 2014 Progress Software Corporation. All rights reserved. 144 256 LRU buffer eviction policy Reducing “Housekeeping” Costs (OE 10.2B06 and OE 11.1) LRU MRU Buffer pool hash table 64 Database Buffer Lookup 96 1024 144 256 LRU buffer eviction policy Primary Buffer Pool -B buffers -lruskips <n> (-lru2skips for alternate buffer pool) (OE 10.2B06 and OE 11.1) • • • Skip LRU maintenance for <n> accesses LRU and LFU (least frequently used) combined Small numbers make a BIG difference (100 works well) Promon, R&D 4. Administrative Functions ... 4. Adjust Latch Options 4. Adjust LRU force skips: 100 26 128 © 2014 Progress Software Corporation. All rights reserved. FIND _Startup. ASSIGN _Startup-LRU-Skips = 100. Alternate Buffer Pool (OE 10.2B) MRU LRU 64 96 1024 144 256 Primary Buffer Pool -B buffers Buffer pool hash table Database Buffer Lookup 128 1 buffer pool cache 1 Hash table Multiple LRU replacement chains Alternate Buffer Pool -B2 buffers LRU 92 64 164 48 132 LRU buffer eviction policy 128 MRU LRU2 buffer eviction policy (only when needed) Object specific data managed separately Data Dictionary (by table/index/LOB) 27 Reduces “housekeeping” cost proutil enableB2 (by storage area) Data can become resident in memory proutil viewB2 option © 2014 Progress Software Corporation. All rights reserved. Private (per user) Read-only Buffers (little known/used category) Database Buffer Lookup 64 Buffer pool hash table LRU 128 96 1024 144 256 LRU buffer eviction policy MRU LRU Primary Buffer Pool -B buffers 104 128 40 200 40 160 MRU LRU 96 80 48 -Bp (per user) eviction policies Alternate Buffer Pool -B2 buffers LRU 92 64 164 48 132 128 Selfless act for table scan users • Will not improve performance of user • Helps others since it avoids fouling of LRU mechanism • Use it for operations like dbanalys or table scan queries Intended to improve performance overall 28 MRU © 2014 Progress Software Corporation. All rights reserved. MRU LRU2 buffer eviction policy (only when needed) Private (per user) Read-only Buffers (little known/used category) Database Buffer Lookup 64 Buffer pool hash table LRU 128 96 1024 144 256 MRU LRU buffer eviction policy MRU LRU Primary Buffer Pool -B buffers 104 128 40 200 40 160 MRU LRU 96 80 -Bp (per user) eviction policies Alternate Buffer Pool -B2 buffers LRU 92 64 164 48 132 128 MRU Values LRU2 buffer eviction policy (only when needed) • Default: 0; Max 64 per client (cumulative total limited to 25% of –B) • Change per client maximum with -Bpmax at db startup • Set # buffers via -Bp at client startup – Can be changed at runtime via _MyConnection._Myconn-NumSeqBuffers 29 48 © 2014 Progress Software Corporation. All rights reserved. Some Tuning Suggestions -lruskips (OE 10.2B06 & OE 11.1) • 100 seems to work well • Only helps if LRU latch contention -B2 (OE 10.2B) • Set large enough to hold very active “small” tables • Include indexes as well as the table -Bp (V 9.0A) • Set to # tables in query + depth of each index used – 2 way join query: 8 to 10 will work well • Use for table scan type queries – Application as well as utilities (dbanalys) -B • Tune to acceptable read I/O rates, not buffer pool hit ratio 30 © 2014 Progress Software Corporation. All rights reserved. Auditing (OE 10.1A) Policy based (flexible) • Change capture internal to database • Audit logs highly protected • Create, update and delete change capture – (no read capture provided) • Application events • Archive for reporting Reclaim Audit Space (OE 11.2) <db>-C-Cauditreconfig auditreconfig [ tablearea <name> ] [ indexarea • proutil proutil <db> [ tablearea <name> ] [ indexarea <name> ]<name> – Re-creates audit tables and indexes in new area – Drops old audit tables and indexes – Existing data is truncated • Carefully use prostrct truncate/remove to reclaim area’s disk space 31 © 2014 Progress Software Corporation. All rights reserved. OpenEdge Replication 32 © 2014 Progress Software Corporation. All rights reserved. Maintain a “hot” standby with OpenEdge Replication (V9.?) 33 © 2014 Progress Software Corporation. All rights reserved. OpenEdge Replication Enhancements Increase max –pica (OE 11.2) • Database Service Communication Area Size (-pica) • Maximum value of increased from 512 to 1,000,000 • Avoid database “slowdown” when target not keeping up JTA & Replication (OE 11.2) • Allow JTA distributed transactions and OpenEdge Replication 34 © 2014 Progress Software Corporation. All rights reserved. OpenEdge Replication Enhancements Table move/index move (OE 11.3) • Now propagated to target -asc <n>: asynchronous level (relatively unknown category) • Default is 10, Max is 500 • Disabled with setting of 501 • Server tells agent # blocks to be sent before ACK • Increasing can greatly improve performance • Set in pmmgr.properties file replserv option [replserv] arguments=-logging 2 –asc 501 35 © 2014 Progress Software Corporation. All rights reserved. Relatively Unknown or Forgotten Features 36 © 2014 Progress Software Corporation. All rights reserved. Misc Parameters -keeptargetlfe (OE 10.2B04) • prorest <target-db> <source-bak> [ -keeptargetlfe ] • Keep target large file enabled (lfe) • Restores a backup without enabling large files • By default: Target = Source on restore 37 © 2014 Progress Software Corporation. All rights reserved. Client Database-Request Statement Caching (OE 10.1C) What is it? • Client execution stack trace dump • Includes: Line #, procedure name, file name When should you use it? • Diagnosing deployed application problems • Local and remote access What’s in a name? • Don’t confuse with other “statement caches” Documentation • OpenEdge Data Management: Database Administration 38 © 2014 Progress Software Corporation. All rights reserved. Enablement – Multi-level activation promon R&D 1. Status Displays ... 18. Client Database-Request Statement Cache ... Activate 1. Activate For Selected Users 2. Activate For All Users 3. Activate For All Future Users 4. Deactivate For Selected Users 5. Deactivate For All Users 6. Deactivate For All Future Users View Location 39 7. View Database-Request Statement Cache 8. Specify Directory for Statement Cache Files © 2014 Progress Software Corporation. All rights reserved. Procedure Call Stack It’s just a LIFO… Top is last procedure executed One time full stack Bottom is first procedure executed Continuous full stack Top down, newest to oldest Continuous current location Top Newest Oldest # Procedure Name 19 : reallyLongNamedInternalProcedure3 12 : reallyLongNamedInternalProcedure2 5 : reallyLongNamedInternalProcedure1 445 : reallyLongNamedInternalProcedure0 1 : /usr1/stmtest/p72340_Untitled1.ped Bottom 40 © 2014 Progress Software Corporation. All rights reserved. File Name proctestb.r proctestb.r proctesta.r proctesta.r Manual Stack Generation Stack also available via “prostack” or “progetstack” • Useful for “non-responsive” connections • Available with 10.1c • Must have access to client’s machine kill –SIGUSR1 <pid> (Don’t forget the dash) • protrace.<pid> – Startup parameters – Execution stack – ** ABL Stack Trace ** (OE 10.2B) – ** Persistent procedures/Classes ** – ** PROPATH ** – ** Databases (logical/type/physical) ** 41 © 2014 Progress Software Corporation. All rights reserved. Mapping Get _StorageObject record Schema 42 © 2014 Progress Software Corporation. All rights reserved. Table D Cust Tbl Index D-2 Index D-1 SRep Idx Cust# Idx Name Idx Index B-1 Index A-1 Table B Logical Table A Object Mapping Cache - OM Latch: OM Lookup and LRU Mapping Hash Table No latching, no paging -omsize Primary Cache Zero contention! Get _StorageObject record OM cache loaded at startup Schema 43 © 2014 Progress Software Corporation. All rights reserved. Table D Cust Tbl Index D-2 Index D-1 SRep Idx Cust# Idx Name Idx Index B-1 Index A-1 Logical Table B Table A Object Mapping Cache - OM Latch: OM Lookup and LRU Table D Cust Tbl Index D-2 Index D-1 SRep Idx Cust# Idx Name Idx Index B-1 Index A-1 Logical Table B Table A Object Mapping Cache - OM Latch: OM Lookup and LRU Mapping Hash Table No latching, no paging Hash Table -omsize Primary Cache Zero contention! Secondary Cache Little contention 2nd cache for over flow & new objects Schema © 2014 Progress Software Corporation. All rights reserved. OM latch needed for paging maintenance Get _StorageObject record OM cache loaded at startup 44 OM How to manage Object Mapping Cache Do I have a problem? define variable prev-latches as integer. • Check latch statistics repeat: find _latch where _latch-name = "MTL_OM". display _Latch-Name _Latch-Lock /* # times latch acquired */ Activity Contention _Latch-Wait /* # time conflict occurred */ _Latch-Lock - prev-latches label "latch/sec". prev-latches = _Latch-Lock. pause 1. end. 45 © 2014 Progress Software Corporation. All rights reserved. How to manage Object Mapping Cache Do I have a problem? • Check latch statistics Identify proper value select count(*) from _storageObject Set it properly on startup to avoid eviction & latch contention •proserv <db> -omsize <n> … Memory used: ~192 bytes per entry Default 1024 = ~384K This is shared memory • 1 primary and 1 secondary cache per database 46 © 2014 Progress Software Corporation. All rights reserved. (V9.0a !) Altering Performance Related Parameters Online Paramater Name recspacesearchdepth lru2Enabled lruskips lru2skips spin nGovernor LGovernor pwqdwlay pwsdelay pwscan pwwmax client trace level client trace request enablelargekeys Nmsgwait prefetchFactor prefetchNumRecs prefetchPriority prefetchDelay Bp (Private Buffers) TableBase IndexBase 47 promon Adjust Latch Options Adjust Latch Options Adjust Latch Options Adjust Latch Options Adjust Latch Options Adjust Governor options Adjust Governor options Adjust Page Writer Options Adjust Page Writer Options Adjust Page Writer Options Adjust Page Writer Options Client DB-Request Statement Cache Client DB-Request Statement Cache Server Options Server Options Server Options Server Options Server Options © 2014 Progress Software Corporation. All rights reserved. VST _ActBuffer. _Buffer-LRUEnabled _startup._Startup-LRU-Skips _startup._Startup-LRU2-Skips _startup._Startup-Spin _startup._Startup-nGovernor _startup._Startup-Lgovernor _startup._Startup-APWQTime _startup._Startup-APWSTime _startup_Startup-APWBuffs _startup._Startup-APWMaxWrites _connect._Connect-CachingType _Database-Feature _startup._Startup-NmsgWait _startup._Startup-Prefetch-Factor _startup._Startup-Prefetch-Num-Recs _startup._Startup-Prefetch-Priority _startup._Startup-Prefetch-Delay _myConnect._MyConn-NumSeqBuffers _statbase._TableBase _statbase._IndexBase Increase Resource Startup Parameters Online (OE 10.1C) proutil <dbname> -C increaseto <- param> <value> Parameter list • -aibuf, -bibufs Increase, not decrease New segment restrictions • -B, -B2 • Kernel restrictions apply • -L • Security restrictions • -omsize • -mtpmsize (OE 11.0) • -Mxs – Servers: automatically attach quickly – Self serve: attach w/db action over time • Minimum segment size created: 128 MB 48 © 2014 Progress Software Corporation. All rights reserved. OpenEdge Database Enhancement Summary Major Enhancements Little known or forgotten • Transparent Data Encryption • Private buffers • Multi-tenancy • Om cache (-omsize) • Changing parameters online Other Enhancements • Runtime tuning • Utility Performance • Application insights • Auditing 50 © 2014 Progress Software Corporation. All rights reserved.