Download Progress Performance Tuning

Progress Performance Tuning
Dan Foreman
Progress Expert for Strategic Information Group
Presenter Intro
• Working with Progress since 1984
• Author of some books (paper & digital):
–
–
–
–
Progress Performance Tuning Guide
Progress Database Administration Guide
Progress System Tables
And more
• Strategic employee since July, 2016
Strategic Information Group Copyright © 2016
May only be reproduced with prior consent from Strategic
Audience Intro
• How many here are true “end users”, i.e. not vendors or
consultants?
• Progress Version?
–
–
–
–
11.*
10.2B*
10.1*
Earlier
• Largest Single DB Size?
• Largest Concurrent User Count?
Other Info
• Phones on silent please
• I don’t discuss syntax in this presentation and how to use
specific startup options…the documentation does that
nicely
– http://documentation.progress.com/#page/progdocindex%2Fopen
edge.html%23wwconnect_header
– http://knowledgebase.progress.com/
Two Main Database Activities
• Reporting, i.e. reading lots of records
• OLTP (Online Transaction Processing), i.e. updating the
database
• Both of these activities exercise the database in somewhat
different ways
• Definitions:
– CRUD: Create, Read, Update, Delete
– CUD: CRUD without the Reads
Generic Tuning (for both OLTP & Reporting)
• Database Buffer Cache (-B)
– Finding data in memory can be thousands of times
faster than retrieving from the database
– General recommendation: make it as large as
possible without causing paging/swapping to occur
– “10% of database size” may or may not be a good
starting place
Generic Tuning – Monitoring the Buffer Cache
• Don’t use the promon ‘Buffer Hits %’ to monitor
– Prior to V10.1B it is buggy and frequently wrong
– A Percentage is an imprecise reporting metric;
especially as the percentage approaches 100%
• Instead use “Buffer Cache Hit Ratio”
• Ratio of: DB Requests / DB Reads
• DB Requests: The number of times a process
requested a database block (not a record)
• DB Reads: The number of times the block
requested was not in memory and had to be
fetched from the database
Generic Tuning – Monitoring the Buffer Cache
• What not use the promon Buffers Hits % ?
• Why is the Hit Ratio Better?
Generic Tuning – Buffer Cache Hit Ratio
• Buffer Cache Hit Ratio Guidelines
– < 100:1 is not good
– 100:1 to 10000:1 is the sweet spot
– > 10000:1 is NOT super excellent and is usually attributed bad
code/indexing
– Remember: databases are not static entities and a Hit Ratio
seldom runs in a flat line
Generic Tuning – Buffer Cache Hit Ratio
• Common Causes of a Poor Hit Ratio
– One report looking at ‘old’ data (e.g. YTD data) can kill a good Hit
Ratio although the “dip” may only be temporary
– Database needs a dump/load (poor Scatter and/or Fragmentation
Factors)
– Type 1 Storage Areas
– Database Block less than 8k
– Record Fragmentation
– -B is too small
– Online utilities (dbanalys, probkup, etc.); use –Bp to reduce the
impact
– Reports with indexing problems
– Hit Ratio was checked too soon after the DB Broker was started
Generic Tuning - Database Block Size
• Recommendation: 8k
• There are some other opinions related to the possibility
of “torn pages” when the OS Block Size is smaller than
the DB Block Size (Linux, Windows, AIX)
• An increase in DB Block Size may mean a reduction in
Database Buffers (–B)
– (DB Block Size) * (-B) = Amount of memory allocated for Cache
• Dump/Load is required to change the Block Size
Generic Tuning – DB Configuration
• Fixed Size Data Extents
– Don’t grow into the Variable Extent for best performance
– But the cost is higher maintenance and potentially more frequent
outages
– Because storage is much faster, it’s not as important now as it
was in the 90’s and early 21st Century
Generic Tuning – DB Configuration
• Type 2 (AKA T2) Storage Areas
– Use them! There were some releases when QAD Structure files did
not utilize T2
• Performance
• Future proofing
• Block CRC improves
• Utilities (indexless binary dump)
– Reduces the frequency (but not the need) of Dump & Load events
• Reduced Scatter Factor
• No impact on Fragmentation
– Use a Large Cluster Size (512) for Tables with a large number of
records (tr_hist for example)
– Smallest Cluster Size (8) for Tables with few records (the
REFERENCE Area for example)
Generic Tuning - Spin locks (-spin)
• General Recommendation: 2000 to 100000
• Don’t go beyond 50000 unless you can prove there is a
benefit
• Re-evaluate when your Progress version changes (major
latching changes have been rolled into service packs)
• Only a Benchmark can give you the Best Value for your
particular DB, HW, Progress Version, and Application(s)
• If set higher than 32767, Virtual System Tables (VSTs)
report as a Negative Value; fixed in V10
Generic Tuning - Spin locks (-spin)
• In V10+ defaults to (# of Cores * 6000)
• Obsolete: (# of Cores * <some constant>)
• Dan’s patented formula:
– (DBA’s Birth Year) * Pi
Generic Tuning – Shared Memory Contention
• Object Cache Size (-omsize)
– One entry for each Object (table, index, LOB)
– Default is 1024
– The primary QAD DB has well over a thousand Objects
• Select count(*) from _storageobject (plus a buffer)
– If the –omsize is large enough, OM Latch activity is eliminated
OLTP – Before Image Cluster Size
• Recommendation: 16-128mb (32-64 being most
common)
• Default Cluster Size is 512k
• Checkpoint - event that occurs when BI Cluster is full
• The larger the Cluster Size, the less frequent the
Checkpoints
• But also: Faster Hardware = More Frequent Checkpoints
• More Frequent Checkpoints = Larger Clusters
• General goal is Checkpoints at 60 seconds or more
OLTP – Checkpoint Issues
• All CUD activity is FROZEN if one of the following occurs
• The duration of the checkpoint can be monitored in promon
• BI Cluster formatting
– Can occur even with fixed size BI extents
– Use proutil bigrow to pre-format a specified number of Clusters
• Sync/Fdatasync System Call
– Promon R&D
• Buffers Flushed
– Avoided with APWs and a proper BI Cluster Size
OLTP – Before Image Tuning
• BI Buffers (-bibufs)
–
–
–
–
–
Recommendation: 50-100
Cost: slight increase in memory
Default is 5 prior to V10.1A; then 20
In rare cases a higher value is necessary
Set After Image Buffers (-aibufs) to the same value
OLTP – Before Image Tuning
• BI Block Size (-biblocksize)
–
–
–
–
–
Recommendation: 8k or 16k
Cost: higher “Partial Writes”
Default is 8k since V8
16k has the best performance
Set the After Image Block Size (-aiblocksize) to the same value
OLTP – Helper Processes
• Async Page Writers (proapw)
– Recommendation: 2 APWs
– Add more if Buffers Flushed is consistently non-zero
– Some older Progress doc’s recommend one APW per disk plus
one - Obsolete
• BI Writer (probiw)
– Can only start one (except a old V8 bug)
• AI Writer (proaiw)
– Can only start one (except a old V8 bug)
Let’s change gears and talk about Reporting
Reporting – Alternate Data Sources
• Run High Volume Time-insensitive Reports against a copy
(or sub-set) of the Production DB
• Options:
– Keep the replicated DB up-to-date with AI Logs from production DB
– OpenEdge Replication Plus
– Progress’s Pro2* Product line
• Also Ad-Hoc Reporting using ODBC/SQL Tools
– Recommendation: never run tools with queries created by users
with no IT background against production DBs
Reporting – Dump & Load
•
•
•
•
To Reduce Scatter & Fragmentation
To Change T2 DCS, RPB, DB Block Size
To verify no DB Corruption Exists
So that if you need to do one in an Emergency, it won’t
be your first time
• Usually much more effective than idxbuild or idxcompact
Reporting – Shared Memory Contention
• LRU Skips (-lruskips)
– LRU Chain is updated every time a record is read
– If the same record (or set of records) is accessed repeatedly
• The data is probably being found in the buffer cache – good
• Latch activity for the LRU chain is very high – bad
– Recommendation: set between 100 and 500
Reporting – Shared Memory Contention
• Alternate Buffer Cache (-B2)
– “ABC” for short
– Can be used to “pin” specific Objects in the DB Buffer Cache
– Will completely eliminate LRU Latch activity IF the Objects in the
ABC completely fit inside (-lruskips only reduces LRU activity)
– Complex to implement
Reporting – Remote Client Connections
• -Mm (applies to Remote Clients only)
–
–
–
–
–
–
–
Progress “Packet” Size
Default is 1024; Sweet spot is 4096 or 8192
Must be set on Database and Client (until V11.5)
Ignored by SQL Clients
Usually good improvement NO-LOCK Reads
Small Performance Penalty for CUD
VERY Network/DB/Application Dependent
Miscellaneous
• Virtual Machines (VM’s) typically offer great convenience &
Disaster Recovery benefits but NOT better performance,
especially when compared to bare metal
• Set –tablerangesize
• Set -indexrangesize
War Stories
•
•
•
•
•
•
•
Dump/Load of Database
DB was on Internal Drives of HP DB Server
Not enough space on Internal Drives to Dump
New SAN was available
Very Slow Dump Performance
Customer said “nothing’s wrong w/SAN”
The SAN Switch was using 10mbs but capable of 1gbs
Storage – RAID Levels
• We all know (or have heard) that RAID 5 is evil, right?
• ;Y
S 7
6 +AY
• Translation: Saru mo ki kara ochiru
• Japanese proverb: even a monkey can fall from a tree
RAID Configuration
•
•
•
•
•
General Rule: RAID 5, 6, DP are EVIL
RAID Levels are not precisely crafted standards
Today’s SANs are complicated devices
Alternative: RAID 10
But RAID 10 requires more disk space for the same
amount of data compared RAID 5
• Sales people will tell you that “their implementation of
RAID 5” doesn’t have performance issues
• Hybrids may work (RAID 5 for DB, RAID 10 for AI/BI)
• YMMV
Disks – How Many?
• Answer: Lots (at least for the database)
• Brief history of disk size / performance
–
–
–
–
1999:
2009:
2014:
2016:
9gb & 9-12ms Average Access (Seek + Latency)
144gb & 6-9ms Average Access
370gb & 4-8ms Average Access (15K disks)
Varies & 0.05-0.5ms (SSD)
• But it is a lot more complicated than a single number
Disks – Separate Device for the AI/BI Files?
• Separation of After Image, Before Image, and Database Disks
– For DB/BI, not as important as it used to be
– In fact, some devices make it difficult to split out individual disks for
performance purposes
– Mainly for Integrity (especially AI)
• Separate physical disk (if possible)
• Separate LUN
• Separate Volume Group
• Separate Logical Volume
• Separate Directory/Folder
– Secondarily for Performance (maybe)
Fix the Application – Part 1
•
•
•
•
Large USA (Ohio) Bank
Project to consolidate several Databases into one
Database Connections increased 4x
The pre-consolidation Databases were well tuned from
an earlier visit
• Good Cache Hit Ratio (100:1 or better)
• But logged over 3 Billion DB Requests in 14 hours
• Performance Unacceptable!
Fix the Application – Part 2
• On a Test DB used Virtual System Tables to capture
Table I/O before & after perceived “Hot Spots”
• Used the Client Parameter (-yx) to identify the Program
(fortunately there was only one)
• Reviewed the Code with the Progress Profiler
• Root cause: Indexing Problem
• Emergency Change Request to add a new index
• After the Index was added: 742 million DB Requests
(versus 3 Billion the day before)
Conclusion
• [email protected]
• Mobile: +1 541-908-3437 (24 hours if you have a Critical
Database problem) but it will not be a free call
• Thank you for attending
• Questions?