Download here - IDERA Community

Edge Case Testing for the
Database Professional
Vicky Harp
About Me – Vicky Harp
 Product Manager at Idera
 Community Manager at Idera
 http://community.idera.com
 Twitter: @vickyharp
 Email: [email protected]
Let’s talk bugs
 All software has failure conditions
 Corruption
 Tampering
 Most software has bugs
 Errors in logic
 Errors in execution
Let’s talk bugs
 Grey areas
 Scalability
 Platform and Hardware Compatibility
 Language and Regional Support
 If a customer experiences a problem, it’s
no longer a grey area – it’s a bug
The Blame Game
 Blame the developer
 All development done as sysadmin on a local
Developer edition server with no test data
 Data access layer design treated as an afterthought
 Cowboy code
 “Cool tricks” that rely on insecure, unstable, or
deprecated functionality
 Continuing to use deprecated features
 Absurdly low fault tolerance
 Overuse of ad hoc SQL
The Blame Game
 Blame the DBA




No access to reasonable development environments
No availability of QA environments
QA environments used as semi-production
Uneducated use of obscure configuration options,
trace flags, and data topology
 Database compatibility modes
 Failure to maintain server as a stable code platform
 Mismatched tempdb collation
 Triggers in msdb
The Blame Game
 Blame the designer
 Preposterous object names
 Trailing spaces
 Special characters
 Over/Under-normalization
 User objects in system databases
 Underdocumentation
The Blame Game
 Blame management





No time allotted for QA
Bugs are expected to be fixed too quickly
Insufficient personnel
Insufficient hardware
Unreasonable expectations
Setting Boundaries
 Define expectations for what is and is not
supported in your application
 Document, communicate, enforce, and
maintain these boundaries
 Example: SQL 2005 SP2 through SQL 2008
R2 on case insensitive, US-regionalized
English language instances
Enter QA
 All software, internal or external, application
or database, needs QA
 Professionals test their code
 Not testing your code is unprofessional
Creating Test Cases
 Most testing done in a small environment is
ad hoc
 Did this change work?
 More robust testing relies on test cases and
use cases
Use Cases
 Expected behavior for the application
 Example:
 When User clicks the Cancel button:
 The window will close
 No data will be changed
 When User clicks the OK button:
 The window will close
 Data will be saved
 If there is an exception, retry twice before returning an
error message
Test Cases
 Test definition with expected output
 Example:
 Precursors: Product is running on a supported
platform but the SQL Server is offline
 User clicks Cancel
 Expected: Window closes
 User clicks OK
 Expected: Exception returned to user
Using Test Cases
 Almost no limit to how highly documented
test cases can be
 If you’re starting from 0, a good start is to
start making checklists
 Things to check before accepting a build
 Things to check after failing over a cluster
Test Plans
 A collection of test cases is a test plan
 The best case scenario is to write the test
plan before the feature
 Realistically this often needs to be done with
an application in-situ
Types of Test Cases




Main Success Cases
Edge Cases
Corner Cases
Others? It depends
Main Success Cases
 What most people think of when they think of
testing
 Test case that supports a use case
 Example:
 Use Case: User runs sales forecasting report
 Main Success Case: Report returns correct data
with no error messages within 30 seconds
Edge Cases
 Test cases which explore the boundaries of
an application
 The boundary may be based on the use case
or based on technology
 Based on use case: User clicks all the buttons on
a view
 Based on technology: What is the behavior of the
application on SQL 2000?
Edge Cases
 Ideally edge cases should be identified at
design time
 This is part of good programming practice,
regardless of language
 DBAs should be aware of where their
configuration is introducing new edge cases
that may not have been accounted for
Edge Cases
 Most common edge for databases is
datatypes
 Name field backed by an nvarchar(100) column
 Edge case: NULL
 Edge case: 0-length string
 Edge case: 100 character Unicode string
 Primary key on a smallint column
 Edge case: key seeded at 0 and 32,768 rows inserted
 Edge case: key seeded at -32,768 and 65536 rows
inserted
Corner Cases
 The intersection of two cases
 Not necessarily the intersection of two edges
 Example:
 Mail merge – inserting a customer’s name into a
message, then saving the text to a column
 Corner cases
 Special character in either name or message, plus
 Extremely long string for either name or message
Corner Cases
 SQL Server is especially prone to corner
cases
 Shared resources are a common source of
corner cases
 Shared disks
 Shared tempdb
 Shared memory
 Be aware that configuration decisions can
introduce corner cases that a developer
cannot possibly anticipate
Common Edge Cases









Data Types
Language and Regionalization
Date/Time Considerations
Performance
Usability/Maintainability
Security
Integration
Recoverability
Compatibility
Data Types
 Column and variable types
 int, bigint, nvarchar, char, xml, etc.
 Most basic of all test cases, so no excuse not
to test this
Data Types




Max and min values
Collation and sort order
Special characters
NULL values
Data Type Overflows
 Overflow during aggregation
 select sum(intcolumn)
 select sum(cast(intcolumn as bigint))
 Overflow or truncation with isnull
 Use same data types or coalesce()
DEMO
Data Types
 Let the engine help you out – use table
constraints!
 Name your constraints such that any error
message returned will help you understand the
problem
Language/Regionalization
 Behavior when using different languages and
regional settings
 Particularly affects numeric and date fields,
may also cause other issues
Language/Regionalization
Afrikaans
Albanian
Amharic
Armenian
Arabic
Assamese
Azeri (Latin)
Bangla (Bangladesh)
Basque
Bengali (India)
Bosnian (Cyrillic)
Bosnian (Latin)
Bulgarian
Catalan
Chinese (Simplified)
Chinese (Traditional)
Croatian
Czech
Danish
Dari
English
Estonian
Finnish
Filipino
French
Galician
Georgian
German
Greek
Gujarati
Hausa (Latin)
Hebrew
Hindi
Hungarian
Icelandic
Igbo
Indonesian
Inuktitut (Latin)
Irish
isiXhosa
isiZulu
Italian
Japanese
Kannada
Kazakh
Khmer
KiSwahili
Konkani
Korean
Kyrgyz
Lao
Latvian
Lithuanian
Luxembourgish
Malay (Brunei
Darussalam)
Malay (Malaysia)
Malayalam
Maltese
Maori
Marathi
Mongolian (Cyrillic)
Nepali
Norwegian (Bokmål)
Norwegian (Nynorsk)
Oriya
Persian
Polish
Portuguese (Brazil)
Portuguese (Portugal)
Punjabi
Quechua
Romanian
Serbian (Cyrillic)
Serbian (Latin)
Sesotho sa Leboa
Setswana (South Africa)
Sinhala
Slovak
Slovenian
Spanish
Swedish
Tamil
Tatar
Telugu
Thai
Turkish
Turkmen
Ukrainian
Urdu
Uzbek (Latin)
Vietnamese
Ting Vit
Welsh
Yoruba
Language/Regionalization
Language/Regionalization
Language/Regionalization
DEMO
Language/Regionalization
 Make your code language and region
agnostic
 2012-01-07 11:00:00 AM
 1000 instead of 1,000
 Use “set language”
 set language us_english
 As a DBA, seriously consider the region
settings for your SQL Server
 Mismatch between SQL and Windows can cause
heartache
Collation Conflicts
 Using different collations on the same database
or on the same server can cause conflicts
 Use a COLLATE statement when comparing two
strings
 Rule of thumb: if you are going to normalize
case with LOWER or UPPER, use COLLATE
 Never assume rows will come back in a
specified order
Date/Time Considerations
 Inaccuracies in scheduling and in date
arithmetic
 Can become a real nightmare on a WAN
Date/Time Considerations
 Daylight savings time
 When DST ends, the clock goes from 1:59 AM 1:00 AM
 SQL Agent will pause for an hour
 February and Leap Day
 Day-of-year calculations after the 60th day
Date/Time Considerations
 Timezones
 Not all timezones are on even hour boundaries so
you need to be able to account for 30 and 15
minute differences
 Date Arithmetic
 Region settings can affect functions like
datepart()
 Date math functions can and do overflow. It takes
less than 25 days in milliseconds.
DEMO
Performance
 This is a very large topic
 DBAs sometimes take this more seriously
than developers
 Making your development environment look
and behave more like production goes a long
way toward identifying these cases
 Unless it is a single-user application, test with
multiple users
Usability/Maintainability
 How hard is it to track down bugs?
 Are error messages helpful?
 Classic problems:




Huge stored procedures
TSQL embedded in other application code
Obfuscated table names
Encrypted stored procedures
Security
 Doing all dev work with an administrator login
is not realistic
 Default databases for logins
 Compliance issues
 xp_cmdshell
Integration
 Does this application play well with others?
 Do two projects that live together in prod
have separate test environments?
 Does maintenance overlap?
Recoverability
 Are you backing up your data? Can it be
restored?
 Is your application mirror or cluster aware?
Compatibility
 Compatibility with SQL Server and Windows
versions
 Non-default configurations
 Database compatibility modes
 The master database will be set at 80 compatibility on
a 2000 instance that has been upgraded to 2005
 DMF functions, COLLATE statements, and JOIN
statements are all subject to failure in certain
compatibility modes
 Know what you support and be sure those collations
exist in your test environment
How To Test?




Dev and Test Environments
Ad Hoc testing
Automated testing
Bug tracking
Dev and Test Environments
 You should have a dev and/or test
environment
 This environment should be more challenging
than production, not less
 Having a difficult dev environment means that
code is more likely to perform in prod
Dev and Test Environments
 Example





Korean language and regionalization
case sensitive collation
large number of databases
many agent jobs
long database and object names with special
characters
Ad Hoc Testing
Set aside time to “poke around” in code
Enter long strings into fields
Input negative numbers
Change the clock and see what happens on
Leap Day
 Set high seed values for primary keys and
change default values for tables




Automated Testing
 Can be as simple as SQL Agent jobs
 Developers can add test cases to their
product
 Many tools
 If you do not have source code, just profile
 Build up a list of test cases that need to be
run before going to production
Bug Tracking




Post it notes won’t cut it
Start with a spreadsheet
Keep track of how long bugs take to fix
Try to prevent bug fixes from taking over your
schedule
Wrap-Up




Define your application boundaries
Define use and test cases
Test your software
Refine, repeat, improve
Discussion
More Information
 Contact me:
 Forums: community.idera.com
 Twitter: @vickyharp
 Email: [email protected]
 Slides will be available at
http://community.idera.com