Download Data Validation and Security

Module 8
*********
Data
Validation &
Security
Workforce Information
Database Training
Last update November 2006
Module 8
Data Validation & Security
Providing Accurate Data Consistently
Data validation allows us to provide data
with confidence in its accuracy, and we
can consistently provide this data by
implementing thorough security.
Module 8
Data Validation & Security
Data Integrity
Validity, consistency, and accuracy of
the data in a database.
•Table-level
•Field-level
•Relationship-level
•Business Rules
Module 8
Data Validation & Security
Data Validation
The process of determining if an
update to a value in a table’s data cell
is within a preestablished range or is a
member of a set of allowable values.
Module 8
Data Validation & Security
What are some common data
quality problems that affect
data integrity and validation?
Module 8
Data Validation & Security
Common Data Quality Problems
•
•
•
•
•
•
•
•
•
Illegal values
Violated attribute dependencies
Uniqueness violation
Referential integrity violation
Missing values
Misspellings
Cryptic values
Embedded values
Misfielded values
Module 8
Data Validation & Security
Common Data Quality Problems
continued….
•
•
•
•
•
•
•
•
Word transpositions
Duplicate records
Contradicting records
Wrong references
Overlapping data/matching records
Name conflicts
Structural conflicts
Inconsistencies
Module 8
Data Validation & Security
What are some methods for
checking for data validity?
Module 8
Data Validation & Security
Methods for checking
for data validity
•
•
•
•
•
•
•
Visual/manual
Aggregation
Reviewers guide
Auto data checks
Record counts
Spell checks
Have data provider review
Module 8
Data Validation & Security
Time to Exercise!!
Module 8
Data Validation & Security
ALMIS Database Security Considerations
The contact in my state is:______________
Security Concerns:
Confidentiality
Integrity
Availability
Web security
Application
security
•Production vs. test
User level
Physical security
(including backup
& restoration plan)
•Network
permissions
Data Storage Access
(Server or PC)
User level
•Suppression flags
•Data aggregation issues
•RDBMS
•ODBC
Network
(LAN/WAN)
security
Application
security
•Production vs. test
User level
Database security prevents unauthorized person(s) from viewing, destroying or altering data within the database.
Module 8
Data Validation & Security
ALMIS Database Security Considerations
The contact in my state is:______________
Security Concerns:
Confidentiality
Integrity
Availability
Web security
Application
security
•Production vs. test
User level
Physical security
(including backup
& restoration plan)
•Network
permissions
Data Storage Access
(Server or PC)
User level
•Suppression flags
•Data aggregation issues
•RDBMS
•ODBC
Network
(LAN/WAN)
security
Application
security
•Production vs. test
User level
Database security prevents unauthorized person(s) from viewing, destroying or altering data within the database.
Module 8
Data Validation & Security
Physical Security
Questions to ask yourself:
• Where is your data actually stored?
• Are you responsible for physical security?
If you are...
Module 8
Data Validation & Security
Physical Security
• What physical security measures do you
have in place?
• Do you have the right hardware?
• Are you using obsolete hardware that is
prone to crashing or hacking? Do you
have a choice?
Module 8
Data Validation & Security
Physical Security
• Do you have a database backup plan?
• Do you have remote backup so if a fire
burns the building you don’t lose both
your primary and backup data at the
same time?
• Do you have a data restoration plan?
Does the backup plan allow for feasible
restoration?
Module 8
Data Validation & Security
Physical Security
Restoration Plan
• Does your restoration plan allow for
the restoration of individual tables
or require the entire database?
• Do you maintain copies of the tables
on your hard drive?
Module 8
Data Validation & Security
Physical Security
• Who has access to the physical storage
location?
• Is your database on a PC or server that
someone could easily kick, spill coffee
on, or accidentally reappropriate?
If you are not responsible for physical
security, ask these questions of those
who are!
Module 8
Data Validation & Security
ALMIS Database Security Considerations
The contact in my state is:______________
Security Concerns:
Confidentiality
Integrity
Availability
Web security
Application
security
•Production vs. test
User level
Physical security
(including backup
& restoration plan)
•Network
permissions
Data Storage Access
(Server or PC)
User level
•Suppression flags
•Data aggregation issues
•RDBMS
•ODBC
Network
(LAN/WAN)
security
Application
security
•Production vs. test
User level
Database security prevents unauthorized person(s) from viewing, destroying or altering data within the database.
Module 8
Data Validation & Security
Data Storage Access
• What software are you using to store the
database?
• SQL Server, Oracle, FoxPro, Access?
Survey says:
SQL Server
48%
Oracle
28%
Other
24%
Module 8
Data Validation & Security
Data Storage Access
• What can be done at the server level to
provide for security?
• Confidentiality, Integrity, Availability
DO apply advanced security to the most
sensitive data, but DON’T apply
advanced security to non-sensitive data
Module 8
Data Validation & Security
Data Storage Access
Security at the server and/or database
level…
• Permissions
• What kind of permissions exist?
• Read only, SA, etc.
• Who sets those permissions?
• Who has those permissions?
Module 8
Data Validation & Security
ALMIS Database Security
Security at the data level…
• Suppression
• Suppression flags
• Suppression can be handled at the
database level or the application level
Module 8
Data Validation & Security
ALMIS Database Data Security
Two approaches:
1. ALMIS Database contains NO confidential
data (all data available for use without
restriction)
2. ALMIS Database contains confidential
data (data access controlled by security
and/or suppression)
Module 8
Data Validation & Security
ALMIS Database Security
Tables that have suppression flags...
ces
iomatrix
stindprj
indprj
occprj
stoccprj
industry
oeswage
NOTE: tables stfirms and wage have no
suppression flags but may contain confidential data
Module 8
Data Validation & Security
ALMIS Database Data Security
Issues to consider regarding suppression:
•
If your database doesn’t contain
suppressed records, detailed data won’t
aggregate to totals (validity checks?)
•
Without proper suppression, confidential
data can be back-calculated
Module 8
Data Validation & Security
ALMIS Database Data Security
A note on confidential data:
If you are checking 202 data (or any other
potentially confidential data) to see that it
loaded right, make sure your EQUI data files
and/or printouts are always secured and
shred all photocopies when finished.
Check with your local BLS personnel or LMI
administrator for specific confidentiality
policies.
Module 8
Data Validation & Security
ALMIS Database Security Considerations
The contact in my state is:______________
Security Concerns:
Confidentiality
Integrity
Availability
Web security
Application
security
•Production vs. test
User level
Physical security
(including backup
& restoration plan)
•Network
permissions
Data Storage Access
(Server or PC)
User level
•Suppression flags
•Data aggregation issues
•RDBMS
•ODBC
Network
(LAN/WAN)
security
Application
security
•Production vs. test
User level
Database security prevents unauthorized person(s) from viewing, destroying or altering data within the database.
Module 8
Data Validation & Security
User Access
Three major types of user access:
• PC - direct to database (ODBC or
RDBMS)
• Network (LAN/WAN) through
application
• Web (passive or active through
application)
Module 8
Data Validation & Security
User Access via the Web
• How do Web users access the data from the
database?
• Passive (static tables automatically
updated to web pages)
• Active (query through application)
Module 8
Data Validation & Security
User Access via the Web
Questions to ask yourself:
• What kind of web server are you using?
What are the security considerations with
that choice?
• What kind of firewall do you have?
Module 8
Data Validation & Security
User Access via the Web
• Does your web interface have
security/confidentiality suppression?
• Do you display suppressed data to certain
users? If so, how is access managed?
Passwords? IP address?
Module 8
Data Validation & Security
User Access via a Network
(LAN/WAN)
Questions to ask yourself:
• What type of network system do you have?
• Who has rights/access to your data via the
network? Through what applications?
• Are confidential data (suppressed records)
available?
Module 8
Data Validation & Security
User Access via a Network
(LAN/WAN)
• How do you control access to confidential
data? Through user Ids? Through server
access permissions?
• DO recognize the important distinction
between network security and data
security.
Module 8
Data Validation & Security
User Access via direct connection
Questions to ask yourself:
• Is local access machine password
protected?
• Who has access to your machine?
• Is there a backup plan for your access
machine?
• Is the source data for your database
secure? Backup plan? Restorable?
Module 8
Data Validation & Security
ALMIS Database Data Security
Documentation
It may prove beneficial to keep detailed
records on…
•
•
•
•
•
How security is done
Where security exists
Who is responsible for security
Who has access/permissions to what
Etc...
Module 8
Data Validation & Security
Data Security
DISCUSSION