Download security

Protecting a Tsunami
of Data in Hadoop
HPE Security – Data Security
March 2017
Increased Adoption of Big Data Across Industries
European Telco: Collecting Call Data Records (CDRs) from
27 countries. Used for fault detection, roaming data, network
optimization. Includes number of PI data elements
Top 5 Global Car Manufacturer: Collecting sensor data from
5 million cars globally to find defects. Includes GPS location
information.
Top retailer: Analyze customer buying patterns, brand loyalty,
detect credit card fraud, etc.
Vast
Quantities
Of
PI Data
Health Insurer: Analysis sensitive customer health insurance
to detect prescription medication fraud, insurance
overpayments and to customize their portal.
(C) 2016 Hewlett Packard Enterprise - Confidential
2
Why is securing Hadoop difficult?
Rapid innovation in a well
funded open source community
Systems such as Hadoop do
not have “Delete” or “Update”
functionality
Multiple feeds of data in real time
from different sources with different
protection needs
Mainframe
RDBMs
MQ
XML
Salesforce
Flat
Files
3
Why is securing Hadoop difficult?
Access by many different users
with varying analytic needs
Automatic replication of data across
multiple nodes once entered into
the HDFS data store
Reduced control if Hadoop
clusters are deployed in a cloud
environment
4
Existing ways to secure Hadoop
– Existing IT security
– Enterprise-scale security for Apache Hadoop
− Network firewalls
− Apache Knox: Perimeter security
− Logging and monitoring
− Kerberos: Strong authentication
− Configuration management
− Apache Ranger: Monitoring and management
Need to augment these with “data-centric” protection of data in use, in motion and at rest
5
Introducing “Data-centric” security
Threats to
Data
Credential
Compromise
Traditional IT
Infrastructure Security
Authentication
Management
Data
Ecosystem
Security
Gaps
Data and applications
Traffic
Interceptors
Middleware
SSL/TLS/firewalls
Security gap
SQL injection,
Malware
Database encryption
Databases
Security gap
Malware,
Insiders
Malware,
Insiders
SSL/TLS/firewalls
File systems
Data security coverage
Security gap
Security gap
Disk encryption
Storage
6
Introducing “Data-centric” security
Threats to
Data
Credential
Compromise
Traditional IT
Infrastructure Security
Authentication
Management
Data
Ecosystem
Security
Gaps
HPE SecureData
Data-centric Security
Data and applications
Middleware
SSL/TLS/firewalls
Security gap
SQL injection,
Malware
Database encryption
Databases
Security gap
Malware,
Insiders
Malware,
Insiders
SSL/TLS/firewalls
File systems
End-to-end Protection
Traffic
Interceptors
Data security coverage
Security gap
Security gap
Disk encryption
Storage
7
Format-Preserving Encryption (FPE)
Meet Requirements for Encryption and Pseudonymisation
Format-Preserving Encryption (FPE)
Credit card
SSN/ID
Email
DOB
4171 5678 8765 4321
934-72-2356
[email protected]
31-07-1966
Full
8736 5533 4678 9453
347-98-8309
[email protected]
20-05-1972
Partial
4171 5681 5310 4321
634-34-2356
[email protected]
20-05-1972
Obvious
4171 56AZ UYTZ 4321
AZS-UD-2356
[email protected]
20-05-1972
8
Before: All applications and users have access to data
HR Application
Name
James Potter
Ryan Johnson
Carrie Young
Brent Warner
Anna Berman
SS#
385-12-1199
857-64-4190
761-58-6733
604-41-6687
416-03-4226
Analysts
ETL Tool
Credit Card #
37123 456789 01001
5587 0806 2212 0139
5348 9261 0695 2829
4929 4358 7398 4379
4556 2525 1285 1830
Help Desk
Mainframe App
Street Address
1279 Farland Avenue
111 Grant Street
4513 Cambridge Court
1984 Middleville Road
2893 Hamilton Drive
DBAs
Customer ID
G8199143
S3626248
B0191348
G8888767
S9298273
Malware
State
NY
NY
CA
CA
KY
Score
100
200
120
120
160
Malicious User
After: Data is protected at source at “Field Level”
ETL Tool
HR Application
Name
SS#
Kwfdv Cqvzgk
Veks Iounrfo
Pdnme Wntob
Eskfw Gzhqlv
Jsfk Tbluhm
161-82-1292
200-79-7127
095-52-8683
178-17-8353
525-25-2125
Analysts
Credit Card #
Payments App
Street Address
3712 3488 7865 1001
2890 Ykzbpoi Clpppn
5587 0876 5467 0139
406 Cmxto Osfalu
5348 9212 3456 2829
1498 Zejojtbbx Pqkag
4929 4356 7432 4379 8261 Saicbmeayqw Yotv
4556 2598 7643 1830 8412 Wbbhalhs Ueyzg
Help Desk
DBAs
Malware
Customer ID
State
Score
G7202483
S0928254
B7265029
G3951257
S6625294
NY
NY
CA
CA
KY
100
200
120
120
160
Malicious User
Malicious users, DBAs and malware: only see protected data
Malware
Name
SS#
Kwfdv Cqvzgk
Veks Iounrfo
Pdnme Wntob
Eskfw Gzhqlv
Jsfk Tbluhm
161-82-1292
200-79-7127
095-52-8683
178-17-8353
525-25-2125
Credit Card #
Street Address
3712 3488 7865 1001
2890 Ykzbpoi Clpppn
5587 0876 5467 0139
406 Cmxto Osfalu
5348 9212 3456 2829 1498 Zejojtbbx Pqkag
4929 4356 7432 4379 8261 Saicbmeayqw Yotv
4556 2598 7643 1830 8412 Wbbhalhs Ueyzg
DBAs
Customer ID
State
Score
G7202483
S0928254
B7265029
G3951257
S6625294
NY
NY
CA
CA
KY
100
200
120
120
160
Malicious User
Help desk and payments apps: operate on partially protected
data
Payments App
Name
SS#
Kwfdv Cqvzgk
Veks Iounrfo
Pdnme Wntob
Eskfw Gzhqlv
Jsfk Tbluhm
161-82-1292
200-79-7127
095-52-8683
178-17-8353
525-25-2125
Credit Card #
Street Address
3712 3488 7865 1001 2890 Ykzbpoi Clpppn
5587 0876 5467 0139
406 Cmxto Osfalu
5348 9212 3456 2829 1498 Zejojtbbx Pqkag
4929 4356 7432 4379 8261 Saicbmeayqw Yotv
4556 2598 7643 1830 8412 Wbbhalhs Ueyzg
Help Desk
Customer ID
State
Score
G7202483
S0928254
B7265029
G3951257
S6625294
NY
NY
CA
CA
KY
100
200
120
120
160
Analysis on de-identified data
Name
SS#
Kwfdv Cqvzgk
Veks Iounrfo
Pdnme Wntob
Eskfw Gzhqlv
Jsfk Tbluhm
161-82-1292
200-79-7127
095-52-8683
178-17-8353
525-25-2125
Card
Average Score
Amex
M/C
Visa
100
160
140
Credit Card #
Street Address
3712 3488 7865 1001 2890 Ykzbpoi Clpppn
5587 0876 5467 0139
406 Cmxto Osfalu
5348 9212 3456 2829 1498 Zejojtbbx Pqkag
4929 4356 7432 4379 8261 Saicbmeayqw Yotv
4556 2598 7643 1830 8412 Wbbhalhs Ueyzg
Class
# of states
State
Average Score
G
S
B
2
2
1
NY
CA
KY
150
120
160
Analysts
Analysts
Customer ID
State
Score
G7202483
S0928254
B7265029
G3951257
S6625294
NY
NY
CA
CA
KY
100
200
120
120
160
Analysts
Authorized applications access real data
Key Management
Authorized HR
Application
Name
SS#
Kwfdv Cqvzgk
Veks Iounrfo
Pdnme Wntob
Eskfw Gzhqlv
Jsfk Tbluhm
161-82-1292
200-79-7127
095-52-8683
178-17-8353
525-25-2125
Credit Card #
Name
James Potter
Ryan Johnson
Carrie Young
Brent Warner
Anna Berman
Street Address
3712 3488 7865 1001
2890 Ykzbpoi Clpppn
5587 0876 5467 0139
406 Cmxto Osfalu
5348 9212 3456 2829 1498 Zejojtbbx Pqkag
4929 4356 7432 4379 8261 Saicbmeayqw Yotv
4556 2598 7643 1830 8412 Wbbhalhs Ueyzg
Authorized Fraud
Analysts
Key Management
Customer ID
State
Score
G7202483
S0928254
B7265029
G3951257
S6625294
NY
NY
CA
CA
KY
100
200
120
120
160
SS#
385-12-1292
857-64-7127
761-58-8683
604-41-8353
416-03-2125
Architectures for Protecting Data in Hadoop
Hadoop Cluster
Upstream
Applications
4
1
Hadoop jobs
and analytics
FPE Encrypt
Data
2
Upstream
Applications
HDFS
5
FPE Encrypt
Data
3
Upstream
Applications
Landing Zone
ETL and
batch
Hadoop jobs
Hive,
MapReduce, etc.
FPE Decrypt
Data
Hadoop jobs
MapReduce,
Sqoop, Flume
FPE Encrypt
Data
6
Applications,
analytics and data
Egress Zone
ETL and
batch
HPE Decrypt
Data
Applications,
analytics and data
15
Conclusion
– Multi-platform enterprises adopting a data lake architecture need a
cross-platform solution for protection of sensitive data
– Big data partners bring comprehensive security within Hadoop, with
core capabilities for authentication, authorization and auditing
– Implementing data-centric security across data stores including
Hadoop—protecting data at rest, in use and in motion, and
maintaining the value of the data for analytics
– Together enabling comprehensive security for the enterprise, and
rapid and successful Hadoop adoption!
16
Thank you
Contact information
17