Download Public Network aka Cluster Interconnect Oracle

Updates from Database Services at CERN
Andrei Dumitru
CERN IT Department / Database Services
Credit: Mariusz Piorkowski
Databases at CERN
~100 Oracle databases, most of them RAC
•
•
Mostly NAS storage plus some SAN with ASM
More than 500 TB of data files for production DBs in total
Example of critical production DBs:
•
LHC logging database ~170 TB, expected growth up to ~70 TB / year
But also as DBaaS, as single instances
•
•
•
MySQL Open community databases
PostgreSQL databases
Oracle 11g
4
Accelerator logging
~ 95% data reduction
Millions of records/min
~ 1 million signals
~ 300 data loading processes
~ 4 billion records/day
~ 160 GB/day  52 TB/year stored
~ 250000 signals
~ 15 data loading processes
~ 5.5 billion records/day
~ 275 GB/day  100 TB/year
throughput
Credit: C. Roderick
5
Administrative systems
•
•
•
•
All applications are based on Oracle as database
Oracle Weblogic server manages numerous HR and administrative
Java-based web applications used by CERN
Java EE and Apex
Oracle HR is E-Business Suite R12
6
Engineering & equipment data
•
•
Managing a million components over a lifecycle of 30 years
Integrated PLM platform by linking together commercial systems
3D CAD
CATIA
Design data
management
Manufacturing
follow-up
Installation
follow-up
Maintenance
management
Data publishing
Workflow actions
Credit: D. Widegren
7
Experiment systems
Online
•
Data-taking operations
•
Rely on SCADA system to store and monitor the detector parameters
(temperatures, voltages, …)
•
Up to 150.000 changes/second stored in Oracle databases
Offline
•
Post data-taking analysis, reconstruction and reprocessing, logging of
critical operations, …
Database replication:
•
Oracle Streams, Oracle Golden Gate, active standby databases
8
New DBs Services
QPSR
•
•
•
•
•
Quench Protection System
will store ~150 K rows/second (64GB per redo log)
1 000 000 rows/second achieved during catchup tests
need to keep data for a few days (~ 50 TB)
Doubtful that previous HW could handle that
SCADAR
•
•
•
Consolidated WinCC/PVSS archive repository
Will store ~50-60K rows/second (may increase in the future)
the data retention varies depending on the application (from a few
days to 5 years)
9
Service Lifecycle
HW Migration
SW upgrade
New systems
Installation
Decomission
Stable
services
Every ~3-4 years
10
Preparation for LHC Run 2
Requirement: changes had to fit LHC schedule
New HW installation on critical power
• Decommission of some old HW
• Critical power move from current location to new
location
Keep up with Oracle SW evolution
Applications’ evolution - more resources needed
Integration with Agile Infrastructure @CERN
LS1: no stop DB services
11
Service Evolution during LS1
Hardware evolution
• New DB servers
Storage evolution
• New generation of Storage servers
Refresh cycle of OS and OS related
• Puppet
• RHEL 6
Database Software evolution
• Upgrade to newer Oracle version
12
Oracle Real Application Clusters - Overview
Node 1
Public Network
Oracle RAC
Instance 1
Oracle
Clusterware
Operating
System
Private Network
a.k.a Cluster Interconnect
Node 2
Oracle RAC
Instance 2
Oracle
Clusterware
Operating
System
Storage Network
Oracle Database
on Shared Storage
Example of 2-node DB cluster
13
Our Deployment Model
Database Clusters with RAC
Servers
•
Running Red Hat Enterprise Linux
Storage
•
•
NAS (Network-attached Storage) from NetApp
High capacity SATA + SSD cache
Network
•
10 Gig Ethernet - for Storage, Interconnect, Users
Number of nodes: 2 – 5
14
New Hardware
Consolidation of HW
•
100 Production Servers
•
•
•
Dual 8 core XEON e5-2650
128GB/256GB RAM
3x10Gb interfaces
Specific network requirements
1.
IP1 (cs network)
2.
ATLAS Pit
3.
Technical network
4.
Routed network accessible from outside of CERN,
5.
Non-routed network only internal to CERN
Credit: Paul Smith
15
Storage Evolution
NetApp FAS8060
NVRAM
1.0 GB
8.0 GB
System memory
8GB
64GB
CPU
1 x 64-bit 4-core 2.33 Ghz
2 x 64-bit 8-core 2.10 Ghz
SSD layer (maximum)
512GB
8TB
Aggregate size
180TB
400TB
OS controller
Data ONTAP®
7-mode
Data ONTAP®
C-mode*
* Cluster made of 8 controllers. Shared with other services.
Credit: Ruben Gaspar
scaling out
scaling up
NetApp FAS3240
16
Storage Evolution
Centrally managed storage
• Monitored: Netapp + home made tools
Enables consolidation
Thin provisioning on file systems
Transparent volume move
More capacity for growth
More SSD => Performance gains for DB service
~2-3 times more of overall performance
Credit: Ruben Gaspar
17
Advantage of the New Hardware
More memory & more CPU
•
•
MEM:
DB Cache:
RAM 48GB -> 128GB / 256GB
20GB -> 86GB / 197GB
Faster storage
•
Storage cache
HW Migration
18
Preparation for LHC Run 2 - software
Available Software releases
•
Production databases on Oracle 11.2.0.3 before LHC LS1
•
All databases were upgraded and migrated to new hardware
Oracle 11g – version 11.2.0.4
•
Terminal patch set of Oracle 11g
•
Extended support ends January 2018
Oracle 12c - versions 12.1.0.1 and 12.1.0.2
•
First release of 12c and the subsequent patch set
•
Users of 12.1.0.1 will have to upgrade to 12.1.0.2 or higher by 2016
No current Oracle version fits well the entire LHC Run 2.
19
Consolidation
Schema based consolidation
•
Many applications share the same RAC cluster
•
Consolidation - per customer and/or functionality
Host based consolidation
•
Run different DB services on the same machine
•
Support for different Oracle homes (versions) on the same host
RAC clusters
•
Load Balancing and possibility to growth
•
High Availability: cluster survives node failures
•
Maintenance: scheduled rolling interventions
20
Replication
Disaster Recovery Data Guard Databases in Wigner
Active Data Guard available to users for read-only operations
Streams to Golden Gate migration completed!
•
Improved scalability – performance better than Streams
•
ATLAS ONLINE – OFFLINE (condDB)
•
ATLAS OFFLINE – TIER1s
•
•
•
•
RAL
IN2P3
TRIUMF
LHCb ONLINE-OFFLIN
21
Scalable Databases
Goal: open and scalable analytic platform for the data currently stored in traditional databases
•
LHClog / control systems archives / other monitoring and auditing systems
Solution - Hadoop cluster
•
shared nothing - scalable
•
open systems - many approaches on storing and processing the data
Conclusions
•
Data processing with Hadoop scales-out - no matter what engine you will use
•
Choosing the right data format for storing certain data is a key to deliver high
performance
All the details in “Evaluation of distributed open source solutions in CERN database use cases”
by Kacper Surdy (Tuesday)
22
Database on Demand
https://cern.ch/DBOnDemand
Database on Demand
Covers a demand from CERN community not addressed by the
Oracle service
•
Different RDBMS: MySQL, PostgreSQL and Oracle
Follows a DBaaS paradigm
Making users database owners - full DBA privileges
No access to underlying hardware
No DBA support or application support
No vendor support (except for Oracle)
Foreseen as single instance service
It provides tools to manage DBA actions: configuration,
start/stop, upgrades, backups & recoveries, instance monitoring
24
Database on Demand
Evolution of the amount of MySQL, Oracle and
PostgreSQL instances in the DBoD service
25
Database on Demand instances per
Database Management System
26
Enhanced Monitoring now available
27
Acknowledgements
Work presented here on behalf of the
•
CERN Database Services group
In particular key contributions to this presentation
from: Marcin Blaszczyk, Ruben Gaspar, Zbigniew
Baranowski, Lorena Lobato Pardavila, Luca
Canali, Eric Grancher
28