Download Slide 1

Активное распределенное
хранилище для многомерных
массивов
Дмитрий Медведев
ИКИ РАН
Scientific data arrays
•
Arrays are widely used in environmental sciences to store modelling results,
satellite observations, raster maps, etc.
•
Datasets can be quite large, up to several terabytes.
•
Most data are stored as file collections in proprietary formats or universally
adopted formats like netCDF, GRIB, HDF5.
•
File access can be problematic:
 Scientists need to know about too many file formats
 Usually files must be completely downloaded before they can be used
 Thousands of files can be processed in one data request; only a small
portion of their contents appears in the result set
•
Currently available database solutions do not have convenient array storage
capabilities.
ActiveStorage
• ActiveStorage is a generic storage for arrays of primitive
data types.
• Its data model is based on the Unidata’s Common Data
Model, used in netCDF, HDF5 and OpenDAP.
• Basically, ActiveStorage is a SQL Server database with CLR
stored procedures and a client library.
• The stored procedures and the client library provide an
abstraction layer for data access.
• Large arrays are split into chunks and can be spread across
several parallel database servers for better performance.
ActiveStorage
RDBMS
Binary data,
metadata
RasDaMan
RDBMS
Binary data,
metadata
Stored
procedures
Middleware
Client library
Client library
SciDB
Common Data Model
Dataset
-name
Group
-name
DataType
Dimension
Attribute
-name
-length
-name
-value
-dataType
Variable
-char
-byte
-short
-int
-long
-float
-double
-String
-name
-shape
-dataType
This is the Common Data Model (CDM) used in the recent versions of OpenDAP, netCDF
and HDF5. Its purpose is the representation of multidimensional scientific data.
Database schema
dimensions
groups
PK
dim_id
PK
group_id
FK1
dim_group
dim_name
dim_length
FK1
group_name
parent_id
grp_attributes
shapes
PK,FK2
PK
FK1
var_id
dim_index
dim_id
dim_type
variables
PK
var_id
FK1
var_group
var_name
var_type
data_table
index_table
vector_size
FK2
PK,FK1
PK
group_id
att_name
FK2
att_type
att_value
types
PK
type_id
servers
PK,FK1
PK
PK
PK
type_name
type_length
var_id
host
port
db_name
login
passwd
var_attributes
PK,FK2
PK
var_id
att_name
FK1
att_type
att_value
data table
data table
data table
PK chunk_key
PK chunk_key
PK chunk_key
chunk
chunk
chunk
directory table
directory table
directory table
PK,FK1 chunk_key
PK,FK1
chunk_key
PK
dim_index
PK,FK1
chunk_key
PK
dim_index
PK
dim_index
key_min
key_min
key_max
key_min
key_max
key_max
Splitting an array into chunks
Non-chunked array
1 seek
8 seeks
Chunked array
• We store chunks in BLOB fields of a database table
• Chunks do not need to be the same size
4 seeks
4 seeks
chunk_key
chunk
0
<Chunk0>
1
<Chunk1>
2
<Chunk2>
3
<Chunk3>
Data and directory tables
ns1_air0_directory
ns1_air0_data
PK
chunk_key
chunk
PK,FK1
PK
chunk_key
dim_index
I1
I1
key_min
key_max
Two tables are automatically created for
each new variable:
• Data table
• Directory table
The data table stores data chunks in BLOB columns.
The directory table contains information about chunk boundaries.
A chunk consists of a header and a data block.
Header
x0min
x0max
...
Data
xn-1min xn-1max
How it works
1. Pass multi-dimensional data
request to the client library
2. Issue commands to
the database server
Application
Client library
4. Assemble the data parts into
one multi-dimensional array
SQL Server
DB
3. Return the data parts
to the client library
3. Select the requested
data from several chunks
Parallel query processing
SQL Server
DB 1
Application
Client library
SQL Server
DB 2
Parallel query performance
1 database server
4 parallel database servers
NCEP/NCAR Weather Reanalysis
• Continually updating gridded data set
• Incorporates observations and global
climate model output
• 74 weather parameters
• 5000 netCDF files, 30 – 500 MB each
Time coverage:
Grids:
• 1948 – 2008
• Regular grid, 2.5 x 2.5 degrees
• 4-hourly values
• T62 Gaussian grid, 192 x 94 points.
Database contents
NCEP/NCAR Weather Reanalysis Database
Group: “ns5”
Group: “ns2”
Group: “ns4”
Group: “ns1”
Group: “ns3”
“time”
“time”
“lat”
data
data
data
“lon”
“lat”
“lon”
“level”
ns1 – Single-layer data on regular grid
ns2 – Single-layer data on Gaussian grid
ns3, ns4, ns5 – Multi-layer data on regular grid
data
data
data
NCDC Integrated Surface Database
Fixed ground stations
Ships
Mobile stations
Buoys
• 1901 – 2008 time coverage.
• 470 000 ASCII files packed with gzip.
• 30 million sensors.
• 50 GB packed; 400 GB unpacked.
• 1.7 billion observations.
When you’ve downloaded and unpacked the data...
Control data section
Mandatory data section
Section marker
Additional data section
0189010020999992007022817004+80050+016250FM-12+000899999V0202201N008019999999N0090001N1+00631+00541098651ADDGA1031+003009999KA1120N+99999...
date time
lat
lon
Group marker
Parameter group
Fixed stations
ActiveStorage database for NCDC data
The main challenges:
• Observation times are irregular
• Observations are distributed unevenly in time and
space
• Different stations have different sets of observed
parameters
• Huge number of observations
M
Modifications to ActiveStorage
ActiveStorage was designed to handle
dense multidimensional arrays, with only a
small number of missing values.
0
It works well for regularly gridded data.
0
N
Some multidimensional data are sparse and
can not be represented by a single data block.
Modifications to ActiveStorage
0
0
1
2
1
0
1
2
2
0
3
1
2
0
1
2
(3,0,x,y,z)
• Sparse arrays can be represented as a tree hierarchy of dense data blocks
• Some data blocks can be empty
• Hierarchy levels are treated as additional dimensions
Modifications to ActiveStorage
data
PK
chunk_key
directory
PK,FK1
PK
chunk_key
dim_index
I1
I1
key_min
key_max
chunk
data
PK
chunk_key
directory
PK,FK1
PK
chunk_key
dim_index
I1
I1
I1
var0
key_min
key_max
chunk
data
PK
chunk_key
directory
PK,FK1
PK
chunk_key
dim_index
I1
I1
I1
I1
var0
var1
key_min
key_max
chunk
Time series representation
Point IDs
Time series
• Time series are stored as a set of 1D arrays
• 1 array → 1 geographical point
• One geographical point may have observations from several sensors
• Sensors can be distinguished by observation parameters (station code,
observation type, call letters, etc.)
Buckets
Bucket IDs
Bucket
latitude
1⁰
2
2
5
9
5
6
9
longitude
Arrays of point IDs
1⁰
• The whole spatio-temporal domain is divided into buckets
• Each bucket contains a subset of observations from several geographical points
• A set of IDs of geographical points is stored as a 1D array
• For each bucket we store only those points that have observations in this bucket
Database contents
NCDC Integrated Surface Database
Group: “mandatory”
Group: additional
“time”
“buckets”
“time”
data
data
data
“buckets”
data
data
data
coords
pointId
PK
lat
lon
I1
I1
The “coords” table helps to select time series by latitude/longitude
Request processing chart
Get bucket ids
Return results
Read point ids
from bucket
Filter points by
coordinates
Data
storage
Read observation
times
Read observation
data
Filter points by
time
for each point
for each point
for each bucket
Request processing times
Location
Sensors
Observations
Time
Moscow
5
53621
1s
Madrid
13
50992
2s
Gulf of Guinea
195
3717
9s
• Moscow, Madrid – fixed stations
• Gulf of Guinea – buoys, ships
 Small number of sensors
 Large number of sensors
 Large number of observations
 Small number of observations
* All requests are 2 x 2 degrees, 01/01/2007 –
12/31/2007
ActiveStorage on Windows Azure
VMs
VMs
VMs
HTTP
Load
Balancer
IIS
Web
Role
Instance
Worker
Role
Instance
Agent
Agent
HTTP
Blobs
Windows Azure Fabric
Application
Storage
Compute
Fabric
…
Tables
Queues
How it works
Queue1
BLOB Storage
Processed chunks
Web Role
Raw chunks
Result
Queue2
Worker Role
Worker Role
Worker Role
ActiveStorage on Windows Azure
Advantages
 Easy and natural implementation of parallel query execution.
 BLOB read rates are quite good: 6.5 MB/s + 0.1 s overhead.
 Very scalable.
CTP problem: replication overhead
 BLOB writes are several times slower than SQL Server.
 Message exchange rate is slow (several seconds).