Download In-memory Spatial database based on key-value pair

IN-MEMORY
SPATIAL DATABASE
BASED ON KEY-VALUE PAIR
Topics
 In-memory Spatial Database based on key-value
pair
 Building Information Model based on relationship
inference
In-memory Spatial Database
based on key-value pair
 Background
 Architectural Overview
 Features
Background
This research is supported to solve kernel issues
during the State Grid information construction.
The achievements have been applied to the power
information systems.
Deploy view
State Grid
Beijing
Grid
User1
User2
Hubei
Province
Grid
Shanghai
Grid
…
User1
User1
User2
…
User2
…
…
User1
User2
…
Technical Architecture
Businesses
Planning
Security
Data
Schedule
Sale
…
Application
Integrate
Systems
ERP
Network
Marking
GIS
…
Product
Power Grid Services
Manage
Analysis
Spatial
…
Monitor
System
Emergency
Database
…
Resource
Graphics
Vector
…
Physical
Network
Storage
Software
…
…
Background
Key problems
Others
Store
Data
Access
Organize Data
Data
Change
High concurrency

notify
Reuse
High throughout

existing power

Large
Partition
scale
based
dataon
 Batch tosuch
submit
data
database,
as Oracle
Key-value
spatial
and
topology
 long period,
offline
Spatial,
SDE, DB2
Spatial
Transaction

logical
data
model
data edit
Extender

Locks
Version
Increment
data query

Data
synchronize

Lazy
History
write
Database
to
DBMS
…
Can’t be accessed

directly by user
 hybrid index based on
spatial and topology
Background
Why not traditional spatial database?
 Why not DBMS, such as Oracle?



 Bad I/O throughout
 Bad performance when
high concurrency and
Why not SDE or ArcServer?
batch edit
Can’t
provide
necessary
 More
slow
than raw
data type,
 Bad
performance
suchfeatures
BLOBare some bugs in
asThere
Why not object-oriented, such as 
Oracle
spatial?
Spatial
has bugs and has to
Linuxindex
be rebuilt
atprovide
fixed period
Can’t
necessary
Spatial
index performance is bad
features
Precise
is evenindex,
slow, such
Why not traditional spatial index,No.
such
asfilter
Quad-tree,
R*-as
We need
a hybrid
sdo_relate
including
spatial, topology and
tree?
Spatial function
is not necessary
attributes
information
Background
Why in-memory spatial database on key-pair
 Why store data in memory? How is the data
persistence?



 in-memory database is
much fast than disk
database
 The changed data will
Why use key-pair? What is your spatial
model?

high
beSimple
writtenmeans
into DBMS
performance!
 In-memory database

We use
the complex
hybrid
doesn’t
need
Do you use spatial index?
index,
searchincluding spatial,
topology
and attribute
 We support
simple
information
compare function ,such
Do you support condition query like SQL?
as greater, equal, less…
d
tp
Data synchronize
ht
ht
tp
Data synchronize
d
d
http
http
S
co erve
mp r
ut
er
S
co erve
mp r
ut
er
er
Serv ter
u
comp
er
Serv ter
u
comp
cluster
cluster
Data synchronize
Server
computer
d
Architectural-Logical View
Server
computer
Apache httpd
La
zy
wr
it
e
L
y
az
i
wr
te
In-memory
Database
httpd
httpd
Server computer
DBMS
Architectural-Framework
Data
synchronize
HTTP protocol
Pool refresh
Pool Fill
Query
Persistence
Access API
Index
Thread
management
Hash
Index
Page
management
Memory
schedule
Spatial
Index
Index library
Cache
Memory management
In-memory database
Connect pool
Data edit
Features-Data Model
Indeed, every record is a memory block!
 Typical data struct
protocol head description field value 1 field value 2
…
 Index for attributes
B-tree index, Hash index
 Spatial index
Spatial index is a hybrid index:
B-tree + grid index + topology based index
field value n
Features-Cluster
In my research, cluster is a group of virtual, loosely coupled inmemory database that work together closely.
 In a cluster, all in-memory databases have the same data.
 For different group, maybe they have intersected data.
 How to partition the power data for cluster is based on the
spatial , topology and attribute information
 When some data are updated in one in-memory database, other
in-memory database which has these data will be synchronized
almost real-time.
Features-Change notify
When data is update, clients will receive the notify.
 The notification granularity is based on object, not
record, which help to
 The notify is based on subscribe & notify model.
 User can only download changed records.
message server
Features-Version & History DB
Provide version mechanism and history
database based on time axis .
 Sometimes, work maybe needs several days,
even a month to finish. These data will be
managed as a version and other client can’t
see them before finishing the work.
BASE
version 1
conflict
resolve
conflict
resolve
 At any time, in-memory can create database
snapshot for archive or future comparative
version 2
analysis.
Snapshot 1
commit
commit
version 3
Snapshot 2
Time axis
2012年4月1日
2012年2月21日
2012年7月1日
2012年8月21日
BIM based on relationship
inference
 Background
 Roadmap
Background
 In common, the life cycle of a building
includes 3 phases:
 IFC is a vendor-neutral BIM data
repository for the semantic
information of building objects,
including geometry, associated
properties, and relationships, to
facilities.
come from T.E. El-Diraby
(ABS)
IfcObject
RelatedObjects S[1:?]
(ABS)
(ABS)
IfcPropertyDefinition
IfcRelationship
1
IFC2…
(ABS)
IfcRelDefines
IfcRoot
1
IFC2x
(ABS)
IFC2.0
IfcPropertySetDefinition
IFC1.5
IFC1.0
HasProperty Sets *L[1:?]
IfcObject
(INV) Def inesTy pe S[0:1]
(ABS)
*IfcTypeObject
IfcRelAssociates
RelatingTy pe
(INV) Ty peObjectOf
IfcProperty
S[0:1]
(ABS)
IfcRelDefines
RelatingProperty Def inition
(ABS)
IfcRelConnects
(INV) PropertyDef 200
initionOfIfcRelationship
0
400
600
S[0:1]
1
1
800
IfcRelDefines
IfcRelDefines
classes in IFC
(ABS)number of ByProperties
(ABS) ByType
IfcRelAssigns
IfcRelDecompose
s
come from T.E. El-Diraby
IfcXxxProperties
IfcPropertySet
Background
Impression
 IFC is a complex, relatively complete and huge specification.
 From the OO point of view, IFC is a well designed and loose coupled
class set.
Background
Problems and Limitations for IFC
 For OO design, message or event driven is an import design pattern,
but it seems there is no definition about it.
 IFC doesn’t define something about analysis, which is important for
management phase during the life cycle of a building.
 The ability of relationship description is much weak and can not
describe the implicit relationship automatically, which is import for
analysis and modeling.
GIST
belong to
OTB
Liu
Liu
belong to
GIST
belong to
OTB
Roadmap
Goals
 Provide the ability of relationship inference.
 Improve the degree of automation during the process of building
information modeling.
 Provide a mechanism for building information analysis
Roadmap-relationship inference
Building Modeling
Building Analysis
IfcXXXRelations (derived from IfcRelationship)
Relationship Operator(aggregation, group, order, add,
subtract,..)
Classificat
ion
relationshi
ps
Constrain
Property
Formal Description
IFC
(IfcRelationship)
OWL
 hasValue
transitive

Classification based on
Classification based on
type

symmetric
the
exists
at of relation
number
 subsumption(subClass occur
inverse

at
 unary
of,
superClass-of,
…)
Binary
mereology(is-part-of,)
equality
maximum
minimum
topology(next-to,
triple

touch-with, …)
….
Roadmap-analysis based on
event driven
Initial thoughts
 Information transmission is modeled a virtual pipeline.
 Every object will get information from pipeline and analyze them. The
result is also pushed into the pipeline.
 The purpose: 1) object is self-contain; 2) object is independent of
other object; 3) combine different analysis
building information pipeline
analysis
IfcObject 1
analysis
IfcObject 2
analysis
IfcObject ...
Thank you