Download - OpenStack

Lockless Upgrades
Upgrading OpenStack without locking up your DB
Alicja Kwaśniewska, Grzegorz Grasza, Roman Dobosz
Agenda
▪ Problem Overview
▪ Various approaches from OpenStack projects
▪ Test setup (POC)
▪ Test results
▪ Q&A
2
Problem Overview
Problem overview
▪ Operator wants a particular feature
▪ This feature exists only in a newer version
of OpenStack
▪ Newer service has changes in the
database schema
▪ But… Operator doesn’t want to have
downtime
4
Ultimate goal:
rolling upgrade
v1
v2
5
Things to consider
▪ Communication protocols compatibility
–RPC (RPC versioning,
Oslo Versioned Objects)
–REST API
▪ Persistence layer compatibility
–Online schema migration
–Data migration
An Operator upgrading glance image API
6
Online Schema Migration requirements
Aim: services are always running and using DB
compatibility between versions has to be preserved
7
Current status
and various approaches
in OpenStack
Current status of upgrades in selected projects
▪Nova – schema and data migrations in place
▪Cinder – online upgrade tech preview
▪Keystone – doubts about DB locks led to the idea of a POC
implementation
▪Neutron – expand-contract approach implemented
▪Heat – code for live db migration in in place but not yet tested
https://etherpad.openstack.org/p/AUS-ops-Liberty-upgrades
9
Keeping the database compatible (cinder approach)
column
write
read
write
app
column
app
read
data
migrate
script
X
new
column
X+1
10
Keeping the database compatible (cinder approach)
write
column
app
read
column
write
read
data
migrate
script
X+1
app
new
column
new
column
X+2
11
Keeping the database compatible (cinder approach)
column
write
read
app
column
removed
write
read
app
new
column
new
column
X+2
X+3
12
Keeping the database compatible (cinder approach)
X(base release)
X+2
• Writes data in old place
• Write data in both places
• Reads data from old place
• Read data from new place
X+1 (change introduction)
X+3
• Write data in both places
• Stop writing data to old place
• Read data from old place
• Drop old (unused) schema
https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/online-schema-upgrades.html
13
Nova approach
schema v1
schema v2
data v1
data v1
schema migration
data migration
schema v2
schema v2
data v1
data v1/v2
schema v2
data v2
drop column
schema v2
data v2
http://www.danplanet.com/blog/2015/10/07/upgrades-in-nova-database-migrations/
14
Altering schema in one release (keystone
approach)
class RoleAssignment(sql.ModelBase, sql.DictBase):
(…)
if CONF.sql.backward_compatible:
# TODO(xek): remove in Newton cycle
role_id = sql.Column(sql.String(64), nullable=False)
role_id_v1 = sql.CopyColumn(sql.NewType(), copy='role_id')
else:
role_id = sql.Column('role_id_v1', sql.NewType(), nullable=False)
(…)
15
Table access locks
Table locks - MySQL internals
read
table1
deleted
table1
write
lock
temp copy
read
table1’
write
table1’
renamed
to table1
17
MySQL 5.6 ALTER operations
In-place?
Allow concurrent
INSERT/UPDATE/
DELETE/?
Allow
concurrent
query?
yes
yes
yes
yes
Using ALGORITHM=INPLACE will prevent
internal table copying
drop column
yes
yes
yes
yes
Dropping a virtual generated column will not
copy the table, all the other column types will
copy
add primary key
yes
yes
yes
yes
Although ALGORITHM=INPLACE is allowed,
the data is reorganized substantially, so it is
still an expensive operation.
convert character
set
yes
no
no
yes
add index
no
yes
yes
yes
Creating first FULLTEXT index involves table
copy
rename column
no
yes
yes
yes
Concurrent DML is available only for
changing name of the column, not it’s type
Table
copy?
add foreign key
Operation
Notes
Table rebuild is performed
https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html
18
PostgreSQL Operations
DDL operation
Doable on large
tables?
Add a new column
yes
Drop a column
yes
Add index concurrently
yes
Drop constraint
yes
Add default to an existing
column
yes
Add a column that is not
nullable
no
Required new table with not null constraint, fill the data from the source table and
switch source with new table
Add column with unique
constraint
no
Add column, add unique index concurrently, then add constraint using alter table
using index
Add foreign key
no
Add it with NOT VALID, then VALIDATE CONSTRAINT (takes a weaker,
ShareUpdateExclusive lock in 9.4)
VACUUM FULL
no
Use pgrepack tool instead
Workaround
(using the CONCURRENTLY keyword)
https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/
19
Experiments
POC
▪ Why? To have a quick benchmark for
– testing the problem with table locks
– testing any solutions which may fix it
14 examples:
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
add column
drop column
add table
drop table
add foreign key
drop foreign key
add constraint
drop constraint
add index
drop index
retype column
change primary key
ascii -> utf8
21
The “DDLoad” tool
schema
initial
workload
test
workload
...
Jinja2
tsung.xml
schema
changes
run tsung
generate
graphs
compare
with
tsplot
22
MySQL 5.5 results - part 1
23
MySQL 5.5 results - part 2
24
PostgreSQL 9.2 results - part 3
25
PostgreSQL 9.2 results - part 1
26
PostgreSQL 9.2 results - part 2
27
POC – first results
▪ Performance problem confirmed
–on MySQL 5.5, but deleting an unused table is ok
–on PostgreSQL 9.2 while adding a foreign key
▪ There are locking errors returned by the DB after a change is triggered
▪ it's not the load – making only one query which locks the table for
a considerable amount of time will be a problem
28
Approaches to Online
Schema Changes
Solution 1: Upgrade database
▪ MySQL 5.6
▪ PostgreSQL 9.4
30
Solution 2: Online Schema Change (triggers)
write
table1
Update rows
using trigger
table1
deleted
read
group copy
write
table1’
read
table1’
renamed
to
table1
31
Tools for MySQL
▪ Percona pt-online-schema-change (Perl)
▪ Facebook OSC - Online Schema Change (PHP)
▪ openark kit (Python)
32
Online Schema Change on PostgreSQL
▪ CONCURRENTLY keyword for adding/dropping indexes
▪ ALTER TABLE ADD CONSTRAINT USING INDEX
▪ VALIDATE CONSTRAINT takes a weaker, ShareUpdateExclusive lock
▪ pgrepack tool for VACUUM FULL
33
Solution 3: Write to two tables
table
write
read
table
write
app
app
read
new
table
data
migrate
script
X
X+1
table
remove
d
table
write
read
write
app
read
app
new
table
X+2
new
table
X+3
34
More results
Time in minutes
36
Time in minutes
37
38
PostgreSQL 9.5 results - adding
39
PostgreSQL 9.5 results - removing
40
41
Key takeouts
● MySQL
▪ Upgrade your database to at least version 5.6
▪ Try pt-online-schema-change
● PostgreSQL
▪ Divide your queries and use the CONCURRENTLY, ALTER WITH
INDEX, NOT VALID, VALIDATE CONSTRAINT keywords
▪ Use transactional DDL, so you can revert long running queries
42
DDLoad – testing your own approach
▪ You can test your own approach with DDLoad:
1. Drop in your schema and schema change
2. Record your sessions with Tsung proxy tool
3. Run the tests
4. Graph and analyze the performance impact and error rates
DDLoad tool: http://github.com/xek/ddload
43
Thank you !
Q&A
DDLoad tool: http://github.com/xek/ddload
44
Legal Notices and Disclaimers
▪ Intel technologies’ features and benefits depend on system configuration and may require enabled
hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.
▪ No computer system can be absolutely secure.
▪ Tests document performance of components on a particular test, in specific systems. Differences in
hardware, software, or configuration will affect actual performance. Consult other sources of
information to evaluate performance as you consider your purchase. For more complete
information about performance and benchmark results, visit http://www.intel.com/performance.
▪ Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
▪ © 2016 Intel Corporation.
45
Upgrading OpenStack? Sit With a Panel of Experts to
Hear Their Stories
Operations
Tuesday, April 26, 4:40pm-5:20pm
Austin Convention Center - Level 1 - Ballroom A
46
###################################
Online Schema Migration Approaches
(operator’s perspective)
▪
Nova: Migrate first, then upgrade code
# db sync
# service start
# nova live-data-migration
▪
Upgrade code, then migrate the database
# service start
# db sync
▪
}
▪
In the future in Neutron, you will be able to do both:
# neutron-db-manage upgrade --expand
# systemctl restart neutron-server
# neutron-db-manage upgrade --contract
In Kolla, the upgrade is as easy as:
changing the openstack_release config, then
# kolla-ansible upgrade
48
37 000 records
DROP INDEX
49