Download NASDAQ Case Study

NASDAQ OMX Group Reduces 2 PB of Data to
500 TB with Microsoft In-Memory Technology
Customer: NASDAQ OMX Group
Website: www.nasdaqomx.com
Customer Size: 2,433 employees
Country or Region: United States
Industry: Financial services—Capital
markets and securities
Customer Profile
National Association of Securities Dealers
Automated Quotation (NASDAQ) OMX
Group is a leading provider of trading,
exchange technology, information, and
public company services across six
continents. Based in New York City, the
company earned US$1.9 billion in
revenue in 2013.
Software and Services
 Microsoft Server Product Portfolio
− Windows Server 2008 R2 Enterprise
− Microsoft SQL Server 2014
Enterprise
“SQL Server 2014 is a game changer for us. Its inmemory columnstore, advanced statistics, and
cardinality estimation are the tools we need to manage
our very large databases for the long term.“
Stanley Orlowski, Director of Database Structures, NASDAQ OMX
NASDAQ OMX Group is a leading global provider of financial
trading services, exchange technology, and market insight. Every
year, it adds more than 500 terabytes (TB) of uncompressed data
to its archives. To better manage data growth, maintain fast
queries, and reduce database maintenance time, NASDAQ OMX
migrated data warehouses to SQL Server 2014. As a result, the
company will save 500 TB of disk space this year by using new
data compression and indexing capabilities. The company is also
saving time and money on data management, improving query
performance, and helping to facilitate long-term goals.
Business Needs
Established in 1971 as the world’s first
electronic stock market, NASDAQ OMX
owns and operates three clearing houses,
five central securities depositories, and 26
markets (including the NASDAQ Stock
Market) with a combined value that
exceeds US$8 trillion. NASDAQ OMX is
also a technology company. Its
infrastructure includes custom applications
and a trading engine that’s used internally
and by 80 global marketplaces.
For more information about other
Microsoft customer successes, please visit:
www.microsoft.com/casestudies
The volume of transactions and data that
NASDAQ OMX manages is impressive.
When markets open, the company
processes more than 1 million messages
per second. In addition, the company
manages many very large databases.
Stanley Orlowski, Director of Database
Structures at NASDAQ OMX, says, “Just
our US Options and Equity data archive
handles billions of transactions per day,
stores multiple petabytes of online data,
and has tables that contain quintillions of
records about business transactions.”
Uncompressed, the Options and Equity
archive measures 2 petabytes (PB). With
page compression, it measures 1.5 PB.
Financial traders need near-real-time insight
into current and historical data to make
decisions, in just seconds, that can involve
millions of dollars. Analysts also need indepth insight to understand trends and
create regulatory reports. To meet these
requirements in the past, IT staff had to
constantly monitor and fine-tune databases
so that they could provide the best possible
performance, especially with uncompressed
data growing by 500 TB each year.
By 2013, NASDAQ OMX recognized that its
existing database solutions couldn’t support
projected long-term growth. Orlowski says,
“Data is the lifeblood of the company—and
managing a petabyte of data takes a lot of
planning. You can’t turn quickly on a dime.”
Solution
When NASDAQ OMX learned about SQL
Server 2014 software and its in-memory
columnstore, IT staff decided to migrate
data warehouses to SQL Server 2014
Enterprise and the Windows Server 2008 R2
Enterprise operating system. By choosing
SQL Server 2014 and taking advantage of
in-memory columnstore—as well as other
new capabilities—the company can better
manage its massive amounts of data, speed
queries, and simplify database maintenance.
Not only does in-memory columnstore
serve as the only index for a table, but also
it can increase data compression ratios, and
it supports live table updates.
NASDAQ OMX can also improve query
performance and simplify maintenance by
taking advantage of incremental statistics
and the changes in the cardinality estimator
in SQL Server 2014. With these new tools,
table statistics can be updated more
frequently, and they can be updated for
individual partitions rather than for an
entire table. As a result, statistics are more
accurate, which helps produce better query
plans and faster query performance. SQL
Server 2014 also supports native backup
encryption so that engineers can encrypt
data in one step rather than two—and
encrypt directly to disk, tape, and the cloud.
Commenting on the company’s continued
use of the Microsoft platform, Orlowski
says, “The Microsoft technology stack
gives us excellent total cost of ownership,
and we have a high degree of confidence
in the platform. We’ve also had SQL Server
in place for well over a decade on missioncritical systems, and it has a positive track
record for delivering reliable performance.”
In 2013, engineers initially migrated the
Nordic OMX data warehouse (which
includes 8 TB of page-compressed data) to
SQL Server 2014. Engineers are currently
upgrading the US Options and Equities
archives to SQL Server 2014. They expect
to complete the database upgrade by the
middle of 2014 and finish converting
tables to in-memory columnstore by the
end of 2014.
Benefits
By moving its data warehouses to SQL
Server 2014, NASDAQ OMX saves time,
money, and disk space—and keeps its
massive data volumes manageable.
Compresses Data to One-Quarter Its
Original Size, Saving 500 TB of Space
In the past, NASDAQ OMX could compress
500 TB to 250 TB with page compression
in SQL Server 2012. However, with SQL
Server 2014, the company can realize even
greater compression levels, reducing data
to one-quarter of its original size. Orlowski
explains, “With SQL Server 2014, we can
reduce 250 TB of page-compressed data
to 125 TB using in-memory columnstore.
After we migrate all of our US data
archives to SQL Server 2014, we expect to
reduce our storage footprint by half a
petabyte. We’ve seen on average a 50
percent reduction in the amount of data
that’s physically stored on disk. And not
only does it help us save money, but it also
helps us more efficiently and effectively
deliver data to our consumers.”
Speeds Queries and Increases Accuracy
NASDAQ OMX also expects to facilitate
faster and more accurate insight into its
This case study is for informational purposes only.
MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
Document published April 2014
massive data stores. “After an initial
migration to SQL Server several years ago,
we were able to rework queries so instead
of taking a day or two to run, these same
queries run in just minutes,” says Andrew
Geaslin, Principal Database Administrator
at NASDAQ OMX. “Upgrading databases
to SQL 2014 and taking advantage of inmemory columnstore will make it easier
for us to hold on to those performance
improvements because statistics
maintenance is easier.”
Saves Time Spent on Maintenance and
Cuts Storage Footprint in Half
IT personnel will also save disk space with
in-memory columnstore and new
encryption capabilities. “SQL 2014 allows
us to save disk space on multiple fronts,
making it possible for us to cut our
existing storage footprint for the Options
and Equities archive in half,” says Geaslin.
“The database itself has a smaller storage
footprint on disk due to in-memory
columnstore, and encrypted backups
reduce the need for swap space on disk
used by third-party tools to encrypt data
before it’s moved to storage.” IT personnel
will also increase efficiency and reduce
complexity. “By consolidating multiple
indexes into a single in-memory
columnstore, we reduce our maintenance
and troubleshooting overhead,” says
Geaslin.
Improves Decision Making and Helps
Facilitate Long-Term Goals
Maintaining its ability to deliver rapid and
accurate insight into current and historical
market data is critical for NASDAQ OMX.
With its new solution, the organization has
the platform it needs to effectively manage
petabytes of information. “SQL Server 2014
is a game changer for us. Its in-memory
columnstore, advanced statistics, and
cardinality estimation are the tools we
need to manage our very large databases
for the long term,” says Orlowski. “By
making data more readily available to our
customers, we can also help them make
better business decisions.”