Download Hive Computing - Tribury Media, LLC

Hive Computing
A New Vision for
Mission Critical Applications
Presented to COOUG
February 11, 2003
January 29, 2003
2003 Tsunami Research, Inc.
Agenda
•
•
•
•
•
•
•
•
•
Definition
Executive Summary
Our History
The Problem
The Five Pillars
Benefits
Demo
The Technology
eXtreme Programming
Definition
HIVE COMPUTING
A revolutionary approach to the
development, deployment, and management
of applications
that involves enabling
large numbers of dedicated, commodity computers
to join together and form
a mission critical computing environment
called a Hive
Executive Summary
The goal of Hive Computing is
to make reliability affordable
In Essence
High Performance
Low Price
Commodity
Computers
In Essence
Performance
- Scalability
Supercomputing
Linux Clusters
Distributed Supercomputers
Beowulf
Unlimited Scale
Qlusters
Platform Computing
United Devices
Entropia
Data Synapse
High Performance
Low Price
Commodity
Computers
In Essence
Performance
- Scalability
Supercomputing
Linux Clusters
Distributed Supercomputers
Beowulf
Unlimited Scale
Qlusters
Platform Computing
United Devices
Entropia
Data Synapse
Reliability + Affordability
High Performance
Low Price
Commodity
Computers
- Scalability
- Availability
- Predictability
- Adaptability
- Maintainability
- Usability
Mission
Critical
Computing
Hive Computing
Tsunami Research
Our History
•
•
•
•
•
Significant transaction processing experience
Core team previously founded PaylinX Corporation
Leading real-time enterprise payment processing solution
PaylinX sold to CyberSource in 2000 for $130+ million
Unfinished business…
–
–
–
–
Scalability
Reliability
Complexity
Cost
The Problem
Mainframes
Fault
Tolerant
Computers
Scalability
Reliability
Availability


Predictability


Application
Servers
Linux
Clusters
Distributed
Supercomputers
Hive








Adaptability
Affordability



Maintainability

Usability
IBM
Unisys
Fujitsu
HP/Tandem
Stratus
BEA
IBM

Beowulf
Qlusters
Unlimited Scale
United Devices
Data Synapse
Entropia
Existing solutions force organizations to make trade-offs
Why Can’t Solutions…
•
•
•
•
•
•
Scale without limits?
Be immune to failure?
Guarantee the job will get done?
Keep up with a changing world?
Maintain themselves?
Be easy to use and develop for?
Old Problems Result From Old Assumptions
• TCP/IP
• Not a better centralized or decentralized network
• The core assumption…
–
•
Networks are unreliable
Design implications
–
–
–
The network could not have any centralized authority
Messages would be divided up into equally-sized, independently-routed packets
The system should be easy to expand by adding additional, inexpensive resources
New Assumptions Create New Possibilities
• The application is the only thing that matters
• Self-organizing
• Hardware and operating system layers relatively unimportant
• Computers will fail
• Self-healing
• Failure happens, deal with it
• Computers are disposable
•
•
•
•
Self-maintaining
Inexpensive
Make it very easy to add and remove hardware
Dedicate hardware to simplify solution & improve results
Application Fabric
Application
Hive
Commodity
Operating System
Hardware
Allows you to leverage commodity hardware
and inexpensive operating systems
(e.g. Windows 2000 and XP, FreeBSD, Linux)
Hive Computing
Application
Hive
Commodity
Computer
Commodity
Computer
Commodity
Computer
Commodity
Computer
Commodity
Computer
Delivers scalability and reliability without the cost
Commodity
Computer
The Five Pillars of Hive Computing
• Collective Intelligence
-
The intelligence of a Hive resides in every member
Peer-to-peer
Bottom-up, not top-down
Role-based
• Commodity Components
– PC-grade computers and blade servers
• Assimilation
– The Hive automatically brings new Workers on line
• Self-Monitoring
– The Hive monitors and recovers all Tasks
• Self-Maintaining
– The Hive maintains own configuration
Benefits
• Reliability :: A Hive has no reason to ever go down
– Scalability
• A Hive can scale in small increments and in a near-linear manner
– Availability
• No single point of failure
– Predictability
• The job gets done in a specified period of time
• Affordability :: Low acquisition, maintenance, & development costs
– Adaptability
• A Hive can easily adapt to changing conditions
– Maintainability
• Dramatically reduced administration and maintenance costs
– Usability
• Developers can focus on the application
Hive Architecture
Logical, not Physical
Sample Architecture
Client
Client
Client
Access Control
Authentication
Page Rendering
Client
Client
Client
Presentation Layer
Business Logic Layer
Application Server
High Value Operations
High Volume Operations
Client
Hive
Database
Client
Client
Sample Architecture
Client
Client
System
Hive
Client
Enterprise Applications
Web Services
Hive
System
Sample Architecture
Client
Hive
Hive
Grid
Resource
Pool
Resource
Pool
Demo
•
•
•
•
Collective Intelligence
Load balancing and fail-over
Fault tolerance
Scalability
Request/Response Loop
Hive Mind
Client
task
task
result
Worker
Machine B
task
request
response
Worker
Machine
task
result
Worker
Machine
Worker
Machine A
Worker
Machine C
Security
• SSL encryption between Workers and the Hive
• Authentication required to access Operations, Process Flows, and
Tasks
• Additional layers of security planned or in the works
Terminology
•
•
•
•
•
•
•
Hive
Worker
Package
Application
Operation
Process Flow
Task
Hive
Worker
Worker
Worker
Package
Application
Operation
Operation
Application
Operation
Operation
Operation
Process Flow
Task
Operation
•
•
•
•
A brokerage firm might perform the following Operations in a Hive...
– GetRealtimeQuoteForCUSIP
– GetDelayedQuoteForCUSIP
– GetChartForCUSIP
– CalculateCurrentPortfolioValue
A credit card company might perform the following Operations in a Hive...
– PerformAuthorization
– PerformFraudCheck
– CaptureTransaction
A mortgage company might perform the following Operations in a Hive...
– GetScoreForApplication
– CalculateMonthlyPayment
– CalculateEscrow
A security application might perform the following Operations in a Hive...
– AnalyzeFingerprint
– AnalyzeFace
– AuthenticateUser
Worker
Process Flow
Task
• An atomic function to be performed by the Hive (e.g. a syntax check
or database lookup)
• A Task can be written in C/C++ or Java
• A Task can optionally implement OnStartup and OnShutdown
functions to initialize memory and or the environment
Task Example
void makeStringUppercase(TASK_HANDLE request)
{
string data = task_getAttribute( request, "argument" );
data = ToUpper(data);
task_setAttribute( request, "argument", data );
task_setResult( request, "COMPLETE" );
}
•
•
•
•
All data passes into the task through the TASK_HANDLE data structure
Hive Developer’s task code reads arguments from the TASK_HANDLE object
The task performs operations as necessary
Any data updates are made to the TASK_HANDLE object
Limited Tasks
• Some tasks should not run on every machine such as connections
to limited resources
• These tasks can be ‘limited’ by the Hive to run on either X total
machines or Y percentage of machines
Rollbacks
• Application level rollbacks are handled via compensating tasks
within the process flow
• Only the application developer is aware of how to “undo” any of the
atomic actions that have been completed
• The Hive’s process flow allows the application developer to catch
errors generated by the task and send the transaction to a task to
undo any actions within the process flow
Databases and a Hive
• Hive Developers should leverage existing database solutions for
data storage
• Database connection code can be written as tasks for the Hive to
manage
• If one machine with a database connection goes down, the Hive can
bring another connection up
Hive API
• The Hive is accessed through thin client side API’s written in C/C++
or Java in Windows or UNIX
• Client API addresses the Hive as a whole, not a single machine
• Client API can use SSL connections when communicating with the
Hive
API Example
HAPI_PROCESS process;
process = hapi_createProcess();
hapi_setAttribute( process, "aString", "ABCDEF");
long rc = hapi_executeProcess(
process,
"integration_test_app",
"CaseProcess",
"224.0.0.1",
31000,
HAPI_NO_ENCRYPTION );
string resultString = hapi_getAttribute( process, "aString" );
string processResult =hapi_getProcessResult( process );
hapi_destroyProcess( process );
Deploying Hive Applications
•
•
•
•
Hivepkg bundles process flows, tasks, and support files and provides syntax
checking before deployment
Hivectl deploys applications to a Hive
Applications can be deployed to a running Hive
The Hive gracefully updates a subset of workers while the majority of the
workers continue processing transactions
eXtreme Programming
• Test-Driven Development
– Automated build process
– Each build is subjected to 3,000+ unit tests
– Extremely tight code
• Paired Programming
• The Planning Game
Suitable Applications
•
Transaction-Oriented applications
–
•
Financial Services & Insurance
–
–
–
•
Credit Card Processors
Trading Systems
Financial and process modeling
Homeland Security, Government, & Defense
–
•
Request/Response loop
Applications where scalability, reliability, & survivability are important
Biotechnology & Life Sciences
–
–
Bioinformatics
In-silico drug discovery
For More Information
Chris O’Leary
314.336.2841 – Direct
[email protected]
Don Mang
314.336.2840 – Direct
[email protected]
www.tsunamiresearch.com