Download Distributed (Operating) Systems -Architectures

Distributed (Operating) Systems
-Communication in Distributed SystemsFall 2011
Kocaeli University Computer
Engineering Department
Communication in
Distributed Systems
• Communication is done through message passing
• Expressing communication through message passing is
harder than using primitives based on shared memory
• Remote Procedure Calls
– Transparency but poor for passing references
– Ideal for client server applications
• Message-oriented Communication
• Stream-oriented Communication
– Continuous media
Communication Between Processes
• Unstructured communication
– Use shared memory or shared data structures
• Structured communication
– Use explicit messages (IPCs)
• Distributed Systems: both need low-level
communication support (why?)
Types of Communication -1
• Transient Communication
– A message is stored by the communication system
only as long as the sending and the receiving
application are executing
– Typically all transport level communication is transient
communication
• Persistent Communication
– A message that has been submitted for transmission is
kept by middleware as long as it takes to deliver it to
the receiver
– Receiving application don’t need to be executing
when the message when the message is submitted
Types of Communication -2
• Asynchronous Communication
– Sender continues immediately after it has submitted
its message for transmission
– It means, message is temporarily stored in
middleware upon submission
• Synchronous Communication
– Sender is blocked until its request is known to be
accepted
– Three points
• Sender informs message is delivered
• Middleware informs message is delivered
• Sender informs with the response after processing
Types of Communication -3
• Connection-oriented (telephone)
– TCP
• Connectionless (mailbox – dropping a letter)
– IP, UDP
Persistence and Synchronicity in
Communication
Persistence
• Persistent communication
– Messages are stored until (next) receiver is ready
– Examples: email, pony express
Transient Communication
• Transient communication
– Message is stored only so long as
sending/receiving application are executing
– Discard message if it can’t be delivered to next
server/receiver
– Example: transport-level communication services
offer transient communication
– Example: Typical network router – discard
message if it can’t be delivered next router or
destination
Synchronicity
• Asynchronous communication
– Sender continues immediately after it has submitted
the message
– Need a local buffer at the sending host
• Synchronous communication
– Sender blocks until message is stored in a local buffer
at the receiving host or actually delivered to sending
– Variant: block until receiver processes the message
• Six combinations of persistence and synchronicity
Persistence and Synchronicity
Combinations
a) Persistent asynchronous communication (e.g., email)
b) Persistent synchronous communication
Persistence and Synchronicity
Combinations
c) Transient asynchronous communication (e.g., UDP)
d) Receipt-based transient synchronous communication
Persistence and Synchronicity
Combinations
e) Delivery-based transient synchronous communication at message
delivery (e.g., asynchronous RPC)
f) Response-based transient synchronous communication (RPC)
Layered Protocols 1
• Communication messaging rules are defined in
protocols
• Due to the absence of shared memory, all
communication in distributed systems is based on
sending and receiving (low-level) messages
• Many different agreements
–
–
–
–
IBM’s EBCDIC and ASCII
How many volts should be used to signal 0
How does the receiver know which is the last bit
How can the receiver detect if a message has been
damaged
Layered Protocols 2
• ISO/OSI never widely used and are essentially
dead.
• In the OSI model communication is divided up
into 7 layers
• OSI layers were never popular. In contrast,
protocols developed for the internet, such as
TCO and IP, are mostly used
Communication Protocols
• There are some rules that communicating processes must adhere to –
known as protocols
• Protocols are agreements/rules on communication
• Protocols could be connection-oriented or connectionless
Layered Protocols
• A typical message as it appears on the network.
Layers -1
• Physical layer
– Transmitting 0s and 1s
– How many bits per sec transfer
– Can transmission take place in both direction
• Data Link Layer
– Puts special bit pattern on the start and end of each
frame
– Computing checksum
• Network Layer
– Routing
Layers -2
• Transport Protocol
– Turns the underlying network into something than an
application developer can use
– Messages from the application is broken into small piecespackets
– Which packets have been sent, which have been received, which
should be retransmitted
– Connection-oriented: messages arrive in the same order (as in
TCP)
– Connectionless: message can arrive in different order (as in
UDP)
– Another example of transport protocol is RTP
– The combination of TCP/IP is now used as a de facto standard
for network communication
Client-Server TCP
Layers -3
• Session Layer
– Enhanced version if transport layer
– Keeps track of which party is currently talking and
it provides a synchronization
– Puts checking points – in case of crash going to
the last check point
• Presentation Layer
– Concerned with the meaning of the bits
Middleware Protocols
• Middleware: layer that resides between an OS and an
application
– May implement general-purpose protocols that warrant their
own layers
OSI Model summary
• In this model, clear distinction between
applications, application-specific protocols and
general purpose protocols.
• Application-specific protocol
– FTP
– HTTP
• General-purpose protocols
– Useful to many applications but cannot be qualified as
transport protocols. These protocols fall into the
category of middleware protocols.
Middleware Protocols 1
• Some services are not tied to any specific
applications, but instead can be integrated
into a middleware system as a general service
– Authentication protocols
• Proof of claimed identity
– Atomicity protocols
• Widely applied in transactions
– Distributed locking protocols
• Resources can be protected against simultaneous
accesses
Middleware Protocols 2
• Also supports high-level communication
services
– Remote Procedure Call (RPC)
– Message Oriented Middleware (MOM)
– RTP: Streams for transferring real-time data, such
as needed for multimedia applications
Remote Procedure Calls
• Goal: Make distributed computing look like
centralized computing
• Allow remote services to be called as procedures
– Transparency with regard to location, implementation,
language
• Issues
– How to pass parameters
– Bindings
– Semantics in face of errors
Example of an RPC
No message passing at all is visible to the programmer.
Divide programs up and add
communication protocols
Client
Server
blah, blah, blah
bar = add(i,j);
blah, blah, blah
Int add(int x, int y ) {
if (x>100)
return(y-2);
else if (x>10)
return(y-x);
else
return(x+y);
}
RPC Semantics
• Principle of RPC between a client and server program [Birrell&Nelson 1984]
Other RPC Models
• Asynchronous RPC
– Request-reply behavior often not needed
– Server can reply as soon as request is received and execute
procedure later
• Deferred-synchronous RPC
– Use two asynchronous RPCs
– Client needs a reply but can’t wait for it; server sends reply
via another asynchronous RPC
• One-way RPC
– Client does not even wait for an ACK from the server
– Limitation: reliability not guaranteed (Client does not know
if procedure was executed by the server).
Asynchronous RPC
a)
b)
The interconnection between client and server in a traditional RPC
The interaction using asynchronous RPC
Deferred Synchronous RPC
• A client and server interacting through two asynchronous RPCs
Conventional Procedure Call
a) Parameter passing in a local
procedure call: the stack
before the call to read
b) The stack while the called
procedure is active
Count = read(fd, buf, nbytes)
Possible Issues
• Calling and called procedures run on different
machines
• They execute in different address spaces
• Parameters and results have to be passed, it can
be complicated when the machines are not
identical.
– How do you represent integers – big-endian littleendian
• Either or both machines can crash and each of
the possible failures causes different problems.
Parameter Passing
• Local procedure parameter passing
– Call-by-value
– Call-by-reference: arrays, complex data structures
• No sense BUT in RMI it is OK
• Remote procedure calls simulate this through:
– Stubs – proxies
– Flattening – marshalling
• Related issue: global variables are not allowed
in RPCs
Client and Server Stubs
• Client makes procedure call (just like a local
procedure call) to the client stub
• Server is written as a standard procedure
• Stubs take care of packaging arguments and
sending messages
• Packaging parameters is called marshalling
• Stub compiler generates stub automatically from
specs in an Interface Definition Language (IDL)
– Simplifies programmer task
Steps of a Remote Procedure Call
1. Client procedure calls client stub in normal way
2. Client stub builds message, calls local OS
3. Client's OS sends message to remote OS
4. Remote OS gives message to server stub
5. Server stub unpacks parameters, calls server
6. Server does work, returns result to the stub
7. Server stub packs it in message, calls local OS
8. Server's OS sends message to client's OS
9. Client's OS gives message to client stub
10. Stub unpacks result, returns to client
Marshalling
• Problem: different machines have different
data formats
– Intel: little endian, SPARC: big endian
• Solution: use a standard representation
– Example: external data representation (XDR)
Binding
• Problem: how does a client locate a server?
– Use Bindings
• Server
– Export server interface during initialization
– Send name, version no, unique identifier, handle
(address) to binder
• Client
– First RPC: send message to binder to import server
interface
– Binder: check to see if server has exported interface
• Return handle and unique identifier to client
Binder: Port Mapper
•
•
•
•
•
Server start-up: create port
Server stub calls svc_register to register prog #, version # with local port mapper
Port mapper stores prog #, version #, and port
Client start-up: call clnt_create to locate server port
Upon return, client can call procedures at the server
Case Study: SUNRPC
•
•
•
•
•
One of the most widely used RPC systems
Developed for use with NFS
Built on top of UDP or TCP
Multiple arguments marshaled into a single structure
At-least-once semantics if reply received, at-least-zero semantics if
no reply. With UDP tries at-most-once
• Use SUN’s eXternal Data Representation (XDR)
– Big endian order for 32 bit integers, handle arbitrarily large data
structures
• XDR was originally designed for specifying eXternal Data
Representation
• XDR has been extended to become Sun RPC IDL
• An interface contains a program number, version number,
procedure definition and required type definitions
Rpcgen: generating stubs
• Q_xdr.c: do XDR conversion
• Detailed example: later in this course