Download Shared Memory IPC

Advanced
Operating Systems
Lecture 5: Interprocess
Communication
University of Tehran
Dept. of EE and Computer Engineering
By:
Dr. Nasser Yazdani
Univ. of Tehran
Distributed Operating Systems
1
How Processes communicate


Improving process communication to make
systems more modular, scalable and
flexable.
References

“improving IPC by Kernel Design”, by Jochen
Liedtke.
Univ. of Tehran
Distributed Operating Systems
2
Outline


Why IPC?
IPC mechanisms







Pipe,
Socket,
Shared memory
Message
RPC (Remote Procedure call)
Universal address space.
How improve IPC through kernel.
Univ. of Tehran
Distributed Operating Systems
3
IPC Fundamentals

What is IPC? Mechanisms to transfer data between
processes





To share resources
Client/server paradigms
Inherently distributed applications
… Other good software engineering reasons
Why is it needed?



Not all important procedures can be easily built in a single
process
To make systems modular, flexible and scalable.
Critical for micro-kernel OS and distributed systems.
Univ. of Tehran
Distributed Operating Systems
4
IPC from the OS Point of
View
Private
address
space
Process A
OS address space
Private
address
space
Process B
•Each process has a private address space
•Normally, no process can write to another process’s
space
•How to get important data from process A to
process
B?
Univ. of Tehran
Distributed Operating Systems
5
Solutions to IPC

Fundamentally, two options
1. Support some form of shared address space

Shared memory
2. Use OS mechanisms to transport data from one
address space to another


Shared memory


Files, messages, pipes, RPC
OS has job of setting it up and perhaps synchronizing But not
transporting data
Messages, etc


OS involved in every IPC step
OS transports data
Univ. of Tehran
Distributed Operating Systems
6
Ideal IPC Characteristics






Fast
Easy to use
Well defined synchronization model
Versatile (Flexible)
Easy to implement
Works remotely
Univ. of Tehran
Distributed Operating Systems
7
IPC and Synchronization

Synchronization is a major concern for
IPC



Allowing sender to indicate when data is
transmitted
Allowing receiver to know when data is ready
Allowing both to know when more IPC is
possible
Univ. of Tehran
Distributed Operating Systems
8
IPC and Connections



IPC mechanisms can be connectionless or
require connection
Connectionless IPC mechanisms require
no preliminary setup
Connection IPC mechanisms require
negotiation and setup before data flows

Sometimes much is concealed from user,
though
Univ. of Tehran
Distributed Operating Systems
9
Connectionless IPC
Data simply flows
 Typically, no permanent data structures
shared in OS by sender and receiver
+ Good for quick, short communication
+ less long-term OS overhead
- Less efficient for large, frequent
communications
- Each communication takes more OS
resources per byte

Univ. of Tehran
Distributed Operating Systems
10
Connection-Oriented IPC



Sender and receiver pre-arrange IPC
delivery details
OS typically saves IPC-related information
for them
Advantages/disadvantages pretty much
the opposites of connectionless IPC
Univ. of Tehran
Distributed Operating Systems
11
IPC Through File System



Sender writes to a file
Receiver reads from it
But when does the receiver do the read?


Often synchronized with file locking or lock files
Special types of files can make file-based IPC
easier
Process A
Univ. of Tehran
Data
Distributed Operating Systems
Process B
12
Message-Based IPC

Sender formats data into a formal
message




With some form of address for receiver
OS delivers message to receiver’s
message input queue (might signal too)
Receiver (when ready) reads a message
from the queue
Sender might or might not block
Univ. of Tehran
Distributed Operating Systems
13
Message-Based IPC
Diagram
OS
Process A
Univ. of Tehran
Data sent
from A to B
B’s message
queue
Distributed Operating Systems
Process B
14
Procedure Call IPC

Interprocess communication uses same
procedure call interface as intraprocess




Data passed as parameters
Information returned via return values
Complicated since destination procedure
is in a different address space
Generally, calling procedure blocks till call
returns
Univ. of Tehran
Distributed Operating Systems
15
Procedure IPC Diagram
main () {
.
call();
.
.
.
Data as parameters
Data as return values
.
.
.
server();
.
.
.
}
Process B
Process A
Univ. of Tehran
Distributed Operating Systems
16
Shared Memory IPC



Processes share a common piece of memory
either physically or virtually
Communications via normal reads/writes
May need semaphores or locks

In or associated with the shared memory
main () {
.
x = 10
.
.
.
Univ. of Tehran
}
write variable x
x: 10
read variable x
.
.
.
print(x);
.
.
.
Process B
Process A
Distributed Operating Systems
17
Synchronizing in IPC


How do sending and receiving process
synchronize their communications?
Many possibilities, based on which process
block when



Blocking Send, Blocking Receive, Often called
message rendezvous
Non-Blocking Send, Blocking Receive, Essentially,
receiver is message-driven
Non-Blocking Send, Non-Blocking Receive, polling
or interrupt for receiver
Univ. of Tehran
Distributed Operating Systems
18
Addressing in IPC



How does the sender specify where the data goes?
The mechanism makes it explicit (shared memory or
RPC)
Or, there are options: Direct Addressing



Sender specifies name of the receiving process using some
form of unique process name
Receiver can either specify name of expected sender or
take stuff from anyone
Indirect Addressing: Data is sent to queues, mailboxes, or
some other form of shared data structure: Much more
flexible than direct addressing
Univ. of Tehran
Distributed Operating Systems
19
Duality in IPC Mechanisms




Many aspects of IPC mechanisms are duals of
each other Which implies that these mechanisms
have the same power
First recognized in context of messages vs.
procedure calls At least, IPC mechanisms can be
simulated by each other
Depends on model of computation And on
philosophy of user
In particular cases, hardware or existing software
may make one perform better
Univ. of Tehran
Distributed Operating Systems
20
UNIX IPC Mechanisms

Different versions of UNIX introduced
different IPC mechanisms






Pipes
Message queues
Semaphores
Shared memory
Sockets
RPC
Univ. of Tehran
Distributed Operating Systems
21
Pipes

Only IPC mechanism in early UNIX
systems (other than files)





Uni-directional
Unformatted
Uninterpreted
Interprocess byte streams
Accessed in file-like way
Univ. of Tehran
Distributed Operating Systems
22
Pipe Details




One process feeds bytes into pipe
A second process reads the bytes from it
Potentially blocking communication
mechanism
Requires close cooperation between
processes to set up

Named pipes allow more flexibility
Univ. of Tehran
Distributed Operating Systems
23
Pipes and Blocking

Writing more bytes than pipe capacity
blocks the sender


Reading bytes when none are available
blocks the receiver


Until the receiver reads some of them
Until the sender writes some
Single pipe can’t cause deadlock
Univ. of Tehran
Distributed Operating Systems
24
UNIX Message Queues


Introduced in System V Release 3 UNIX Like
pipes, but data organized into messages
Message component include:



Type identifier
Length
Data
Univ. of Tehran
Distributed Operating Systems
25
Semaphores


Also introduced in System V Release 3
UNIX
Mostly for synchronization only


Since they only communicate one bit of
information
Often used in conjunction with shared
memory
Univ. of Tehran
Distributed Operating Systems
26
UNIX Shared Memory




Also introduced in System V Release 3
Allows two or more processes to share
some memory segments
With some control over read/write
permissions
Often used to implement threads
packages for UNIX
Univ. of Tehran
Distributed Operating Systems
27
Sockets



Introduced in 4.3 BSD
A socket is an IPC channel with
generated endpoints
Great flexibility in it characteristics


Intended as building block for
communication
Endpoints established by the source and
destination processes
Univ. of Tehran
Distributed Operating Systems
28
UNIX Remote Procedure
Calls

Procedure calls from one address space
to another




On the same or different machines
Requires cooperation from both processes
In UNIX, often built on sockets
Often used in client/server computing
Univ. of Tehran
Distributed Operating Systems
29
Problems with Shared
Memory



Synchronization
Protection
Pointers
Univ. of Tehran
Distributed Operating Systems
30
Synchronization



Shared memory itself does not provide
synchronization of communications
Except at the single-word level
Typically, some other synchronization
mechanism is used


E.g., semaphore in UNIX
Events, semaphores, or hardware locks in
Windows NT
Univ. of Tehran
Distributed Operating Systems
31
Protection



Who can access a segment? And in what
ways?
UNIX allows some read/write controls
Windows NT has general security
monitoring based on the object-status of
shared memory
Univ. of Tehran
Distributed Operating Systems
32
A Troublesome Pointer
a: __
z: __
b: __
y: 20
x: 10
w: 5
Process B
Process A
Univ. of Tehran
Distributed Operating Systems
33
How share pointers?

Copy-time translation




Reference-time translation



Encode pointers in shared memory segment as pointer surrogates,
typically, as offsets into some other segment in separate contexts. So
each sharer can have its own copy of what is pointed to
Slow, pointers in two formats
Pointer Swizzling





Locate each pointer within old version of the data Then translate
pointers are required
Requires both sides to traverse entire structure
Not really feasible for shared memory
cache results in the memory location, only first reference is expensive
But each sharer must have his own copy
Must “unswizzle” pointers to transfer data outside of local context
Stale swizzled pointers can cause problems
All involve somehow translating pointers at some point
before they are used
Univ. of Tehran
Distributed Operating Systems
34
Shared Memory in a Wide
Virtual Address Space


When virtual memory was created, 16 or
32 bit addresses were available
Reasonable size for one process



But maybe not for all processes on a
machine
And certainly not for all processes ever on a
machine.
How a global address space, e.x 64 bit
address space. (Opal system)
Univ. of Tehran
Distributed Operating Systems
35
Implications of Single Shared
Address Space

IPC is trivial



Shared memory, RPC
Separation of concepts of address space
and protection domain
Uniform address space
Univ. of Tehran
Distributed Operating Systems
36
Address Space and
Protection Domain

A process has a protection domain


The data that cannot be touched by other
processes
And an address space


The addresses it can generate and access
In standard systems, these concepts are
merged
Univ. of Tehran
Distributed Operating Systems
37
Uniform-Addressing Allows
Easy Sharing

Any process can issue any address



So any data can be shared
All that’s required is changing protection
to permit desired sharing
Suggests programming methods that
make wider use of sharing
Univ. of Tehran
Distributed Operating Systems
38
Windows NT IPC

Inter-thread communications


Local procedure calls


Within a single process
Between processes on same machine
Shared memory
Univ. of Tehran
Distributed Operating Systems
39
Windows NT and
Client/Server Computing




Windows NT strongly supports the
client/server model of computing
Various OS services are built as servers,
rather than part of the kernel
Windows NT needs facilities to support
client/server operations
Which guide users to building
client/server solution
Univ. of Tehran
Distributed Operating Systems
40
Client/Server Computing
and RPC


In client/server computing, clients
request services from servers
Service can be requested in many ways


But RPC is a typical way
Windows NT uses a specialized service for
single machine RPC
Univ. of Tehran
Distributed Operating Systems
41
Local Procedure Call (LPC)




Similar in many ways to RPC, but optimized
to only work on a single machine
Primarily used to communicate with
protected subsystems
It also provides a true RPC facility
Application calls routine in an application
programming interface Which is usually in a
dynamically linked library Which sends a
message to the server through a messaging
mechanism
Univ. of Tehran
Distributed Operating Systems
42
Windows NT LPC Messaging
Mechanisms



Messages between port objects
Message pointers into shared memory
Using dedicated shared memory
segments
Univ. of Tehran
Distributed Operating Systems
43
Port Objects



Windows NT is generally object-oriented
Port objects support communications
Two types:

Connection ports




Used to establish connections between clients and servers
Named, so they can be located
Only used to set up communication ports
Communication ports




Used to actually pass data
Created in pairs, between given client and given server
Private to those two processes
Destroyed when communications end
Univ. of Tehran
Distributed Operating Systems
44
Windows NT Port Example
Connection port
Server process
Client process
Univ. of Tehran
Distributed Operating Systems
45
Windows NT Port Example
Connection port
Server process
Client process
Univ. of Tehran
Distributed Operating Systems
46
Windows NT Port Example
Connection port
Communication ports
Server process
Client process
Univ. of Tehran
Distributed Operating Systems
47
Windows NT Port Example
Connection port
Send request
Communication ports
Server process
Client process
Univ. of Tehran
Distributed Operating Systems
48
Message Passing through
Port Object Message Queues

One of three methods in Windows NT to
pass messages
1. Client submits message to OS
2. OS copies to receiver’s queue
3. Receiver copies from queue to its own
address space
Univ. of Tehran
Distributed Operating Systems
49
Characteristics of Message
Passing via Queues


Two message copies required
Fixed-sized, fairly short message


Port objects stored in system memory


~256 bytes
So always accessible to OS
Fixed number of entries in message
queue
Univ. of Tehran
Distributed Operating Systems
50
Message Passing Through
Shared Memory


Used for messages larger than 256 bytes
Client must create section object



Shared memory segment of arbitrary size
Message goes into the section
Pointer to message sent to receiver’s
queue
Univ. of Tehran
Distributed Operating Systems
51
Setting up Section Objects




Pre-arranged through OS calls
Using virtual memory to map segment
into both sender and receiver’s address
space
If replies are large, need another
segment for the receiver to store
responses
OS doesn’t format section objects
Univ. of Tehran
Distributed Operating Systems
52
Characteristics of Message
Passing via Shared Memory


Capable of handling arbitrarily large
transfers
Sender and receiver can share a single
copy of data


i.e., data copied only once
Requires pre-arrangement for section
object
Univ. of Tehran
Distributed Operating Systems
53
Server Handling of
Requests



Windows NT servers expect requests
from multiple clients
Typically, they have multiple threads to
handle requests
Must be sufficiently general to handle
many different ports and section objects
Univ. of Tehran
Distributed Operating Systems
54
Message Passing Through
Quick LPC



Third way to pass messages in Windows NT
Used exclusively with Win32 subsystem
Like shared memory, but with a key difference

Dedicated resources
Univ. of Tehran
Distributed Operating Systems
55
Use of Dedicated Resources
in Quick LPC

To avoid overhead of copying





Notification messages to port queue
And thread switching overhead
Client sets up dedicated server thread
only for its use
Also dedicated 64KB section object
And event pair object for synchronization
Univ. of Tehran
Distributed Operating Systems
56
Characteristics of Message
Passing via Quick LPC




Transfers of limited size
Very quick
Minimal copying of anything
Wasteful of OS resources
Univ. of Tehran
Distributed Operating Systems
57
Shared Memory in
Windows NT



Similar in most ways to other shared
memory services
Windows NT runs on multiprocessors,
which complicates things
Done through virtual memory
Univ. of Tehran
Distributed Operating Systems
58
Section Views




Given process might not want to waste
lots of its address space on big sections
So a process can have a view into a
shared memory section
Different processes can have different
views of same section
Or multiple views for single process
Univ. of Tehran
Distributed Operating Systems
59
Shared Memory View
Diagram
view 1
view 2
Section
view 3
Process B
Process A
Physical memory
Univ. of Tehran
Distributed Operating Systems
60
Improving IPC
Experiments based on L3 operating system
which is micro-kernel system.
 A null IPC

Thread A (user mode) Kernel
Thread B (user mod)
Load ID of B
Set Msg length to 0
Call Kernel
Insepct received msg
Access Thread B
Switch Stack pointer
Switch Address Space
Load id of A
Return to user
On intel 486, 50 Mhz, time for null IPC is 172 cycles (3.5 µ)
 Entering and Leaving kernel 2 µ
 Our aim was 350 or 7 µ and achieved 250 or 5 µ
Univ. of Tehran
Distributed Operating Systems
61
Improving IPC

System calls: simply entering and leaving kernel
mode cost 2 µ




Reduce the cost of system call.
For one msg and reply we need 4 kernel crossing.
Reduce it by implementing Reply and receive.
Less Copies: User -> kernel -> user then, at
least two copies.


Copying n bytes cost 20 + 0.75n cycles plus additional TLB
misses
In L3 kernel determine the destination address and maps it
into communication window and directly copy msg there.
Univ. of Tehran
Distributed Operating Systems
62
Improving IPC

Combine Sequences of send operations into one
message


Reduce the cost of system call.
Thread Control Blocks (TCB) are treated as
objects and kept in a virtual array.

Use 64 bit theard id to find TCB in the virtual array.
Queues in Kernel implemented as double linked
list for wakeup, etc.
 Lazy scheduling: instead of entering
queue/delete
change
thread status.
Univ. of Tehran
Distributed Operating Systems
63

Improving IPC

Use registers for IPC

For passing parameter.
Reduce cache misses by concentrating
frequently used data in few cache lines.
 Put IPC related kernel codes in one page to
reduce TLB misses

Univ. of Tehran
Distributed Operating Systems
64
Next Lecture
Scheduling
 References



Surplus Fair Scheduling: A ProportionalShare CPU Scheduling Algorithm for
Symmetric Multiprocessors
Scheduler Activations: Effective Kernel
Support for User-Level Management of
Parallelism",
Univ. of Tehran
Distributed Operating Systems
65