Download Experiment: Step by Step

Experiment: Step by Step
Author: Anna Bekkerman
[email protected]
Setup
Node
Client
Server
Target system
Data
LMM
Node
LMM
Node
Client
LMM
Configuration File

Describes an experiment
–
Nodes

–
–
–
–
Commands to start/stop involved processes
Collected metrics (CPU/memory utilization, etc.)
Monitored processes
Net control parameters

–
IP addresses, types (SOCC node/radar node), etc.
Delays, drop rates
Refresh rates
Start LMMs

When started, RAPIDS server:
–
Grabs two ports:


–
–
–
–
49162 - to communicate with LMMs
8888 - to communicate with RAPIDS clients FIXME: Server
Reads a configuration file
Starts LMMs on all nodes through SSH
connections
Waits for ack signals from all LMMs
Starts setting LMMs up according to the
configuration file
will wait
indefinitely for
the acks from
all LMMs.
A time-out
mechanism
should be
introduced.
Set LMMs Up


Home-made protocol is used to set up LMM
parameters
Examples of commands sent from the server to
LMMs:
–
–
–
–
–


STM
STP
STE
STT
SPP
set metric
set monitored process
set start-up command
start
stop
When a parameter is set, LMM sends an ack signal
back to the server
At the end of each step, server waits for acks from
all LMMs
Start Monitoring

When LMM receives the start command:
–
If needed, network control application is started
• Network control application runs only if
iptables are turned on.
• iptables select IP packets (as specified in
iptables rules) and queue them for
processing by the application.
• The application introduces delays and/or
drops packets according to the settings in
the configuration file.
Start Monitoring

When LMM receives the start command:
–
–
If needed, network control application is started
RAPIDS Message Queues (RMQ) are initialized
• A mechanism used for
communication between RAPIDS and
monitored applications.
• See more in the “RMQ” section.
Start Monitoring

When LMM receives the start command:
–
–
–
If needed, network control application is started
RAPIDS Message Queues (RMQ) are initialized
Heartbeat applications are started
• Send “I’m alive” signals from radar
nodes to SOCC nodes.
• If a signal has not been received,
RAPIDS reports link failure.
FIXME: Timeout mechanism should
be added to minimize false alarms.
Start Monitoring

When LMM receives the start command:
–
–
–
–
If needed, network control application is started
RAPIDS Message Queues (RMQ) are initialized
Heartbeat applications are started
Processes are started

Commands are specified by user in the configuration file
Start Monitoring

When LMM receives the start command:
–
–
–
–
If needed, network control application is started
RAPIDS Message Queues (RMQ) are initialized
Heartbeat applications are started
Processes are started

–
Commands are specified by user in the configuration file
“Collection sessions” are started every t seconds

According to the refresh rates provided by user in the
configuration file
Collection Session

During each collection session LMM:
–
–
–

Collects metrics
Reads events accumulated in RMQ
Sends the metrics and events to the RAPIDS
server
More details in the “LMM” section
Stop Monitoring


When the server is stopped, it sends stop
commands to all LMMs
Upon receiving the stop signal, LMM:
–
–
Stops launching collection sessions
Stops processes

–
–
–
Using the commands specified by user in the
configuration file
Heartbeat applications are stopped
RMQ is deleted
Network control applications are stopped
What Might Go Wrong?


When the server is stopped, it sends stop
commands to all LMMs
Upon receiving the stop signal, LMM:
–
–
If “untrappable” signals (SIGKILL
Stops launching
collection sessions
and SIGSTOP) are used to kill the
Stops processes
server, the shut-down procedures

–
–
–
not be executed!
Using thewill
commands
specified by user in the
configuration file
Heartbeat applications are stopped
RMQ is deleted
Network control applications are stopped
What Might Go Wrong?
• If commands provided by user do not stop all processes, LMM
will hang waiting for their termination.
 When
theanserver
is stopped,
it sends
stop with
• While
LMM is hanging
the port used
for communication
the server remains
unreleased, which means that the new
commands
to all LMMs
experiment cannot be started until LMMs are stopped and all
 Uponnecessary
receiving
the
stop have
signal,
LMM:
clean-up
procedures
been completed.
–
–
Stops launching collection sessions
Stops processes

–
–
–
Using the commands specified by user in the
configuration file
Heartbeat applications are stopped
RMQ is deleted
Network control applications are stopped
What Might Go Wrong?


When the server is stopped, it sends stop
commands toFIXME:
all LMMs
• These
applications
do not always
react
Upon receiving
the
stop
signal,
LMM:
to the termination signal properly.
–
–
Stops launching
collection
sessions
• Symptom:
sometimes
a number of
zombie processes appear
Stops processes

–
–
–
Using the commands specified by user in the
configuration file
Heartbeat applications are stopped
RMQ is deleted
Network control applications are stopped