Download 1 Project definition

‫ מכון טכנולוגי לישראל‬- ‫הטכניון‬
TECHNION - ISRAEL INSTITUTE OF TECHNOLOGY
‫הפקולטה להנדסת חשמל‬
‫המעבדה לבקרה ורובוטיקה‬
PROJECT SUBJECT :
HUMAN-TRACKING USING
A ZEPPELIN AND A PC-STATION
ACCOMPLISHED BY :
Ronen Horovitz
Uri Gover
INSTRUCTORS :
Koby Kohai
David Gidony
DATE :
Winter 2003
Table of contents:
1
1 Project definition ............................................................ 3
2 Introduction ..................................................................... 4
2.1
Project specifications & requirements ................................... 4
2.1.1
2.1.2
2.1.3
2.1.4
CCD digital camera requirements: ........................................................................ 5
Frame Grabber requirements: ............................................................................... 5
Static PC station requirements: ............................................................................. 5
Blimp requirements: ............................................................................................... 6
3 System review ................................................................. 7
3.1
Hardware review ................................................................... 8
3.1.1
Top......................................................................................................................... 9
3.1.1.1
Transmit-unit top ............................................................................................. 10
3.1.1.2
Board-unit top .................................................................................................. 12
3.1.1.2.1
Rec&enc-unit .............................................................................................. 14
3.1.1.2.2
Controller top .............................................................................................. 16
3.1.1.2.2.1 Pulse-generator...................................................................................... 18
3.1.1.2.2.2 Cmd-decoder ......................................................................................... 19
3.1.1.2.3
Drivers-unit .................................................................................................. 20
3.2
3.3
photos of the units .............................................................. 22
Software review .................................................................. 25
3.3.1
3.3.2
3.3.2.1
3.3.2.2
3.3.2.3
3.3.2.4
3.3.2.5
3.3.2.6
3.3.2.7
S.w. general review ............................................................................................. 25
S.W. algorithm review .......................................................................................... 25
Position selection ............................................................................................ 25
Segmentation Preprocessing .......................................................................... 25
Color extraction ............................................................................................... 26
Color Correlation & Threshold ......................................................................... 26
Morphological Filtering .................................................................................... 27
Blob Analysis & Distance measure ................................................................. 27
Color/Position Updating .................................................................................. 27
4 Operating instructions ................................................. 28
2
1 Project definition
The purpose of the project is to make a feasibility test and an
implementation for a new indoor surveillance system implemented by
a flying blimp carrying a digital camera and required RF
communication accessories.
The blimp is fully controlled ( direction & speed ) through a PC station
which also receives video transmission from the blimp.
The image processing is done on the PC station equipped with a
frame-grabber and a smart tracking algorithm.
MOBILE
BLIMP
STATIC
PC
STATION
TARGET
3
2 Introduction
 The project’s concept is a light-weight flying blimp platform equipped with a
video camera and communication devices capable of tracking human
targets in an indoor scene using an image processing algorithm in real-time
 The video camera sends its video signal using wireless communication from the
blimp to a static PC station.
 The user in the PC station decides to track a suspect, clicks on the suspect and the
blimp starts to track the suspect.
 The tracking is based on commands sent from the station to the motors of the
blimp.
2.1 Project specifications & requirements
The purpose of this part is to analyze the requirements from such a system and
to decide, consequent upon, which components are required to fulfill the task
presented by it.
The requirements for the system:
1. The tracking will be implemented by an image processing algorithm.
2. The image of the object will be transmitted to a PC station so that the
image processing will be implemented solely on the PC station.
3. The flying platform will be controlled from the PC station.
4. the blimp will be able to gain as much air time as needed,
therefore the whole system will be designed as
low P-dissipation.
5. The system will be used indoors only.
6. The system should be light-weight in order to achieve maneuvering
capabilities.
The following components are required:
 Blimp
 3 DC motors + propellers
 Motor drivers for 3 motors
 CCD digital camera
 RF transmitter/receiver for the controlling of the blimp
 RF Audio/Video transmitter
 FrameGrabber
 A strong enough PC station
 A low dissipation power supply battery
 HW board to implement transmitter-unit from PC-station
 HW board to implement receiver-unit on Blimp
4
2.1.1 CCD digital camera requirements:
The tracking algorithm requires a high level image processing and therefore
requires good image quality. The following specifications should be met:




Minimum resolution of 640x480.
Color camera for higher dimensionality feature extraction.
High SNR for easier pre-processing.
High frame rate for motion detection.
The need for a light-weight system and low P-dissipation also requires:


Low power supply consumption.
Light-weight as possible
After testing several options and a thorough research in the internet we found
that camera weights are ranging from ~50g with poor quality images up to ~350g
with superb image quality and decided that the Watec 207A (~76g) model would
stand a good option because of its light-weight/image quality compromise and its
availability.
2.1.2 Frame Grabber requirements:
The frame grabber should be able to attach to an Audio/Video RF receiver for the
capture of color image information.
It should also be compatible with the Matrox Imaging Library used for the
implementation of the tracking algorithm.
The Matrox Meteor II frame grabber will be used for its availability.
2.1.3 Static PC station requirements:
The station will be responsible for the execution of the tracking algorithm based
on the images received from the frame grabber.
The station will also be responsible for the control of the blimp by transmitting the
proper movement command.





A very high performance station – Pentium IV equipped with a
minimum of 256MB RAM
I/O card installed
Windows 2000 OS
Visual C++ installed
Matrox Imaging Library installed
5
2.1.4 Blimp requirements:
The blimp must have enough payload lifting capacity for the carrying of some of
these components however it must be both small in dimensions and made from a
durable material and economically worthwhile.
A conflict arises when trying to maintain high payload lifting capacity whilst
striving for a durable and small balloon since a durable material costs more and
weighs more.
The calculation for the blimp dimensions in dependence of the required net
payload lifting:







~2Kg net payload lifting capacity
Cylinder shaped for a better volume-payload ratio
Dimensions: 3.5m x 1.3m
Price < 1000$
Stabilizing wingies (optional)
Gondola scotch straps
Low Helium dissipation
6
3 System review
We can examine the system from 3 different points of view :
 PC-APPLICATION :
a suspect that was marked upon video-stream received from zeppelin, will be
tracked by a s.w. algorithm. PC-station will send steering commands to
zeppelin in order to track suspect.
 VIDEO-TRANSMISSION :
a camera will send video-stream to a video-transmitter , both placed on the
zeppelin . PC-station will receive the video-stream through a video-receiver .
 COMMANDS-TRANSMISSION :
PC-station will transmit commands to zeppelin through commands-transmitter
. These commands will be received by the zeppelin and control its
3 motors : right, left and up-down motors
Mobile zeppelin
Video-stream
commands
Static PC-station
moving target
7
3.1 Hardware review
THE TASK :
commands from PC-station will be transmitted using a transmitter-unit.
It will be received and decoded on the zeppelin , to control its motors .
SOLUTION REQUIRMENTS :
 Support a wide/flexible range of commands
 Reliable
 Lightweight
 Low power consumption
 Small dimensions
PROPOSED SOLUTION :
We will use a P.W.M control . but not a full one. Meaning :
The motors will be controlled by pulses , with modulated width .
The width of the pulse will be defined by the H.W. on the board-unit.
The interval between the pulses is defined by the S.W. – the frequency of the
Each command coming from the pc-unit invokes a pulse to the requested
Motors . a stream of commands will create a stream of pulses.
In addition to that – each command can be transmitted continuously and
Produce a steady action of the motors (not a P.W.M)
proposed solution advantages :
Support a wide/flexible range of commands :
 10 commands , pulse-freq. determined by s.w.
Reliable
 enc/dec units , pulse-width is fixed,smaller implementation = less noise
Lightweight & Small dimensions
 smaller implementation
Low power consumption
 smaller implementation , using 3V-logic (74hcXX family)
8
3.1.1 Top
The top unit is the system which its inputs is the pc-output and its outputs are the motors
anode & cathode.
9
3.1.1.1 Transmit-unit top
This unit encode and transmit data from pc-station to the board-unit.
 Parallel-connector : leads data from pc to the encoder on the pc-unit.
2 bits from the pc also controls the enable-signals of the encoder and the
transmitter
 Encoder : the encoder takes the parallel data of command (4 bits), adds a
signature ( which the decoder on the board-unit expects to find ) , and drives the
overall 9 bits (4 command + 5 signature) serially into the transmitter.
 Transmitter : takes the serial data and transmit it on a 433 MHz carrier
TRANSMITTER-UNIT
HIERARCHEY
PC APPLICATION TO
WRITE DATA TO LPT
9 BIT TRANSMITTED
AS SEEN ON THE
SCOPE
A ‘1’ AND A ‘0’ IS
DISTINGUISHED BY
THE PULSE WIDTH
10
PC-UNIT ARCHITECTURE:
As we can see , the encoder owns a set of a capacitor and 2 resistor , which defines the
clock of the transmitted data ( and hence the speed of transmittion ) it is aimed so it
would be low enough not to corrupt the receiver-processing and high enough so
commands frequency could be at least fast as command-executing.
11
3.1.1.2 Board-unit top
the receive unit gets the command transmission , and drives current to the motors
:
 Rec&dec unit : gets the RF-TX and translate it into a parallel command
 Controller : takes the 4bit command and translate it into an output of
selected motors and their direction
 Driver : takes the signals onf selected motors and their direction and
drives current to the right motors at the right polarity.
BOARD-UNIT
HIERARCHEY
12
BOARD-UNIT ARCHITECTURE:
We can see that the rec&dec unit gets the rf-tx and translate it into a 4bit command and a control signalvt (valid tx ). When the controller senses a high vt it commands the driver with a selection & direction of
motors . the driver than operates the motors.
Notice that we also have a power-suplly unit which supplies 2 voltage levels to the system.
Most of the logic gets a 5v level , but the controller also gets a 3.3v for a special logic it contains ,
In order to save power where we can.
13
3.1.1.2.1 Rec&enc-unit
This unit takes the rf-tx and outputs 4b command :
 Receiver : takes the rf-data transmitted by the pc-unit and translate it into a serial
data
 Decoder : takes the serial data and checks if it is
REC&DEC-UNIT
HIERARCHEY
A SEQUENCE OF WORDS
(EACH CONTAINS 9BIT)
AT OUTPUT OF RECEIVER
A SEQUENCE OF THE 9BITS
THAT A WORD CONTAINS
AT OUTPUT OF RECEIVER
14
REC&DEC-UNIT ARCHITECTURE:
We can see the rf-data-in into the transmitter and the serial out going inti the decoder.
The decoder contains of a 5-fixed bits that should match the signature that is being sent.
It also have a 2 sets of a capacitor and a resistor , to define typical time-constant of a bit and of a word ,
So it can sense each bit and know when a word is done.
When data inputs the decoder , it first check the signature – if it was found correct , it matches the
current 9 bits to the previous 9 bits ( the tx is always in 2 identical words ) and if they matched ,
I latches out the 4 bits of data and sets vt high. Vt resets as soon as the decoder notices incorrectness of
signature or mismatch of the current & previous word.
15
3.1.1.2.2 Controller top
The controller takes the 4 bit command and translate it to selctiom & direction of motors .
 Mux: takes the 4b and translate it to the correspond decimal output (0-15)
 Cmd-decoder : takes the outputs of the mux and selects & directs the motors
 Pulse generator : generates a fixed pulse width when a valid command arrives
CONTROLLER-UNIT
HIERARCHEY
AN EXAMPLE OF
MODIFIED INPUT
SIGNAL AND THE
FIXED PULSE WIDTH
IT GENERATES
16
CONTROLLER ARCHITECTURE:
We can see the 4 bit muxed out to a 16 bit output , these bits sets through the controller the motors.
All these signals are enable through the and-gates stage only when a pulse is generate from the pulsegenerator. Notice that the pulse-generator is invoked by the vt signal.
17
3.1.1.2.2.1 Pulse-generator
The pulse generator inputs the vt signal ,but inverted, because it is ivoked on active-low.
The capacitor and resistor defines the pulse width. When the time interval ends , the
capacitor is being discharge through the discharge-input. Note that this path has no
resistor and hence happens at no-time. This feature allows pulses as close to each other as
needed.
18
3.1.1.2.2.2 Cmd-decoder
The command decoder translates its 16 inputs into a command of motor selection and
direction.
Each command sets only one of the input bits . the rest are ‘0’ . this bit goes through the
or-gates to operate the needed signals.
19
3.1.1.2.3 Drivers-unit
The driver takes the motor commands and translate it into a current.
There are 2 drivers used , each one can drive 2 motors with diverse direction foreach.
one is responsible for the right and left motors and the other for the elevation motor
DRIVER-UNIT
HIERARCHEY
20
DRIVER ARCHITECTURE:
Each output to motor requires inputs of the chip_inhibit ( enable ) and 2 inputs for polarity of motor.
The polarity inputs should be negated each other for a motor to spin , this is generated by inputing the
direction and the inverted signal of it.
The ouputs of this unit is actually the outputs of the board-unit.
21
3.2 photos of the units
22
23
24
3.3 Software review
3.3.1 S.w. general review
The task is defined as follows:
A user clicks on a human or an object in the screen.
The algorithm analyses the object to be tracked and continuously finds it in the
next video frame along the video stream, marking it as well.
 Color is a very powerful attribute which distinguishes an object from its
environment, and is easy to represent.
 Objects change their position in a very smooth and continuous way so an
object’s local position is a very important attribute which can help
distinguish an object from other similar or exact objects. It is also an easy
attribute to represent.
 Due to these arguments we decided to combine the vast information
inherent in the color attribute with the position attribute and came up with
 To use the algorithm the user should only click on a colorful segment of
the target such as a shirt. The color information along with the position of
the target will be enough for the algorithm to fulfill its tracking task.
3.3.2 S.W. algorithm review
Position selection -> Segmentation Preprocessing -> Color extraction ->
Color Correlation -> Threshold -> Morphological Filtering -> Blob Analysis ->
Distance measure -> Color/Position Updating
3.3.2.1 Position selection
In this stage the user clicks on the target he wants to follow.
We now keep the initial target position.
This position will be referred to as X0, Y0
3.3.2.2 Segmentation Preprocessing
in order to overcome the problem of inhomogeneous illumination caused
by the non-rigidity of the clothes and folds in the clothes we could have
used Filtering/Segmentation We need an operation that will match
areas with similar colors to one color and preserve the edges as well. The
purpose is to facilitate the next color correlation stage by reducing the
number of colors in the image and their smoothing their texture
 The main problem with all these preprocessing operators is their
computational complexity!
 A local solution is an option.
25
3.3.2.3 Color extraction
 first frame only – we keep the color information in small window around
the specified target location X0, Y0.
 Update every N frames – we update the color information in a small
window around the changing target location X0, Y0, for robustness against
shadows.
 We keep the mean color in the small window as the representative color.
 We also tried using the median in 3 different channels since it is not
defined for RGB color coordinates.
 The mean gave better results since the median created new colors.
 We call it C0.
We tried different color spaces for the color representation and correlation
such as RGB, HSV, YCbCr and found that the RGB gives the best results.
 2 color coordinates Instead of 3 ???
3.3.2.4 Color Correlation & Threshold
 We check the color correlation of a window from the next frame around
X0, Y0 with C0.
 We threshold the result with a fixed threshold value for finding similar color
regions in the next frame.
 We use a normalized L2 distance similarity measure for the correlation:
1
2
2
1   RC 0  R    GC 0  G    BC 0  B  


2
We also tried different similarity measures for the correlation stage such as:
 The classic mathematical correlation
 Euclidean L1 norm
 Euclidean L2 norm
We finally chose the normalized L2 norm which results with a number between
[0, 1] (zero match – perfect match).
For the thresholding step we figured that a fixed >0.95 threshold is a good
choice after running exhaustive experiments.
26
3.3.2.5 Morphological Filtering
We now have several regions or “candidates” in the thresholded
correlation image so we perform a few morphological filtering steps for
achieving smooth and closed regions.
3.3.2.6 Blob Analysis & Distance measure
Now we decide which region is the closest region to the last position of
the target, hence we label the regions and using a geodetic distance function
we pick the closest one according to he closest centroid.
3.3.2.7 Color/Position Updating
We update the new position according to the centroid of the closest region.
We can also update the color in that position for robustness against
Illumination change as stated earlier
27
4 Operating instructions



run from a win98 platform the LPT control program : ‘parallel port monitor’
connect the PC-unit to the PC through the LPT connector
apply power of 4.5 – 5.5 volt. Check signal of encoder-output to assure supply
voltage is enough ( could change from computer to another )
 set LPT through ‘parallel port monitor’ according to :
pin 9 – transmitter enable. active high.
pin 8 – encoder enable. active low.
pin 7 – command bit 3 (according to tablet below)
pin 6 – command bit 2 (according to tablet below)
pin 5 – command bit 1 (according to tablet below)
pin 4 – command bit 0 (according to tablet below)
pin 3 – decode bit 1 ( should be 0 )
pin 2 – decode bit 0 ( should be 1 )
pin 1 – RESERVED. LPT–control bit. Not applicable.
Command tablet :
Command bits
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
Others



Motors & directions
Right forward
Right back
Left forward
Left back
Elevation forward
Elevation back
Right & Left forward
Right foreward Left back
Right back Left forward
Right & left back
No operation
connect motors to the board-unit
apply power of 5v
use leds as status indication
28