Download BitTorrent - VoD Framework

VoD BitTorrent Framework
Background
• BT is a very popular peer to peer protocol
with many implementations:
– http://en.wikipedia.org/wiki/Comparison_of_Bit
Torrent_clients
• The first one was created in Python by
Bram Cohen.
Real world motivation
• BitTorrent is common and Free
• Good chance there is a swarm for your movie
• VoD servers are expensive and complicated
• Its better to use P2P for VoD
Some definitions
• Delay (D) = How much time I am allowed to hold before
starting to play M (Movie).
• Prefetch Time (P) = How much time before I watch byte
B from M, it should arrive.
• Playback Rate (R) = How many bytes/second I watch
• B(t) – the index of the bit that must arrive by t.
• B(t) = R *(P+t-D)
Rules and Examples
• B(t) = R * (P+t-D)
• If R=5 KB/sec and D=6 seconds and P=2
seconds, than:
– At t=6 seconds, I must have 10KB continuous from
the start of the movie.
– At t=18 seconds I must have 70KB continuous from
the start of the movie.
What does one VOD client do
• Your client tries to get from the Bittorent clients
pieces of data needed to play the video.
• Each piece needs to arrive before its deadline.
• If the piece does not arrive by the deadline you
can download it “instantly” from the “server”.
• Each VOD client counts the data from server
(DFS).
What is my goal
• Each byte taken from the server costs
money, therefore you need to minimize
DFS.
• Goal - Minimize the accumulated DFS
of all the VOD clients that run
concurrently.
How do I create my client
•
Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this
Code” link in the course web page.
– It is also in /users/courses/hanochworkshop/group_vod
•
GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to
monitor DFS. Any changed section is marked ###### GROUP VOD
######
•
Modify it to minimize acumulated DFS.
•
Do not change StreamWatched.py.
Standard BT parameters
• --saveas F
• --max_upload_rate
• F.torrent
VOD client additional params
• SET = defined inside run_all.sh, not a parameter.
• delay – Wait before prefetch, in seconds. SET TO 20
• prefetchT – How much time in advance the byte must arrive. In
percent of movie length. SET TO 5%
• rate – VOD rate in KB/sec. SET TO 90 KB.
• order – How many run when client spawned. SET BY run_all.sh.
•
•
•
•
•
•
•
Parameter of run_multiple.sh.
gap – How much to wait between spawns – unit is pewfetchT.
alg – Name of the algorithm. Default is ORIG.
verbose – Print and kill the whole swarm when complete.
peers_num – how many regular peers.
seeds_num – how many seeds.
Vod_num – how many VoD peers.
How do I test my idea
• cd
/users/courses/hanochworkshop/voddiversegroup/vod_
tests
• run_multiple.sh
– your_client - example:
~/GROUP_VOD_CODE/btdownloadheadless.py
– Gap – example: 0
– output_location example: ~/bt_out
– iteration example: 4
– empty/full example: empty
– peers_num example: 16
– seeds_num example: 2
– vod_num example: 14
Execution example
run_multiple.sh
Tracker
BT
BT
BT
BT
BT
run-all.sh
BT
BT
BT
BT
VOD
VOD
BT
VOD
BT
VOD
BT
BT
BT
How does it work?
• Run tracker
• Run the “world”
• Run your clients
How does it work?
• Both tracker and the “world” run on the same
computer.
• Running a test:
– Reserve time on pc-hanoch-w3/w4 in:
• http://bt-p2p-vod.wikispaces.com/
• Or from course www - "Scheduling computers for tests.“
Output to screen
==================================================================================
total time:
1 min 07 sec
share rating:
0.904 (4816.0 KB up / 5328.0 KB down)
saving:
picture-100.bmp (27.5 MB)
percent done:
18.9
time left:
4 min 52 sec
download to:
/home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp
download rate:
93.4 kB/s
upload rate:
80.6 kB/s
seed status:
2 seen now, plus 0.340 distributed copies
peer status:
28 seen now, 10.7% done at 1220.3 kB/s
--------------------------------StreamWatcher------------------------------------Csv stats:
~/out/1/statistics-order-4-gap-0.csv
DFS is:
8372224 bytes
DFS/Total is:
29 %
FullPieces:
63 / 220
DirtyPieces:
1 / 220
FaildHashChecks:
0
Prefetching
5 %
Playing point:
56 / 220 ( 25 %)
Only last VOD client prints
Output to .csv
• In ~/out you will find vod_num directories: /1, /2,
…/iteration.
• In each directory, you have vod_num files.
• Statistics in - order-<1..vod_num>-gap-<GAP>.csv
• Order is how many VOD clients ran before.
• <GAP> is the parameter Gap.
.csv file
alg,dfs,p2p
ORIG,5,0
ORIG,10,2
ORIG,15,6
ORIG,19,11
ORIG,23,16
ORIG,28,20
ORIG,31,25
ORIG,34,31
ORIG,37,36
ORIG,39,41
.
.
.
File percentage
total DFS and total P2P
GAP = 2
File percentage
GAP = 0
Local DFS GAP = 2
GAP = 0
run_all.sh defaults
• Start
– Use in run_multiple.sh
• Stop
– Kill all python threads
• Kill
– Kill all python threads and scripts.
To run client separately:
• PATH_TO_YOUR_VOD_CLIENT/btdownloadheadless.py
--saveas F
--delay D
--prefetchT P
--rate R
--out_dir O
Can add the parameters
--order X
in the IDE
--gap G
--group_size S
--alg ‘ALG'
--verbose
F.Torrent
Python
• Very common script language
• IDE:
– PyDev
– WINGWARE Python IDE – not free, but very good.
• References:
– python.org :http://docs.python.org/reference/
– http://wiki.python.org/moin/BeginnersGuide/NonProgrammers
– http://wiki.python.org/moin/BeginnersGuide/Programmers
• The book "Think like a Computer Scientist in Python" has a very
good reputation:
Getting started
• Run the environment
–
–
–
–
ssh pc-hanoch-w3 or pc-hanoch-w4
cd /users/courses/hanochworkshop/group_vod/vod_tests
run_multiple.sh MyClient GAP output-directory
Note tracker prints a message for each GET.
• You can connect to the tracker, from the CS network,
with a browser at:
– http://pc-hanoch-wk:6970/
• Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see
downloaded files
Directory Structure
• Root –
– high level application
• Root/BitTornado –
– services
• Root/BitTornado/BT1 –
– Actual work
Client Operation
• btdownloadheadless.py – First to run.
– Performs initializations.
•
•
•
•
get_response – parse the .torrent
Infohash – the ID of the .torrent
startEngine – Start connection with peers
startRerequester - Start connection with tracker
– Run rawserver.listen_forever from RawServer.py.
– listen_forever is:
• Polling port
• executing func() from a task queue.
Receiving messages
• From listen_forever loop it goes to:
– BT1/Connecter.py/got_message
•
•
•
•
•
•
•
•
•
CHOKE
UNCHOKE
INTERESTED
NOT_INTERESTED
BITFIELD
REQUEST
CANCEL
PIECE
HAVE
Influencing decisions
• Can be done in one of the callbacks which
spring from Connecter.py / got_message:
• Downloader.py
– HAVE->got_have->send_interested
• Tune for VoD?
BT tracker
• Send GET request in Rerequester.py /
announce
• Receive GET data in thread in
Rerequester.py / _rerequest
Policy
• Choker.py/_round_robin – decide who to
choke / optimistically un choke /
send_have to.
• Downloader.py / _request_more.
• PiecePicker.py / next - rarest piece, etc.
Tune for VoD?
• got_unchoke->_request_more->next…
More about the code
• Start your project from “BitTornado - Start
with this Code” link.
• Click “BitTornado - Legacy Code Guide”
for basic explanations about the code.