Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
VoD BitTorrent Framework Background • BT is a very popular peer to peer protocol with many implementations: – http://en.wikipedia.org/wiki/Comparison_of_Bit Torrent_clients • The first one was created in Python by Bram Cohen. Real world motivation • BitTorrent is common and Free • Good chance there is a swarm for your movie • VoD servers are expensive and complicated • Its better to use P2P for VoD Some definitions • Delay (D) = How much time I am allowed to hold before starting to play M (Movie). • Prefetch Time (P) = How much time before I watch byte B from M, it should arrive. • Playback Rate (R) = How many bytes/second I watch • B(t) – the index of the bit that must arrive by t. • B(t) = R *(P+t-D) Rules and Examples • B(t) = R * (P+t-D) • If R=5 KB/sec and D=6 seconds and P=2 seconds, than: – At t=6 seconds, I must have 10KB continuous from the start of the movie. – At t=18 seconds I must have 70KB continuous from the start of the movie. What does one VOD client do • Your client tries to get from the Bittorent clients pieces of data needed to play the video. • Each piece needs to arrive before its deadline. • If the piece does not arrive by the deadline you can download it “instantly” from the “server”. • Each VOD client counts the data from server (DFS). What is my goal • Each byte taken from the server costs money, therefore you need to minimize DFS. • Group - all the VOD clients that run concurrently. • Goal - Minimize the DFS for the group. How do I create my client • Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this Code” link in the course web page. – It is also in /users/courses/hanochworkshop/group_vod • GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked ###### GROUP VOD ###### • Modify it to have lower group DFS • Do not change StreamWatched.py. Standard BT parameters • --saveas F • --max_upload_rate • F.torrent My VOD client new parameters • delay – Wait before prefetch, in seconds. • prefetchT – How much time in advance the byte must arrive. In percent of movie length. • rate – VOD rate in KB/sec • out_dir – Where to place the .csv files. • order – How many run when client spawned. • gap – How much to wait between spawns – unit is pewfetchT. • group_size – How many concurrent clients. • alg – Name of the algorithm. Default is ORIG. • Verbose – Print and kill the whole swarm when complete. How do I test my idea • Run the environment: ./run_multiple.sh ~/GROUP_VOD_CODE/btdownloadheadless.py <GAP> ~/out • ~/GROUP_VOD_CODE/btdownloadheadless.py: Your client. • <GAP>: An integer. How many prefetchT between spawns of your clients. • ~/out: .csv files Execution example run_multiple.sh Tracker BT BT BT BT BT run-all.sh BT BT BT BT VOD VOD BT VOD BT VOD BT BT BT How does it work? • Run tracker • Run the “world” • Run your clients How does it work? • Both tracker and the “world” run on the same computer. • Running a test: – Reserve time on pc-hanoch-w3/w4 in: • http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“ – cd /users/courses/hanochworkshop/group_vod/vod_tests – ./run_multiple.sh ~/btdownloadheadless.py <GAP> ~/out Output to screen ================================================================================== total time: 1 min 07 sec share rating: 0.904 (4816.0 KB up / 5328.0 KB down) saving: picture-100.bmp (27.5 MB) percent done: 18.9 time left: 4 min 52 sec download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp download rate: 93.4 kB/s upload rate: 80.6 kB/s seed status: 2 seen now, plus 0.340 distributed copies peer status: 28 seen now, 10.7% done at 1220.3 kB/s --------------------------------StreamWatcher------------------------------------Csv stats: ~/out/1/statistics-order-4-gap-0.csv DFS is: 8372224 bytes DFS/Total is: 29 % FullPieces: 63 / 220 DirtyPieces: 1 / 220 FaildHashChecks: 0 Prefetching 5 % Playing point: 56 / 220 ( 25 %) Only last VOD client prints Output to .csv • In ~/out you will find 4 directories: /1, /2, /3 and /4. • In each directory, you have 4 files • statistics-order-<1..4>-gap-<GAP>.csv • Order is how many VOD clients ran before. • <GAP> is the parameter <GAP>. .csv file alg,dfs,p2p ORIG,5,0 ORIG,10,2 ORIG,15,6 ORIG,19,11 ORIG,23,16 ORIG,28,20 ORIG,31,25 ORIG,34,31 ORIG,37,36 ORIG,39,41 . . . File percentage Required plot 1 - total DFS and total P2P – GAP = 2 File percentage GAP = 0 Required plot 2 - local DFS GAP = 2 GAP = 0 The main script - run_multiple.sh • Location: – /users/courses/hanochworkshop/group_vod/vod_tests • run_multiple.sh: – Runs run_all.sh four times. – Parameters: • Your client – PATH/btdownloadheadless.py • Gap – wait Gap*PrefetchT between VOD spawns. • Csv location – where to store the .csv files. run_all.sh defaults • Start – VODClient: – Max_upload rate = 90 – prefetchT = 5% – Rate = 90 – Alg = ORIG – Verbose = TRUE # For last client in group • Stop – Kill all python threads • Kill – Kill all python threads and scripts. To run client separately: • PATH/btdownloadheadless.py --saveas F --delay D --prefetchT P --rate R --out_dir O Can add the parameters --order X in the IDE --gap G --group_size S --alg ‘ALG' --verbose F.Torrent Python • Very common script language • IDE: – PyDev – WINGWARE Python IDE – not free, but very good. • References: – python.org :http://docs.python.org/reference/ – http://wiki.python.org/moin/BeginnersGuide/NonProgrammers – http://wiki.python.org/moin/BeginnersGuide/Programmers • The book "Think like a Computer Scientist in Python" has a very good reputation: Getting started • Run the environment – – – – ssh pc-hanoch-w3 or pc-hanoch-w4 cd /users/courses/hanochworkshop/group_vod/vod_tests run_multiple.sh MyClient GAP output-directory Note tracker prints a message for each GET. • You can connect to the tracker, from the CS network, with a browser at: – http://pc-hanoch-wk:6970/ • Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files Directory Structure • Root – – high level application • Root/BitTornado – – services • Root/BitTornado/BT1 – – Actual work Client Operation • btdownloadheadless.py – First to run. – Performs initializations. • • • • get_response – parse the .torrent Infohash – the ID of the .torrent startEngine – Start connection with peers startRerequester - Start connection with tracker – Run rawserver.listen_forever from RawServer.py. – listen_forever is: • Polling port • executing func() from a task queue. Receiving messages • From listen_forever loop it goes to: – BT1/Connecter.py/got_message • • • • • • • • • CHOKE UNCHOKE INTERESTED NOT_INTERESTED BITFIELD REQUEST CANCEL PIECE HAVE Influencing decisions • Can be done in one of the callbacks which spring from Connecter.py / got_message: • Downloader.py – HAVE->got_have->send_interested • Tune for VoD? BT tracker • Send GET request in Rerequester.py / announce • Receive GET data in thread in Rerequester.py / _rerequest Policy • Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to. • Downloader.py / _request_more. • PiecePicker.py / next - rarest piece, etc. Tune for VoD? • got_unchoke->_request_more->next… More about the code • Start your project from “BitTornado - Start with this Code” link. • Click “BitTornado - Legacy Code Guide” for basic explanations about the code.