Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
VoD BitTorrent Framework Background • BT is a very popular peer to peer protocol with many implementations: – http://en.wikipedia.org/wiki/Comparison_of_Bit Torrent_clients • The first one was created in Python by Bram Cohen. Real world motivation • BitTorrent is common and Free • Good chance there is a swarm for your movie • VoD servers are expensive and complicated • Its better to use P2P for VoD Some definitions • Delay (D) = How much time I am allowed to hold before starting to play M (Movie). • Prefetch Time (P) = How much time before I watch byte B from M, it should arrive. • Playback Rate (R) = How many bytes/second I watch • B(t) – the index of the bit that must arrive by t. • B(t) = R *(P+t-D) Rules and Examples • B(t) = R * (P+t-D) • If R=5 KB/sec and D=6 seconds and P=2 seconds, than: – At t=6 seconds, I must have 10KB continuous from the start of the movie. – At t=18 seconds I must have 70KB continuous from the start of the movie. What does one VOD client do • Your client tries to get from the Bittorent clients pieces of data needed to play the video. • Each piece needs to arrive before its deadline. • If the piece does not arrive by the deadline you can download it “instantly” from the “server”. • Each VOD client counts the data from server (DFS). What is my goal • Each byte taken from the server costs money, therefore you need to minimize DFS. • Goal - Minimize the accumulated DFS of all the VOD clients that run concurrently. How do I create my client • Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this Code” link in the course web page. – It is also in /users/courses/hanochworkshop/group_vod • GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked ###### GROUP VOD ###### • Modify it to minimize acumulated DFS. • Do not change StreamWatched.py. Standard BT parameters • --saveas F • --max_upload_rate • F.torrent VOD client additional params • SET = defined inside run_all.sh, not a parameter. • delay – Wait before prefetch, in seconds. SET TO 20 • prefetchT – How much time in advance the byte must arrive. In percent of movie length. SET TO 5% • rate – VOD rate in KB/sec. SET TO 90 KB. • order – How many run when client spawned. SET BY run_all.sh. • • • • • • • Parameter of run_multiple.sh. gap – How much to wait between spawns – unit is pewfetchT. alg – Name of the algorithm. Default is ORIG. verbose – Print and kill the whole swarm when complete. peers_num – how many regular peers. seeds_num – how many seeds. Vod_num – how many VoD peers. How do I test my idea • cd /users/courses/hanochworkshop/voddiversegroup/vod_ tests • run_multiple.sh – your_client - example: ~/GROUP_VOD_CODE/btdownloadheadless.py – Gap – example: 0 – output_location example: ~/bt_out – iteration example: 4 – empty/full example: empty – peers_num example: 16 – seeds_num example: 2 – vod_num example: 14 Execution example run_multiple.sh Tracker BT BT BT BT BT run-all.sh BT BT BT BT VOD VOD BT VOD BT VOD BT BT BT How does it work? • Run tracker • Run the “world” • Run your clients How does it work? • Both tracker and the “world” run on the same computer. • Running a test: – Reserve time on pc-hanoch-w3/w4 in: • http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“ Output to screen ================================================================================== total time: 1 min 07 sec share rating: 0.904 (4816.0 KB up / 5328.0 KB down) saving: picture-100.bmp (27.5 MB) percent done: 18.9 time left: 4 min 52 sec download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp download rate: 93.4 kB/s upload rate: 80.6 kB/s seed status: 2 seen now, plus 0.340 distributed copies peer status: 28 seen now, 10.7% done at 1220.3 kB/s --------------------------------StreamWatcher------------------------------------Csv stats: ~/out/1/statistics-order-4-gap-0.csv DFS is: 8372224 bytes DFS/Total is: 29 % FullPieces: 63 / 220 DirtyPieces: 1 / 220 FaildHashChecks: 0 Prefetching 5 % Playing point: 56 / 220 ( 25 %) Only last VOD client prints Output to .csv • In ~/out you will find vod_num directories: /1, /2, …/iteration. • In each directory, you have vod_num files. • Statistics in - order-<1..vod_num>-gap-<GAP>.csv • Order is how many VOD clients ran before. • <GAP> is the parameter Gap. .csv file alg,dfs,p2p ORIG,5,0 ORIG,10,2 ORIG,15,6 ORIG,19,11 ORIG,23,16 ORIG,28,20 ORIG,31,25 ORIG,34,31 ORIG,37,36 ORIG,39,41 . . . File percentage total DFS and total P2P GAP = 2 File percentage GAP = 0 Local DFS GAP = 2 GAP = 0 run_all.sh defaults • Start – Use in run_multiple.sh • Stop – Kill all python threads • Kill – Kill all python threads and scripts. To run client separately: • PATH_TO_YOUR_VOD_CLIENT/btdownloadheadless.py --saveas F --delay D --prefetchT P --rate R --out_dir O Can add the parameters --order X in the IDE --gap G --group_size S --alg ‘ALG' --verbose F.Torrent Python • Very common script language • IDE: – PyDev – WINGWARE Python IDE – not free, but very good. • References: – python.org :http://docs.python.org/reference/ – http://wiki.python.org/moin/BeginnersGuide/NonProgrammers – http://wiki.python.org/moin/BeginnersGuide/Programmers • The book "Think like a Computer Scientist in Python" has a very good reputation: Getting started • Run the environment – – – – ssh pc-hanoch-w3 or pc-hanoch-w4 cd /users/courses/hanochworkshop/group_vod/vod_tests run_multiple.sh MyClient GAP output-directory Note tracker prints a message for each GET. • You can connect to the tracker, from the CS network, with a browser at: – http://pc-hanoch-wk:6970/ • Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files Directory Structure • Root – – high level application • Root/BitTornado – – services • Root/BitTornado/BT1 – – Actual work Client Operation • btdownloadheadless.py – First to run. – Performs initializations. • • • • get_response – parse the .torrent Infohash – the ID of the .torrent startEngine – Start connection with peers startRerequester - Start connection with tracker – Run rawserver.listen_forever from RawServer.py. – listen_forever is: • Polling port • executing func() from a task queue. Receiving messages • From listen_forever loop it goes to: – BT1/Connecter.py/got_message • • • • • • • • • CHOKE UNCHOKE INTERESTED NOT_INTERESTED BITFIELD REQUEST CANCEL PIECE HAVE Influencing decisions • Can be done in one of the callbacks which spring from Connecter.py / got_message: • Downloader.py – HAVE->got_have->send_interested • Tune for VoD? BT tracker • Send GET request in Rerequester.py / announce • Receive GET data in thread in Rerequester.py / _rerequest Policy • Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to. • Downloader.py / _request_more. • PiecePicker.py / next - rarest piece, etc. Tune for VoD? • got_unchoke->_request_more->next… More about the code • Start your project from “BitTornado - Start with this Code” link. • Click “BitTornado - Legacy Code Guide” for basic explanations about the code.