Download Project Presentation: KAD Crawler

A way to vizualize KAD network
Elena Digor,
May, 4. 2009
Jacobs University, Bremen
Crawling KAD
Table of content
●
●
●
Introduction
–
P2P networks
–
KAD network
–
Related work
Approach
–
General description
–
Analysis
–
Has been done
Conclusions
Introduction
(P2P networks)
●
Objectives:
–
Decentralization
–
Scalability
–
Fault tolerance
–
Load balance
Introduction
(KAD network)
●
Open source from aMule/eMule
●
Decentralized
–
●
●
DHT network, based on XOR metrics
Uses UDP protocol (messaging port)
–
Quick messages
–
Handles churn
TCP protocol (service port)
–
uploading/downloading files
Introduction
(KAD network)
●
Routing table: (src/kademlia/routing)
–
Binary tree
●
●
Leaves: k-buckets of contacts (k=10)
Nodes – the routing zones (distances to
contacts)
Related work
●
We've seen Kademlia description
●
We've seen KAD vizualization
●
No official KAD specification
●
No open source crawler for KAD
Approach
●
Create our own crawler
–
Analyze aMule code
–
Modify to get more living contacts
–
Dump contacts in a database
MySQL Setup
Field
Type
Null
Key
Default
KAD_ID
varchar(16)
NO
PRI
NULL
IP
varchar(16)
NO
NULL
START_TIME
timestamp
NO
CURRENT_TIMESTAMP
END_TIME
timestamp
NO
0000-00-00 00:00:00
DEAD
int(11)
YES
NULL
STATIC
tinyint(1)
YES
NULL
KADEMLIA_REQ
int(11)
YES
NULL
Setup
●
aMule from sources
●
Monitored under OS X and Linux
●
Wireshark was used for grabbing packets
●
UDP packets were further analyzed
Analysis (wireshark)
Analysis (Wireshark)
●
Initial Handshake
●
Packets' structure:
–
<ID, opcode, data*>
–
ID: 0xE4 or 0xE5
–
Opcode : operation
Analysis (source)
●
aMule 2.2.4
–
Hybrid: e2dk and KAD
–
Uses WXwidgets
–
“\kademlia”, “\include”, and common files
Analysis (source)
●
Kademlia:
–
Main file: Kademlia.cpp
–
Routing table: (RoutingZone and RoutingBin)
–
KademliaUDPListener.cpp
–
SearchManager.cpp and Search.cpp
Analysis (source)
●
Important OP-codes:
–
KADEMLIA_HELLO_REQ
= 0x10, // <PEER (sender) [25]>
–
KADEMLIA_HELLO_RES
–
KADEMLIA_REQ
= 0x20, // <TYPE [1]> <HASH (target)
[16]> <HASH (receiver) 16>
–
KADEMLIA_RES
<PEER [25]>*(CNT)
–
KADEMLIA_SEARCH_REQ
= 0x30, // <HASH (key) [16]>
<ext 0/1 [1]> <SEARCH_TREE>[ext]
–
KADEMLIA_SEARCH_RES
= 0x38, // <HASH (key) [16]>
<CNT1 [2]> (<HASH (answer) [16]> <CNT2 [2]>
<META>*(CNT2))*(CNT1)
–
etc
= 0x18, // <PEER (receiver) [25]>
= 0x28, // <HASH (target) [16]> <CNT>
Analysis (source)
●
New nodes are generated from:
–
Passive incoming requests
–
Bootstrap requests
●
–
Get 20 new nodes
Requests for finding target
●
Gets 16 closer nodes
KADEMLIA_BOOTSTRAP_REQ
●
KADEMLIA_BOOTSTRAP_REQ = 0x00
–
●
<PEER_HASH (sender) [16], IP [4],
UDP_PORT [2], TCP_PORT[2], Type [1]>
KADEMLIA_BOOTSTRAP_RES = 0x08
–
<20*(PEERs)>
Has been done
●
●
●
Analyzed the source code and algorithms
Modified “KademliaUDPListener.cpp” to spy
on incoming “*_RES”.
Dump info from incoming requests into the
database
ToDo
●
Force more “bootstrap” messages
●
Dump info in the database
●
Vizualize the data from database
●
Run more than one clients in parallel
Conclusions
●
aMule/eMule – hybrids (use KAD and e2dk)
●
Act differently on different OSes
●
Modifying the code helps to build a basic
crawler
Thank you!
Questions? :)