* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Slides
Survey
Document related concepts
Distributed firewall wikipedia , lookup
TCP congestion control wikipedia , lookup
SIP extensions for the IP Multimedia Subsystem wikipedia , lookup
Server Message Block wikipedia , lookup
Deep packet inspection wikipedia , lookup
Cracking of wireless networks wikipedia , lookup
Dynamic Host Configuration Protocol wikipedia , lookup
Zero-configuration networking wikipedia , lookup
Recursive InterNetwork Architecture (RINA) wikipedia , lookup
Internet protocol suite wikipedia , lookup
Remote Desktop Services wikipedia , lookup
Transcript
Chapter 2: Application layer 2.1 Principles of network applications 2.2 Web and HTTP 2.3 FTP 2.4 Electronic Mail 2.6 P2P applications SMTP, POP3, IMAP 2.5 DNS 2: Application Layer 1 Some network apps e-mail voice over IP web real-time video instant messaging remote login P2P file sharing multi-user network games streaming stored video clips conferencing cloud computing A Simple Survey of four most frequently used websites requiring login and password http://www.surveymon key.com/s/785PGPV 2: Application Layer 2 Creating a network app write programs that run on (different) end systems communicate over network e.g., web server software communicates with browser software No need to write software for network-core devices Network-core devices do not run user applications applications on end systems allows for rapid app development, propagation application transport network data link physical application transport network data link physical application transport network data link physical 2: Application Layer 3 Application architectures Client-server Peer-to-peer (P2P) Hybrid of client-server and P2P 2: Application Layer 4 Client-server architecture server: always-on host permanent IP address server farms for scaling clients: client/server communicate with server may be intermittently connected may have dynamic IP addresses do not communicate directly with each other 2: Application Layer 5 Google Data Centers Estimated cost of data center: $600M Google spent $2.4B in 2007 on new data centers, each using 50-100 megawatts Pure P2P architecture no always-on server arbitrary end systems directly communicate peer-peer peers are intermittently connected and change IP addresses Highly scalable but difficult to manage 2: Application Layer 7 Hybrid of client-server and P2P Skype voice-over-IP P2P application centralized server: finding address of remote party: client-client connection: direct (not through server) Instant messaging chatting between two users is P2P centralized service: client presence detection/location • user registers its IP address with central server when it comes online • user contacts central server to find IP addresses of buddies 2: Application Layer 8 some jargon related to applicationlevel protocols Process: program running user agent: implements within a host. user interface & application-level processes running in protocol different hosts Web: browser IE communicate by (implements HTTP) exchanging messages E-mail: PINE, Outlook with an application-layer (a reading agent protocol, e.g., HTTP implements POP3 and a (for web), SMTP, POP3 sending agent (for email access) implements SMTP) Transport Layer 3-9 A Big Picture: Processes Communication via Sockets (API provided by OS) users user agent API IP address Port # (e.g., 80 for HTTP, and 25 for SMTP Transport Layer 3-10 Socket Programming (More in Recitation/Project #1) App processes communicate with each other through sockets socket Socket API introduced in BSD4.1 UNIX, 1981 explicitly created, used, released by apps client/server paradigm two types of transport service via socket API: unreliable datagram reliable, byte streamoriented a host-local, application-created, OS-controlled interface (a “door”) into which application process can both send and receive messages to/from another application process A socket is associated with one port # on a host. Transport Layer 3-11 App-layer protocol defines Public-domain protocols: exchanged, e.g., request defined in RFCs & response messages allows for interoperability Syntax of message types: what fields in E.g., HTTP, SMTP messages & how fields uses well-defined port # are delineated (0 to 1023) Semantics of the fields, Proprietary protocols: ie, meaning of can’t use well-define information in fields port #s Rules for when and how E.g. Skype processes send & respond to messages Types of messages Transport Layer 3-12 Transport vs. network layer network layer: logical communication between hosts may not be reliable transport layer: logical communication between processes uses, enhances, network layer services Provides either reliable or unreliable services to app processes Use IP Address and Port #s. Household analogy: 12 kids in one house sending letters to 12 kids in another house processes = kids in each room app messages = letters in envelopes 2 hosts = 2 houses transport protocol = Ann and Bill (2 mail collectors in 2 houses) network-layer protocol = postal service Transport Layer 3-13 Socket-programming Socket: a door between application process and end-end-transport protocol (UDP or TCP) controlled by application developer controlled by operating system process process socket TCP with buffers, variables host or server internet socket TCP with buffers, variables controlled by application developer controlled by operating system host or server Transport Layer 3-14 Socket programming with UDP UDP: no “connection” between client and server no handshaking sender explicitly attaches IP address and port of destination to each packet OS attaches IP address and port of sending socket to each segment server must extract IP address, port of client from received packet UDP: transmitted data may be received out of order, or lost Client must contact server •server process must first be running •server must have created socket (door) that welcomes client’s contact Client contacts server by: •creating client-local socket •sending a UDP datagram specifying IP address, port number of server process application viewpoint UDP provides unreliable transfer of groups of bytes (“datagrams”) between client and server Transport Layer 3-15 1 Client/1 server with UDP socket Server (running on hostid) create socket, port=x, for incoming request: serverSocket = DatagramSocket(port) read request from serverSocket write reply to serverSocket specifying client host address, port number (y) Client create socket, clientSocket = DatagramSocket() (at port y) Create, address (hostid, port=x) send datagram request using clientSocket read reply from clientSocket close clientSocket Transport Layer 3-16 Example: Java client (UDP) Receive the converted characters and display them Process UDP packet receivePacket Send to a server for uppercase conversion Client input stream sendPacket Get keyboard input (in lowercase) from users monitor inFromUser keyboard UDP packet clientSocket UDP socket to network from network 2: Application Layer 17 Example: Java server (UDP) import java.io.*; import java.net.*; Create datagram socket at port 9876 class UDPServer { public static void main(String args[]) throws Exception { DatagramSocket serverSocket = new DatagramSocket(9876); byte[] receiveData = new byte[1024]; byte[] sendData = new byte[1024]; while(true) { Create space for received datagram Receive datagram DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); serverSocket.receive(receivePacket); Transport Layer 3-18 Example: Java server (UDP), cont String sentence = new String(receivePacket.getData()); Get IP addr port #, of Client/requester InetAddress IPAddress = receivePacket.getAddress(); int port = receivePacket.getPort(); String capitalizedSentence = sentence.toUpperCase(); sendData = capitalizedSentence.getBytes(); Create datagram to send to client DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, port); Write out datagram to socket serverSocket.send(sendPacket); } } } End of while loop, loop back and wait for another datagram Transport Layer 3-19 Example: Java client (UDP) import java.io.*; import java.net.*; Create input stream Create client socket Translate server name to IP address using DNS class UDPClient { public static void main(String args[]) throws Exception { BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in)); DatagramSocket clientSocket = new DatagramSocket(); InetAddress IPAddress = InetAddress.getByName("hostname"); byte[] sendData = new byte[1024]; byte[] receiveData = new byte[1024]; String sentence = inFromUser.readLine(); sendData = sentence.getBytes(); Transport Layer 3-20 Example: Java client (UDP), cont. Create datagram with data-to-send, length, IP addr, port DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, 9876); Send datagram to server clientSocket.send(sendPacket); Read datagram from server clientSocket.receive(receivePacket); DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); String modifiedSentence = new String(receivePacket.getData()); System.out.println("FROM SERVER:" + modifiedSentence); clientSocket.close(); } } Transport Layer 3-21 Socket programming with TCP Client must contact server server process must first be running server must have created socket (door) that welcomes client’s contact Client contacts server by: creating client-local TCP socket specifying IP address, port number of server process When client creates socket: client TCP establishes connection to server TCP To support multiple clients, server TCP can create new socket for each client All subsequent data exchanges over the new socket Similar concept applies to UDP too application viewpoint TCP provides reliable, in-order transfer of bytes (“pipe”) between client and server Transport Layer 3-22 Client/server socket interaction: TCP Server (running on hostid) Client create socket, port=x, for incoming request: welcomeSocket = ServerSocket(port) TCP wait for incoming connection request connection connectionSocket = welcomeSocket.accept() read request from connectionSocket write reply to connectionSocket close connectionSocket create socket, setup connect to hostid, port=x clientSocket = Socket(hostid, port) send request using clientSocket read reply from clientSocket close clientSocket Transport Layer 3-23 Example: Java client (TCP) import java.io.*; import java.net.*; class TCPClient { public static void main(String argv[]) throws Exception { String sentence; String modifiedSentence; Create input stream Create client socket, connect to server Create output stream attached to socket BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in)); Socket clientSocket = new Socket("hostname", 6789); DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream()); 2: Application Layer 24 Example: Java client (TCP), cont. Create input stream attached to socket BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream())); sentence = inFromUser.readLine(); Send line to server outToServer.writeBytes(sentence + '\n'); Read line from server modifiedSentence = inFromServer.readLine(); System.out.println("FROM SERVER: " + modifiedSentence); clientSocket.close(); } } 2: Application Layer 25 Example: Java server (TCP) import java.io.*; import java.net.*; class TCPServer { Create welcoming socket at port 6789 Wait, on welcoming socket for contact by client Create input stream, attached to socket public static void main(String argv[]) throws Exception { String clientSentence; String capitalizedSentence; ServerSocket welcomeSocket = new ServerSocket(6789); while(true) { Socket connectionSocket = welcomeSocket.accept(); BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream())); 2: Application Layer 26 Example: Java server (TCP), cont Create output stream, attached to socket DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream()); Read in line from socket clientSentence = inFromClient.readLine(); capitalizedSentence = clientSentence.toUpperCase() + '\n'; Write out line to socket outToClient.writeBytes(capitalizedSentence); } } } End of while loop, loop back and wait for another client connection 2: Application Layer 27 TCP Sockets vs UDP Sockets TCP socket identified by 4-tuple: source IP address source port number dest IP address dest port number Dest IP and port are not explicitly attached to segment. Server has two types of sockets: When client knocks on serverSocket’s “door,” server creates connectionSocket and completes TCP conx. Web servers have different sockets for each connecting client non-persistent HTTP will have different socket for each request Transport Layer 3-28 What transport service does an app need? Data loss some apps (e.g., audio) can tolerate some loss other apps (e.g., file transfer, telnet) require 100% reliable data transfer Timing some apps (e.g., Internet telephony, interactive games) require low delay to be “effective” Throughput some apps (e.g., multimedia) require minimum amount of throughput to be “effective” other apps (“elastic apps”) make use of whatever throughput they get Security Encryption, data integrity, … 2: Application Layer 29 Transport service requirements of common apps Application Data loss file transfer e-mail Web documents real-time audio/video no loss no loss no loss loss-tolerant stored audio/video interactive games instant messaging loss-tolerant loss-tolerant no loss Throughput Time Sensitive no elastic no elastic no elastic audio: 5kbps-1Mbps yes, 100’s msec video:10kbps-5Mbps yes, few secs same as above yes, 100’s msec few kbps up yes and no elastic 2: Application Layer 30 Internet transport protocols services TCP service: connection-oriented: setup required between client and server processes reliable transport between sending and receiving process flow control: sender won’t overwhelm receiver congestion control: throttle sender when network overloaded does not provide: timing, minimum throughput guarantees, security UDP service: unreliable data transfer between sending and receiving process does not provide: connection setup, reliability, flow control, congestion control, timing, throughput guarantee, or security Q: why bother? Why is there a UDP? 2: Application Layer 31 Internet apps: application, transport protocols Application e-mail remote terminal access Web file transfer streaming multimedia Internet telephony Application layer protocol Underlying transport protocol SMTP [RFC 2821] Telnet [RFC 854] HTTP [RFC 2616] FTP [RFC 959] HTTP (eg Youtube), RTP [RFC 1889] SIP, RTP, proprietary (e.g., Skype) TCP TCP TCP TCP TCP or UDP typically UDP 2: Application Layer 32 Chapter 2 outline 2.1 Principles of app layer protocols 2.6 P2P file sharing clients and servers app requirements 2.2 Web and HTTP 2.3 FTP 2.4 Electronic Mail SMTP, POP3, IMAP 2.5 DNS Transport Layer 3-33 Web and HTTP First some jargon Web page consists of objects Object can be HTML file, JPEG image, Java applet, audio file,… Web page consists of base HTML-file which includes several referenced objects Each object is addressable by a URL Example URL: www.someschool.edu/someDept/pic.gif host name path name 2: Application Layer 34 HTTP connections Nonpersistent HTTP At most one object (e.g., a HTML file, or a jpeg image but not both!) is sent over a TCP connection. HTTP/1.0 uses nonpersistent HTTP Persistent HTTP Multiple objects can be sent over single TCP connection between client and server. HTTP/1.1 uses persistent connections in default mode Transport Layer 3-35 Non-Persistent HTTP Response time modeling Definition of RRT: time to send a small packet to travel from client to initiate TCP server and back. connection Response time: RTT request one RTT to initiate TCP file connection RTT one RTT for HTTP request file and first few bytes of received response to return One file/one object time transmission time total = 2RTT+transmit time time to transmit file time Transport Layer 3-36 Persistent HTTP Persistent without pipelining: client issues new request only when previous response has been received one RTT for each referenced object Persistent with pipelining: default in HTTP/1.1 client sends requests as soon as it encounters a referenced object as little as one RTT for all the referenced objects HTTP is “stateless” server maintains no information about past client requests Transport Layer 3-37 User-server interaction: authorization Authorization : control access to server client server content usual http request msg authorization credentials: typically name, password 401: authorization req. WWW authenticate: stateless: client must present authorization in each request authorization: header line in each request if no authorization: header, server refuses access, sends WWW authenticate: header line in response usual http request msg + Authorization: <cred> usual http response msg usual http request msg + Authorization: <cred> usual http response msg time Transport Layer 3-38 Cookies: keeping “state” (privacy issue?) client Cookie file server usual http request msg usual http response + ebay: 8734 Cookie file amazon: 1678 ebay: 8734 Set-cookie: 1678 usual http request msg cookie: 1678 usual http response msg one week later: Cookie file amazon: 1678 ebay: 8734 usual http request msg cookie: 1678 usual http response msg server creates ID 1678 for user amazon cookiespecific action cookiespectific action Transport Layer 3-39 Cookies (continued) What cookies can bring: authorization shopping carts recommendations user session state (Web e-mail) aside Cookies and privacy: cookies permit sites to learn a lot about you you may supply name and e-mail to sites How to keep “state”: protocol endpoints: maintain state at sender/receiver over multiple transactions cookies: http messages carry state 2: Application Layer 40 Web caches (proxy server) Goal: satisfy client request without involving origin server user sets browser: Web accesses via cache browser sends all HTTP requests to cache If object in cache: cache returns object else cache requests object from origin server, then returns object to client origin server client client Proxy server origin server Transport Layer 3-41 Content distribution networks (CDNs) The content providers (e.g., foxnews.com) own the original server Content replication CDN company (e.g., Akamai) installs hundreds of CDN servers throughout Internet in lower-tier ISPs, close to users CDN replicates its customers’ content in CDN servers. When provider updates content, CDN updates servers origin server in North America CDN distribution node CDN server in S. America CDN server in Europe CDN server in Asia Transport Layer 3-42 More on CDN goto www.cdn.com goto the nearest CDN server not just Web pages streaming stored audio/video streaming real-time audio/video CDN nodes create application-layer overlay network Transport Layer 3-43 FTP: separate control, data connections TCP control connection port 21 FTP client contacts FTP server at port 21, specifying TCP as transport protocol Client obtains authorization over control connection Client browses remote directory by sending commands over control connection. When server receives a command for a file transfer, the server opens a TCP data connection to client After transferring one file, server closes connection. FTP client TCP data connection port 20 FTP server Server opens a second TCP data connection to transfer another file. Control connection: “out of band” FTP server maintains “state”: current directory, earlier authentication Transport Layer 3-44 Electronic Mail outgoing message queue user mailbox user agent Three major components: user agents mail servers mail server SMTP simple mail transfer protocol: SMTP User Agent a.k.a. “mail reader” composing, editing, reading mail messages e.g., Eudora, Outlook, elm, Mozilla Thunderbird outgoing, incoming messages stored on server SMTP SMTP mail server user agent user agent mail server user agent user agent user agent 2: Application Layer 45 Scenario: Alice sends message to Bob 1) Alice uses UA to compose message to bob 2) Alice’s UA sends message to her mail server; message placed in message queue 3) Client side of SMTP opens TCP connection with Bob’s mail server 4) SMTP client sends Alice’s message over the TCP connection 5) Bob’s mail server places the message in Bob’s mailbox 6) Bob invokes his user agent to read message Access to folders: Yes with IMAP or HTTP; No with POP3 1 user agent SMTP 2 HTTP mail server 3 SMTP 4 mail server 5 POP3 IMAP 6 user agent HTTP Transport Layer 3-46 SMTP: final words SMTP establishes a direct client-server (persistent) TCP connection SMTP requires message (header & body) to be in 7bit ASCII For non-ASCII data, use Multipurpose Internet Mail Extension (MIME) encoding Encoding method (e.g., base64) and type of the encoded data (e.g., jpeg) are sent in the header Comparison with HTTP: HTTP: pull (client request) SMTP: push (client send) both have ASCII command/response interaction, status codes HTTP: each object encapsulated in its own response msg SMTP: multiple objects sent in multipart msg Transport Layer 3-47 DNS: Domain Name System Maps between a host’s Local (default), Root and name and its IP address Other functions Authoritative name servers host and mail server aliasing (with multiple names) Load balancing with replicated web servers (one name maps to a set of IP addresses distributed database implemented with a hierarchy and caching Transport Layer 3-48 P2P: centralized directory original “Napster” design 1) when a user (peer) connects, it informs central server: Bob centralized directory server 1 peers 1 IP address content 2) Alice queries for a file 3) Alice requests/gets file from Bob 4) A node = client + server 5) failure, bottleneck, copyright infringement 3 1 2 1 Alice Transport Layer 3-49 P2P: decentralized directory KaZaA/FastTrack use a bootstrap node Each peer is either a group leader or assigned to a group leader. Group leader tracks the content in all its group members. Peer queries group leader; group leader may query other group leaders. ordinary peer group-leader peer Bootstrap node neighoring relationships in overlay network Transport Layer 3-50 P2P: Query flooding Gnutella no hierarchy/group leaders use bootstrap node to learn about others send join message Send query to neighbors Neighbors forward query If queried peer has object, it sends message back to querying peer join Transport Layer 3-51 P2P: more on query flooding Pros no group leaders (which maybe overburdened) Cons even more difficult to maintain (when nodes leave) may generate excessive query traffic Solution: limit the number of “hops” or query radius query radius: may not find content when present bootstrap node Transport Layer 3-52 File Distribution: Server-Client vs P2P Question : How much time to distribute file from one server to N peers? us: server upload bandwidth Server us File, size F dN uN u1 d1 u2 ui: peer i upload bandwidth d2 di: peer i download bandwidth Network (with abundant bandwidth) 2: Application Layer 53 File distribution time: server-client server sequentially sends N copies: NF/us time client i takes F/di time to download Time to distribute F to N clients using client/server approach Server F us dN u1 d1 u2 d2 Network (with abundant bandwidth) uN = dcs = max { NF/us, F/min(di) } i increases linearly in N (for large N) 2: Application Layer 54 File distribution time: P2P server must send one copy: F/us time client i takes F/di time to download NF bits must be uploaded and downloaded (aggregate) Server F us dN u1 d1 u2 d2 Network (with abundant bandwidth) uN fastest possible upload rate: us + Sui Download faster than upload (so won’t be the bottleneck) dP2P = max { F/us, F/min(di) , NF/(us + Sui) } i 2: Application Layer 55 Server-client vs. P2P: example Client upload rate = u, F/u = 1 hour, us = 10u, dmin ≥ us Minimum Distribution Time 3.5 P2P Client-Server 3 2.5 2 1.5 1 0.5 0 0 5 10 15 20 25 30 35 N 2: Application Layer 56 File distribution: BitTorrent P2P file distribution torrent: group of peers exchanging chunks of a file tracker: tracks peers participating in torrent obtain list of peers trading chunks peer 2: Application Layer 57 BitTorrent (1) file divided into 256KB chunks. peer joining torrent: has no chunks, but will accumulate them over time registers with tracker to get list of peers, connects to subset of peers (“neighbors”) while downloading, peer uploads chunks to other peers. peers may come and go once peer has entire file, it may (selfishly) leave or (altruistically) remain 2: Application Layer 58 BitTorrent (2) Pulling Chunks at any given time, different peers have different subsets of file chunks periodically, a peer (Alice) asks each neighbor for list of chunks that they have. Alice sends requests for her missing chunks rarest first Sending Chunks: tit-for-tat Alice sends chunks to four neighbors currently sending her chunks at the highest rate re-evaluate top 4 every 10 secs every 30 secs: randomly select another peer, starts sending chunks newly chosen peer may join top 4 “optimistically unchoke” 2: Application Layer 59 BitTorrent: Tit-for-tat (1) Alice “optimistically unchokes” Bob (2) Alice becomes one of Bob’s top-four providers; Bob reciprocates (3) Bob becomes one of Alice’s top-four providers With higher upload rate, can find better trading partners & get file faster! 2: Application Layer 60 Distributed Hash Table (DHT) DHT = distributed P2P database Database has (key, value) pairs; key: ss number; value: human name key: content type; value: IP address Peers query DB with key DB returns values that match the key Peers can also insert (key, value) peers DHT Identifiers Assign integer identifier to each peer in range [0,2n-1]. Each identifier can be represented by n bits. Require each key to be an integer in same range. To get integer keys, hash original key. eg, key = h(“Led Zeppelin IV”) This is why they call it a distributed “hash” table How to assign keys to peers? Central issue: Assigning (key, value) pairs to peers. Rule: assign key to the peer that has the closest ID. Convention in lecture: closest is the immediate successor of the key. Ex: n=4; peers: 1,3,4,5,8,10,12,14; key = 13, then successor peer = 14 key = 15, then successor peer = 1 Circular DHT (1) 1 3 15 4 12 5 10 8 Each peer only aware of immediate successor and predecessor. “Overlay network” Circle DHT (2) O(N) messages on avg to resolve query, when there are N peers 0001 Who’s resp for key 1110 ? I am 0011 1111 1110 0100 1110 1110 1100 1110 1110 Define closest as closest successor 1010 1110 1000 0101 Peer Churn 1 3 15 4 12 •To handle peer churn, require each peer to know the IP address of its two successors. • Each peer periodically pings its two successors to see if they are still alive. 5 10 8 Peer 5 abruptly leaves Peer 4 detects; makes 8 its immediate successor; asks 8 who its immediate successor is; makes 8’s immediate successor its second successor. What if peer 13 wants to join? P2P Case study: Skype Skype clients (SC) inherently P2P: pairs of users communicate. proprietary Skype login server application-layer protocol (inferred via reverse engineering) hierarchical overlay with SNs Index maps usernames to IP addresses; distributed over SNs Supernode (SN) 2: Application Layer 67 Peers as relays Problem when both Alice and Bob are behind “NATs”. NAT prevents an outside peer from initiating a call to insider peer Solution: Using Alice’s and Bob’s SNs, Relay is chosen Each peer initiates session with relay. Peers can now communicate through NATs via relay 2: Application Layer 68