Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
HTTP Programming Copyright (C) 2009 Dr.Yong Uk Song Yonsei University Wonju Campus What is HTTP? Hypertext Transfer Protocol (HTTP) is an application-level (application-layer) protocol for distributed, collaborative, hypermedia information systems, i.e. the World Wide Web. HTTP development was coordinated by the World Wide Web Consortium and the Internet Engineering Task Force (IETF), culminating in the publication of a series of Requests for Comments (RFCs), most notably RFC 2616 (June 1999), which defines HTTP/1.1, the version of HTTP in common use. HTTP is a request/response standard of a client and a server. A client is the end-user, the server is the web site. The client making a HTTP request - using a web browser, spider, or other end-user tool - is referred to as the user agent. The responding server - which stores or creates resources such as HTML files and images - is called the origin server. TCP/IP & OSI 7 Layer OSI 7 Layer Application Presentation Session Internet Protocol Suite BGP · DHCP · DNS · FTP · GTP · HTTP · IMAP · IRC · Megaco · MGCP · NNTP · NTP · POP · RIP Application · RPC · RTP · RTSP · SDP · SIP · SMTP · SNMP · SOAP · SSH · Telnet · TLS/SSL · XMPP · (more) Transport Network TCP ICMP Transport UDP IP ARP RARP Internet Data Link H/W interface: Ethernet, Token-Ring … Physical Link Architecture of WWW user input GUI user output content File Input request with arguments Web Browser HTTP response Web Server HTML file CGI input CGI CGI output CGI program History of HTTP HTTP/0.9 : 1991 HTTP/1.0 : 1992, RFC1945 HTTP/1.1 : Jan. 1997, RFC2068 HTTP/1.1 : Jun. 1999, RFC2616 What is in HTTP? Dialogue-layer protocol Presentation-layer protocol Other protocols HTTP: Dialogue-layer Protocol (1) It is based on the Request/Response paradigm. A connection is established. A client sends a request to a server. The server sends a response to the client. The connection is ended. In terms of TCP/IP programming: After socket connection, A client sends (prints) a message to a server. The server receives (reads) the message from the client. The server makes a response message based on the message from the client. The server sends (prints) the response message to the client. The client receives (reads) the message from the server. The socket connection is ended. The client processes the message: prints out on window, or, …. HTTP: Dialogue-layer Protocol (2) * Client * Server … String request = "…"; output.println(request); … input.readLine(); … input.readLine(); String response = "…"; output.println(response); … HTTP: Presentation-layer protocol (1) It defines the format of request and response. Definitions HTTP-message Request = = Response = Request | Response Request-Line * (General-Header | Request-Header | Entity-Header) crlf [Entity-Body] Status-Line * (General-Header | Response-Header | Entity-Header) crlf [Entity-Body] Legend: | - or * - multiple [ ] - optional HTTP: Presentation-layer protocol (2) Example : Request (1) GET http://www.miami.muohio.edu/ HTTP/1.1 Host: www.example.com crlf Example : Request (2) POST /HTTP2Server/login.jsp HTTP/1.1 Host: www.example.com Content-Type: application/x-www-form-urlencoded Content-Length: 30 crlf username=yusong&password=10000 HTTP: Presentation-layer protocol (3) Example : Response HTTP/1.1 200 OK Date: Wed, 08 Jul 2009 21:39:26 GMT Server: Apache/2.0.46 (Red Hat) Content-Language: en-US Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8 crlf <!DOCTYPE html> <html> <head> <title>Hello HTML</title> </head> <body> <p>Hello World!</p> </body> </html> Digression URL URL (1) A Uniform Resource Locator (URL) is a type of Uniform Resource Identifier (URI) that specifies where an identified resource is available and the mechanism for retrieving it. In its current strict technical meaning, a URL is a URI that, in addition to identifying a resource, provides a means of locating the resource by describing its primary access mechanism (e.g., its network 'location'). In popular language, a URL is also referred to as a Web address. URL (2) Format resource_type://domain:port/filepathname?query_string#anchor e.g. http://dragon.yonsei.ac.kr/~yusong/index.html http://yonsei.ac.kr:8000/~yusong/a.cgi?username=yusong&b=b ftp://dragon.yonsei.ac.kr/~yusong/sample.file telnet://dragon.yonsei.ac.kr rlogin://[email protected] Tim Berners-Lee regrets: http://www.serverroute.com/path/to/file.html http://com/serverroute/www/path/to/file.html HTTP (1) : Client using GET Method HTTP (1) : Client using GET Method HTTP1Client import java.io.*; import java.net.*; try { Socket socket = new Socket("www.miami.muohio.edu", 80); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("GET http://www.miami.muohio.edu/ HTTP/1.1"); output.println("Host: www.example.com"); output.println(""); String response; for (; (response = input.readLine()) != null;) { Host Port System.out.println(response); } output.close(); input.close(); socket.close(); Request } catch (Exception e) { System.err.println(e); } Response HTTP (1) : Client using GET Method Request Message GET http://www.miami.muohio.edu/ HTTP/1.1 Host: www.example.com crlf HTTP (1) : Client using GET Method Response Message HTTP/1.1 200 OK Date: Wed, 22 Jul 2009 18:03:36 GMT Server: Apache/2.0.46 (Red Hat) Set-Cookie: SaneID=134.53.7.196-1248285816897469; path=/; expires=Sat, 26-Jul-14 18:03:36 GMT; domain=.muohio.edu Content-Language: en-US Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8 crlf <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta name="keywords" content="miami,university,ohio,college,public" /> <meta name="description" content="Located in Oxford, Ohio, Miami University is a public university of 16,000 students offering strong academics, a vibrant residence life program, and faculty who love to teach and mentor students." /> <title>Miami University</title> … </html> HTTP (1) : Client using GET Method Exercise Make a Java program such that: Send a request to the Web page of "http://www.muohio.edu:8080/", and print out the response from the Web server. Project name HTTP1X1Client Digression Website (1) : Login Website (1) : Login HTTP2Server/login.html <html> <body> <h1>Login</h1> <form method="post" action="login.jsp"> <table> <tr> <td>User name:</td> <td><input type="text" name="username"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="password"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Login"></td> </tr> </table> </form> </body> </html> Website (1) : Login HTTP2Server/login.jsp <% String username = request.getParameter("username"); String password = request.getParameter("password"); %> <html> <head> <title>Login</title> </head> <body> <h1>Hello "<%= username%>"</h1> <h3>You typed the password as "<%= password%>".</h3> </body> </html> HTTP (2) : Client using POST Method HTTP (2) : Client using POST Method HTTP2Client import java.io.*; import java.net.*; try { Socket socket = new Socket("localhost", 8080); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("POST /HTTP2Server/login.jsp HTTP/1.1"); output.println("Host: www.example.com"); output.println("Content-Type: application/x-www-form-urlencoded"); output.println("Content-Length: 30"); output.println(""); output.println("username=yusong&password=10000"); Host Port String response; for (; (response = input.readLine()) != null;) { System.out.println(response); } output.close(); Request input.close(); socket.close(); } catch (Exception e) { System.err.println(e); } Response HTTP (2) : Client using POST Method HTTP2Client import java.io.*; import java.net.*; try { Socket socket = new Socket("localhost", 8080); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("POST /HTTP2Server/login.jsp HTTP/1.1"); output.println("Host: www.example.com"); output.println("Content-Type: application/x-www-form-urlencoded"); output.println("Content-Length: " + "username=yusong&password=10000".length()); output.println(""); output.println("username=yusong&password=10000"); String response; for (; (response = input.readLine()) != null;) { System.out.println(response); } output.close(); input.close(); socket.close(); } catch (Exception e) { System.err.println(e); } HTTP (2) : Client using POST Method Request Message POST /HTTP2Server/login.jsp HTTP/1.1 Host: www.example.com Content-Type: application/x-www-form-urlencoded Content-Length: 30 crlf username=yusong&password=10000 HTTP (2) : Client using POST Method Response Message HTTP/1.1 200 OK X-Powered-By: JSP/2.1 Server: Sun GlassFish Enterprise Server v2.1 Set-Cookie: JSESSIONID=39d808dbc6c8600ab0b0786f6fbb; Path=/HTTP2Server Content-Type: text/html;charset=UTF-8 Content-Length: 282 Date: Wed, 22 Jul 2009 17:58:44 GMT crlf <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>Login</title> </head> <body> <h1>Hello "yusong"</h1> <h3>You typed the password as "10000".</h3> </body> </html> HTTP (2) : Client using POST Method Exercise Make a Java program such that: Send a request to the Web page at "http://localhost:8080/HTTP2X1Server/register.jsp", with an entity body of "name=X&username=X&password=X&password2=X&address=X&zip =X". Print out the response from the Web server. Project names HTTP2X1Server/register.html (see next slide) HTTP2X1Server/register.jsp (see next slide) HTTP2X1Client Website (2) : Register HTTP2X1Server/register.html <html> <body> <h1>Register</h1> <form method="post" action="login.jsp"> <table> <tr> <td>Name:</td> <td><input type="text" name="name"></td> </tr> <tr> <td>User name:</td> <td><input type="text" name="username"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="password"></td> </tr> <tr> <td>Password (retype):</td> <td><input type="password" name="password2"> </td> </tr> <tr> <td>Address:</td> <td><input type="text" name="address"></td> </tr> <tr> <td>ZIP code:</td> <td><input type="text" name="zip"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Register"></td> </tr> </table> </form> </body> </html> Website (2) : Register HTTP2X1Server/register.jsp <% String name = request.getParameter("name"); String username = request.getParameter("username"); String password = request.getParameter("password"); String password2 = request.getParameter("password2"); String address = request.getParameter("address"); String zip = request.getParameter("zip"); %> <html> <head> <title>Login</title> </head> <body> <h1>Hello "<%= name %>"!</h1> <h3>Your information (<%= username %>, <%= password %>, <%= password2 %>, <%= address %>, <%= zip %>) will be processed later.</h3> </body> </html> HTTP (3) : Web Server HTTP (3) : Web Server HTTP3Server import java.io.*; import java.net.*; try { ServerSocket serverSocket = new ServerSocket(8000); Socket clientConn = serverSocket.accept(); BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream())); PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true); String request; for (; (request = input.readLine()).isEmpty() == false;) { System.out.println(request); } output.println("HTTP/1.1 200 OK"); output.println(""); output.println("<HTML>"); output.println("<BODY><H1>This is a stupid Web server.</H1></BODY>"); output.println("</HTML>"); output.close(); input.close(); Request clientConn.close(); serverSocket.close(); } catch (Exception e) { System.err.println(e); } Response Port HTTP (3) : Web Server Response Message HTTP/1.1 200 OK crlf <HTML> <BODY><H1>This is a stupid Web server.</H1></BODY> </HTML> HTTP (3) : Web Server for Redirect Exercise Make a Java program such that: Respond to any request from any Web browser. Print out the request header from the Web browser. Send a response like below: HTTP/1.1 302 Found Location: http://www.miami.muohio.edu/ crlf We moved. Project name HTTP3X1Server HTTP (3) : Web Server for Redirect HTTP3X1Server import java.io.*; import java.net.*; try { ServerSocket serverSocket = new ServerSocket(8000); Socket clientConn = serverSocket.accept(); BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream())); PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true); String request; for (; (request = input.readLine()).isEmpty() == false;) { System.out.println(request); } output.println("HTTP/1.1 302 Found"); output.println("Location: http://www.miami.muohio.edu/"); output.println(""); output.println("We moved."); output.close(); input.close(); clientConn.close(); Request serverSocket.close(); } catch (Exception e) { System.err.println(e); } Response Port HTTP (3) : Web Server for Redirect Response Message HTTP/1.1 302 Found Location: http://www.miami.muohio.edu/ crlf We moved. Encoding Note on Encoding Encoding is the process of transforming information from one format into another. ASCII : alphabet → integer (e.g. A → 65, …) Presentation-layer protocols are also encoding standards. Request/response message format in HTTP HTML … Encoding is used everywhere. What was the first encoding method in human history? ? Numbers Arabic numeral : 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 Roman numeral : I, II, III, IV, V, VI, VII, VIII, IX, X, XI, XI, … Letters Meaning : Chinese Syllable : Japanese Phoneme : English, Korean, … (Alphabet, consonant, vowel) Digression HTML HTML HTML, an initialism of HyperText Markup Language, is the predominant markup language for web pages. A markup language is a set of annotations to text that describe how it is to be structured, laid out, or formatted. Hypertext refers to text on a computer that will lead the user to other, related information on demand. Rather than remaining static like traditional text, hypertext makes possible a dynamic organization of information through links and connections (called hyperlinks). HTML HTML provides a means to describe the structure of text- based information in a document — by denoting certain text as links, headings, paragraphs, lists, and so on — and to supplement that text with interactive forms, embedded images, and other objects. HTML is written in the form of tags, surrounded by angle brackets. Simple Example <HTML> <HEAD> <TITLE>Simple HTML</TITLE> </HEAD> <BODY> <H1>Head</H1> First paragraph<P> Second paragraph<P> </BODY> </HTML> Tags Angle brackets Heading Paragraphs HTML: Markup Tag Formats <tag-name> e.g. <br> <tag-name> text </tag-name> e.g. <b>Bold</b>, <TITLE> … </TITLE>, <BODY> … </BODY> <tag-name attribute=argument> e.g. <img src="a.jpg"> <tag-name attribute=argument> text </tag-name> e.g. <a href="http://www.muohio.edu">Miami University</a> Note Tag-names are case-insensitive. <HTML>, <html>, <Html>, <hTmL> Multiple consecutive whitespaces (space, tab, EOLN) are regarded as one space. HTML: Text paragraph : <P> ... [</P>] line break : <BR> centering : <CENTER> ... </CENTER> bold : <B> ... </B> italic : <I> ... </I> headline : <Hn> ... </Hn> Many tags are deprecated now. (no more standard) HTML: Special Characters Format &character-name; Special characters < > " & → → → → < > "E; & HTML: Hyperlink Anchor Tag <A HREF = "address"> ... </A> e.g. <A HREF="http://www.apple.com">Apple</A> Relative Path vs. Absolute Path HREF = "abc.html" HREF = "../ddd/abc.html" HREF = "http://www.joongang.co.kr" HTML: List Unnumbered Lists <UL> <LI> ... <LI> ... </UL> Numbered Lists <OL> <LI> ... <LI> ... </OL> Definition Lists <DL> [<DT> ...] <DD> ... [<DT> ...] <DD> ... </DL> HTML: Image <IMG SRC = "address/... .gif" ALIGN = TOP/MIDDLE/BOTTOM/LEFT/RIGHT> <IMG SRC = "address/... .jpg"> e.g. <IMG SRC = "picture.gif"> HTML: Table <TABLE BORDER[ = n]> <CAPTION> ... </CAPTION> <TR> <TH> ... </TH> <TD> ... </TD> </TR> <TR> <TD> ... </TD> <TD> ... </TD> </TR> </TABLE> e.g. <TABLE BORDER> <TR> <TD> A11 </TD> <TD> A12 </TD> </TR> <TR> <TH> B11 </TH> <TD> B12 </TD> </TR> </TABLE> HTML: Form Will be addressed later in JSP Exercise 1: Text and Hyperlink <a href="…">…</a> Exercise 2: Numbered List <ol><li>…<li>…</ol> Exercise 3: Numbered List and Hyperlink <ol><li><a href="…">…</a><li>…</ol> Exercise 4: Table <table><tr><td>…</td>...</tr>…</table> Exercise 5: Table and Hyperlink <table><tr><td><a href="…">…</a></td>...</tr>…</table> Exercise 6: Table and Height <td height="50"> Exercise 7: Table and Width <td width="200"> Exercise 8: Table and Size <td height="50" width="200"> Exercise 9: Image <img src="a.jpg"> Exercise 10: Image and Hyperlink <a href="…"><img src="…"></a> Exercise 11: Table, Image and Hyperlink <table><tr><td>…</td><td><a href="…"><img src="…"></a></td></tr></table> HTML Exercise Make your own homepage. Project name HTML1X1 Demo: A Web Server Supporting HTML Files Demo (1) Project name HTTPD1Server Testing Scenarios Prepare an HTML file to browse. Access the Demo Web server using a Web browser. Demo (1) : Architecture user input GUI user output content File Input request with arguments Web Browser HTTP response Web Server HTML file CGI input CGI CGI output CGI program Demo (1) import java.io.*; import java.net.*; import java.util.*; public class Main { private static String getFilename(String request) { StringTokenizer t = new StringTokenizer(request, " "); t.nextToken(); return ("." + t.nextToken()); } public static void main(String[] args) { try { ServerSocket serverSocket = new ServerSocket(8080); Socket clientConn = serverSocket.accept(); BufferedReader ninput = new BufferedReader(new InputStreamReader(clientConn.getInputStream())); PrintWriter noutput = new PrintWriter(clientConn.getOutputStream(), true); String request = ninput.readLine(); String filename = getFilename(request); System.out.println(request); System.out.println("Filename - " + filename); BufferedReader finput = new BufferedReader(new InputStreamReader(new FileInputStream(filename))); noutput.println("HTTP/1.1 200 OK\n"); String line; while ((line = finput.readLine()) != null) { noutput.println(line); } finput.close(); noutput.close(); ninput.close(); clientConn.close(); serverSocket.close(); } catch (Exception e) { System.err.println(e); } } } Java URL & HttpURLConnection Class Java URL & HttpURLConnection Class As Internet sockets were provided for TCP/IP programming, Java provides URL class and HttpURLConnection class for HTTP programming. The URL class is used for the requests using GET method. Simple to use, but limited in functionality The HttpURLConnection class is used for a general HTTP programming. Complicated to use, but has full functionalities The action of sending and receiving messages to and from Web servers is called Web Scrapping, and URL class and HttpURLConnection class are the basic tools for Web Scrapping. Java URL & HttpURLConnection Class import java.net.*; import java.io.*; public URL(String fullURL) throws MalformedURLException public InputStream openStream( ) throws IOException public URLConnection openConnection( ) throws IOException public abstract HttpURLConnection(URL url) public void setDoOutput(boolean dooutput) public OutputStream getOutputStream( ) throws IOException public String getHeaderField(String name) public InputStream getInputStream( ) throws IOException HTTP (4) : URL Class HTTP4Client import java.io.*; import java.net.*; try { URL url = new URL("http://www.miami.muohio.edu/"); BufferedReader input = new BufferedReader(new InputStreamReader(url.openStream())); String line; for (; (line = input.readLine()) != null;) { System.out.println(line); URL } } catch (Exception e) { System.err.println(e); } Response Entity Body HTTP (4) : URL Class Request Message (automatically generated) GET http://www.miami.muohio.edu/ HTTP/1.1 Host: www.example.com crlf HTTP (4) : URL Class Response Message input.readLine() HTTP/1.1 200 OK Date: Wed, 22 Jul 2009 18:03:36 GMT Server: Apache/2.0.46 (Red Hat) Set-Cookie: SaneID=134.53.7.196-1248285816897469; path=/; expires=Sat, 26-Jul-14 18:03:36 GMT; domain=.muohio.edu Content-Language: en-US Transfer-Encoding: chunked Content-Type: text/html; charset=UTF-8 crlf <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta name="keywords" content="miami,university,ohio,college,public" /> <meta name="description" content="Located in Oxford, Ohio, Miami University is a public university of 16,000 students offering strong academics, a vibrant residence life program, and faculty who love to teach and mentor students." /> <title>Miami University</title> … </html> HTTP (1) : Client using GET Method import java.io.*; import java.net.*; try { Socket socket = new Socket("www.miami.muohio.edu", 80); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("GET http://www.miami.muohio.edu/ HTTP/1.1"); output.println("Host: www.example.com"); output.println(""); String response; for (; (response = input.readLine()) != null;) { Host Port System.out.println(response); } output.close(); input.close(); socket.close(); Request } catch (Exception e) { System.err.println(e); } Response HTTP (4) : URL Class Exercise Make a Java program such that: Send a request to the Web page of "http://www.muohio.edu:8080/", and print out the response from the Web server. Project name HTTP4X1Client HTTP (5) : HttpURLConnection Class HTTP5Client import java.io.*; import java.net.*; try { URL url = new URL("http://localhost:8080/HTTP2Server/login.jsp"); HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection(); // Request urlConnection.setRequestMethod("POST"); urlConnection.setFollowRedirects(true); URL urlConnection.setDoOutput(true); PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true); output.print("username=yusong&password=10000"); output.flush(); // Response String res = urlConnection.getResponseMessage(); System.out.println("Response message - " + res); // OK Request String enc = urlConnection.getContentType(); System.out.println("Content Type - " + enc); // text/html;charset=UTF-8 BufferedReader input = new BufferedReader(new InputStreamReader(urlConnection.getInputStream())); String line; for (; (line = input.readLine()) != null;) { System.out.println(line); } urlConnection.disconnect(); } catch (Exception e) { Response Entity Body System.err.println(e); } HTTP (5) : HttpURLConnection Class Request Message (automatically generated) POST /HTTP2Server/login.jsp HTTP/1.1 Host: www.example.com Content-Type: application/x-www-form-urlencoded Content-Length: 30 crlf username=yusong&password=10000 HTTP (5) : HttpURLConnection Class Response Message input.readLine() HTTP/1.1 200 OK X-Powered-By: JSP/2.1 Server: Sun GlassFish Enterprise Server v2.1 Set-Cookie: JSESSIONID=39d808dbc6c8600ab0b0786f6fbb; Path=/HTTP2Server Content-Type: text/html;charset=UTF-8 Content-Length: 282 Date: Wed, 22 Jul 2009 17:58:44 GMT crlf <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>Login</title> </head> <body> <h1>Hello "yusong"</h1> <h3>You typed the password as "10000".</h3> </body> </html> HTTP (2) : Client using POST Method import java.io.*; import java.net.*; try { Socket socket = new Socket("localhost", 8080); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("POST /HTTP2Server/login.jsp HTTP/1.1"); output.println("Host: www.example.com"); output.println("Content-Type: application/x-www-form-urlencoded"); output.println("Content-Length: 30"); output.println(""); output.println("username=yusong&password=10000"); Host Port String response; for (; (response = input.readLine()) != null;) { System.out.println(response); } output.close(); Request input.close(); socket.close(); } catch (Exception e) { System.err.println(e); } Response HTTP (5) : HttpURLConnection Class Exercise Make a Java program such that: Send a request to the Web page at "http://localhost:8080/HTTP2X1Server/register.jsp", with an entity body of "name=X&username=X&password=X&password2=X&address=X&zip =X". Print out the response from the Web server. Project names HTTP2X1Server HTTP2X1Server HTTP5X1Client Application 1: Search Engine Application 1 Exercise Make a search engine such that: Step 1) Set a seed URL as CurrentURL. Step 2) Get the document at CurrentURL by an HTTP request. Step 3) Find all hyperlinked documents' URLs from the document, and put them in URLPool. Step 4) Take out a URL from URLPool and set the URL as CurrentURL. Go to Step 2. Project name HTTPA1Client Application 1 : Amazon.com http://www.amazon.com/access http://www.amazon.com/ http://www.amazon.com/gp/yourstore?ie=UTF8&signIn=1 http://www.amazon.com/gp/flex/signout.html?ie=UTF8&path=%2Fgp%2Fyourstore&signIn=1&useRedirectOnSuccess=1&action=sign-out http://www.amazon.com/gp/product/B00154JDAI http://www.amazon.com/gp/site-directory http://www.amazon.com/gp/cart/view.html http://www.amazon.com/wishlist http://www.amazon.com/books-used-books-textbooks/b?ie=UTF8&node=283155 http://www.amazon.com/Kindle-Books/b?ie=UTF8&node=1286228011 http://www.amazon.com/New-Used-Textbooks-Books/b?ie=UTF8&node=465600 http://www.amazon.com/magazines/b?ie=UTF8&node=599858 http://www.amazon.com/dvds-used-hd-action-comedy-oscar/b?ie=UTF8&node=130 http://www.amazon.com/Blu-ray-movies-discs-store-deals/b?ie=UTF8&node=193640011 http://www.amazon.com/Video-On-Demand/b?ie=UTF8&node=16261631 http://www.amazon.com/music-rock-classical-pop-jazz/b?ie=UTF8&node=5174 http://www.amazon.com/MP3-Music-Download/b?ie=UTF8&node=163856011 http://www.amazon.com/musical-instruments-accessories-sound-recording/b?ie=UTF8&node=11091801 http://www.amazon.com/computer-video-games-hardware-accessories/b?ie=UTF8&node=468642 http://www.amazon.com/Game-Downloads/b?ie=UTF8&node=979455011 http://www.amazon.com/kindle-store-ebooks-newspapers-blogs/b?ie=UTF8&node=133141011 http://www.amazon.com/amazon-shorts-digital-shorts/b?ie=UTF8&node=13993911 http://www.amazon.com/Kindle-Amazons-Wireless-Reading-Generation/dp/B00154JDAI http://www.amazon.com/gp/product/B0015TCML0 http://www.amazon.com/Kindle-Accessories/b?ie=UTF8&node=1268192011 http://www.amazon.com/Kindle-Newspapers/b?ie=UTF8&node=1263068011 http://www.amazon.com/Kindle-Magazines/b?ie=UTF8&node=1263069011 … Application 2: Automatic Registration Application 2 Exercise Make client and server programs such that: The server is a JSP Web site for registration. The client is a registration software such that: It generates user names like a, b, …, z, aa, ab, …, zz. It sends registration requests for each of the generated user names. Project names Client - HTTPA2Client Server - HTTPA2Server Testing Scenarios Run both of client and server program on your own computer. Make a group of two students and assign a role of client and that of server to each student respectively. After testing, switch the roles and do the same things again. Something wrong? Try to use the debugger. Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : Web Server Setup Application 2 : New Project Application 2 : New Project Application 2 : New Project Application 2 : New Project Application 2 : Source Code Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : DSN Setup Application 2 : Run HTTPA2Server Application 2 : New Project Application 2 : New Project Application 2 : Source Code Application 2 : Run HTTPA2Client Application 2 : Run HTTPA2Client a -> Internal Server Error aa -> Internal Server Error ab -> OK ac -> OK ad -> OK ae -> OK af -> OK ag -> OK ah -> OK ai -> OK aj -> OK ak -> OK al -> OK am -> OK an -> OK ao -> OK ap -> OK aq -> OK ar -> OK as -> OK at -> OK au -> OK av -> OK aw -> OK ax -> OK ay -> OK az -> OK b -> OK ba -> OK bb -> OK … Application 2 : Simple Version import java.io.*; import java.net.*; public class Main { private static void register(String username) { try { URL url = new URL("http://localhost:8080/EF01/register.jsp"); HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection(); // Request urlConnection.setRequestMethod("POST"); urlConnection.setFollowRedirects(true); urlConnection.setDoOutput(true); PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true); output.print("username=" + username + "&password=1234&f_name=Song&l_name=Song&address=Wonju&zip=12345&gender=M&agree=N"); output.flush(); // Response String res = urlConnection.getResponseMessage(); System.out.println(username + " - " + res); urlConnection.disconnect(); } catch (Exception e) { System.err.println(e); } } public static void main(String[] args) { char c; for (c = 'a'; c <= 'z'; c++) { register("" + c); // "" + c : to convert char to String } } } Business model: Agents Agents A software agent is a piece of software that acts for a user or other program in a relationship of agency. "Action on behalf of" implies the authority to decide which (and if) action is appropriate. Related and derived concepts include Intelligent agents (in particular exhibiting some aspect of Artificial Intelligence, such as learning and reasoning), autonomous agents (capable of modifying the way in which they achieve their objectives), distributed agents (being executed on physically distinct computers), multi-agent systems (distributed agents that do not have the capabilities to achieve an objective alone and thus must communicate), and mobile agents (agents that can relocate their execution onto different processors). Agents in Electronic Commerce Search Engines Product Search Engines (Price Comparison Engines) … Search Engines Directory service Know-how → Know-where Yahoo directory service Early search engines To demonstrate its speed by DEC (Digital Corp.) Google To improve quality Send requests to several search engines, and merge the responses. Altavista See Architecture next page Automatic, but low speed, low quality Meta-search engines Good quality Labor-intensive Good quality, good speed Word of mouth by counting the number of hyperlinks Note: Search engines in Korea - www.naver.com, www.daum.net, … Provide human-made results for the most popular 100 keywords. (labor-intensive) Provide "Keyword Hot 100" (like Billboard Hot 100) Search Engines: Architecture Keyword Web Server Web Server … Web Server Search Agent Web Site Web Scrapping Web Browser DB Keyword Web Browser URL miami http://www.muohio.edu miami http://www.miami.edu/ song http://web.yonsei.ac.kr/yusong song http://en.wikipedia.org/wiki/Song … … … Web Browser AltaVista (Late 1990s) Yahoo (Late 1990s) Yahoo (07/21/2009) Google (07/21/2009) Product Search Engines BargainFinder Andersen Consulting (now Accenture) Compare CD album prices among album shops like CDnow, … Failed due to blocking by album shops Jango To overcome the blocking by album shops A client software to send and receive messages to and from album shops Inconvenient for users Product Search Engines (nowadays) Data gathering Web scraping DB sharing Danawa (http://www.danawa.com) Web-hosting service for shops in electronics mall in Seoul Use the data in Web-hosting DB for price comparison CDnow (Late 1990s) Product Search Engines : Danawa Building Blocks for HTTP Programming Building Block: URL Format resource_type://domain:port/filepathname?query_string#anc hor e.g. http://dragon.yonsei.ac.kr/~yusong/index.html http://yonsei.ac.kr:8000/~yusong/a.cgi?user=yusong&b=b ftp://dragon.yonsei.ac.kr/~yusong/sample.file telnet://dragon.yonsei.ac.kr rlogin://[email protected] HTTP (1) : Client using GET Method Request Message GET http://www.miami.muohio.edu/ HTTP/1.1 Host: www.example.com crlf HTTP (1) : Client using GET Method HTTP1Client import java.io.*; import java.net.*; try { Socket socket = new Socket("www.miami.muohio.edu", 80); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("GET http://www.miami.muohio.edu/ HTTP/1.1"); output.println("Host: www.example.com"); output.println(""); String response; for (; (response = input.readLine()) != null;) { Host Port System.out.println(response); } output.close(); input.close(); socket.close(); Request } catch (Exception e) { System.err.println(e); } Response HTTP (2) : Client using POST Method Request Message POST /HTTP2Server/login.jsp HTTP/1.1 Host: www.example.com Content-Type: application/x-www-form-urlencoded Content-Length: 30 crlf username=yusong&password=10000 HTTP (2) : Client using POST Method HTTP2Client import java.io.*; import java.net.*; try { Socket socket = new Socket("localhost", 8080); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("POST /HTTP2Server/login.jsp HTTP/1.1"); output.println("Host: www.example.com"); output.println("Content-Type: application/x-www-form-urlencoded"); output.println("Content-Length: 30"); output.println(""); output.println("username=yusong&password=10000"); Host Port String response; for (; (response = input.readLine()) != null;) { System.out.println(response); } output.close(); Request input.close(); socket.close(); } catch (Exception e) { System.err.println(e); } Response HTTP (3) : Web Server Response Message HTTP/1.1 200 OK crlf <HTML> <BODY><H1>This is a stupid Web server.</H1></BODY> </HTML> HTTP (3) : Web Server HTTP3Server import java.io.*; import java.net.*; try { ServerSocket serverSocket = new ServerSocket(8000); Socket clientConn = serverSocket.accept(); BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream())); PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true); String request; for (; (request = input.readLine()).isEmpty() == false;) { System.out.println(request); } output.println("HTTP/1.1 200 OK"); output.println(""); output.println("<HTML>"); output.println("<BODY><H1>This is a stupid Web server.</H1></BODY>"); output.println("</HTML>"); output.close(); input.close(); Response clientConn.close(); serverSocket.close(); } catch (Exception e) { System.err.println(e); } Request Port HTTP (3) : Web Server for Redirect Response Message HTTP/1.1 302 Found Location: http://www.miami.muohio.edu/ crlf We moved. HTTP (3) : Web Server for Redirect HTTP3X1Server import java.io.*; import java.net.*; try { ServerSocket serverSocket = new ServerSocket(8000); Socket clientConn = serverSocket.accept(); BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream())); PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true); String request; for (; (request = input.readLine()).isEmpty() == false;) { System.out.println(request); } output.println("HTTP/1.1 302 Found"); output.println("Location: http://www.miami.muohio.edu/"); output.println(""); output.println("We moved."); output.close(); input.close(); clientConn.close(); Request serverSocket.close(); } catch (Exception e) { System.err.println(e); } Response Port HTTP (4) : URL Class HTTP4Client import java.io.*; import java.net.*; try { URL url = new URL("http://www.miami.muohio.edu/"); BufferedReader input = new BufferedReader(new InputStreamReader(url.openStream())); String line; for (; (line = input.readLine()) != null;) { System.out.println(line); URL } } catch (Exception e) { System.err.println(e); } Response HTTP (5) : HttpURLConnection Class HTTP5Client import java.io.*; import java.net.*; try { URL url = new URL("http://localhost:8080/HTTP2Server/login.jsp"); HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection(); // Request urlConnection.setRequestMethod("POST"); urlConnection.setFollowRedirects(true); URL urlConnection.setDoOutput(true); PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true); output.print("username=yusong&password=10000"); output.flush(); // Response String res = urlConnection.getResponseMessage(); System.out.println("Response message - " + res); // OK Request String enc = urlConnection.getContentType(); System.out.println("Content Type - " + enc); // text/html;charset=UTF-8 BufferedReader input = new BufferedReader(new InputStreamReader(urlConnection.getInputStream())); String line; for (; (line = input.readLine()) != null;) { System.out.println(line); } urlConnection.disconnect(); } catch (Exception e) { Response System.err.println(e); } Exercises Exercise (1) Exercise Make a Java program such that: The client sends a GET request to and receives the response from the URL "http://www.muohio.edu". Use the Socket-related classes only. (Do not use URL or HttpURLConnection classes.) Project name HTTPX1Client Exercise (2) Exercise Make a Java program such that: The client sends a POST request to and receives the response from the URL "http://localhost:8080/HTTPX2Server/login.jsp". The "login.jsp" corresponds to the "login.html" on the next slide. Use "yusong" as the username and "12345" as the password. Use the Socket-related classes only. (Do not use URL or HttpURLConnection classes.) Project name HTTPX2Client Exercise (2) HTTPX2Server/login.html <html> <body> <h1>Login</h1> <form method="post" action="login.jsp"> <table> <tr> <td>User name:</td> <td><input type="text" name="UN"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="PW"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Login"></td> </tr> </table> </form> </body> </html> Exercise (3) Exercise Make a Java program such that: The client sends a POST request to and receives the response from the URL "http://localhost:8080/HTTPX3Server/login.jsp". The "login.jsp" corresponds to the "login.html" on the next slide. Use "Yong Uk" as the username and "1 2$3" as the password. Note that you have to encode the above values by URL encoding rule. ASCII code for '$' is 24 in hexadecimal. Use the Socket-related classes only. (Do not use URL or HttpURLConnection classes.) Project name HTTPX3Client Exercise (3) HTTPX3Server/login.html <html> <body> <h1>Login</h1> <form method="post" action="login.jsp"> <table> <tr> <td>User name:</td> <td><input type="text" name="UN"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="PW"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Login"></td> </tr> </table> </form> </body> </html> Exercise (4) Exercise Make a Web server which always responds with the HTML document on the next slide. Use a "for" statement for the Web server to run permanently. Project name HTTPX4Server Exercise (4) <HTML> <HEAD> <TITLE> Exercise</TITLE> </HEAD> <BODY> Welcome to the world of HTTP. </BODY> </HTML> Exercise (5) Exercise Make a Web server which always responds with a redirect message of which HTML document appears on the next slide. The new address is "http://www.muohio.edu". Use a "for" statement for the Web server to run permanently. Project name HTTPX5Server Exercise (5) <HTML> <HEAD> <TITLE> Exercise</TITLE> </HEAD> <BODY> We moved to <A HREF="http://www.muohio.edu"> a new address</A> </BODY> </HTML> Exercise (6) Exercise Make a Java program such that: The client sends a GET request to and receives the response from the URL "http://www.muohio.edu". Use the URL classes. Project name HTTPX6Client Exercise (7) Exercise Make a Java program such that: The client sends a POST request to and receives the response from the URL "http://localhost:8080/HTTPX7Server/login.jsp". The "login.jsp" corresponds to the "login.html" on the next slide. Use "yusong" as the username and "12345" as the password. Use the HttpURLConnection class. Project name HTTPX7Client Exercise (7) HTTPX7Server/login.html <html> <body> <h1>Login</h1> <form method="post" action="login.jsp"> <table> <tr> <td>User name:</td> <td><input type="text" name="UN"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="PW"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Login"></td> </tr> </table> </form> </body> </html> Exercise (8) Exercise Make a Java program such that: The client sends a POST request to and receives the response from the URL "http://localhost:8080/HTTPX3Server/login.jsp". The "login.jsp" corresponds to the "login.html" on the next slide. Use "Yong Uk" as the username and "1 2$3" as the password. Note that you have to encode the above values by URL encoding rule. ASCII code for $ - 24 (in hexadecimal) Use the HttpURLConnection class. Project name HTTPX8Client Exercise (8) HTTPX8Server/login.html <html> <body> <h1>Login</h1> <form method="post" action="login.jsp"> <table> <tr> <td>User name:</td> <td><input type="text" name="UN"></td> </tr> <tr> <td>Password:</td> <td><input type="password" name="PW"></td> </tr> <tr> <td colspan="2"><input type="submit" value="Login"></td> </tr> </table> </form> </body> </html> Examinations Examination (1) Examination Make a Java program such that: The client sends a GET request to and receives the response from the URL "http://web.yonsei.ac.kr/yusong/index.htm". Use the Socket-related classes only. (Do not use URL or HttpURLConnection classes.) Project name EM05 Examination (1) EM05.Main import java.io.*; import java.net.*; try { Socket socket = new Socket("web.yonsei.ac.kr", 80); BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream())); PrintWriter output = new PrintWriter(socket.getOutputStream(), true); output.println("GET /yusong/index.htm HTTP/1.1"); output.println("Host: www.example.com"); output.println(""); String response; for (; (response = input.readLine()) != null;) { System.out.println(response); } output.close(); input.close(); socket.close(); } catch (Exception e) { System.err.println(e); }