Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Web Mining and Link Analysis Programming Collective Intelligence – Segaran Padhraic Smyth notes KDNuggets course notes Data Mining - Volinsky - Fal 2011 - Columbia University 1 Web Mining v. Data Mining Web Mining is: Discovering useful information from the World-Wide Web and its usage patterns • Structure (or lack of it) – Textual information and linkage structure – unstructured data • Scale – Data generated per day is comparable to largest conventional data warehouses • Speed – Often need to react to evolving usage patterns in real-time (e.g., merchandising, web security) Data Mining - Volinsky - Fal 2011 - Columbia University 2 What is “the Web”? • The WWW is huge, widely distributed, global information service center for – Information services: news, advertisements, consumer information, financial management, education, government, e-commerce, etc. • Hyper-link structure is what makes it so useful • provides rich sources for data mining • Essentially, infinite size (>20B pages) – With lots of duplication Data Mining - Volinsky - Fal 2011 - Columbia University 3 Why Web Mining? • Useful to study human digital behavior, e.g. search engine data can be used for – Exploration e.g. # of queries per session? – Modeling e.g. any time of day dependence? – Prediction e.g. which pages are relevant? • Applications – – – – Understand social implications of Web usage Design of better tools for information access E-commerce applications Advertising is a key driver of online business Data Mining - Volinsky - Fal 2011 - Columbia University 4 Advertising Applications • Revenue of many internet companies is driven by advertising • Key problem: – Given user data: • Pages browsed • Keywords used in search • Demographics – Determine the most relevant ads (in real-time) – Includes bidding/pricing of ads • Another major problem: “click fraud” – AdSense – place Google ads on your web site – AdWords – buy “keywords” to put on Google search – Determine fraudulent usage through data mining • Understanding the user is key to these types of applications Data Mining - Volinsky - Fal 2011 - Columbia University 5 Data Sources for Web Mining • Web content – Text and HTML content on Web pages – User generated content: Blogs, microblogs (Twitter), social networks • Web connectivity – Hyperlink/directed-graph structure of the Web • Web user data – Data on how users interact with the Web • • • • Navigation data, aka “clickstream” data Search query data (keywords for users) Online transaction data Who has this data? Data Mining - Volinsky - Fal 2011 - Columbia University 6 Accessing data for web mining • Scripting languages like Perl or Python make web scraping access easy. • “user-generated” content is meant to be consumed! • Many websites have APIs for access to data – If there is an API, please follow it! – Can be open: wikipedia, imdb – Can be restricted: facebook, ebay, amazon • If you are interested, a good book is Data Mining - Volinsky - Fal 2011 - Columbia University 7 Examples of Web Mining Viz • Volume of data along with useful APIs makes a lot of data available for visualization and analysis. • Twitter happiness metric • Blogpulse.com Data Mining - Volinsky - Fal 2011 - Columbia University 8 Analyzing User navigation • Web logs – Record activity between client browser and a specific Web server – Easily available – Collected by server, ISP • Search engine records – Text in queries, which pages were viewed, which snippets were clicked on, etc • Client-side browsing records – Automatically recorded by client-side software – Harder to obtain, but much more accurate than server-side logs • Other sources – Cookies: collected on client/browser, readable by server – Web site registration, purchases, email, etc Data Mining - Volinsky - Fal 2011 - Columbia University – ISP recording of Web browsing 9 Example of Web Log entries Apache web log: 207.237.112.68 - - [25/Oct/2009:06:13:30 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" 207.237.112.68 - - [25/Oct/2009:06:13:35 -0400] "GET /~volinsky/DataMining/HW/HW4.html HTTP/1.1" 304 "http://www.research.att.com/~volinsky/DataMining/Columbia.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" 69.86.67.231 - - [25/Oct/2009:10:10:36 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042700 SUSE/3.0.10-1.1.1 Firefox/3.0.10" 66.234.60.140 - - [25/Oct/2009:10:21:29 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 200 11527 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729)" 66.234.60.140 - - [25/Oct/2009:10:21:33 -0400] "GET /~volinsky/DataMining/HW/HW4.html HTTP/1.1" 200 4584 "http://www.research.att.com/~volinsky/DataMining/Columbia.html" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729)" 68.239.18.39 - - [25/Oct/2009:11:03:11 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9" 68.239.18.39 - - [25/Oct/2009:12:09:47 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9" 66.65.114.97 - - [25/Oct/2009:12:17:15 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 200 11527 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3" 128.59.154.126 - - [25/Oct/2009:13:14:18 -0400] "GET /~volinsky/DataMining/Columbia.html HTTP/1.1" 200 11527 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)" 128.59.154.126 - - [25/Oct/2009:13:14:21 -0400] "GET /~volinsky/DataMining/HW/HW4.html HTTP/1.1" 200 4584 "http://www.research.att.com/~volinsky/DataMining/Columbia.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729 66.249.65.210 - - [01/Oct/2009:05:52:03 -0400] "GET /~volinsky/myrefs.html HTTP/1.1" 200 21698 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Data Mining - Volinsky - Fal 2011 - Columbia University 10 Routine Server Log Analysis • Typical statistics/histograms that are computed – – – – – – Most and least visited web pages Entry and exit pages Referrals from other sites or search engines What are the searched keywords How many clicks/page views a page received Error reports, like broken links • Many software products that produce standard reports of this type of data – e.g., are there clusters/groups of users that use the site in different ways? Data Mining - Volinsky - Fal 2011 - Columbia University 11 Descriptive Summary Statistics Data Mining - Volinsky - Fal 2011 - Columbia University 12 Web data measurement issues • Important to understand how data is collected • Web data is collected automatically via software logging tools – Advantage: • No manual supervision required – Disadvantage: • Data can be skewed (e.g. due to the presence of robot traffic) • Important to identify robots (also known as crawlers, spiders) Data Mining - Volinsky - Fal 2011 - Columbia University 13 Robot / human identification • • Removal of robot data is important preprocessing step before any clickstream analysis Robots come in all shapes and sizes – Good: Google wants to map the net to provide good search – Bad: Competitor is scraping your web site to see what you are up to (or steal data) • Robot page-requests often identified using a variety of heuristics – e.g. some robots self-identify themselves in the server logs • Robots.txt • Also, robots should identify themselves via the User Agent field in page requests – Patterns of access • How would you detect robots? How would you escape detection? • Tan and Kumar (Journal of Data Mining and Knowledge Discovery, 2002) provide a detailed description of using classification techniques to learn how to detect robots Data Mining - Volinsky - Fal 2011 - Columbia University 14 A time-series plot of UCI Website data Number of page requests per hour as a function of time from page requests in the www.ics.uci.edu Web server logs during the first week of April 2002. Data Mining - Volinsky - Fal 2011 - Columbia University 15 From Tan and Kumar, 2002 Overall accuracies of around 90% were obtained using decision tree classifiers, Like spam, identifying bots (like spam) is a constant arms race Data Mining - Volinsky - Fal 2011 - Columbia University 16 Sessionizing • Aggregating clicks into sessions can be useful – e.g., what did you do when you sat at the computer? • how might you determine this? Data Mining - Volinsky - Fal 2011 - Columbia University 17 Client-side data • Advantages of collecting data at the client side: – Direct recording of page requests (eliminates ‘masking’ due to caching) – Recording of all browser-related actions by a user (including visits to multiple websites) – More-reliable identification of individual users (e.g. by login ID for multiple users on a single computer) • Preferred mode of data collection for studies of navigation behavior on the Web • Companies like ComScore and Nielsen use client-side software to track home computer users – but with what biases? Data Mining - Volinsky - Fal 2011 - Columbia University 18 comScore Report 2008 • 185 million U.S. people age 2+ online in a month, spending an average of 29 hours online per person* • 80% of 824 million global Internet users now outside of U.S. • 99% of online population search in a month, conducting 22 searches per searcher** • 75% of online population stream a video, viewing an average of 70 videos per viewer per month*** – Up 36% vs YA • 66% of online population visit a social networking site, spending 4 hours per month per visitor* • 40% of online population visit a blog site in a month* February 2008, U.S., comScore Media Metrix ** February 2008, U.S., comScore qSearch 2.0 *** January 2008, U.S., comScore Video Metrix Data Mining - Volinsky - Fal 2011 - Columbia University 19 Modeling Clickrate Data • Data – goal is to build a time-series model that characterizes user click rates – Usually: cluster data into user types Data Mining - Volinsky - Fal 2011 - Columbia University 20 Markov models for page prediction • Why would we want to predict where a user is surfing? – pre-cached web pages save time • General approach is to use a finite-state Markov chain – Each state can be a specific Web page or a category of Web pages – If only interested in the order of visits (and not in time), each new request can be modeled as a transition of states • For simplicity, consider order-dependent, time-independent finite-state Markov chain with M states Data Mining - Volinsky - Fal 2011 - Columbia University 21 Markov models for page prediction • Let s be a sequence of observed states of length L. e.g. s = ABBCAABBCCBBAA with three states A, B and C. st is state at position t (1<=t<=L). In general, L P( s ) P( s1 ) P( st | st 1 ,..., s1 ) t 2 • first-order Markov assumption P(st | st 1 ,..., s1 ) P(st | st 1 ) • This provides a simple generative model: L P ( s ) P ( s1 ) P ( st | st 1 ) t 2 Data Mining - Volinsky - Fal 2011 - Columbia University 22 Markov models for page prediction • If we denote Tij = P(st = j|st-1 = i), we can define a P x P transition matrix • Each page is a “state”: P can be of the order 105 to 106 • If P is large, we might cluster P pages into M clusters, which now become the states in the Markov model Data Mining - Volinsky - Fal 2011 - Columbia University 23 Markov models for page prediction • Tij = P(st = j|st-1 = i) represents the probability that an individual user’s next request will be from state j, given they were in state i • We can add E, an end-state to the model • E.g. for three categories with end state: P(1 | 1) P(2 | 1) P(3 | 1) P( E | 1) P ( 1 | 2 ) P ( 2 | 2 ) P ( 3 | 2 ) P ( E | 2 ) T P(1 | 3) P(2 | 3) P(3 | 3) P( E | 3) 0 0 0 1 • Rows sum to 1 • E denotes the end of a sequence, and start of a new sequence Data Mining - Volinsky - Fal 2011 - Columbia University 24 Markov models for page prediction • First-order Markov model assumes that the next state is based only on the current state • This is a strong assumption! – Doesn’t consider ‘long-term memory’ • We can try to capture more memory with kth-order Markov chain (increased complexity) P(st | st 1 ,.., s1 ) P(st | st 1 ,.., st k ) Data Mining - Volinsky - Fal 2011 - Columbia University 25 Transition probability estimates for Markov model T ij nij ni • Where nij is the number of cases that go from state i to state j. ni is the number of cases starting in state i. • Smoothed parameter estimates: T ij nij aqij ni a • qij is a prior transition matrix • If nij = 0 for some transition (i, j), smoothed version allows prior knowledge to be incorporated, instead of having a parameter estimate of0. • If nij > 0, we get a smooth combination of the data-driven information (nij) and the prior. a determines how much the prior (qij) matters Data Mining - Volinsky - Fal 2011 - Columbia University 26 Ranking Web Pages Data Mining - Volinsky - Fal 2011 - Columbia University 27 Ranking web pages • Web pages are not equally “important” • How do you determine the “importance” of a web page? • Big Idea: Inlinks are a measure of importance. – Virtualstapler.com = 178 – Nytimes.com = 13,000 • Are all inlinks equal? – They are important if linked to by many important sites – Recursive question! Data Mining - Volinsky - Fal 2011 - Columbia University 28 Simple recursive formulation • Each link’s vote is proportional to the importance of its source page – if pages link to me, my links count more • If page P with importance x has n outlinks, each link gets x/n votes Data Mining - Volinsky - Fal 2011 - Columbia University 29 Simple “flow” model courtesy Rajaraman, Ullman y a/2 Yahoo y/2 y = y /2 + a /2 a = y /2 + m m = a /2 y/2 m M’soft Amazon a a/2 m Data Mining - Volinsky - Fal 2011 - Columbia University 30 Solving the flow equations • 3 equations, 3 unknowns, – y+a+m = 1 – Solution: y = 2/5, a = 2/5, m = 1/5 • Nice for a small example, but need something more general Data Mining - Volinsky - Fal 2011 - Columbia University 31 Matrix formulation • • Matrix M has one row and one column for each web page Suppose page j has n outlinks – – – • If j links to i, then Mij=1/n Else Mij=0 Columns sum to 1 Suppose r is a vector with one entry per web page – – ri is the importance score of page i Call it the rank vector Then, The flow equations can be written r = Mr • So the rank vector is an eigenvector of the web matrix Data Mining - Volinsky - Fal 2011 - Columbia University 32 Example y a y 1/2 1/2 a 1/2 0 m 0 1/2 Yahoo m 0 1 0 r = Mr Amazon M’soft y = y /2 + a /2 a = y /2 + m m = a /2 Data Mining - Volinsky - Fal 2011 - Columbia University y 1/2 1/2 0 a = 1/2 0 1 m 0 1/2 0 y a m 33 Power Iteration method • • • • • Simple iterative scheme Suppose there are N web pages Initialize: r0 = [1/N,….,1/N] Iterate: rk+1 = Mrk Stop when |rk+1 - rk|1 < Data Mining - Volinsky - Fal 2011 - Columbia University 34 Power Iteration Example y a y 1/2 1/2 a 1/2 0 m 0 1/2 Yahoo Amazon y a = m m 0 1 0 M’soft 1/3 1/3 1/3 1/3 1/2 1/6 5/12 1/3 1/4 3/8 11/24 . . . 1/6 Data Mining - Volinsky - Fal 2011 - Columbia University 2/5 2/5 1/5 35 Random Walk Interpretation • Imagine a random web surfer – At any time t, surfer is on some page P – At time t+1, the surfer follows an outlink from P uniformly at random – Ends up on some page Q linked from P – Process repeats indefinitely • Let p(t) be a vector whose ith component is the probability that the surfer is at page i at time t – p(t) is a probability distribution on pages Data Mining - Volinsky - Fal 2011 - Columbia University 36 The stationary distribution • Where is the surfer at time t+1? – Follows a link uniformly at random – p(t+1) = Mp(t) • Suppose the random walk reaches a state such that p(t+1) = Mp(t) = p(t) – Then p(t) is called a stationary distribution for the random walk • Our rank vector r satisfies r = Mr – So it is a stationary distribution for the random surfer – (also, r is an eigenvector of M) Data Mining - Volinsky - Fal 2011 - Columbia University 37 Existence and Uniqueness A central result from the theory of random walks (aka Markov processes): For graphs that satisfy certain conditions, the stationary distribution is unique and eventually will be reached no matter what the initial probability distribution at time t = 0. Data Mining - Volinsky - Fal 2011 - Columbia University 38 Spider traps • A group of pages is a spider trap if there are no links from within the group to outside the group – Random surfer gets trapped • Spider traps violate the conditions needed for the random walk theorem • Solution for traps: – At every step, with probability , follow a link at random – With probability 1-, jump to some page uniformly at random (teleport) – Common values for are in the range 0.8 to 0.9 • This is the essence of Google’s PageRank algorithm Data Mining - Volinsky - Fal 2011 - Columbia University 39 Matrix formulation • Suppose there are N pages – Consider a page j, with set of outlinks O(j) – We have Mij = 1/|O(j)| when j links to i and Mij = 0 otherwise – The random teleport is equivalent to • adding a teleport link from j to every other page with probability (1-)/N • reducing the probability of following each outlink from 1/|O(j)| to /|O(j)| Data Mining - Volinsky - Fal 2011 - Columbia University 40 Previous example with traps Yahoo y a y 1/2 1/2 a 1/2 0 m 0 1/2 m 0 0 1 M’soft Amazon y a = m 1/3 1/3 1/3 1/3 1/6 1/2 1/4 1/6 7/12 5/24 1/8 2/3 Data Mining - Volinsky - Fal 2011 - Columbia University ... 0 0 1 41 Previous example with =0.8 1/2 1/2 0 0.8 1/2 0 0 0 1/2 1 Yahoo Amazon y a = m M’soft 1/3 1/3 1/3 0.33 0.28 0.20 0.2 0.46 0.52 1/3 1/3 1/3 + 0.2 1/3 1/3 1/3 1/3 1/3 1/3 y 7/15 7/15 1/15 a 7/15 1/15 1/15 m 1/15 7/15 13/15 0.24 0.17 0.58 Data Mining - Volinsky - Fal 2011 - Columbia University ... 0.212 0.152 0.636 42 • • • The Google Model Google uses a combination of tools: – – – TFIDF from query to page retrieval PageRank to upweight important pages Link text info Problems: – – Biased against topic-specific authorities Ambiguous queries e.g., jaguar, spears Susceptible to Link spam – – Artificial links created in order to boost page rank called Google Bombing “miserable failure” Data Mining - Volinsky - Fal 2011 - Columbia University 43 Other measures of importance • Hubs and Authorities (Klienberg) – PageRank model works on the assumption that important pages link to imporant pages – Kleinberg notes that important sites might not link to each other – authorities • pages which are prominent for a given topic – hubs • assemble high-quality guides and direct users to authorities – A good hub page is one that points to many good authority pages, A good authority page is one that is pointed to by many good hub pages – each page gets a hub score and an authority score…this helps also in defining web communities Data Mining - Volinsky - Fal 2011 - Columbia University 44 HITS: Hubs and Authorities • The HITS algorithm has two basic steps: – Authority Update: Update each node's Authority score to be equal to the sum of the Hub Scores of each node that points to it. – Hub Update: Update each node's Hub Score to be equal to the sum of the Authority Scores of each node that it points to. • • • • • • • Let a be the vector of authority scores and h be the vector of hub scores a=[1,1,....1], h = [1,1,.....1] ; do a=MTh; h=Ma; Normalize a and h; Repeat until a and h converge The vectors a* and h*represent the authority and hub weights Data Mining - Volinsky - Fal 2011 - Columbia University 45 Web Advertising and Auction Models Data Mining - Volinsky - Fal 2011 - Columbia University 46 History of web advertising • Banner ads (1995-2001) – Initial form of web advertising – Popular websites charged X$ for every 1000 “impressions” of ad • Called “CPM” rate • Modeled similar to TV, magazine ads – Untargeted to demographically targeted – Low clickthrough rates • low ROI for advertisers Data Mining - Volinsky - Fal 2011 - Columbia University 47 Performance-based advertising • Introduced by Overture around 2000 – Advertisers “bid” on search keywords – “second price” auction (why?) – When someone searches for that keyword, the highest bidder’s ad is shown – Advertiser is charged only if the ad is clicked on • Google’s version came out in 2000 – Called “Adwords” Data Mining - Volinsky - Fal 2011 - Columbia University 48 Ads vs. search results Data Mining - Volinsky - Fal 2011 - Columbia University 49 Web 2.0 • Performance-based advertising works! – Multi-billion-dollar industry – Ad server is incented to provide best ads for a given search - they only get paid if successful! – auction model is sensible…what are you willing to pay? – Top words : • http://www.cwire.org/highest-paying-search-terms/ • Interesting problems – What ads to show for a search? – If I’m an advertiser, which search terms should I bid on and how much to bid? Data Mining - Volinsky - Fal 2011 - Columbia University 50 Adwords problem • A stream of queries arrives at the search engine – q1, q2,… • Several advertisers bid on each query • When query qi arrives, search engine must pick a subset of advertisers whose ads are shown • Goal: maximize search engine’s revenues Data Mining - Volinsky - Fal 2011 - Columbia University 51 Greedy algorithm • Simplest algorithm is greedy - highest bidder wins! • Complications: – Each ad has a different likelihood of being clicked (quality) • • • • • advertiser 1 bids $2, click probability = 0.1 Advertiser 2 bids $1, click probability = 0.5 Click probability based on historic data and statistical model Maximize both ad revenue and user relevance Google solution: use the “expected revenue per click” as a ranking Data Mining - Volinsky - Fal 2011 - Columbia University 52 Adwords model • • • • • • Max Cost per Click (CPC) determined by advertiser Quality Score (of ad) is determined by Google Minimim bid is determined by popularity of search term and by Quality Score of ad Rank is calculated, and determines position Actual CPC is the answer to this question: “ what is the lowest amount I can pay, and still retain my rank? Actual cost is one cent more than needed to retain rank – • e.g. if A had bid 0.36, she would have been tied with B for Rank, so, charge A $0.37 (as long as this is above the Min Bid Google also takes into account advertiser budget, and diversity of rankings Data Mining - Volinsky - Fal 2011 - Columbia University 53 References • A nice overview of web mining topics, models and issues: – http://users.atw.hu/ignatius/mining.pdf • “Discovery of Web Robot Sessions…” – Tan and Kumar (2002) • Book : Programming Collective Intelligence • Original PageRank paper, published by Brin and Page when they were in graduate school • Markov Chains for Link Prediction: paper • Kleinberg’s HITS Algorithm: paper Data Mining - Volinsky - Fal 2011 - Columbia University 54