Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Automation and Customization of Rendered Web Pages Rob Miller User Interface Design Group MIT Computer Science & Artificial Intelligence Lab [email protected] http://uid.csail.mit.edu/ Joint work with Michael Bolin, Greg Little, Eric Lieberman, Marcos Ojeda, Matt Webber, Philip Rha, Tom Wilson, Darris Hupp, Ricarose Roque Web Applications • The Web is now an application platform – Many web sites are now rich applications – As we spend more time using them, we want to customize them for our own needs MIT Computer Science and Artificial Intelligence Laboratory 2 NPUC July 24, 2006 Web Automation & Customization • Automate repetitive operations – Bookmark my latest bank statement – Click on the nearest download mirror site – Print or save a list of web pages – Fill in defaults on a web form MIT Computer Science and Artificial Intelligence Laboratory 3 NPUC July 24, 2006 Web Automation & Customization • Automate repetitive operations • Transform appearance – Change color schemes for better contrast – Concatenate a sequence of pages into one page – Simplify site to remove unnecessary or distracting features MIT Computer Science and Artificial Intelligence Laboratory 4 NPUC July 24, 2006 Web Automation & Customization • Automate repetitive operations • Transform appearance • Integrate multiple web sites (“mashups”) – Bookstore has links for New Books, Used Books, Auctions… but it doesn’t know about books in my local library – Real estate site has lots of data about houses for sale… but it can’t compute the length of my trip to work MIT Computer Science and Artificial Intelligence Laboratory 5 NPUC July 24, 2006 Web Apps Are Inherently Open • Web applications have built-in hooks for scripting and modification can be automated HTTP requests HTML/XML replies can be parsed for data HTML/CSS display can be modified to change appearance • Desktop applications don’t have these hooks unless the developer intentionally provides an API • So it’s possible for an end user to change a web application – Greasemonkey for Firefox, User Javascript for Opera – But it’s not easy MIT Computer Science and Artificial Intelligence Laboratory 6 NPUC July 24, 2006 Problem: It’s Hard to Customize • Web applications are complex under the hood • The user has to know how to program var xpath = "//text()[(parent::" + allowedParents.join(" or parent::") + ") and " + "contains(translate(., 'HTTP', 'http'), 'http')]"; var candidates = document.evaluate(xpath, document, null, XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null); for (var cand = null, i = 0; (cand = candidates.snapshotItem(i)); i++) { if (urlRegex.test(cand.nodeValue)) { http://www.mit.edu/... var span = document.createElement("span"); var source = cand.nodeValue; MIT Computer Science and Artificial Intelligence Laboratory 7 NPUC July 24, 2006 Our Approach • Avoid complexity by looking at the user interface, not the HTML code • Avoid learning a programming language by using keywords rather than language syntax click feeling lucky button push “I’m Feeling Lucky” feeling lucky MIT Computer Science and Artificial Intelligence Laboratory 8 NPUC July 24, 2006 Outline Introduction • Web automation – Demo – How it works • Keyword commands – Motivation – Demo – How it works • Discussion MIT Computer Science and Artificial Intelligence Laboratory 9 NPUC July 24, 2006 Web Automation • Chickenfoot is our new web automation system – User shouldn’t have to look at HTML source to customize the Web – Runs in Mozilla Firefox web browser – Available for download from http://uid.csail.mit.edu/chickenfoot MIT Computer Science and Artificial Intelligence Laboratory 10 NPUC July 24, 2006 Web Automation Demo DEMO MIT Computer Science and Artificial Intelligence Laboratory 11 NPUC July 24, 2006 Web Automation Commands • Page navigation go(url) openTab(url) fetch(url) • Clicking and form manipulation click(button-or-link) enter([textbox], value) check(checkbox) pick([listbox], choice) • Pattern matching find(pattern) • Page modification insert(pattern, html) remove(pattern) replace(pattern, html) • Widgets & input handling Button(label, action) Link(label, action) onClick(pattern, action) MIT Computer Science and Artificial Intelligence Laboratory 12 NPUC July 24, 2006 Keyword Patterns • Keywords + component type feeling lucky button depart textbox search web form • Component type is optional if it can be inferred – For example, click() command assumes a button or a link MIT Computer Science and Artificial Intelligence Laboratory 13 NPUC July 24, 2006 Pattern Matching Algorithm Keyword pattern google search button Web page Weighted list of components 1.0 Matcher 0.5 MIT Computer Science and Artificial Intelligence Laboratory 14 NPUC July 24, 2006 Pattern Matching Algorithm Keyword pattern google search button Web page Weighted list of components 1.0 Matcher 0.5 Steps 1. Find labels matching the keywords – Text in the page, button labels, menu items, image ALT attributes – Matching ignores word ordering, punctuation, capitalization MIT Computer Science and Artificial Intelligence Laboratory 15 NPUC July 24, 2006 Pattern Matching Algorithm Keyword pattern google search button Web page Weighted list of components 1.0 Matcher 0.5 Steps 1. Find labels matching the keywords 2. Find components matching label MIT Computer Science and Artificial Intelligence Laboratory 16 NPUC July 24, 2006 Pattern Matching Algorithm Keyword pattern google search button Web page Weighted list of components 1.0 Matcher 0.5 Steps 1. Find labels matching the keywords 2. Find components matching label 3. Rank the components & pick the best match – How well do the keywords match the label? (# of matching keywords) – How well does the label match the component? (pixel distance) MIT Computer Science and Artificial Intelligence Laboratory 17 NPUC July 24, 2006 Outline Introduction Web automation • Keyword commands – Motivation – Demo – How it works • Discussion MIT Computer Science and Artificial Intelligence Laboratory 18 NPUC July 24, 2006 Problem: How Do I Learn the Language? tbox = find('textbox'); text = tbox.element.value; click('advanced image search'); enter('all of the words', text); pick('small'); pick('GIF'); click('search button'); button = new Button('Icon Search', doIconSearch); insert(after('search images button'), button); MIT Computer Science and Artificial Intelligence Laboratory 19 NPUC July 24, 2006 Keyword Commands • Save the user from learning details of syntax – Don’t worry about precise command names, punctuation, parameter ordering Chickenfoot go(“google.com”); enter(“npuc 2006”); click(“feeling lucky button”); • Divide command into keywords • Search for keywords in context Keyword Commands go to google.com type “npuc 2006” click feeling lucky button google.com npuc 2006 feeling lucky – Some keywords are found in user interface; others are command names • Generate a ranked list of possible interpretations – Punctuation and keyword ordering affect ranking, but are not required • Use the highest-ranked interpretation – If ambiguous, ask the user to choose an interpretation MIT Computer Science and Artificial Intelligence Laboratory 20 NPUC July 24, 2006 Keyword Commands Demo DEMO MIT Computer Science and Artificial Intelligence Laboratory 21 NPUC July 24, 2006 Advantages of Keyword Commands • Is this natural? – Yes: this is how my 2-year-old son talks, and I can understand him – Most users are at the same level when it comes to talking to a computer in a formal language – Our computers should become smart enough to understand us when we talk this way • Similar to keyword matching – e.g., web search engines • Benefits novice users and expert users differently – Novices may use verbose, almost natural language commands – But experts can be very concise • Keyword commands vs. point & click interfaces – More appropriate in some contexts (especially if combined with speech) – May even be faster (no hunting through menus and dialog boxes) • Keyword commands vs. macro recording – Macro recordings don’t capture the user’s intention, but keyword programs do – Keyword programs are text, which can be easily modified and shared with others MIT Computer Science and Artificial Intelligence Laboratory 22 NPUC July 24, 2006 How Keyword Commands Work Keyword command press feeling lucky button Function Database button click Program code Translator keywords textbox string enter click, press, push, … MIT Computer Science and Artificial Intelligence Laboratory url keywords findButton findTextbox go button textbox 23 NPUC July 24, 2006 How Keyword Commands Work Keyword command press feeling lucky button Translator keywords Function Database Program code click(“feeling lucky”) findButton button click MIT Computer Science and Artificial Intelligence Laboratory 24 NPUC July 24, 2006 Outline Introduction Web automation Keyword commands • Discussion – Robustness – Security – Resistance from web application developers MIT Computer Science and Artificial Intelligence Laboratory 25 NPUC July 24, 2006 Robustness • When web applications change, user customizations and mashups break – Customizations need to be robust to change • Open question: How does the abstraction level of a customization affect its robustness? Rendered web page DOM or raw HTML Web service <SOAP-ENV:Body> <ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <key xsi:type="xsd:string">00000000000000000000000000000000</key> <q xsi:type="xsd:string">shrdlu winograd maclisp teletype</q> MIT Computer Science and Artificial Intelligence Laboratory 26 NPUC July 24, 2006 Security • Can user customizations open security holes for malicious web sites to exploit? – Greasemonkey’s GM_xmlhttprequest was an unexpected hole – Chickenfoot has to open carefully-controlled holes in the browser’s security model to make certain things possible https:// page file:// images localUrl(“file://...”) “chicken-bypass:4hXFDh:file://…” • Can users be fooled into installing malicious customizations? – Yes: spyware is a case in point – Making customizations transparent (easy to read and understand) might help MIT Computer Science and Artificial Intelligence Laboratory 27 NPUC July 24, 2006 Resistance from Web App Developers • “Don’t muck around with my DOM!” – – – – Design: user customizations may make the site ugly or less usable Financial: customizations that hide or remove ads Server load Copyright • We’ve been down this road before – Ad blockers – Putting frames around other sites’ content – Deep linking • Web sites that fight their users will lose their users MIT Computer Science and Artificial Intelligence Laboratory 28 NPUC July 24, 2006 Conclusion • Two major barriers to automation and customization – Complexity of programming system – Difficulty of language syntax • We can tackle these barriers with two new techniques – Programming by looking at the user interface go(“google.com”); enter(“npuc 2006”); click(“feeling lucky button”); – Eliminating syntax from commands click feeling lucky button push “I’m Feeling Lucky” feeling lucky • Chickenfoot is available for download from http://uid.csail.mit.edu/chickenfoot MIT Computer Science and Artificial Intelligence Laboratory 29 NPUC July 24, 2006