Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Scenario-Based Requirements for Web Macro Tools Christopher Scaffidi, Allen Cypher, Sebastian Elbaum, Andhy Koesnandar, Brad Myers PBE and browser automation • Programming-by-example (PBE) is a success story – Research systems: Eager, Marquise – Systems in widespread use: Microsoft Office macros • Researchers have applied PBE to browser automation – e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise – These tools… 1. Watch the user perform a task 2. Infer the user’s intent 3. Record a web macro representing the intent 4. Allow the user to replay the macro on new data 2 introduction scenarios requirements Robofox conclusion Step 1: Open http://www.oanda.com/converter/classic Step 2: Submit the amount, date, foreign currency type, and target currency. Step 3: Retrieve the result in the target currency. PBE web macro tools are not widely adopted • The web context introduces new challenges to PBE: – Web site changes can break macros without warning Need features for detecting and responding to site change – Office tasks involve not just browsers but also spreadsheets and other applications Need integration with other applications • What other unmet requirements are there? 4 introduction scenarios requirements Robofox conclusion Missing: a methodical characterization of requirements for web macro tools • Many papers use scenarios to explain and motivate a PBE tool’s features. • Such scenarios… – Are retrospective: The scenarios show the features that actually work, rather than opportunities for future work. – Lack pedigree: Are the scenarios hypothetical or based on a user study? Will the tool features be useful in practice? 5 introduction scenarios requirements Robofox conclusion Outline • We have… – Analyzed 10 scenarios where web macros would be desirable – Characterized 25 requirements that tools must support to be useful for many real-world tasks – Used these requirements as a benchmark to identify opportunities for enhancing a particular tool (Robofox) 6 introduction scenarios requirements Robofox conclusion Catalog of scenarios • Using a web site to look up information – 6 scenarios – Currency converter, Package tracker, Per diem lookup, Staff lookup, Stock analysis, Watcher for eBay – Typical users: office workers • Using a web form to order products – 1 scenario – Path to procurement – Typical users: online shoppers • “Scraping” a large part of a site – 3 scenarios – Peoplesoft scraper, Person locator scraper, Scraper for CMS – Typical users: System administrators 7 introduction scenarios requirements Robofox conclusion Scenarios came from a variety of sources • Contextual inquiry – 3 scenarios – Currency converter, Per diem lookup, Staff lookup • Co-workers – 2 scenarios – Path to procurement, Peoplesoft scraper • Our own experiences – 2 scenarios – Package tracker, Scraper for CMS • Online sources – 3 scenarios – Person locator scraper, Stock analysis, Watcher for eBay 8 introduction scenarios requirements Robofox conclusion What is in a scenario? * Title Pithy name to make it easy to refer to the scenario * Typical User Tells what type of person is likely to perform the task * Overview Context and forces that would prompt a user to do the task * Starting Conditions Pre-conditions that hold true before the scenario * Result Post-conditions that should hold true after the scenario * Actions Steps that the user performs (maybe with a tool) Action Details Screenshots or other clarifying information Variations Discusses likely tweaks of the main scenario Macro Maintenance Examines how scenario evolution prompting macro changes * Scenario Source 9 Summarizes empirical data that generated this scenario * = required section introduction scenarios requirements Robofox conclusion Requirements: Triggering macros • On-demand execution – Required to support users’ work tasks • Scheduled execution – Especially for scrapers • Event-based triggers – Watcher for eBay executes when user visits Google homepage • Subroutines – Especially for lookup macros 10 introduction scenarios requirements Robofox conclusion Requirements: Using objects on pages • Text snippets – Almost all scenarios require reading text • Tabular information - – Almost all scenarios read tabular data • Web form widgets – Read/write textbox, dropdown, checkbox widgets • Other HTML structures – Watcher for eBay 11 introduction scenarios requirements Robofox conclusion Requirements: Adapting to site changes • Adaptation to changing page layout – Changes to text location, text color, page structure • Adaptation to changing form fields - – Changes to field names and field values • Adaptation to changing URLs – “This page has moved to X. Please update your links.” 12 introduction scenarios requirements Robofox conclusion Requirements: Reading and writing data outside pages • Browser APIs – Read/write highlighted text, show popup window • Spreadsheets and other files - – Often one record per spreadsheet row or XML node • Parameters containing user input – Prompting for an input (e.g. password) during playback 13 introduction scenarios requirements Robofox conclusion Requirements: Transforming data • Reformat to equivalent value - – For comparing values and for making outputs consistent • Extracting values’ parts – For comparing values • Combining values – Such as arithmetic and string concatenation 14 introduction scenarios requirements Robofox conclusion Requirements: Executing control structures • Looping operations - – Especially in scrapers • Conditional operations – Macro behavior depends on data provided by user as well as data returned by web sites 15 introduction scenarios requirements Robofox conclusion Requirements: Recovering from failure • Partial restarts – Especially for scrapers • Exception handlers – Integration with assertions 16 introduction scenarios requirements Robofox conclusion Requirements: Supporting macro maintenance • User-understandable representation – For maintenance and for sharing macros with other users • Editable macros – Analogous to features supported by textual editors • Features for debugging – Traces, breakpoints, step-by-step execution, etc • Maintenance at runtime – On error, enter maintenance mode 17 introduction scenarios requirements Robofox conclusion Example benchmark use: Robofox Triggering macros On-demand execution Yes Scheduled execution Yes Event-based triggers Yes Subroutines No Using objects on web pages Text snippets Yes Tabular information No Web form widgets Yes Other HTML structures No • Inserts “sanity checks” to detect changing page structure. • Includes heuristics for “refinding” page elements based on color, label, DOM location. • Some changes (e.g.: form field renaming) are not handled automatically. Adapting to site changes Adaptation to changing page layout Limited Adaptation to changing form fields Yes Adaptation to changing URLs No 18 introduction scenarios requirements Robofox conclusion Example benchmark use: Robofox Reading and writing data outside pages Browser APIs Limited Spreadsheets and other files Limited Parameters containing user input Limited Transforming data Reformat to equivalent value No Extracting values’ parts Yes Combining values No • Can read/write spreadsheets, but only if they have a specific layout • Can do a “foreach” loop but not a “while(condition)” loop Executing control structures Looping operations Conditional operations 19 Limited Yes introduction scenarios requirements Robofox conclusion Example benchmark use: Robofox Recovering from failure Partial restarts No Exception handlers Yes Supporting macro maintenance User-understandable representation Yes Editable macros Yes Features for debugging Yes Maintenance at runtime Yes 20 • User-defined exception handlers can fire when assertions fail introduction scenarios requirements Robofox conclusion Summary of Robofox limitations • Of 25 requirements… – Supported: 13 – Limited support: 5 – Unsupported: 7 • Of 10 scenarios… – Supported: 3 – Limited support: 2 (unsupported scenario variations) – Unsupported: 5 21 introduction scenarios requirements Robofox conclusion Opportunities to enhance Robofox • Adding support for 2 requirements… – Tabular information – Reformat to equivalent value • … would leave only… – 2 unsupported scenarios – 2 unsupported scenario variations 22 introduction scenarios requirements Robofox conclusion Extending the benchmark • These 25 requirements are “Version 1” of a benchmark for evaluating web macro tools. • As researchers meet existing requirements, new scenarios and new requirements will appear. • We created a wiki where YOU can upload scenarios. – http://softwaresurvey.cs.cmu.edu/wmcorpus.html 23 introduction scenarios requirements Robofox conclusion Thank You… • To VL/HCC for the opportunity to present. • To EUSES Consortium members for suggestions. • To NSF and EUSES for funding (CCF-0325273, CCF-0438929, CCF-0324861). 24 introduction scenarios requirements Robofox conclusion