Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
What’s in your customer’s wallet? C. Perlich, R. Lawrence, S. Rosset, I. Khabibrakhmanov, S. Mahatma, S. Weiss Predictive Modeling Group Mathematical Sciences Department IBM T.J. Watson Research Center Media 6 Degrees © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Sales Alignment Task Slide 2 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Market Alignment Project (MAP) Objective: Allocate resources so as to Focus the marketing and sales efforts on customers with high growth potential Reduce risk of exposure to limited set of customers by identifying new valuable customers Wallet/Opportunity Definitions 1. 2. 3. The total spending by a customer in a particular set of products in a given time Total IBM attainable spending of a customer Realistic IBM-attainable spending Slide 3 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research We cannot directly observe the Opportunity Company Revenue (D&B) We observe this in the data Wallet/Opportunity IBM Sales to this Company But we do not observe this How can we make this a data mining problem? Slide 4 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research We formulate the problem as Quantile Estimation Imagine 1,000 customers with identical customer features Consider the distribution of the IBM Sales to these customers: Best Customers IBM Sales Opportunity is High Quantile Slide 5 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research How do we estimate the Quantile? Linear regression estimates the conditional mean by minimizing sum of squared error Quantile regression estimates the conditional quantile p by minimizing asymmetric loss function p ( y yˆ ) L p ( y, yˆ ) (1 p) ( yˆ y ) if y yˆ if yˆ y y observed IBM Sales yˆ predicted Revenue Opportunit y Lp is optimized in expectation for Quantile p Quantile loss is used for estimation and evaluation Implemented several variants (Linear, Decision trees, …) KDD 2007 Paper Slide 6 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Linear Quantile Regression (Koenker) 9 8 Opportunity for C 2 IBMRevenue Revenue IBM 7 6 Opportunity for C 1 Opportunity for C 1 5 4 C2 3 C C1 2 1 10 Slide 7 20 30 40 50 60 Company Firm Sales Sales 70 80 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Explanatory features are extracted from multiple sources Dun & Bradstreet (D&B) Data IBM Client Transactions Entity Matching Feature Extraction D&B Features Industry Revenue (Rank) Employees State D&B Structure Code … IBM Transactional Features Prior-year revenue in other product brands Long-term revenue in other product brands … Train model against current year revenue based on previous year Apply model by rolling forward to current year and predicting future opportunity Slide 8 © Copyright IBM Corporation 2010 Predictive Modeling Group –aMathematical – IBM Research The MAP process provides unique Sciences integration of OR and expert insight MAP Workshops IBM Sales Team Interviews MAP Web Interface Model Estimates Expert Feedback Modeled Opportunity MAP Models Integrated Data Data Model Validated Opportunity Realign Sales Resources Slide 9 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research MAP Validation and Expert Feedback Expert-validated Opportunity (log) Validates Opportunity Expert 20 Experts accept opportunity (45%) 18 16 Increase (17%) 14 12 Experts change opportunity (40%) 10 Decrease (23%) 8 6 4 2 0 0 2 4 6 8 10 12 14 16 18 20 Experts reduced opportunity to 0 (15%) MODEL_OPPTY ModelkNN Opportunity (log) Opportunity Slide 10 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematicaland Sciences – IBM Research In 2008 MAP covered 50+ countries ~100% of IBM revenue and opportunity g 2005 2006 2007 , 2008 Resources shifted to high growth Markets and Accounts Shifted resources performed >10 pts better Slide 11 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Scope and some of the tedious details • 3 Million customers • 20 Brands (Product categories) • 4 Markets • Annual model refresh • The Quantile is chosen for each brand and market separately based on market insights on IBM market share • Whitespace model for customers with no prior IBM revenue are build using the same methodology but only D&B features • Entity matching between IBM customer records and D&B hierarchy is HARD • Evaluation remains somewhat subjective and we collect feedback Slide 12 © Copyright IBM Corporation 2010 Validated Revenue Opportunity Modeling Group –segmentation Mathematical Sciencesand – IBM Research MAP outputPredictive drives account resource allocation decisions Invest High growth potential Opportunistic Small Accounts Core Growth Modest growth potential Sellers shifted Resource implications Shift resources to Core Growth and Invest Accounts Reduce resource overlap 8,000 sellers shifted (2006 – 2009 ) Core Optimize Flat or declining Prior Year Actual Revenue Slide 13 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research Validated Revenue Opportunity MAP drove significant revenue impact in 2008 Invest Core Growth $53B of Revenue 3,000 sellers shifted (2008) 30,000 sellers Opportunistic Core Optimize $9B of Revenue Prior Year Actual Revenue [3,000 Sellers] x [$2M Revenue / Seller] x [10% Performance Improvement] = $600M (2008 Revenue Impact) Slide 14 © Copyright IBM Corporation 2010 Predictive Modeling Group – Mathematical Sciences – IBM Research MAP Take away Interesting predictive modeling task that calls for an unorthodox loss function Combination of data mining AND expert feedback Integration into the annual sales management cycle Significant effort on data collection and preparation Many additional analytical tools were build on top of MAP Territory definition and assignment Quota assignment Substantial impact on the bottom line © Copyright IBM Corporation 2010