Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Questionnaire Design, Survey Methods, Sampling and Causal Research Market Intelligence Julie Edell Britton Session 5 September 4, 2009 1 Today’s Agenda Announcements Comparative Advertising, Measurement Scales & Data Analysis Introduction to Survey Research Sampling Procedures Causal Research - maybe 2 Announcements For Sat prepare Milan Food case – download data (Milan.sav) from the platform, please post your responses by 8 pm tonight – no slides needed. For Sat prepare WSJ/ Harris Survey – no slides 3 Comparative Advertising Measurement Scales & Data Analysis Page 52 packet What do you conclude? Remember – Percentage change or difference can only be calculated with Ratio scales This is an interval scale (at best). 4 Descriptive Survey Research Surveys usually used for descriptive research Provide a snapshot at a point in time Most analyses univariate or bivariate (but can do elaboration model with control variables) Would you recommend National to a friend interested in insurance services? Yes 1 No 2 Bivariate allows for hypothesis testing Hypothesis: Less educated people more likely to recommend Descriptive, not causal Recommendation could be driven by some 3rd factor correlated with education such as income 5 Sources of Survey Errors Population definition Representativeness of the sample frame Sampling Procedure Used Respondent Participation: Willing to participate (Do Not Call) Comprehend questions Have knowledge, opinions Willing & able to respond (language or memory) Interviewer understands & records accurately 6 Raising Willingness to Participate A good response rate requires persuasion Survey Introduction Phone or send letter in advance Introduce self, give affiliation unless this would bias Describe purpose briefly, w/o making survey sound threatening or demanding Make respondent feel that s/he is getting chance to provide opinions that will influence market offerings & that her/his cooperation is extremely important 7 Comprehends Questions? Advice on Question Wording Be simple and precise Give clear instructions Check for question applicability respondent screening question branching based on prior answers Avoid leading & double barrel questions 8 What’s the Problem? “Laws should be passed to eliminate the possibilities of special interests giving huge sums of money to candidates” “Laws should be passed to prohibit interest groups from contributing to campaigns, as groups do not have the right to contribute to candidates they support?” 9 Comprehends Questions? Literacy, translation considerations Conversational Norms How demanding was Term 3? How demanding was Core Finance? How demanding was Core Finance? How demanding was Term 3? How demanding was Managerial Accounting? How demanding was Core Finance? How demanding was Global Economic Environment of the Firm? How demanding was Term 3? “Question order effects in measuring service quality,” by DeMoranville and Bienstock, in International Journal of Research in Marketing, September, 2003 10 Do Respondents Have Knowledge? Retrieve answer from memory vs. construct it on spot Constructed answers are more likely to be influenced by question wording & prior questions. When answering later questions or engaging in later behavior, likelihood of using earlier answer input A: positively related to accessibility of A positively related to diagnosticity (relevance) of A negatively related to accessibility, diagnosticity of alternative inputs B, C, etc. (Feldman & Lynch) e.g., when political poll respondents asked: issue opinion A, presidential voting intention, issue opinion B, answers to A predict intention, but only for those who did not vote for either candidate in primary 11 Survey Best Practices: Survey Content, Question Order Survey Questions First figure out what questions are needed! Then order Lead with interesting, nonthreatening, easy questions Do you like to play golf? Have you ever travelled with your clubs? Can you remember the last time you traveled with your clubs? Put difficult or sensitive questions well into the interview How many times did you have to see your doctor for your reconstructive surgery? What is the size of your company (revenue)? Usually use funnel order (general to specific) Use product category? Brand X? Do you like Brand X? Why? 12 12 Question Order (Cont.) Survey Questions (cont.) Inverted funnel (specific to general) for complex topics. Is your company considering offering training courses on word processing over the Internet? Database? Spreadsheets? In general, how big is the untapped market for your software training courses if offered over the Internet? Group questions in logical order All questions about one subject together, with transitional phrases in between, “Now I’m going to ask you about agricultural applications of GPS systems...” 13 Survey Best Practices: Question Order (cont.) Demographics Questions Put last—these are less sensitive to prior questions Seem nosy if put first Rely on standard approaches for assessing http://www.norc.org/GSS+Website/ The Process of Survey Design Use Backwards Marketing Research to decide what is “need to know” Draft the survey Pretest for time, clarity, variability in responses Revise and retest Field the survey and keep an eye open for problems 14 15 Survey Best Practices: Choosing a Survey Method Mail, phone, web, in person? Cost Complexity of inquiries (branching) Need for aids Issue sensitivity Control over sample 16 Web and Telephone Web surveys now dominate. To compare web, in person, phone, mail, see http://knowledge-base.supersurvey.com/ 17 17 Free to Fuqua students: Qualtrics http://www.qualtrics.com/duke#submit Set up an account Build surveys Allows for complex designs Available to you during this course 18 Multi-Attribute Attitude Model (MAAM) Liking for a product as a whole = sum of liking for component parts Attitude toward brand j = (sum from i = 1 to n for salient attributes) Importance of Attributei * Evaluationij Importance 0 – 100 (allocate 100 points across attributes) Rating on 1 (unimportant) to 7 (very important) where 0 undefined but implicitly entirely unimportant ) Evaluation of brand j on attribute I -4 = poor to +4 = excellent 19 MAAM and SUVs Attribute Sporty Styling Handling Cost Ruggedness Off-Road Ability Total Attitude Importance 1=unimp, 7= important 6 5 2 4 2 Land Rover RAV Land Rover RAV Brand Evaluation -4 = poor, +4 = excellent Imp*Eval Imp*Eval 1 2 6 12 0 1 0 5 -2 3 -4 6 4 2 16 8 2 4 4 8 22 39 20 Diagnostics of Advantage 21 Measure Types Revisited Nominal (Unordered Categories) Just need unique number for each category Ordinal: ranking scale, intervals not assumed equal Interval: Intervals assumed equal, zero is arbitrary Ratio: Intervals assumed equal, zero means zero To multiply X * Y, (e.g., importance * evaluation), both X and Y must be on ratio scales. If X1*Y1 > X2*Y2 (XYbrand 1 >XYbrand 2), it does NOT follow that (X1+a)*Y1 > (X1+a)*Y2…. e.g., 2*2 > 2*(-2), but (2-4)*2 < (2-4)*(-2) To say % change in Y, Y must be on ratio scales 22 More on Scaling To multiply importance x evaluation for each attribute, both must be on ratio scales 0 on scale must be 0 of underlying quantity Importance unipolar (all positive). Completely unimportant = 0 weight Evaluation bipolar (negative to positive). To multiply, must code “neutral” as zero. 23 Improper Rescaling Attribute Sporty Styling Handling Cost Ruggedness Off-Road Ability Total Attitude Importance -3 =unimp, +3 = imp 2 1 -2 0 -2 Land Rover RAV Evaluation -4 = poor, +4 = excellent 1 0 -2 4 2 Land Rover Imp*Eval 2 1 3 2 4 RAV Imp*Eval 2 0 4 0 -4 2 Diff 4 1 -6 0 -8 -9 2 1 -10 0 -4 -11 I got these by subtracting 4 from the values three slides back 24 Consumer Attitudes We want to be able to predict consumer behavior However, instead of examining behavior directly (e.g., choice modeling), we often measure attitudes because… Measuring attitudes is sometimes easier than observing choice Attitudes are more diagnostic Attitudes are sometimes easier to interpret Attitudes can be reasonable predictors of behavior Attitudes toward products or brands typically derive from beliefs, actions, and perceptions 25 Types of Attitude Scales Semantic differential Colgate Combo is: low quality unappealing __:__:__:__:__:__:__ appealing __:__:__:__:__:__:__ high quality Constant sum (e.g., Importance) Purchase intent Likert scale (Agree-Disagree) 26 Recap Survey Design: responses constructed on the spot Moving parts of a good survey Population definition, choosing a survey method, determining what information needed Order of questions Attitude Measurement & multi-attribute attitude model To multiply or examine percentage differences, data must be on ratio scales 27 Sampling Terminology Elements -- Units about which information is sought Population -- Elements we want to generalize to Census -- Collect data on all elements in population Sampling Frame -- At each stage, list of all elements Sample – Collect data on subset of elements in pop. 28 Sampling Process Define population Elements, extent, time Identify a good sampling frame costly to create for yourself Determine sample size budget, accuracy needs Select sampling procedure way to select elements from the frame Physically select sample 29 Probability Samples Each element in population has known, nonzero chance of being sampled Simple random sample: all elements have 1/n chance of being sampled (e.g., cold caller) Systematic sample: start with randomly selected element and take every nth element (e.g., teams in this class) Cluster sampling: pick groups of elements (city blocks, census tracts, schools) then randomly select n elements from each cluster Stratified sampling: divide frame into strata according to a characteristic (e.g., gender), then sample randomly from each strata 30 Complex Sampling Procedures Simple random sampling almost never used in practice Stratified Sampling -- Lowers error Cluster Sampling -- Lowers cost of getting frames and of data collection 31 Stratified Random Sample Have frames sorted on some stratification variable believed to influence the variable you are estimating. Lower variance within each subgroup than across population in general By ensuring that each subgroup is represented in right mix, extreme overall means less likely -- i.e., smaller std. error. 32 Steps for Stratified Random Sample Divide Population into mutually exclusive and exhaustive categories. Decide what sampling fraction f = n / N to use. Draw an independent simple random sample of size f * N(stratum) from each stratum. Compute stratum mean for each Estimate overall pop mean as weighted average of stratum means Estimate SE as weighted combo of SEs in each 33 Cluster Sampling Typically “clusters” are geographic territories. Start with list of clusters, randomly select subset, and survey only subset. Cheaper travel cost, cost per interview Loss of effective sample size if people in cluster more alike than if in different cluster 34 Non-Probability Samples Convenience Judgment Pick especially informative elements Quota Sample matches population on key control characteristics correlated with behavior under study. Match only really matters for control variables related to thing you are trying to estimate. 35 Sampling quiz: Proportion of WEMBA students interested in changing jobs? Sample of 50 How would you do: Convenience Sample Judgment Quota sample (by male / female) Simple random Systematic Stratified (by male / female) 36 Key Sampling Takeaways Probability v Non-Probability Samples For Probability Samples, Standard Error is the measure of precision Precision increases with square root of N More precision with Stratified if and only if stratifier is correlated with thing estimated Same principal for Quota samples. Quotas only help if correlated with variable 37 Experiments Best way to test causal hypotheses Independent Variable = hypothesized cause Manipulated by the researcher/manager Example: Send a color or black and white brochure Dependent Variable = effect Measured (observed) by researcher/manager Example: New accounts secured Random assignment of subjects to conditions Example: receive color or receive b&w brochure 38 Pre-experimental Designs One group, after-only design One group, before & after design Unmatched control group design Matched control group design All have threats to validity not present in a true experiment with random assignment to treatments. 39 Validity The strength of our conclusions i.e., Is what we conclude from our experiment correct? Threats to Validity History: an event occurring around same time as treatment that has nothing to do with treatment Maturation: people change pre to post Testing: pretest causes change in response Instrumentation: measures changed meaning Statistical Regression: Original measure was due to a random peak (SI Cover Curse) or valley 40 One Group After Only We propose a change in MBA Core, to move Finance and Marketing up to Term 2 from their position in Term 3. One major motive for this is that students interview for internships in Term 3, and if they want jobs in marketing or finance, they have no background at the time of the interview. Thus, we perceive that we are at a competitive disadvantage because those courses are in Term 3. EG X O (Mean = 50%) X = Marketing Term 3, O = Did/Did Not Get Desired Internship Key: Lacks a baseline, so worthless. 41 One Group Pre-Post Design Breckenridge Brewery wants to assess the efficacy of TV ad spots for its new amber ale. Time 1 (O1): Duke undergrads are brought to the lab and asked to rate their frequency of buying a series of brands in various categories over the past week. The list includes Breckenridge Amber Ale. Mean = 0.2 packs per week. Time 2 (X): Two weeks of ads for Breckenridge Ale. Time 3 (O2): Same Duke undergrads brought back to lab to rate frequency of buying same set of brands over past week. Mean = 1.3 packs per week. 1.3 - 0.2 = 1.1. We attribute an increase of 1.1 packs per week to the ad. 42 Online Investor Performance X = brick and mortar brokerage customer moves online to trade in 1999 O = Annualized turnover 1998 – 40% annualized turnover 2000 – 100% annualized turnover Did going online cause people to trade more actively? Threats with one-group pre-post? 43 Quasi-Experimental Designs: Interrupted Time Series Same as one-group pretest posttest, but observations at many points in time before and after key treatment for same people: EG O1 O2 O3 X O4 O5 O6 Extra time periods help control for history, maturation, testing. “Quasi-experiment” 44 Online Investor Performance Gross Returns 10 Net Returns 5 36 33 30 27 24 21 18 15 12 9 6 3 0 -3 -6 -9 -12 -15 -18 -21 -24 -27 -30 -33 0 -36 Cumulative Market Adjusted Return (%) 15 -5 -10 Event Month (0 = month of first online trade) 45 Portfolio Turnover 120% Size-Matched 100% Online 80% 60% 40% 20% 24 21 18 15 12 9 6 3 0 -3 -6 -9 -1 2 -1 5 -1 8 -2 1 0% -2 4 Annualized Turnover (%) 140% Event Month (0 is month of first online trade) 46 2 Groups: Unmatched Control Group (Effect of Prior Knowledge on Search) Hypothesis: People with little knowledge about cars search less online because they are overwhelmed 100 Durham residents who are in the market for a car EG CG X1 (Auto Shop Course) O1 (6 hrs online) --------------------------------------------------------X2 (Electronics Course) O2 (3 hrs online) 47 2 Groups: Matched Control Group (True Experiment) EG R X1 (Auto Shop Course) O1 (6 hrs) --------------------------------------------------------- CG R X2 (Electronics Course) O2 (3 hrs) Control for Selection Threat. Key Point: For causal research, chance (not respondent) must determine respondent assignment to condition. 48 Breckenridge Brewery Ads Breckenridge Brewery wants to assess the efficacy of TV ad spots for its new amber ale. Time 1 (O1): Duke undergrads are brought to the lab and asked to rate their frequency of buying a series of brands in various categories over the past week. The list includes Breckenridge Amber Ale. Mean = 0.2 packs per week. Time 2 (X): Two weeks of ads for Breckenridge Ale. Time 3 (O2): Same Duke undergrads brought back to lab to rate frequency of buying same set of brands over past week. Mean = 1.3 packs per week. 1.3 - 0.2 = 1.1 increase in number of packs per week. 49 2-group Before-After Design • Now add a randomly assigned “Control” group with mean scores O1 = 0.3, O2 = 0.5. Experimental O1 X O2 Control O1 O2 O1 O2 O2 - O1 Difference 0.2 1.3 1.1 0.3 0.5 0.2 50 Factorial Designs Independent Variable: Factor manipulated by the researcher Dependent Variable: Effect or response measured by researcher Factorial Design: 2 or more independent variables, each with two or more levels. All possible combinations of levels of A & levels of B. 51 Oreo Promotion Experiment Kroger: Supporting a discount on Oreo cookies Factor A: Ads in local paper a1 = no ads a2 = ad in Thursday local paper Factor B: Display location b1 = regular shelf b2 = end aisle 52 Oreo Promotion Experiment (Expenditures/customer/2 wks) a1 = no ads a2 = ads Row Ave b1 = regular shelf .60 .90 .75 b2 = end aisle .85 .95 .90 Col. Ave .725 .925 53 Sales of Oreos on Promotion as function of Local Advertising, Display Location 54 Oreo Example, No Interaction Main Effect of A (Ads)? Main Effect of B (Display Location)? No AxB (say A by B) interaction. Effect of changing A (Ads) is independent of level of B (Display Location). Sales go up by $0.30 when you advertise, regardless of location. Implies that Ad & Display decisions can be decoupled…they influence sales additively. 55 Managerial Implications of Interactions If two controllable marketing decision variables interact (e.g., advertising x display), implication is that you can’t decouple decisions; must coordinate. If A is a controllable decision variable and B is a potential segmentation variable (e.g., ads x urban/suburban), interaction means that segments respond differently to this lever. 56 Interactions and segmentation c Coupons have a bigger effect in the suburbs Psychology of Consumers Exposure, Attention, & Perception 57 Takeaways for the Day Survey Design: responses constructed on the spot? Order of questions Attitude Measurement & multiattribute attitude model Probability v Non-Probability Samples More precision with Stratified if and only if stratifier is correlated with thing estimated Threats to validity in pre-experimental and quasiexperimental designs Factorial Designs – Main effects and interactions 2 marketing tactics interact coordinate Marketing tactic interacts with customer classification implies classification a potential basis for segmentation…different sensitivities to some marketing mix variable 58