Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Session 5: Causal-comparative & Correlational Research & Survey Methods Today! NIH CITI training modules ◦ Moved back to August 5th : Annotated Bibliography ◦ ◦ Make sure you do BASIC/REFRESHER course Each individual reviews 3 research articles regarding their topic See Example: http://rxsped596.pbworks.com/w/file/fetch/54804527/Example%20Annotated%20Bibliogra phy.pdf Today! : Conceptual Framework ◦ ◦ ◦ ◦ Group submits short summary of literature and presents a conceptual framework for theories that drive their proposal. Group submits short summary of literature and presents a conceptual framework for theories that drive their proposal. Group composes a one page double-spaced document---can be used within your final proposal Use your articles from your Annotated Bibliography to form a framework for your research proposal Explain to the reader why should the intervention or the questions you are asking make sense (based on the literature) Cancelled: Review of Research Article August 12th : Written Research Proposal (group assignment) August 14th: Presentation of Research Proposal (group assignment) Using Single Subject Research to Establish “Evidence-based Practices” A “practice” may be considered “evidence-based” when: The practice is operationally defined, and implemented with fidelity. The outcomes associated with the practice are operationally defined. The context in which the practice in use is operationally defined Results from the single subject studies used to assess the practice demonstrate experimental control. The effects are replicated across 5 single subject studies conducted in at least 3 locations, and with at least 20 different participants. Dependent and independent variables Dependent variable (DV) – the behavior (measure) that you are analyzing You want to produce change (variability) in the dependent variable Studies may have multiple DVs Independent variable (IV) – the variable (event, intervention, condition) that is of experimental interest and that the researcher manipulates in an experimental research design Studies may have multiple IVs Phase A Phase B Phase A Immediacy of Effect Level Phase B Variability Trend Overlap Research Question??? In SSD, a Functional Relationship/Experimental Control has occurred when There are 3 demonstrations of an effect at 3 points in time. Effect could be: change in trend or level Also want to see immediacy of effect Good research has at least 5 data points in each phase to establish a consistent pattern in the data. Defining Features of Multiple Baseline Designs A multiple baseline design involves three or more AB interventions (series) with phase changes staggered across at least three points in time. Key Features Series are independent of each other People, places, materials, behaviors/skills The same IV is applied in each series Staggered implementation of IV BL Lollipop for R+ Treatment 6 100 80 60 Percentage of Correct Responding 40 20 Vivian 0 Lollipop for R+ 100 80 60 40 20 Tammy 0 Lollipop for R+ 100 80 60 40 20 Dr. Cathy 0 10 20 30 40 Sessions 50 60 70 Defining features of withdrawal and reversal designs Sequential phases of data collection involving the implementation and withdrawal of an independent variable(s) within each phase, multiple data points are collected to establish a representative pattern of behavior phase change should occur only after stability of behavior within the phase is established traditionally, the first phase is Baseline, followed by implementation of the IV (Intervention) this is not required, however, as you may begin a study with an intervention phase 4B Baseline FCT Baseline FCT 6 Total SIB per minute 5 4 3 2 1 0 1 5 10 15 Sessions 20 25 30 35 Alternating Treatment Designs Alternating Treatment Designs employ rapid phase reversals across 2 or more conditions to assess sensitivity of change in the dependent variable to change in condition. Student 1 Hypothesis: Escape Math Work Percent Intervals with Occurrence of Problem Behavior 100% 90% 80% 70% 2. Is Esc different than Attn? Control Condition Escape Condition 60% Attention Condition 50% IOA 1. Is Esc different than Control? 40% 30% 20% 10% 0% 1 2 3 Sessions 4 5 15 Descriptive Statistics Who is in your data? sample population Inferential Statistics What your sample says about the population sample population Mean, Median, Mode, standard deviation, variance Prepared by M. Hara ([email protected]) Tests of significance (t-, F-Tests) Central Tendency ◦ Mean- average ◦ Median- midpoint in distribution of scores ◦ Mode- most frequent score Variability ◦ Range- total extension of the data (e.g., 1-10) ◦ Standard Deviation- sum of deviations from the mean squared. How well the mean summarizes the data. ◦ Variance- standard deviation squared. Used in sophisticated analyses Inferential Statistics What your sample says about the population sample population Prepared by M. Hara ([email protected]) T tests- used when have two groups to compare. ◦ Independent samples t- if groups are independent Different people in each group ◦ Dependent samples t-: if two sets of scores are available for the same people Matched groups ANOVA (analysis of variance)- when you have more than 2 groups to compare OR more than one independent variable (reports an F-statistic, which is basically a t-value squared) ANCOVA (analysis of covariance)- ANOVA that allows for control of the influence of an IV (e.g., characteristics of people) that may vary between your groups before treatment is introduced. ◦ Post-hoc method for matching groups on variables such as age, prior education, SES, or a measure of performance Statistical analyses to determine whether a difference is statistically significant (probability for result to occur by chance). Yes or No answer Alpha level (p=) ◦ An established probability level which serves as the criterion to determine whether to accept or reject the null hypothesis ◦ Common levels in education .01 .05 .10 Objectives 4.1 & 6.1 Addressing “WHAT” questions? Depth of Information Representative, Generalizability Details, Depth, and Variability Quantitative Data • Survey Prepared by M. Hara ([email protected]) • Large Scale Assessments Variable Type Example Nominal • Gender • Yes/No Prepared by M. Hara ([email protected]) Male (0) Female (1) No (0) Yes (1) 24 Check All that Apply Q. Which of the following applications have you used with your students? (Please check ALL that apply) MS Word iMovie MS Excel iDVD MS PowerPoint iTunes SPSS iWeb Prepared by M. Hara ([email protected]) 25 Variable Type Example Nominal • Gender • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Prepared by M. Hara ([email protected]) Male Disagree (2) Agree (3) Strongly Agree (4) 26 Likert-scale Strongly Somewhat Somewhat Disagree Agree Disagree Disagree Agree Strongly Agree Q1. Overall, I have a good Parentteacher Relationship. Prepared by M. Hara ([email protected]) 27 Likert-scale Strongly Somewhat Somewhat Disagree Agree Disagree Disagree Agree Q1. Overall, I have a good Parentteacher Relationship. Strongly Agree 1 2 3 4 5 6 NOTE: Distinguishable Prepared by M. Hara ([email protected]) 28 Multiple Categories Ethnicity African American Asian Caucasian Hispanic Other ____________ Education Completed High School Some College BA/BS Master’s Doctoral Decline to state Prepared by M. Hara ([email protected]) 29 Multiple Categories Ethnicity NOTE: Mutually Exclusive, Exhaustive, and Distinguishable African American 1 Asian 2 Caucasian 3 Hispanic 4 Other ____________ 5 Education Completed High School 1 Some College 2 BA/BS 3 Master’s 4 Doctoral 5 Decline to state Prepared by M. Hara ([email protected]) 888 30 Variable Type Example Nominal • Gender • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Scale Male Disagree (2) Agree (3) Strongly Agree (4) • Age, Annual Income, Test-score (Interval/Ratio) 31 Numbers with Different Meaning Example Nominal • Gender Numerical/ Continuous Categorical Variable Type • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Scale Male Disagree (2) Agree (3) Strongly Agree (4) • Age, Annual Income, Test-score (Interval/Ratio) Prepared by M. Hara ([email protected]) 32 Numbers with Different Meaning Example Nominal • Gender Numerical/ Continuous Categorical Variable Type • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Scale Male Disagree (2) Agree (3) Strongly Agree (4) • Age, Annual Income, Test-score (Interval/Ratio) Prepared by M. Hara ([email protected]) 33 Numbers with Different Meaning Example Nominal • Gender Numerical/ Continuous Categorical Variable Type • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Scale Male Disagree (2) Agree (3) Strongly Agree (4) • Age, Annual Income, Test-score (Interval/Ratio) Prepared by M. Hara ([email protected]) 34 Variable Types and Analysis Dependent Variable (a.k.a., Outcome) Prepared by M. Hara ([email protected]) Is there an association? Independent Variable (a.k.a., Predictor) 35 Variable Types and Analysis Dependent Variable (a.k.a., Outcome) Is there an association? Independent Variable (a.k.a., Predictor) Where differences culminate Prepared by M. Hara ([email protected]) 36 Variable Types and Analysis Dependent Variable (a.k.a., Outcome) Where differences culminate Prepared by M. Hara ([email protected]) Is there an association? Independent Variable (a.k.a., Predictor, Intervention) Contributing Factors 37 Variable Types and Analysis Dependent Variable Is there an association? Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 38 Variable Types and Analysis Dependent Variable Independent Variable Chi-square test Or χ²-test Categorical Numerical/ Continuous Prepared by M. Hara ([email protected]) Contingency Tables (a.k.a. Cross-tabs) Categorical Numerical/ Continuous 39 Variable Types and Analysis Dependent Variable Categorical Annual Salary Prepared by M. Hara ([email protected]) Independent Variable Contingency Tables (a.k.a. Cross-tabs) Categorical Gender 40 Variable Types and Analysis Dependent Variable Categorical Independent Variable Contingency Tables (a.k.a. Cross-tabs) Categorical Gender Male Female 12% 4% 26K - 35K 18% 6% 36K - 45K 24% 11% 46K - 55K 36% 39% 56K - 65K 8% 28% 66K and up 2% 12% Annual Salary 25K or below Prepared by M. Hara ([email protected]) 41 Your Research Topic??? Dependent Variable Categorical Independent Variable Contingency Tables (a.k.a. Cross-tabs) Categorical Annual Salary Gender Your DV? Your IV? Prepared by M. Hara ([email protected]) 42 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 43 Variable Types and Analysis Dependent Variable t-test or F-test Numerical/ Continuous Analysis of Variance (a.k.a. ANOVA) 0.45 Males 0.45 0.05 0 0 -0 -0 -0 -0 -1 -1 -1 -1 -1 -2 -2 -2 -2 -2 -3 Prepared by M. Hara ([email protected]) Categorical Females SAT9 Math Score .2 0. 0 0. 2 0. 4 -30 .06 -20 .8 -21 .60 -21 .42 -21 .24 -21 .06 -1 .8 -12 .60 -12 .42 -12 .24 -12 .06 -02 .8 -03 .60 -0 .4 -0 .2 0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 1. 2 1. 4 1. 6 1. 8 2. 0 2. 2 2. 4 2. 6 2. 8 3. 0 0.05 .4 0.1 .6 0.1 .8 0.15 .0 0.15 .2 0.2 .4 0.2 .6 0.25 .8 0.25 .0 0.3 .2 0.3 .4 0.35 .6 0.35 .8 0.4 .0 0.4 Independent Variable 44 Your Research Topic?? Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 45 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Prepared by M. Hara ([email protected]) Regression Numerical/ Continuous 46 Variable Types and Analysis Dependent Variable Numerical/ Continuous Independent Variable Regression Numerical/ Continuous SAT9 Math Score Household Income Prepared by M. Hara ([email protected]) 47 Your Research Topic?? Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Prepared by M. Hara ([email protected]) Regression Numerical/ Continuous 48 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 49 Variable Types and Analysis Dependent Variable Independent Variable Logistic Regression Categorical Numerical/ Continuous Pass High school Exit Exam Probability of passing h.s. exam based on SAT-9 score Fail SAT 9 Math Prepared by M. Hara ([email protected]) 50 Your Research Topic?? Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 51 Numbers with Different Meaning Example Nominal • Gender Numerical/ Continuous Categorical Variable Type • Yes/No Ordinal (0) Female (1) No (0) Yes (1) • Likert-scale Strongly Disagree (1) Scale Male Disagree (2) Agree (3) Strongly Agree (4) • Age, Annual Income, Test-score (Interval/Ratio) Prepared by M. Hara ([email protected]) 52 Variable Types and Analysis Dependent Variable Independent Variable Chi-square test Or χ²-test Categorical Numerical/ Continuous Prepared by M. Hara ([email protected]) Contingency Tables (a.k.a. Cross-tabs) Categorical Numerical/ Continuous 53 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 54 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Prepared by M. Hara ([email protected]) Regression Numerical/ Continuous 55 Variable Types and Analysis Dependent Variable Independent Variable Categorical Categorical Numerical/ Continuous Numerical/ Continuous Prepared by M. Hara ([email protected]) 56 Statistics that function to describe the strength and direction of a relationship between two or more variables ◦ Simple correlation coefficient (r=) ◦ Coefficient Determination- (r-squared). Amount of variance that is accounted for by the explanatory (independent or predictor) variable in the response variable (criterion variable). ◦ Multiple regression- to indicate the mount of variance that all of the predictor variables explain. Way of quantifying the difference between two groups. Not just was there an effect, but the magnitude of the effect. Many ways to calculate ES= [Mean of experimental group] – [Mean of control group]/Standard Deviation R-squared, Cohens-D Standard deviation is how well the mean summarizes the data Investigators attempt to determine the cause of Describes conditions that already exist (a.k.a. ex post facto). differences that already exist between or among groups of individuals. The group difference variable is either a variable that cannot be manipulated or one that might have been manipulated but for one reason or another, has not been. Studies in medicine and sociology are causalcomparative in nature, as are studies of differences between men and women. Similarities and Differences Between CausalComparative and Correlational Research • Similarities – Ex Post Facto research – Attempt to explain phenomena of interest – Seek to identify variables that are worthy of later exploration through experimental research – Neither permits the manipulation of variables – Attempt to explore causation • Differences – Causal studies compare two or more groups of subjects – Causal studies involve at least one categorical variable – Causal studies often compare averages or use tables instead of scatterplots and correlation coefficients Similarities and Differences Between CausalComparative and Experimental Research • Similarities – Require at least one categorical variable – Both compare group performances to determine relationships – Both compare separate groups of subjects • Differences – In experimental research, the independent variable is manipulated – Causal studies are likely to provide much weaker evidence for causation – In experimental studies, researchers can assign subjects to treatment groups – The researcher has greater flexibility in formulating the structure of the design in experimental research Problem Formulation Sample Instrumentation Design The first step is to identify and define the particular phenomena of interest and consider possible causes Selection of the sample of individuals to be studied by carefully identifying the characteristics of select groups There are no limits on the types of instruments that are used in Causal-comparative studies The basic design involves selecting two or more groups that differ on a particular variable of interest and comparing them on another variable(s) without manipulation The Basic Causal-Comparative Designs Independent variable Dependent variable I C (Group possesses characteristic) O (Measurement) II –C (Group does not possess characteristic) O (Measurement) I C1 (Group possesses characteristic 1) O (Measurement) II C2 (Group possesses characteristic 2) O (Measurement) Group (a) (b) Subject Characteristics The possibility exists that the groups are not equivalent on one or more important variables One way to control for an extraneous variable is to match subjects from the comparison groups on that variable Creating or finding homogeneous subgroups would be another way to control for an extraneous variable The third way to control for an extraneous variable is to use the technique of statistical matching Other Threats • • • • • Loss of subjects Location Instrumentation History Maturation • Data collector bias • Attitude In a Causal-Comparative Study, the first step is to construct frequency charts & graphs. Means and SD are usually calculated if the variables involved are quantitative. The most commonly used inference test is a t-test for differences between means. ANCOVAs are useful for these types of studies. Results should always be interpreted with caution since they do not prove cause and effect. PSU uses a free web-based survey software called Qualtrics (available for student use) http://oit.pdx.edu/node/908 Is there another way of collecting the information? Is this method the most efficient and cost effective? Will it provide you with the information you want in a valid manner? Have specific goals for the survey Consider alternatives to using a survey to collect information. Select samples that well represent the population studied. Use designs that balance costs with errors. Take great care in matching wording to concepts being measured and the population being studied Pretest questionnaires and procedures to identify problems prior to the survey Construct quality checks for each stage of the survey Maximize cooperation or response rates within the limits of ethical treatment. Carefully develop and fulfill pledges of confidentiality given to respondents. Disclose all methods of the survey to permit evaluation and replication Step Step Step Step Step Step Step 1234567- Determine Purpose Identify a Sampling Plan & Mode Design survey instrument Test survey instrument Send out a letter of transmittal Deliver the survey Analyze data from survey State specific objectives Consider the types of information needed Use your purpose to guide each of your other steps and check back to this purpose often! ◦ To make sure you are answering research questions Simple descriptive- one-shot to describe moment in time Cross-sectional- several groups at one point in time (e.g., 1st, 3rd, 5th grade students) Longitudinal- cohort at different points (e.g., year 1, 2…after leaving school Strengths and limitations of each?? Identify respondents (based on population you want to sample) ◦ Think external validity…way you will generalize your results. probability or nonprobability methods ◦ Probability- each person has a probability of being surveyed. random sampling, systematic sampling, and stratified sampling ◦ Nonprobability methods convenience sampling, judgment sampling, quota sampling, and snowball sampling Based on what know about population decide how collect data: ◦ Email, Web-based surveys, mail, telephone, personal interviews, etc. Random sampling- Each member of the population has an equal and known chance of being selected Systematic sampling- called an Nth name selection technique Stratified sampling- (1) Identify a subset of the population that share at least one common characteristic (males & females; managers & non-managers) and (2) their actual representation in the population, then (3) random sampling is used to select a sufficient number of subjects from each stratum. Convenience sampling- selected because they are convenient. often used during preliminary research efforts to get a gross estimate of the results, without incurring the cost or time required to select a random sample. Judgment sampling- sample based on judgment. For example, draw the entire sample from one "representative" city, even though the population includes all cities. When using this method, the researcher must be confident that the chosen sample is truly representative of the entire population. Quota sampling- like stratified sampling: (1) identify like characteristics and percentage of population, (2) convenience or judgment to select participants to represent characteristics and meet quota to represent population Snowball sampling- Snowball sampling relies on referrals from initial subjects to generate additional subjects. Review the literature There may be a survey that matches what you need (then use and cite OR modify and cite) Determine question format ◦ Open-ended, close-ended, likert scale (0-5; never, sometimes, always) Avoid sensitive questions Be very clear Short items are better Avoid negative wording OR use positive wording Avoid asking about more than one idea…stay away from AND…make another question to ask for additional information Avoid jargon/big words Emphasize …. (underline) critical words Test instrument on yourself and others (not part of your sample!) Pilot test to similar population. Use feedback to improve administration of survey Cover letter for mail surveys or as introductory letter/email for a phone, web, email, or personal interview survey. Increases participation and limits incomplete surveys Deliver the survey according to timeline Web-based or email surveys may have a start date and date for completion by. Send follow up emails/letter/calls Monitor data as they come in to check for errors----sooner than later. Send thank you emails/letters/calls Follow up with non-respondents so you have some ideas why there was no response ◦ You can then compare respondents vs nonrespondents Decide how to handle missing data Complete descriptive statistics: frequency, percentages, mean, median, etc. Look for interesting patterns in the data Do sub-group analyses if possible Display data in tables & graphs Get together with your research group Based on your overall topic and possible research questions...outline survey methods that you might give to participants or other stakeholders for the purpose of your research.