Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
SWK 707 Research for Social Work Practice Nechama Sammet Moring CLASS 10 Plan for tonight 6:00-6:10 logistics, check in 6:10-6:25 literature review lecture 6:25-7:00 literature review activity 7:00-7:30 peer review your introduction and recommendations 7:30-7:40 break 7:40-8:30 evaluation design lecture and worksheets/ post its 8:30-8:50 collecting evaluation data 8:50 evaluations, I promise Notes on assignment 2 Overall, excellent and you’re in great shape for assignment 3 Take credit for your clinical experience-name yourself as a source! Integrate this experience into your evaluation of studies-you know what you’re doing! Operationalize, operationalize, operationalize! (what does “successful” or “it worked” mean?) Please use page #s Please proofread Selecting studies Not the first 10 you find! You are evaluating YOUR INTERVENTION, not the causes of your social problem. OK to accept some things as givens (i.e. some kids dislike school, for potentially many reasons, but this is not super relevant to a program to improve grades) Bench science vs social science: the chemistry of how prenatal vitamins are absorbed vs answering “do prenatal vitamins reduce low birthweight?” Be outcome focused! Literature reviews and systematic reviews: describe findings but evaluate how the authors selected and evaluated studies-do you agree or disagree with their methods and analysis? Strengths and limitations Rationale for why the study was done is important but don’t dwell on it. Instead, focus on integrating findings to answer YOUR research question Not agreeing with you is not a limitation-it’s a consideration. Agreeing with you or supporting your desired intervention is also not a strength-it’s a consideration Your task is to evaluate the QUALITY of the evidence, not simply amass it. Mechanism of action and direction of association Which came first? Any confounders? (I didn’t take prenatal vitamins because I couldn’t afford them (or food) my baby was low birth weight?) Unless the RESEARCH specifically evaluates mechanism of action (how it works) or direction, we just don’t know. Avoid assumption Avoiding assumption part 1: ordering pizza I like mushrooms and pepperoni on my pizzadoes that sound ok to you all? I’m not really a mushroom fan. Could we do just pepperoni? You and your friends order pizza Pepperoni is awesome! Ok with you? We probably need a large for all of us, huh? Cool, I’ll call Dominos. They have great breadsticks-I’ll get some I might like mushrooms but I haven’t examined my Let’s bias about operation that alize pizza toppings. Do we understand the time sequence of mushrooms becoming pizza toppings? Researchers try to order pizza We should determine a causal pathway before we proceed Avoiding assumptions Research is full of assumptions, because people are full of assumptions, because assumptions helped us navigate avoiding wooly mammoth when we were cave people Good research/researchers question ALL the assumptions. Not all research is good You, as the research consumer, are evaluating how well the researchers avoided assumption, and if they made any assumptions that impact the quality or validity of of their findings Problems you see here: “The sample consisted of Latino children from immigrant families who attended a neighborhood Head Start program. Children’s social interactions were evaluated for appropriateness and were ranked either 1, appropriate, or 2, inappropriate.” “survivors of domestic violence benefit from internet-based information about the importance of getting primary care, including cervical and breast cancer screening” “As treatment is a desired resource in low-income communities, we made counseling (provided by a medical student) available at community health centers.” “Children who have experienced negative events, like divorce of their parents, or who come from non-intact families/lack fathers are at higher risk of dental cavities” Avoid assumption and operationalize If researchers haven’t operationalized their variables, take note Watch “assumption words” like negative/positive, inappropriate/appropriate, safe/unsafe, healthy/unhealthy, complete/intact, progress What does progress mean? WHO DEFINES positive, inappropriate, healthy etc? Who is left out of this definition? Other tips for avoiding assumption Know your own assumptions and bias. Journal/talk to a friend and articulate your own beliefs. Your beliefs are incredibly valid. But separate YOUR beliefs from proven research facts/correlations/causation Remember you never know the direction of the relationship (did the pizza or the pepperoni come first?) consider correlation vs causation 3 tests needed to prove causation: 1. are the 2 variables correlated? 2. Did the independent variable come first? (time) 3. Have you removed all confounding factors that could explain the relationship? Literature review Organizing and synthesizing information Don’t go down the rabbit hole Audience: intelligent but uninformed about your topic adult (me) The Brilliant Best Friend System For each article, write the MAIN findings/take home messages ONLY on a separate index card as well as the author’s name so you can track it later Once you’ve prepared your index cards, sort them into piles by theme. This is a lot like grounded coding Ideas that are interesting but beyond the scope of your paper go onto a post-it and into your ziplock for later. It’s ok to put index cards into your ziplock, but don’t “disappear” findings that contradict your intervention-these are important to address. Example: Managing disclosures of abuse by research participants with IDD (piles) Higher rates of sexual violence against people with disabilities higher likelihood of enrolling survivors of violence Barriers to reporting abuse Barriers to justice for abuse survivors with IDD (and impact on abuse prevalence) Research as a safe environment for disclosures (or not) Appropriate responses by professionals (not researchers) to disclosures Wildcard but important Ziplock for later: LGBTQ people w/IDD completely absent from literature; impact of unaddressed trauma on “functioning”; representation, visibility and feminism/why this isn’t a feminist issue; historical connection between rape, incest and eugenics movement Further sorting Now that all your literature is in piles, approach each pile and do a further sort. Identify main themes within each pile For each pile and each theme, refer back to your assignment 2 to think about the quality of the evidence. Are there a lot of studies that agree/show the same thing? Take advantage of all the work you already did evaluating the quality of the evidence. Write a summary paragraph for each pile. Provide an outline of main findings ACROSS STUDIES and discuss quality of evidence. Using a bias chart (see handout) will help you assess quality of the evidence. For each pile, is any information missing? (these are the gaps you want to identify) Example: Combining, evaluating, providing selective detail, identifying gaps in each pile Higher rates of sexual violence against people with disabilities higher likelihood of enrolling survivors of violence. COMBINING: There is extensive, high quality evidence showing that people with disabilities experience much higher rates of sexual violence victimization (Swango-Wilson, 2009; McCarthy 2011; McCormack et al, 2006; Jones et al, 2016). SELECTIVE DETAIL: People with intellectual disabilities are even more likely than their counterparts with other kinds of disabilities to experience sexual violence (Horner-Johnson & Drum, 2006). We know that people with ID are consensually participating in research at higher rates than ever before (Ianoco, 2005). GAPS: While common sense suggests that this higher rate of sexual violence victimization means that researchers who enroll people with IDD in their studies are more likely to encounter research participants who have survived sexual violence, this has not yet been studied. Though some preliminary studies suggest that research participants with IDD view participatory action research as an opportunity to reduce sexual violence in their communities (Northrup et al, 2010), we lack understanding of researchers’ role in responding to sexual violence against people with IDD. Let’s try it with 2 articles Using assignment 2, work with a peer to identify themes/categories (piles) and put important findings (only) from your first study onto cards Use your ziplocks as needed to discard interesting but less relevant information for later Switch and help your peer put findings from her first study onto cards and identify themes/categories for her piles Sort your cards into piles If you have time, put findings from your second study onto cards and add these findings to your piles Summarize each pile Buy an index card sorter (or some envelopes) and continue your sorting at home Peer review: read and discuss each partner’s work, especially: Introduction Is their writing respectful and professional? Is their writing clear and concise? Do they have a clear population? Do they have a clear SOCIAL problem? Do they clearly convey why the social problem is important? (the so what test)? Do they give you just enough information about their agency that you know why the agency is a good fit for offering the intervention? Recommendation Is their writing respectful and professional? Clear and concise? Is their intervention clear, with enough details to picture it, but not overwhelming? Does their intervention address their social problem? Is their intervention attentive to human rights, client choice and ethics? Does this intervention have clear goals that are measurable and specific? Does the intervention consider the 3 “wheels” of evidence based practice? Evaluation Social problem • Has real, negative impacts on individuals and society • Is action-able • Is contexualized Intervention • Specifically designed to address the social problem or lessen its negative impacts in your population • Tailored to your population and feasible evaluation • Concretely measure if the intervention is working • Does the intervention effectively address or lessen impacts of the social problem? In other words, how do we know we’re meeting our goals? Program evaluation Process evaluation Outcome evaluation What questions do we want to ask? What’s important? (this shapes the evaluation) What new skills do children gain from this program? Which skills are children using 3 months from now? How satisfied are people with our program? How effectively are we reaching underserved populations? How supported do our clients feel? Are we perpetuating systemic bias or oppression in any way? Example evaluation questions Meditation classes for kids with behavioral problems and their parents: Process: Do parents and kids feel welcome at the classes? Do they want to be there? Are they making friends and feeling supported? Are the teachers actually teaching the content (fidelity)? Can they teach? Outcome: Are kids using the meditation skills to manage anger? Do they have fewer negative behaviors? Has parental stress levels changed? Over time, is there a reduction in child abuse rates and/or incarceration rates? Cancer screening van: Process: do people know about the van? Do they trust the van? Are van working treating people well? Outcome: How many people are screened in the van? Does van screening increase the overall % of people in the neighborhood who are screened? Does this increase reduce the screening disparity between PoC and white people? Goal setting Review the goals you identified for your intervention with your neighbor Are they realistic? Are they closely related to your social problem? When you are satisfied with your goals, write them on your worksheet and on a post it Process evaluation: is our program going the way we thought it would? Fidelity, and barriers to fidelity (shared decision making and DMH example) Basics like are people coming to work? ie Does your research methods teacher show up? Does she do more than feed you candy? Does class stick to content or frequently get derailed with dog pictures and the need to unionize zombies? Provider client relationship quality? Trust? Mistrust? Equity and human rights? Is your program free of coercion? Are the human rights of all people being respected? Are there any internal inequities, structural oppression etc? Outcome evaluation: is our program meeting the goals we thought it would meet? Learning objectives: can students successfully perform the skills and competencies listed on the syllabus? What is the effect of our program, on the areas we think are important? How are these effects measured? WHO do you need to talk to / collect data from in order to understand the effects of your program? Time frame What are the short-term impacts, and on whom? Tend to be immediate process measures (60% more kids are accessing reading room support / 2/3 of shelter residents have an effective safety plan) What are the intermediate impacts, and on whom? Tend to be process and outcome measures related to participants (kids gain reading skills and progress by 1 reading level, children of residents witness fewer instances of DV) What are the long term impacts, and on whom? Tend to be outcome measures on participants and society (reading scores across the school are up, and further long-term the drop out rate for the school is down, children experience less PTSD, improved school performance, and further long-term are less likely to perpetrate abuse against their own intimate partners) Realistic goals, and proxies We probably won’t know if our program for preschool aged children is enhancing likelihood of going to college any time soon. But we can know if children tell us they like school, right now or when they get to 1st grade Some things are really difficult to measure (happiness, good relationships, self- determination, cultural competent care). Instead, proxies or proxy indicators are often used taking prenatal vitamins is often used as a proxy for engagement with prenatal care; days sober is a proxy for success of treatment Evaluation questions and indicators Very tight, operationalized questions If X (as defined very specifically by…) happens, we will know that the program is working the way we think it will If Y (as defined very specifically by…) happens, we will know that the program is meeting the goals we think it should meet Operationalizing is key Hard to evaluate operationalized Consistent attendance, defined as once a month for an “improved” “responsible” “proper” “adequate” “appropriate” “reasonable” uninterrupted period of 12 months Demonstrating equal turn-taking in conversations with a peer Reading scores increase to the child’s grade level Use of one of the following 5 coping strategies for managing stress 10% fewer cigarettes per day/calls to a crisis hotline/placement changes indicators For each goal, identify at least 1 indicator that this goal is being reached. In other words, how will you know that you are successful? What information do you need? Goals should be Measurable Operationalized Specific Time-frame Workshop your indicators with your neighbor and write them on a post-it when you are both happy with them Save yourself some work: Standardized instruments Exist in many fields, about many questions What have researchers used in the past (from your lit review) Pros: established way of evaluating a construct of interest Cons: may or may not be appropriate for your population, accessible; may or may not measure what you want it to Consider reliability Measuring your indicators What do you need to know to measure your indicators? How will you collect information about your indicators? From whom? Which stakeholders need to provide information? What will you do with the information you collect? How will you analyze it? Workshop your table of information needed with your neighbor Adding in research design Qualitative, quantitative or mixed methods? Longitudinal or cross sectional? Data collection? Comparison group? Sampling strategy? Sample size? Analysis strategy? (inductive vs deductive) Specifics? Bias and limitations Limitations are fine, but must be identified Fill out the bias table at home and let me know if you have any questions Practice data collection session Next week CITI training (don’t worry if you don’t understand everything. And save your certificate!!) Chapter 2 Extra credit for posting an article about a research abuse (past or present) in the piazza area of canvas. If you can’t access piazza, just email it to me and I’ll post. If I can :> 4 basic types of statistical tests: Description •Mean, standard deviation •Median, Mode •Percentage, frequency Correlation •Pearson’s correlation Comparison •Student’s t tests •Chi-square tests •ANOVA •Odds ratios Prediction •OLS regression •Logit regression Review Descriptive statistics describe patterns in your data; does NOT show relationships. Usually univariate, identifies what’s typical and outliers Kinds of description: frequency, central tendency (mean, median, mode); variance (dispersion); standard deviation Correlation-measures relationship between 2 continuous variables. Can be none, positive, negative, curvilinear More review Statistical significance: p=value, confidence intervals T-tests: measure whether there is statistically significant differences in the MEAN of 2 groups; gives you a p-value and a t-statistic. Higher t-statistics = more difference between means ANOVA-like t-tests but compare the means of multiple groups. Gives you a p- value and an f-statistic. Assignment 3 Assumptions Your agency just got a grant to serve your specific population of interest Based on your clinical experience and knowledge of the population, they have asked you to recommend a SPECIFIC intervention (i.e. wrap around services, peer support, foster care, guardianship etc) In order for the grant to get reviewed, the funder is requiring a program evaluation of the intervention (NOT the agency as a whole). You will describe how the intervention should be evaluated in part 4 Identifying a problem, research question, intervention, evaluation research design Social problem • Has real, negative impacts on individuals and society • Is action-able • Is contexualized Intervention • Specifically designed to address the social problem or lessen its negative impacts in your population • Tailored to your population and feasible evaluation • Concretely measure if the intervention is working • Does the intervention effectively address or lessen impacts of the social problem? Midwife example: Prenatal vitamins Theorized to improve outcomes, like low birth weight but no real evidence for this Recorded on birth certificate Often used as a proxy for access to care, emotional investment in pregnancy Can make people feel nauseous I want my clients to take them because my boss wants me to encourage my clients to take them and is pissed when I check no on the birth certificate NOT TAKING THEM IS NOT A SOCIAL PROBLEM. HOWEVER, LACK OF ACCESS TO ADEQUATE NUTRITION FOR PREGNANT PEOPLE IS A SOCIAL PROBLEM-WHY? Things that are not social problems Disability (but ableism is a social problem. Lack of equitable job opportunities and resultant poverty among people with disabilities is a social problem) Non-compliance (what you want a person or group to do may not be what they themselves want to do; from their perspective, not complying is potentially supportive, life-saving, preferred etc) Personal inconvenience or individual goals, such as me personally losing some weight, cuz I don’t want diabetes (but lack of access to veggies in my neighborhood is a social problem, because it means that people in my zip code are collectively less healthy than people in Newton, and inequity is always a social problem) Lack of your intervention your intervention is great, but must be focused on reducing a social problem. Not having it can’t be the social problem People. People are never social problems. The marginalization that people experience based on their identity is a social problem. So people being transgender or Black are not social problems, but the extremely high rate of violence against Black trans people is a social problem. Medical issues/problems. No one likes cancer, but cancer is a medical problem. Like disability, biomedical realities are not SOCIAL problems, though they may cause suffering (i.e. cancer) In other words, social problems are: Inequitable in some way-people are not being treated the same Things/conditions that attempt to limit human dignity, autonomy and human rights Replicate larger systems of oppression (sexism/racism/ableism is real in the world, and in our classrooms) Often about access or systems Have very real consequences that negatively impact individuals and/or groups Are contexualized and avoid generalization (person in the environment) For this assignment, are somewhat actionable (i.e. you are not gonna cure cancer, but you can reduce feelings of isolation among cancer survivors) Examples Prostate cancer screening (public health). People not getting screened as recommended by the ACS by itself isn’t a social problem-no one likes getting their prostate checked. However, men of color are 4X more likely than their white counterparts to not get screened, and therefore their cancers are found later and are deadlier. This inequity is a social problem because early preventable death has huge social impacts. Intellectual and developmental disability. Has existed throughout time. Disability is a normal variation of the human condition. However, adults with intellectual disabilities face social exclusion, isolation, poverty, unemployment and preventable negative health outcomes at much higher rates than their non-disabled peers. Children with autism are more often bullied in school and report higher rates of depression Children with behavioral health problems. See above; normal variation of the human experience. Parents of children with behavioral health problems face stigma and report less support and more parenting stress than their counterparts without behavioral health problems. Parenting stress is associated with child abuse. This disparity in child abuse rates is a social problem because abuse has long lasting impacts on families and communities. Untreated, kids are higher risk than their peers for being incarcerated, which impacts families and communities and is preventable. Non-compliance with treatment plans. What is the goal of the treatment plan? (i.e. prevent involuntary hospitalization, harm to the person or society)-in this case the consequences of not accessing treatment is the social problem. Does the person want to treatment? Do they have a choice? Removal of choice is a social problem that disproportionally affects some groups. The so what test? People of color have less access to cancer screening Children with behavioral health conditions tend to have fewer friends Masshealth does not cover play therapy. Therefore, low income children with autism do not have access to play therapy. Write down your social problem What are the direct harms of the social problem? Does it pass the so-what test? Workshop your social problem with your neighbor When you are both satisfied, write the social problems on a post it Intervention This is your recommendation for how your agency should address the social problem among your population Remember, for this assignment, your agency does NOT yet offer the intervention. So you get to be in charge of how it should be implemented Your intervention should be based on the 3 wheels of evidence based practice Your intervention should have clear, measurable goals that relate directly to the social problem 50 EBPs in social work: integrate these in your intervention Interventions should address or lessen the impacts of the social problem. Examples Mobile screening van to increase access to cancer screening in a community of color to reduce the disparity in screening rates. Measured by increase in # of screenings approaching those of a white community and fewer cancer deaths over time In-home play therapy with kids with autism to teach them strategies for making friends and coping with bullying. Measured by increased use of coping strategies, higher # of friends, higher happiness Make sure you have a clear mechanism of action; a path from point A (your intervention) to point B (your goal) and measurable outcomes More example of intervention definition and goals, measurement Meditation classes for kids with behavioral health challenges and their parents to provide support. Measured by decreased parental stress, increased reported support and use of positive behavior management techniques by parents, less child abuse over time; and by increased use of coping skills by kids, less incarceration over time Designing an app for people with depression to break up large tasks into steps and provide reminders. Measured by client satisfaction with the app, increased feelings of support, self-efficacy for managing life tasks, despite the biological impacts of depression on motivation, increased skills, decreased job loss over time Your intervention Name of intervention How it addresses the social problem What are the goals of this intervention Workshop this with your neighbor When you are both satisfied, write the intervention and it’s goals on a post-it Still with me? Evaluation How will we know that your intervention is working? How will we know it is meeting its goal? What needs to be happening for the intervention to have a chance of meeting its goals? (this is your process evaluation question) What is the outcome (goal) that you need to see happening? (this is your outcome evaluation question) Example evaluation questions Meditation classes for kids with behavioral problems and their parents: Process: Do parents and kids feel welcome at the classes? Do they want to be there? Are they making friends and feeling supported? Are the teachers actually teaching the content (fidelity)? Can they teach? Outcome: Are kids using the meditation skills to manage anger? Do they have fewer negative behaviors? Has parental stress levels changed? Over time, is there a reduction in child abuse rates and/or incarceration rates? Cancer screening van: Process: do people know about the van? Do they trust the van? Are van working treating people well? Outcome: How many people are screened in the van? Does van screening increase the overall % of people in the neighborhood who are screened? Does this increase reduce the screening disparity between PoC and white people? Evaluation questions Must relate back to the goals (which in turn relate to the social problem) Write an evaluation question for each goal. How will you know if the goal is being accomplished? Is this a process or outcome evaluation question? Both? Workshop with your neighbor. Write your evaluation questions down on a post-it note when you are satisfied Study design Brainstorm in your groups about HOW to answer your evaluation questions What study design does your question suggest? At home, continue to think through study design, using your worksheet from last week. EMAIL me with questions, for phone appointments Notes on assignment 2 Overall, excellent and you’re in great shape for assignment 3 Take credit for your clinical experience-name yourself as a source! Integrate this experience into your evaluation of studies-you now what you’re doing! Operationalize, operationalize, operationalize! (what does “successful” or “it worked” mean?) Please use page #s Please proofread Selecting studies Not the first 10 you find! You are evaluating YOUR INTERVENTION, not the causes of your social problem. OK to accept some things as givens (i.e. some kids dislike school, for potentially many reasons, but this is not super relevant to a program to improve grades) Bench science vs social science: the chemistry of how prenatal vitamins are absorbed vs answering “do prenatal vitamins reduce low birthweight?” Be outcome focused! Literature reviews and systematic reviews: describe findings but evaluate how the authors selected and evaluated studies-do you agree or disagree with their methods and analysis? Strengths and limitations Rationale for why the study was done is important but don’t dwell on it. Instead, focus on integrating findings to answer YOUR research question Not agreeing with you is not a limitation-it’s a consideration. Agreeing with you or supporting your desired intervention is also not a strength-it’s a consideration Your task is to evaluate the QUALITY of the evidence, not simply amass it. Mechanism of action and direction of association Which came first? Any confounders? (I didn’t take prenatal vitamin s because I couldn’t afford them (or food) my baby was low birth weight?) Unless the RESEARCH specifically evaluates mechanism of action (how it works) or direction, we just don’t know. Avoid assumption Literature review Organizing and synthesizing information Don’t go down the rabbit hole Audience: intelligent but uninformed about your topic adult (me) The Brilliant Best Friend System For each article, write the MAIN findings/take home messages ONLY on a separate index card as well as the author’s name so you can track it later Once you’ve prepared your index cards, sort them into piles by theme. This is a lot like grounded coding Ideas that are interesting but beyond the scope of your paper go onto a post-it and into your ziplock for later. It’s ok to put index cards into your ziplock, but don’t “disappear” findings that contradict your intervention-these are important to address. Example: Managing disclosures of abuse by research participants with IDD (piles) Higher rates of sexual violence against people with disabilities higher likelihood of enrolling survivors of violence Barriers to reporting abuse Barriers to justice for abuse survivors with IDD (and impact on abuse prevalence) Research as a safe environment for disclosures (or not) Appropriate responses by professionals (not researchers) to disclosures Wildcard but important Ziplock for later: LGBTQ people w/IDD completely absent from literature; impact of unaddressed trauma on “functioning”; representation, visibility and feminism/why this isn’t a feminist issue; historical connection between rape, incest and eugenics movement Further sorting Now that all your literature is in piles, approach each pile and do a further sort. Identify main themes within each pile For each pile and each theme, refer back to your assignment 2 to think about the quality of the evidence. Are there a lot of studies that agree/show the same thing? Take advantage of all the work you already did evaluating the quality of the evidence. Write a summary paragraph for each pile. Provide an outline of main findings ACROSS STUDIES and discuss quality of evidence. Using a bias chart (see handout) will help you assess quality of the evidence. For each pile, is any information missing? (these are the gaps you want to identify) Example: Combining, evaluating, providing selective detail, identifying gaps in each pile Higher rates of sexual violence against people with disabilities higher likelihood of enrolling survivors of violence. COMBINING: There is extensive, high quality evidence showing that people with disabilities experience much higher rates of sexual violence victimization (Swango-Wilson, 2009; McCarthy 2011; McCormack et al, 2006; Jones et al, 2016). SELECTIVE DETAIL: People with intellectual disabilities are even more likely than their counterparts with other kinds of disabilities to experience sexual violence (Horner-Johnson & Drum, 2006). We know that people with ID are consensually participating in research at higher rates than ever before (Ianoco, 2005). GAPS: While common sense suggests that this higher rate of sexual violence victimization means that researchers who enroll people with IDD in their studies are more likely to encounter research participants who have survived sexual violence, this has not yet been studied. Though some preliminary studies suggest that research participants with IDD view participatory action research as an opportunity to reduce sexual violence in their communities (Northrup et al, 2010), we lack understanding of researchers’ role in responding to sexual violence against people with IDD. Let’s try it with 2 articles Using assignment 2, work with a peer to identify themes/categories (piles) and put important findings (only) from your first study onto cards Use your ziplocks as needed to discard interesting but less relevant information for later Switch and help your peer put findings from her first study onto cards and identify themes/categories for her piles Sort your cards into piles If you have time, put findings from your second study onto cards and add these findings to your piles Summarize each pile Buy an index card sorter (or some envelopes) and continue your sorting at home Next week: Highlight on data collection Bring your evaluation questions! We will be designing, doing and evaluating a brief qualitative interview about your evaluation question topic. We will have about 30 minutes for peer review. Bring either your study design outline or your lit review outline Reading: Get started on your CITI training for week 11