EXA

Exa - Management
PLACE ORDER
Answer Part 2) and Part 3) Only. No Outside References. Use the attached lectures and the readings pdf.  Due in 5 hours. PH 30004 Exam 2 Questions on this exam come from class meetings, guest lectures, the visit to the DI Hub, provided videos and assigned chapter readings in Social Science Research, Testing Treatments, the IDEO book on human-centered design, and any other readings in Canvas. The exam is due at the end of the day on Friday, October 22, 2021. Respond to the questions on this page or use a separate document. If you do not use the test itself, please be certain to number your responses. I suggest you respond to every item; Part 1 Multiple Choice – 10 questions; 2 points each Choose the best response 1. In the book Testing Treatments, the authors recommend that in cases where the effect of a treatment is inconsistent or uncertain: a. Practitioners express confidence so patients remain optimistic and trusting b. Practitioners should discuss uncertainty with patients, and encourage further discussion about options, taking patient preferences into account c. Practitioners should avoid any treatments that do not have immediate and clear benefits d. Practitioners should recommend unproven treatments but not tell patients these are unproven as this knowledge might reduce the benefit or cause undue stress. e. This is not actually a problem as uncertain treatments are never legally available to practitioners. 2. In the book Social Science Research, the author argues for use of a/an ____ number of values (choices) in a Likert-type scale because _________ : a. Odd; this allows inclusion of a neutral choice since people may be neutral in their view on something b. Even; this forces respondents to identify whether they agree or disagree c. Larger; this allow participants to express more range in their responses d. Smaller; this helps counter the ordinal nature of these items so improves analysis e. Varying; this helps improve reliability of responses 3. According to information in the IDEO book (“Field Guide to Human-Centered Design”), design mindsets include or encompass: a. Optimism and embracing ambiguity b. Empathy, learning from failure, and creative confidence c. Iterating d. Making things e. A and B f. A, B, C g. A, B, C, and D 4. In simple and explicit terms, p > 0.05 means (see Testing Treatments, chapter 7): a. The researcher has achieved statistical significance b. Given the data provided, the probability of something occurring is .05 \% c. Given the data provided, the probability of something occurring is less than 1 in 20 d. Fail to reject the null hypothesis 5. According to information provided in the video about the Ottawa Charter, action areas to achieve health include: a. Reorienting health services; creating supportive environments; providing national health care; developing personal skills b. Strengthening community action; creating supportive environments; developing personal skills; reorienting health services c. Reorienting health services; creating supportive environments; subsidizing costs for healthy food; subsidizing costs for access to health-promoting physical activity. d. Reorienting health services; developing personal skills; providing national health care; subsidizing costs for healthy food; subsidizing costs for access to health-promoting physical activity. e. None of these is correct 6. In the book Testing Treatments, the authors identify which potential issues that potentially undermine (interfere with) the credibility of a systematic review of prior research: a. Conflicts of interests such as funding provided by a treatment manufacturer b. Planning to publish the review in a high impact journal c. Failure to plan a systematic review using a specific research protocol that specifies eligibility criteria, selection process, and other details d. Failure to identify all of the relevant evidence, such as occurs when unsuccessful trials are not published e. Use of a large research team to conduct a review f. All of the above g. A, B, C and D h. A, C and D 7. Ms. Emily Nelson, doctoral candidate and guest lecturer, demonstrated use of Microsoft Excel to: a. Track time/duration of excerpts of text/meaning units b. Assign descriptions to meaning units c. Categorize meaning units based on participant values or priorities d. Calculate p values from qualitative data transformed into categories e. All of the above f. A, B, and D only g. A, B, and C Only 8. According to information in the IDEO book (“Field Guide to Human-Centered Design”), the three phases of any design challenge include: a. Inspiration, ideation, integration b. Inspiration, implementation, evaluation c. Inspiration, ideation, evaluation d. Integration, evaluation, iteration e. None of these 9. According to the guest lecture during class, what is the biggest weakness or limitation as respects Community Health Assessment (CHA) information for older adults, currently available to the Kent City Health Department: a. Older adults are less likely to use the internet so might not refer to the Kent Health Dept Facebook page b. The Health Dept’s new location at the top of the PARTA Gateway is less convenient than the old location on Main St. c. CHA surveys track adult responses from ages 18 + as a single category d. Surveys, focus group and key informant interviews are all used and it is difficult to take time to analyze the large amount of data 10. In the Youtube video about Needs Analysis, the presenter (Tim Slade) asserts: a. The solution to most problems is to develop a learning or educational intervention b. Results of a needs assessment might show that a learning intervention is not going to address the problem in a given context c. The primary role of a needs assessment is to identify content for a learning or educational intervention d. It is a mistake to build a training course to address an issue without first conducting a needs assessment e. A and C f. B and D Part 2 Short Answer – 4 questions; 4 points each Provide a brief, complete response as requested for each item. Partial credit may be given for partially correct responses. 1. List and describe two differences between the “scientific method” and human-centered design, as used in design innovation methods. 2. You created a survey with 32 items. In one subscale (section) of 16 items, you found respondents provided inconsistent answers (i.e., they agreed with something one time and disagreed with the same thing in a later question). You reduced the length of the subscale to 8 items and found you had more consistent responses. Which potential cause of methods bias was most likely responsible for the inconsistent answers? (refer to the list of items in the article by Podsakoff et al., in Table 2 on page 882). 3. Review the sample wording, following, in italic font, related to research methods. Within this wording, what quality criteria efforts were used or identified by the authors? You should identify at minimum 5 items. You do not have to define or describe what the quality criteria mean. We gathered primary qualitative and quantitative data and integrated analysis of secondary sources. We conducted interviews with 16 purposively selected individuals who were experiencing depression. Interviewees’ mental health was verified through review of their scores on the Beck Depression Inventory (BDI). The BDI has been shown to be reliable and have construct validity in multiple contexts. We additionally administered a newly developed 45-item survey to 300 participants identified by a mental health counseling service provider. Exploratory factor analysis of the results indicated there were three distinct constructs measured in the survey. Item loadings ranged from .21 to .89. Interview transcripts were created by one author and checked for accuracy by another. Surveys were administered in paper format; results were entered into Microsoft Excel by a research assistant. Following, a random sample of 25\% of data points was checked for accuracy by one of the authors. 4. Describe an example where a measurement, such as a survey item, might be reliable but not accurate. (Refer to Chapter 7 in Social Science Research for helpful information). Part 3 Longer Response – 2 questions; 7 points each. Please read the instructions carefully. Partial credit is available for partially correct responses. 1. . Use the design method “banned” – where you imagine “a world if a product, service system or experience did not exist and how people would possibly adapt” (Curedale, 2013, p. 52) and, in 3-4 sentences of bullet points, describe what the consequences might be if there was no government or private collection of garbage and recyclable items available. 2. Assume you have been asked to conduct a community needs assessment related to health needs of older adults living in Kent, Ohio. Provide a response to these items: a. How would you define older adults? What criteria or sources would you use to develop this definition? What possible problems might arise from using a specific age or age range to define older adults? b. How would you find out how many older adults are in Kent, Ohio? c. If you wanted to recruit participants for interview or survey research, what are three methods or sources, other than social media, you might use to communicate with older adults in Kent, Ohio? PH 30004 Week 4 September 20, 2021 Update/schedule Exams Card Data gathering/analysis assignment due 9/29 – no class meeting Will review today Secondary data – in class today Qualtrics – variations in questions – in class Friday Kent Health Dept speaker Monday 9/27 DI Hub Friday 10/1 Miro – invitations sent – explore as you like; class time eventually MS Teams – plan to use in class at some point for screen sharing Making questions into researchable questions What is the relationship between sleep and mental health? What is the relationship between online school and mental health? What is the correlation between gut health and mental health? What is the impact of COVID-19 on mental health? Does marijuana usage have an affect on physical health? Is working from home responsible for people’s weight gains during the COVID-19 pandemic? Precision and measurability Designing informational/interview questions Finding data Working through one example How do people perceive their risk of injury due to motor vehicle accidents? People – who – anyone, or just people who drive? If people who drive, how much or under what circumstances? Comparing groups leads to a different question but might be interesting Risk, risk perception – defined how? Is there a risk perception instrument available for use? Injury – any injury, any physical injury, or another definition? Motor vehicle – cars, e bikes, others? Accident – collision? Single car accident? Figure out what terms need to be defined, described, or limited Decide on specificity Other research Other resources (medical, government/census) – generally use scientific or academic, not informal language Identify the areas of question Share Time permitting – revise the question Update Kent HD – Monday a.m. No class 9/29 – assignment due DI Hub Friday at 10 Today Qualtrics activity Introduce ICPRS extra credit Review assignment details Due 9/29/2021 – end of day. I will pre-review your draft and respond to your questions! ICPSR https://www.icpsr.umich.edu/web/pages/instructors/icsc/ Read (or skim) about the project/introduction/social capital/datasets Go to exercises – search for the variable: CLUBMEET in the DDB and LEADGRP in the GSS Screenshot the frequency distribution table for each See Canvas for more Qualtrics Demo DI survey plus skip logic Qualtrics Go to Qualtrics Create a survey – alone or in groups Use one of these questions to guide your survey What is the impact of COVID-19 on depression? Does marijuana usage have an affect on back pain? What is the relationship between sleep and anxiety? Create 4 different types of questions – ex. Multiple choice is one type; text entry is another (optional) use one instance of skip logic Share question types with class – see how many unique types there are PH 30004 Fall Semester 2021 Week of October 4 Sequence Review schedule changes Content and assignment due dates Review, recap, and contextualization of content to date Begin measurement and evaluation Innovation in research design (Wed, next week if needed) Schedule - important dates and changes October 8, 2021 – guest speaker KHD Wed, October 13, 2021 – new due date for next assignment Review details on 10/06 October 15, 2021 – no class meeting Fall break October 22, 2021 – Exam 2 (take home) November 12, 2021 – Exam 3 (take home) November 22, 2021 – new due date for assignment December 10, 2021 – final proposal December 17, 10:15 – 12:30 – final exam (take home) Review, recap, contextualization PH research is: examination or study Of hypotheses, premises, or theories, Conducted by scientists, scholars and lay persons, through use of systematic processes or tests to identify discoveries, develop solutions, and derive conclusions That are credible and useful For knowledge and benefit of communities, scientists, academics, professionals and other impacted populations Review, recap, contextualization Definition developed by a content analysis process Process to identify relevant terms: assembling data Clustering similar terms and naming clusters: data analysis via category development or thematic analysis, using provided/ a priori categories Agreement via voting on best terms: interpretation via arriving at consensus based on individual expertise/preferences Transformation of terms into narrative text: presentation of findings Review, recap, contextualization Review of common and useful research designs Trial and experimental designs Useful when comparing a treatment to an alternative Applicable for human science and other interests (e.g., working with samples or non-humans) Difficult to identify and remove other influences; limiting generalizability Efficacy focused so tend to de-emphasize behavioral/individual aspects Field surveys and focus groups Useful to gather data from many responses Can quickly key in on priorities or trends Online survey programs do data analysis (surveys) Difficult to obtain random sample so difficult to generalize Results can be skewed – bots (surveys), groups that attempt to engineer responses (surveys or FGs), dominant individuals (FGs) Review, recap, contextualization Case research or ethnographic methods Can capture in depth detail about best, worst, or middle performing systems or entities Time and labor intensive Generalizability limited to similar contexts Action research More likely to address participant-identified priorities These may differ from researcher priorities In conflict with scholarly or funding aims of research - interest in ongoing improvement rather than specific findings Review, recap, contextualization Secondary data analysis Researcher or funder-driven Incentives to work with and promote some datasets Convenient sometimes instant way to conduct longitudinal/panel research Large, random and/or population-based data Review, recap, contextualization Secondary data analysis Not necessarily likely to address one’s specific interest, requiring revision of research questions or aims Access may be difficult or limited Data are rarely current; typically 18 months or older Systematic reviews, meta-summaries, and meta-analyses are also types of secondary data analysis – data are results of prior research Review, recap, contextualization Priority on most frequent responses Priority on treatments or solutions that are: Convenient or Profit-making or Trendy or Currently among government/agency funding priorities Cost money to carry out Major contributors of published research: Students doing doctoral dissertations Faculty seeking tenure Review, recap, contextualization Research in generaL Short duration projects (10 – 30 weeks) are most appealing Emphasis on problem identification rather than development of solutions Design innovation Aspects of action/community-based research Human science methods Adds innovative processes, i.e., variations on brainstorming Prioritizes sustainability, even circular design Emphasis on development of solutions Review, recap, contextualization Data – qualitative, quantitative, mixed/blended Associated with some designs Field survey – fixed responses Focus group – interview recordings or transcripts Trial – scores or efficacy measures Transformed data Quantitizing – words to categories Mean centering; logarithmic transformations Additional data types can be incorporated into most designs Counter small sample size Address ambiguous results Explain unexpected findings Review, recap, contextualization Research design is the framework for answering your question or addressing your interest Data are the bits of information, assembled within the framework, which undergo processes of compilation, combination, aggregation, transformation, and interpretation to provide an integrated and/or consensus response Design should be focused and adequate to answer the question Data should be sufficient and credible enough to answer the question Data and design should be coherent Schedule - important dates and changes October 8, 2021 – guest speaker KHD Wed, October 13, 2021 – new due date for next assignment Review details on 10/06 No class meeting October 15, 2021 – no class meeting Fall break October 22, 2021 – Exam 2 (take home) November 12, 2021 – Exam 3 (take home) November 22, 2021 – new due date for assignment Measurement Biological/physiological Blood pressure Heart rate Absence/presence of disease Challenges? Consistency Other factors (“white coat” syndrome) False positives, false negatives Interpretation Measurement Knowledge Symptoms Health literacy Challenges? Interpretation, i.e., what constitutes irregular shape or change in appearance? Language – what does “usually” or “often” mean? What is the difference between “agree,” “somewhat agree,” and “strongly agree”? ”Good” health compared to what? Measurement Traits, concepts, constructs and other subjectively defined things Self-report Diagnostic contributions (observation and results of interviews, e.g., intake survey for mental health treatment) Challenges No universally agreed upon definition for complex constructs Many are context-specific Many are individual-specific Many are both context and individual-specific Measurement Standards Measurement Mathematical Reliability Validity Interpretive Saturation Validity Consensus coding Quality Checking entry/checking transcripts Assessment of normality Model fit indices Member checking Interrater reliability Quality efforts during scale development Factor analysis – exploratory and confirmatory Identify all potential items Administer survey Use mathematical/statistical methods to cluster scores Look at clusters and assign a name (similar to thematic analysis) Delphi / expert panels Identify items OR screen and comment on items Add, delete, modify items Rank and score items Approve final instrument or measurement process https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-018-5638-8 Examples of several methods used to develop a scale Common methods bias List of issues Remedies Multi-method and multi-trait Comparative case studies Triangulation Multiple analysts Useful information for assessing community needs https://www.youtube.com/watch?v=G2quVLcJVBk This video ties together individual health promotion, public health, and healthcare in an holistic approach to needs assessment https://www.youtube.com/watch?v=-gIpE8kxFPk This video presents a common sense discussion about how to determine whether needs will be met through informational or educational programming Assessing Community Health: Collecting Data, Sampling, and Getting Accurate Data Mike Anguilano, Accreditation Coordiantor/Public Information Officer Kent City Health Department September 27, 2021 Collecting Data at the State Level State health departments must do a State Health Assessment (SHA) and a corresponding State Health Improvement Plan (SHIP) Involves surveys, focus groups, key informant interviews, and other community events to collect qualitative and quantitative data From the results, the state determines focal points for local health departments (LHDs) Local health departments must align their goals and programs to be in sync with the SHA findings Collecting Data at the Local Level LHDs planning to pursue accreditation (mandatory in Ohio) must have a community health assessment (CHA) completed every three years In Portage County, the CHA steering committee and community partners gather to create a survey with core CDC-required questions and county-specific questions Three surveys are created for each age group (0-11, 12-17, 18+) Cross-sectional randomized sampling conducted in the county Every school district participates in the survey Kent State students are not included in the CHA – why? Kent and the Community Health Assessment Kent is the largest city in Portage County – for 10 out of 12 months Very transient population makes it difficult to get representative results for the county KSU students can skew the results The Kent population turns over every year - this makes it tough to determine what the actual problems are For the 2016 CHA, Kent was oversampled to get a more comprehensive overview of the city Kent State University holds its own college health assessment, though it is not as robust as the county one Other Forms of Data Collection While CHAs are the biggest public health endeavor in the county, they are not the only time to collect data In recent years KCHD has taken steps to collect more primary data specific to the City of Kent: Held focus groups relating to LGBTQ health care access, tobacco control, COVID-19 (virtual), and vaccinations (pre-COVID) Utilize surveys for program development, post-program assessment, and community perceptions Key informant interviews – speak with key personnel or potential partners on projects and programs; specific questions for a select group of people What are the Best Methods for Data Collection? Depends on several factors What is the goal/what will you do with the data? How much data do you need to collect/what is a representative sample? What kind of data will you need to collect? (qualitative, quantitative) Is there a budget or limited number of resources to collect the data? Who is the target audience? (ex. Older adults, youth, certain cultures or ethnicities) Not really any right or wrong methods, but some are better than others and utilize resources better Determining, and Reaching, Your Audience Regardless of which method used, you need to consider the target audience and barriers in reaching them Know your audience Consider technological, cultural, age, ethnic, transportation, and socioeconomic barriers when choosing a sampling/data collection method Plan for attrition in the sampling process – if you send out 30 surveys, plan on getting 15 back Ethical considerations for research purposes must be followed This is especially important for children, older adults, disadvantaged/disenfranchised, and disabled individuals Review Sampling is important – you may only get one opportunity Data collection is an ongoing process and oftentimes slow - Rome was not built in a day and neither will your research project There are a lot of things to consider with regards to barriers – don’t hamstring your project early on, as making changes can be difficult Try not to lose sight of your goal – why are you sampling a certain population? What is the ultimate goal? Consider your time, money, and other resources – they go quickly if not used properly Questions? Contact Info: Mike Anguilano III, BSPH Accreditation Coordinator/Public Information Officer 330-678-8109 ext. 5204 PLoS Medicine | www.plosmedicine.org 0696 Essay Open access, freely available online August 2005 | Volume 2 | Issue 8 | e124 Published research fi ndings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies [1–3] to the most modern molecular research [4,5]. There is increasing concern that in modern research, false fi ndings may be the majority or even the vast majority of published research claims [6–8]. However, this should not be surprising. It can be proven that most claimed research fi ndings are false. Here I will examine the key factors that infl uence this problem and some corollaries thereof. Modeling the Framework for False Positive Findings Several methodologists have pointed out [9–11] that the high rate of nonreplication (lack of confi rmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research fi ndings solely on the basis of a single study assessed by formal statistical signifi cance, typically for a p-value less than 0.05. Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values. Research fi ndings are defi ned here as any relationship reaching formal statistical signifi cance, e.g., effective interventions, informative predictors, risk factors, or associations. “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null fi ndings. As has been shown previously, the probability that a research fi nding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical signifi cance [10,11]. Consider a 2 × 2 table in which research fi ndings are compared against the gold standard of true relationships in a scientifi c fi eld. In a research fi eld both true and false hypotheses can be made about the presence of relationships. Let R be the ratio of the number of “true relationships” to “no relationships” among those tested in the fi eld. R is characteristic of the fi eld and can vary a lot depending on whether the fi eld targets highly likely relationships or searches for only one or a few true relationships among thousands and millions of hypotheses that may be postulated. Let us also consider, for computational simplicity, circumscribed fi elds where either there is only one true relationship (among many that can be hypothesized) or the power is similar to fi nd any of the several existing true relationships. The pre-study probability of a relationship being true is R⁄(R + 1). The probability of a study fi nding a true relationship refl ects the power 1 − β (one minus the Type II error rate). The probability of claiming a relationship when none truly exists refl ects the Type I error rate, α. Assuming that c relationships are being probed in the fi eld, the expected values of the 2 × 2 table are given in Table 1. After a research fi nding has been claimed based on achieving formal statistical signifi cance, the post-study probability that it is true is the positive predictive value, PPV. The PPV is also the complementary probability of what Wacholder et al. have called the false positive report probability [10]. According to the 2 × 2 table, one gets PPV = (1 − β)R⁄(R − βR + α). A research fi nding is thus The Essay section contains opinion pieces on topics of broad interest to a general medical audience. Why Most Published Research Findings Are False John P. A. Ioannidis Citation: Ioannidis JPA (2005) Why most published research fi ndings are false. PLoS Med 2(8): e124. Copyright: © 2005 John P. A. Ioannidis. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abbreviation: PPV, positive predictive value John P. A. Ioannidis is in the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America. E-mail: [email protected] Competing Interests: The author has declared that no competing interests exist. DOI: 10.1371/journal.pmed.0020124 Summary There is increasing concern that most current published research fi ndings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientifi c fi eld. In this framework, a research fi nding is less likely to be true when the studies conducted in a fi eld are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater fl exibility in designs, defi nitions, outcomes, and analytical modes; when there is greater fi nancial and other interest and prejudice; and when more teams are involved in a scientifi c fi eld in chase of statistical signifi cance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientifi c fi elds, claimed research fi ndings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research. It can be proven that most claimed research fi ndings are false. PLoS Medicine | www.plosmedicine.org 0697 more likely true than false if (1 − β)R > α. Since usually the vast majority of investigators depend on α = 0.05, this means that a research fi nding is more likely true than false if (1 − β)R > 0.05. What is less well appreciated is that bias and the extent of repeated independent testing by different teams of investigators around the globe may further distort this picture and may lead to even smaller probabilities of the research fi ndings being indeed true. We will try to model these two factors in the context of similar 2 × 2 tables. Bias First, let us defi ne bias as the combination of various design, data, analysis, and presentation factors that tend to produce research fi ndings when they should not be produced. Let u be the proportion of probed analyses that would not have been “research fi ndings,” but nevertheless end up presented and reported as such, because of bias. Bias should not be confused with chance variability that causes some fi ndings to be false by chance even though the study design, data, analysis, and presentation are perfect. Bias can entail manipulation in the analysis or reporting of fi ndings. Selective or distorted reporting is a typical form of such bias. We may assume that u does not depend on whether a true relationship exists or not. This is not an unreasonable assumption, since typically it is impossible to know which relationships are indeed true. In the presence of bias (Table 2), one gets PPV = ([1 − β]R + uβR)⁄(R + α − βR + u − uα + uβR), and PPV decreases with increasing u, unless 1 − β ≤ α, i.e., 1 − β ≤ 0.05 for most situations. Thus, with increasing bias, the chances that a research fi nding is true diminish considerably. This is shown for different levels of power and for different pre-study odds in Figure 1. Conversely, true research fi ndings may occasionally be annulled because of reverse bias. For example, with large measurement errors relationships are lost in noise [12], or investigators use data ineffi ciently or fail to notice statistically signifi cant relationships, or there may be confl icts of interest that tend to “bury” signifi cant fi ndings [13]. There is no good large-scale empirical evidence on how frequently such reverse bias may occur across diverse research fi elds. However, it is probably fair to say that reverse bias is not as common. Moreover measurement errors and ineffi cient use of data are probably becoming less frequent problems, since measurement error has decreased with technological advances in the molecular era and investigators are becoming increasingly sophisticated about their data. Regardless, reverse bias may be modeled in the same way as bias above. Also reverse bias should not be confused with chance variability that may lead to missing a true relationship because of chance. Testing by Several Independent Teams Several independent teams may be addressing the same sets of research questions. As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation. An increasing number of questions have at least one study claiming a research fi nding, and this receives unilateral attention. The probability that at least one study, among several done on the same question, claims a statistically signifi cant research fi nding is easy to estimate. For n independent studies of equal power, the 2 × 2 table is shown in Table 3: PPV = R(1 − βn)⁄(R + 1 − [1 − α]n − Rβn) (not considering bias). With increasing number of independent studies, PPV tends to decrease, unless 1 − β < α, i.e., typically 1 − β < 0.05. This is shown for different levels of power and for different pre-study odds in Figure 2. For n studies of different power, the term βn is replaced by the product of the terms β i for i = 1 to n, but inferences are similar. Corollaries A practical example is shown in Box 1. Based on the above considerations, one may deduce several interesting corollaries about the probability that a research fi nding is indeed true. Corollary 1: The smaller the studies conducted in a scientifi c fi eld, the less likely the research fi ndings are to be true. Small sample size means smaller power and, for all functions above, the PPV for a true research fi nding decreases as power decreases towards 1 − β = 0.05. Thus, other factors being equal, research fi ndings are more likely true in scientifi c fi elds that undertake large studies, such as randomized controlled trials in cardiology (several thousand subjects randomized) [14] than in scientifi c fi elds with small studies, such as most research of molecular predictors (sample sizes 100- fold smaller) [15]. Corollary 2: The smaller the effect sizes in a scientifi c fi eld, the less likely the research fi ndings are to be true. Power is also related to the effect size. Thus research fi ndings are more likely true in scientifi c fi elds with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20), than in scientifi c fi elds where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1–1.5) [7]. Modern epidemiology is increasingly obliged to target smaller Table 1. Research Findings and True Relationships Research Finding True Relationship Yes No Total Yes c(1 − β)R/(R + 1) cα/(R + 1) c(R + α − βR)/(R + 1) No cβR/(R + 1) c(1 − α)/(R + 1) c(1 − α + βR)/(R + 1) Total cR/(R + 1) c/(R + 1) c DOI: 10.1371/journal.pmed.0020124.t001 Table 2. Research Findings and True Relationships in the Presence of Bias Research Finding True Relationship Yes No Total Yes (c[1 − β]R + ucβR)/(R + 1) cα + uc(1 − α)/(R + 1) c(R + α − βR + u − uα + uβR)/(R + 1) No (1 − u)cβR/(R + 1) (1 − u)c(1 − α)/(R + 1) c(1 − u)(1 − α + βR)/(R + 1) Total cR/(R + 1) c/(R + 1) c DOI: 10.1371/journal.pmed.0020124.t002 August 2005 | Volume 2 | Issue 8 | e124 PLoS Medicine | www.plosmedicine.org 0698 effect sizes [16]. Consequently, the proportion of true research fi ndings is expected to decrease. In the same line of thinking, if the true effect sizes are very small in a scientifi c fi eld, this fi eld is likely to be plagued by almost ubiquitous false positive claims. For example, if the majority of true genetic or nutritional determinants of complex diseases confer relative risks less than 1.05, genetic or nutritional epidemiology would be largely utopian endeavors. Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientifi c fi eld, the less likely the research fi ndings are to be true. As shown above, the post-study probability that a fi nding is true (PPV) depends a lot on the pre-study odds (R). Thus, research fi ndings are more likely true in confi rmatory designs, such as large phase III randomized controlled trials, or meta-analyses thereof, than in hypothesis-generating experiments. Fields considered highly informative and creative given the wealth of the assembled and tested information, such as microarrays and other high-throughput discovery- oriented research [4,8,17], should have extremely low PPV. Corollary 4: The greater the fl exibility in designs, defi nitions, outcomes, and analytical modes in a scientifi c fi eld, the less likely the research fi ndings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results, i.e., bias, u. For several research designs, e.g., randomized controlled trials [18–20] or meta-analyses [21,22], there have been efforts to standardize their conduct and reporting. Adherence to common standards is likely to increase the proportion of true fi ndings. The same applies to outcomes. True fi ndings may be more common when outcomes are unequivocal and universally agreed (e.g., death) rather than when multifarious outcomes are devised (e.g., scales for schizophrenia outcomes) [23]. Similarly, fi elds that use commonly agreed, stereotyped analytical methods (e.g., Kaplan- Meier plots and the log-rank test) [24] may yield a larger proportion of true fi ndings than fi elds where analytical methods are still under experimentation (e.g., artifi cial intelligence methods) and only “best” results are reported. Regardless, even in the most stringent research designs, bias seems to be a major problem. For example, there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem even for randomized trails [25]. Simply abolishing selective publication would not make this problem go away. Corollary 5: The greater the fi nancial and other interests and prejudices in a scientifi c fi eld, the less likely the research fi ndings are to be true. Confl icts of interest and prejudice may increase bias, u. Confl icts of interest are very common in biomedical research [26], and typically they are inadequately and sparsely reported [26,27]. Prejudice may not necessarily have fi nancial roots. Scientists in a given fi eld may be prejudiced purely because of their belief in a scientifi c theory or commitment to their own fi ndings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifi cations for promotion or tenure. Such nonfi nancial confl icts may also lead to distorted reported results and interpretations. Prestigious investigators may suppress via the peer review process the appearance and dissemination of fi ndings that refute their fi ndings, thus condemning their fi eld to perpetuate false dogma. Empirical evidence on expert opinion shows that it is extremely unreliable [28]. Corollary 6: The hotter a scientifi c fi eld (with more scientifi c teams involved), the less likely the research fi ndings are to be true. This seemingly paradoxical corollary follows because, as stated above, the PPV of isolated fi ndings decreases when many teams of investigators are involved in the same fi eld. This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fi elds that draw wide attention. With many teams working on the same fi eld and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly Table 3. Research Findings and True Relationships in the Presence of Multiple Studies Research Finding True Relationship Yes No Total Yes cR(1 − βn)/(R + 1) c(1 − [1 − α]n)/(R + 1) c(R + 1 − [1 − α]n − Rβn)/(R + 1) No cRβn/(R + 1) c(1 − α)n/(R + 1) c([1 − α]n + Rβn)/(R + 1) Total cR/(R + 1) c/(R + 1) c DOI: 10.1371/journal.pmed.0020124.t003 DOI: 10.1371/journal.pmed.0020124.g001 Figure 1. PPV (Probability That a Research Finding Is True) as a Function of the Pre-Study Odds for Various Levels of Bias, u Panels correspond to power of 0.20, 0.50, and 0.80. August 2005 | Volume 2 | Issue 8 | e124 PLoS Medicine | www.plosmedicine.org 0699 alternating extreme research claims and extremely opposite refutations [29]. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics [29]. These corollaries consider each factor separately, but these factors often infl uence each other. For example, investigators working in fi elds where true effect sizes are perceived to be small may be more likely to perform large studies than investigators working in fi elds where true effect sizes are perceived to be large. Or prejudice may prevail in a hot scientifi c fi eld, further undermining the predictive value of its research fi ndings. Highly prejudiced stakeholders may even create a barrier that aborts efforts at obtaining and disseminating opposing results. Conversely, the fact that a fi eld is hot or has strong invested interests may sometimes promote larger studies and improved standards of research, enhancing the predictive value of its research fi ndings. Or massive discovery- oriented testing may result in such a large yield of signifi cant relationships that investigators have enough to report and search further and thus refrain from data dredging and manipulation. Most Research Findings Are False for Most Research Designs and for Most Fields In the described framework, a PPV exceeding 50\% is quite diffi cult to get. Table 4 provides the results of simulations using the formulas developed for the infl uence of power, ratio of true to non-true relationships, and bias, for various types of situations that may be characteristic of specifi c study designs and settings. A fi nding from a well-conducted, adequately powered randomized controlled trial starting with a 50\% pre-study chance that the intervention is effective is eventually true about 85\% of the time. A fairly similar performance is expected of a confi rmatory meta-analysis of good-quality randomized trials: potential bias probably increases, but power and pre-test chances are higher compared to a single randomized trial. Conversely, a meta-analytic fi nding from inconclusive studies where pooling is used to “correct” the low power of single studies, is probably false if R ≤ 1:3. Research fi ndings from underpowered, early-phase clinical trials would be true about one in four times, or even less frequently if bias is present. Epidemiological studies of an exploratory nature perform even worse, especially when underpowered, but even well-powered epidemiological studies may have only a one in fi ve chance being true, if R = 1:10. Finally, in discovery-oriented research with massive testing, where tested relationships exceed true ones 1,000- fold (e.g., 30,000 genes tested, of which 30 may be the true culprits) [30,31], PPV for each claimed relationship is extremely low, even with considerable Box 1. An Example: Science at Low Pre-Study Odds Let us assume that a team of investigators performs a whole genome association study to test whether any of 100,000 gene polymorphisms are associated with susceptibility to schizophrenia. Based on what we know about the extent of heritability of the disease, it is reasonable to expect that probably around ten gene polymorphisms among those tested would be truly associated with schizophrenia, with relatively similar odds ratios around 1.3 for the ten or so polymorphisms and with a fairly similar power to identify any of them. Then R = 10/100,000 = 10−4, and the pre-study probability for any polymorphism to be associated with schizophrenia is also R/(R + 1) = 10−4. Let us also suppose that the study has 60\% power to fi nd an association with an odds ratio of 1.3 at α = 0.05. Then it can be estimated that if a statistically signifi cant association is found with the p-value barely crossing the 0.05 threshold, the post-study probability that this is true increases about 12-fold compared with the pre-study probability, but it is still only 12 × 10−4. Now let us suppose that the investigators manipulate their design, analyses, and reporting so as to make more relationships cross the p = 0.05 threshold even though this would not have been crossed with a perfectly adhered to design and analysis and with perfect comprehensive reporting of the results, strictly according to the original study plan. Such manipulation could be done, for example, with serendipitous inclusion or exclusion of certain patients or controls, post hoc subgroup analyses, investigation of genetic contrasts that were not originally specifi ed, changes in the disease or control defi nitions, and various combinations of selective or distorted reporting of the results. Commercially available “data mining” packages actually are proud of their ability to yield statistically signifi cant results through data dredging. In the presence of bias with u = 0.10, the post- study probability that a research fi nding is true is only 4.4 × 10−4. Furthermore, even in the absence of any bias, when ten independent research teams perform similar experiments around the world, if one of them fi nds a formally statistically signifi cant association, the probability that the research fi nding is true is only 1.5 × 10−4, hardly any higher than the probability we had before any of this extensive research was undertaken! DOI: 10.1371/journal.pmed.0020124.g002 Figure 2. PPV (Probability That a Research Finding Is True) as a Function of the Pre-Study Odds for Various Numbers of Conducted Studies, n Panels correspond to power of 0.20, 0.50, and 0.80. August 2005 | Volume 2 | Issue 8 | e124 PLoS Medicine | www.plosmedicine.org 0700 standardization of laboratory and statistical methods, outcomes, and reporting thereof to minimize bias. Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias As shown, the majority of modern biomedical research is operating in areas with very low pre- and post- study probability for true fi ndings. Let us suppose that in a research fi eld there are no true fi ndings at all to be discovered. History of science teaches us that scientifi c endeavor has often in the past wasted effort in fi elds with absolutely no yield of true scientifi c information, at least based on our current understanding. In such a “null fi eld,” one would ideally expect all observed effect sizes to vary by chance around the null in the absence of bias. The extent that observed fi ndings deviate from what is expected by chance alone would be simply a pure measure of the prevailing bias. For example, let us suppose that no nutrients or dietary patterns are actually important determinants for the risk of developing a specifi c tumor. Let us also suppose that the scientifi c literature has examined 60 nutrients and claims all of them to be related to the risk of developing this tumor with relative risks in the range of 1.2 to 1.4 for the comparison of the upper to lower intake tertiles. Then the claimed effect sizes are simply measuring nothing else but the net bias that has been involved in the generation of this scientifi c literature. Claimed effect sizes are in fact the most accurate estimates of the net bias. It even follows that between “null fi elds,” the fi elds that claim stronger effects (often with accompanying claims of medical or public health importance) are simply those that have sustained the worst biases. For fi elds with very low PPV, the few true relationships would not distort this overall picture much. Even if a few relationships are true, the shape of the distribution of the observed effects would still yield a clear measure of the biases involved in the fi eld. This concept totally reverses the way we view scientifi c results. Traditionally, investigators have viewed large and highly signifi cant effects with excitement, as signs of important discoveries. Too large and too highly signifi cant effects may actually be more likely to be signs of large bias in most fi elds of modern research. They should lead investigators to careful critical thinking about what might have gone wrong with their data, analyses, and results. Of course, investigators working in any fi eld are likely to resist accepting that the whole fi eld in which they have spent their careers is a “null fi eld.” However, other lines of evidence, or advances in technology and experimentation, may lead eventually to the dismantling of a scientifi c fi eld. Obtaining measures of the net bias in one fi eld may also be useful for obtaining insight into what might be the range of bias operating in other fi elds where similar analytical methods, technologies, and confl icts may be operating. How Can We Improve the Situation? Is it unavoidable that most research fi ndings are false, or can we improve the situation? A major problem is that it is impossible to know with 100\% certainty what the truth is in any research question. In this regard, the pure “gold” standard is unattainable. However, there are several approaches to improve the post-study probability. Better powered evidence, e.g., large studies or low-bias meta-analyses, may help, as it comes closer to the unknown “gold” standard. However, large studies may still have biases and these should be acknowledged and avoided. Moreover, large-scale evidence is impossible to obtain for all of the millions and trillions of research questions posed in current research. Large-scale evidence should be targeted for research questions where the pre-study probability is already considerably high, so that a signifi cant research fi nding will lead to a post-test probability that would be considered quite defi nitive. Large-scale evidence is also particularly indicated when it can test major concepts rather than narrow, specifi c questions. A negative fi nding can then refute not only a specifi c proposed claim, but a whole fi eld or considerable portion thereof. Selecting the performance of large-scale studies based on narrow-minded criteria, such as the marketing promotion of a specifi c drug, is largely wasted research. Moreover, one should be cautious that extremely large studies may be more likely to fi nd a formally statistical signifi cant difference for a trivial effect that is not really meaningfully different from the null [32–34]. Second, most research questions are addressed by many teams, and it is misleading to emphasize the statistically signifi cant fi ndings of any single team. What matters is the Table 4. PPV of Research Findings for Various Combinations of Power (1 − β), Ratio of True to Not-True Relationships (R), and Bias (u) 1 … PH 30004 Fall 2021 Oct 11 Schedule reminders No meeting on Oct 13 – assignment due end of day No meeting Oct 15 – Fall break Exam #2 Friday Oct 22 (study guide provided on Oct 18) – take home exam; no class meeting Today – DI activity; brief lecture on human-centered design as compared to “scientific method” Activity Questions: What is the relationship between sleep and mental health What is the relationship between online school and mental health What is the correlation between gut health and mental health What is the impact of COVID-19 on mental health Does marijuana usage have an affect on physical health Is working form home responsible for people’s weight gains during the COVID-19 pandemic Activity Questions as problems: What is the relationship between sleep and mental health What is the relationship between online school and mental health What is the correlation between gut health and mental health What is the impact of COVID-19 on mental health Does marijuana usage have an affect on physical health Is working form home responsible for people’s weight gains during the COVID-19 pandemic Scientific method v human centered Emphasis on what is “known” Consult expert sources Bias reduction Hierarchy of methods Linear process Key concern: is it correct? Emphasis on people’s needs Consult impacted people Prioritize preference and practicality Creative, emergent methods Non-linear process (iterative, circular, etc.) Key concern: does it help? Scientific method v human centered Framed within postpositivism – there is/are, out there, correct answers or solutions, that can be identified through systematic exploration, and assessing and rejecting possibilities.* The end goal is identification of the answer/solution. While context specific modifications are acknowledged, revision is typically triggered by system failure and is a reactive process *standards for rejection are clear while standards for acceptance are less clear Framed within social constructivism and pragmatism – the best solutions are useful, will be used, and are resource effective and sustainable. The need to modify processes to adapt for changes in contexts and to accommodate new innovations is expected; the DI process does not end 1774867819059 ISBN 978-1-905177-48-6 51795 > ConneCt with this book HEALTH/MEDICINE www.pinterandmartin.com UK £9.99 US $17.95 Recommended retail price Published by Pinter & Martin Ltd Cover design by Klor For more great books visit pinterandmartin.com Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou Better research for Better healthcare foreword by Ben Goldacre — author of Bad Science SECond EdITIon SECond EdITIon TESTIn G Tr Ea Tm En TS Im o g en Evans, H azel Tho rnto n, Iain C halm ers and P aul G laszio u How do we know whether a particular treatment really works? How reliable is the evidence? And how do we ensure that research into medical treatments best meets the needs of patients? These are just a few of the questions addressed in a lively and informative way in Testing Treatments. Brimming with vivid examples, Testing Treatments will inspire both patients and professionals. Building on the success of the first edition, Testing Treatments has now been extensively revised and updated. The second edition includes a thought- provoking account of screening, explaining how early diagnosis is not always better, and a new chapter exploring how over-regulation of research can work against the best interests of patients. Another new chapter shows how robust evidence from research can shape the practice of healthcare in ways that allow treatment decisions to be reached jointly by patients and clinicians. Testing Treatments urges everyone to get involved in improving current research and future treatment, and outlines practical steps that patients and doctors can take together. Better research for Better healthcare tt_cover_135x216x16_3.indd 1 07/09/2011 16:24 To buy the paperback edition of Testing Treatments, please visit the Pinter & Martin website at www.pinterandmartin.com Enter the code TT25 at checkout to get 25\% off and free UK p&p on all Pinter & Martin titles Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou Foreword Ben Goldacre We dedicate this book to William Silverman (1917–2004), who encouraged us repeatedly to challenge authority. Testing Treatments Better Research for Better Healthcare First published in 2006 by The British Library This second edition first published 2011 by Pinter & Martin Ltd Copyright © 2011 Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou Foreword © 2011 Ben Goldacre Foreword to the first edition © 2006 Nick Ross The authors have asserted their moral right to be identified as the authors of this work in accordance with the Copyright, Designs and Patents Act of 1988 All rights reserved British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 978-1-905177-48-6 This book is sold subject to the condition that it shall not, by way of trade and otherwise, be lent, resold, hired out, or otherwise circulated without the publisher’s prior consent in any form or binding or cover other than that in which it is published and without a similar condition being imposed on the subsequent purchaser Printed and bound in Great Britain by TJ International Ltd., Padstow, Cornwall This book has been printed on paper that is sourced and harvested from sustainable forests and is FSC accredited Pinter & Martin Ltd 6 Effra Parade London SW2 1PS www.pinterandmartin.com Testing Treatments Interactive: www.testingtreatments.org v Contents About the authors vi Acknowledgements vii Foreword by Ben Goldacre ix Foreword to the first edition by Nick Ross xiii Preface xvii Introduction xix 1 New – but is it better? 1 2 Hoped-for effects that don’t materialize 13 3 More is not necessarily better 21 4 Earlier is not necessarily better 31 5 Dealing with uncertainty about the effects of treatments 50 6 Fair tests of treatments 64 7 Taking account of the play of chance 85 8 Assessing all the relevant, reliable evidence 92 9 Regulating tests of treatments: help or hindrance? 105 10 Research – good, bad, and unnecessary 115 11 Getting the right research done is everybody’s business 130 12 So what makes for better healthcare? 143 13 Research for the right reasons: blueprint for a better future 160 References 169 Additional resources 182 List of Vignettes 185 List of Key Points 190 Index 193 TT_text_press.indd 5 22/09/2011 10:02 vi About the authors Imogen Evans practised and lectured in medicine in Canada and the UK before turning to medical journalism at The Lancet. From 1996 to 2005 she worked for the Medical Research Council, latterly in research ethics, and has represented the UK government on the Council of Europe Biomedical Ethics Committee. Hazel Thornton, after undergoing routine mammography, was invited to join a clinical trial, but the inadequate patient information led to her refusal. However, it also encouraged her advocacy for public involvement in research to achieve outcomes relevant to patients. She has written and spoken extensively on this topic. Iain Chalmers practised medicine in the UK and Palestine before becoming a health services researcher and directing the National Perinatal Epidemiology Unit and then the UK Cochrane Centre. Since 2003 he has coordinated the James Lind Initiative, promoting better controlled trials for better healthcare, particularly through greater public involvement. Paul Glasziou is both a medical researcher and part-time General Practitioner. As a consequence of observing the gap between these, he has focused his work on identifying and removing the barriers to using high-quality research in everyday clinical practice. He was editor of the BMJ’s Journal of Evidence- Based Medicine, and Director of the Centre for Evidence-Based Medicine in Oxford from 2003 to 2010. He is the author of several other books related to evidence-based practice. He is currently the recipient of a National Health and Medical Research Council Australia Fellowship which he commenced at Bond University in July, 2010. TT_text_press.indd 6 22/09/2011 10:02 vii Acknowledgements We thank the following people for their valuable comments and other contributions that have helped us to develop the second edition of Testing Treatments: Claire Allen, Doug Altman, Patricia Atkinson, Alexandra Barratt, Paul Barrow, Ben Bauer, Michael Baum, Sarah Boseley, Joan Box, Anne Brice, Rebecca Brice, Amanda Burls, Hamish Chalmers, Jan Chalmers, Yao-long Chen, Olivia Clarke, Catrin Comeau, Rhiannon Comeau, Katherine Cowan, John Critchlow, Sally Crowe, Philipp Dahm, Chris Del Mar, Jenny Doust, Mary Dixon-Woods, Ben Djulbegovic, Iain Donaldson, George Ebers, Diana Elbourne, Murray Enkin, Chrissy Erueti, Curt Furberg, Mark Fenton, Lester Firkins, Peter Gøtzsche, Muir Gray, Sally Green, Susan Green, Ben Goldacre, Metin Gülmezoğlu, Andrew Herxheimer, Jini Hetherington, Julian Higgins, Jenny Hirst, Jeremy Howick, Les Irwig, Ray Jobling, Bethan Jones, Karsten Juhl Jørgensen, Bridget Kenner, Debbie Kennett, Gill Lever, Alessandro Liberati, Howard Mann, Tom Marshall, Robert Matthews, Margaret McCartney, Dominic McDonald, Scott Metcalfe, Iain Milne, Martin McKee, Sarah Moore, Daniel Nicolae, Andy Oxman, Kay Pattison, Angela Raffle, June Raine, Jake Ranson, James Read, Kiley Richmond, Ian Roberts, Nick Ross, Peter Rothwell, Karen Sandler, Emily Savage-Smith, Marion Savage-Smith, John Scadding, Lisa Schwartz, Haleema Shakur, Ruth Silverman, Ann Southwell, Pete Spain, Mark Starr, Melissa Sweet, Tilli Tansey, Tom Treasure, Ulrich Tröhler, Liz Trotman, Liz Wager, Renee Watson, James Watt, Hywel Williams, Norman Williams, Steven Woloshin, Eleanor Woods, and Ke-hu Yang. Iain Chalmers and Paul Glasziou are grateful to the National Institute for Health Research (UK) for support. Paul Glasziou TT_text_press.indd 7 22/09/2011 10:02 viii is also grateful to the National Health and Medical Research Council (Australia). And a special thank you to our publisher, Martin Wagner, of Pinter & Martin for his forbearance, cheerful encouragement, and cool head at all times. TT_text_press.indd 8 22/09/2011 10:02 ix Foreword Medicine shouldn’t be about authority, and the most important question anyone can ask on any claim is simple: ‘how do you know?’ This book is about the answer to that question. There has been a huge shift in the way that people who work in medicine relate to patients. In the distant past, ‘communications skills training’, such as it was, consisted of how not to tell your patient they were dying of cancer. Today we teach students – and this is a direct quote from the hand-outs – how to ‘work collaboratively with the patient towards an optimum health outcome’. Today, if they wish, at medicine’s best, patients are involved in discussing and choosing their own treatments. For this to happen, it’s vital that everyone understands how we know if a treatment works, how we know if it has harms, and how we weigh benefits against harms to determine the risk. Sadly doctors can fall short on this, as much as anybody else. Even more sadly, there is a vast army out there, queuing up to mislead us. First and foremost in this gallery of rogues, we can mislead ourselves. Most diseases have a natural history, getting better and worse in cycles, or at random: because of this, anything you do, if you act when symptoms are at their worst, might make a treatment seem to be effective, because you were going to get better anyway. The placebo effect, similarly, can mislead us all: people really can get better, in some cases, simply from taking a dummy pill with no active ingredients, and by believing their treatments to be effective. As Robert M Pirsig said, in Zen and the Art of Motorcycle Maintenance: ‘the real purpose of the scientific method is to make sure nature hasn’t misled you into thinking you know something you actually don’t know’. But then there are the people who brandish scientific studies. If there is one key message from this book – and it is a phrase I TT_text_press.indd 9 22/09/2011 10:02 x TESTING TREATMENTS have borrowed and used endlessly myself – it is the concept of a ‘fair test’. Not all trials are born the same, because there are so many ways that a piece of scientific research can be biased, and erroneously give what someone, somewhere thinks should be the ‘right’ answer. Sometimes evidence can be distorted through absent- mindedness, or the purest of motives (for all that motive should matter). Doctors, patients, professors, nurses, occupational therapists, and managers can all become wedded to the idea that one true treatment, in which they have invested so much personal energy, is golden. Sometimes evidence can be distorted for other reasons. It would be wrong to fall into shallow conspiracy theories about the pharmaceutical industry: they have brought huge, lifesaving advances. But there is a lot of money at stake in some research, and for reasons you will see in this book, 90\% of trials are conducted by industry. This can be a problem, when studies funded by industry are four times more likely to have a positive result for the sponsor’s drug than independently funded trials. It costs up to $800m to bring a new drug to market: most of that is spent before the drug comes to market, and if the drug turns out to be no good, the money is already spent. Where the stakes are so high, sometimes the ideals of a fair test can fail.1 Equally, the way that evidence is communicated can be distorted, and misleading. Sometimes this can be in the presentation of facts and figures, telling only part of the story, glossing over flaws, and ‘cherry picking’ the scientific evidence which shows one treatment in a particular light. But in popular culture, there can be more interesting processes at play. We have an understandable desire for miracle cures, even though research is frequently about modest improvements, shavings of risk, and close judgement calls. In the media, all too often this can be thrown aside in a barrage of words like ‘cure’, ‘miracle’, ‘hope’, ‘breakthrough’, and ‘victim’.2 At a time when so many are so keen to take control of their own lives, and be involved in decisions about their own healthcare, it is sad to see so much distorted information, as it can only disempower. Sometimes these distortions are around a TT_text_press.indd 10 22/09/2011 10:02 xi FOREWORD specific drug: the presentation in the UK media of Herceptin as a miracle cure for breast cancer is perhaps the most compelling recent example.3 Sometimes, though, in promoting their own treatments, and challenging the evidence against them, zealots and their friends in the media can do even greater damage, by actively undermining the public’s very understanding of how we know if something is good for us, or bad for us. Homoeopathy sugar pills perform no better than dummy sugar pills when compared by the most fair tests. But when confronted with this evidence, homoeopaths argue that there is something wrong with the whole notion of doing a trial, that there is some complicated reason why their pills, uniquely among pills, cannot be tested. Politicians, when confronted with evidence that their favoured teaching programme for preventing teenage pregnancy has failed, may fall into the same kind of special pleading. In reality, as this book will show, any claim made about an intervention having an effect can be subjected to a transparent fair test.4 Sometimes these distortions can go even deeper into undermining the public’s understanding. A recent ‘systematic review’ of all the most fair and unbiased tests showed there was no evidence that taking antioxidant vitamin pills can prolong life (in fact, they may even shorten it). With this kind of summary – as explained beautifully in this book – clear rules are followed, describing where to look for evidence, what evidence can be included, and how its quality should be assessed. But when systematic reviews produce a result that challenges the claims of antioxidant supplement pill companies, newspapers and magazines are filled with false criticisms, arguing that individual studies for the systematic review have been selectively ‘cherry picked’, for reasons of political allegiance or frank corruption, that favourable evidence has been deliberately ignored, and so on.5 This is unfortunate. The notion of systematic review – looking at the totality of evidence – is quietly one of the most important innovations in medicine over the past 30 years. In defending their small corner of retail business, by undermining the public’s access to these ideas, journalists and pill companies can do us all a great disservice. TT_text_press.indd 11 22/09/2011 10:02 xii TESTING TREATMENTS And that is the rub. There are many reasons to read this book. At the simplest level, it will help you make your own decisions about your own health in a much more informed way. If you work in medicine, the chapters that follow will probably stand head and shoulders above any teaching you had in evidence-based medicine. At the population level, if more people understand how to make fair comparisons, and see whether one intervention is better than another, then as the authors argue, instead of sometimes fearing research, the public might actively campaign to be more involved in reducing uncertainties about the treatments that matter to them. But there is one final reason to read this book, to learn the tricks of our trade, and that reason has nothing to do with practicality: the plain fact is, this stuff is interesting, and beautiful, and clever. And in this book it’s explained better than anywhere else I’ve ever seen, because of the experience, knowledge, and empathy of the people who wrote it. Testing Treatments brings a human focus to real world questions. Medicine is about human suffering, and death, but also human frailty in decision makers and researchers: and this is captured here, in the personal stories and doubts of researchers, their motivations, concerns, and their shifts of opinion. It’s rare for this side of science to be made accessible to the public, and the authors move freely, from serious academic papers to the more ephemeral corners of medical literature, finding unguarded pearls from the discussion threads beneath academic papers, commentaries, autobiographies, and casual asides. This book should be in every school, and every medical waiting room. Until then, it’s in your hands. Read on. Ben Goldacre August 2011 TT_text_press.indd 12 22/09/2011 10:02 xiii Foreword to the first edition This book is good for our health. It shines light on the mysteries of how life and death decisions are made. It shows how those judgements are often badly flawed and it sets a challenge for doctors across the globe to mend their ways. Yet it accomplishes this without unnecessary scares; and it warmly admires much of what modern medicine has achieved. Its ambitions are always to improve medical practice, not disparage it. My own first insight into entrenched sloppiness in medicine came in the 1980s when I was invited to be a lay member of a consensus panel set up to judge best practice in the treatment of breast cancer. I was shocked (and you may be too when you read more about this issue in Chapter 2 [now Chapter 3]). We took evidence from leading researchers and clinicians and discovered that some of the most eminent consultants worked on hunch or downright prejudice and that a woman’s chance of survival, and of being surgically disfigured, greatly depended on who treated her and what those prejudices were. One surgeon favoured heroic mutilation, another preferred simple lump removal, a third opted for aggressive radiotherapy, and so on. It was as though the age of scientific appraisal had passed them by. Indeed, it often had, and for many doctors it still does. Although things have improved, many gifted, sincere and skilful medical practitioners are surprisingly ignorant about what constitutes good scientific evidence. They do what they do because that is what they were taught in medical school, or because it is what other doctors do, or because in their experience it works. But personal experience, though beguiling, is often terribly misleading – as this book shows, with brutal clarity. Some doctors say it is naïve to apply scientific rigour to the treatment of individual patients. Medicine, they assert, is both a science and an art. But, noble as that sounds, it is a contradiction TT_text_press.indd 13 22/09/2011 10:02 xiv TESTING TREATMENTS in terms. Of course medical knowledge is finite and with any individual the complexities are almost infinite, so there is always an element of uncertainty. In practice, good medicine routinely requires good guesswork. But too often in the past many medical professionals have blurred the distinction between guessing and good evidence. Sometimes they even proclaim certainty when there is really considerable doubt. They eschew reliable data because they are not sure how to assess them. This book explains the difference between personal experience and more complex, but better ways of distinguishing what works from what does not and what is safe from what is not. Insofar as it can, it avoids technical terms, and promotes plain English expressions like ‘fair tests’. It warns that science, like everything else in human affairs, is prone to error and bias (through mistakes, vanity or – especially pernicious in medicine – the demands of commerce); but it reminds us that, even so, it is the meticulous approach of science that has created almost all of the most conspicuous advances in human knowledge. Doctors (and media-types, like me) should stop disparaging clinical research as ‘trials on human guinea-pigs’; on the contrary there is a moral imperative for all practitioners to promote fair tests to their patients and for patients to participate. This is an important book for anyone concerned about their own or their family’s health, or the politics of health. Patients are often seen as the recipients of healthcare, rather than participants. The task ahead is as much for us, the lay public in whose name medicine is practised and from whose purse medical practitioners are paid, as for doctors and medical researchers. If we are passive consumers of medicine we will never drive up standards. If we prefer simplistic answers we will get pseudoscience. If we do not promote the rigorous testing of treatments we will get pointless and sometimes dangerous treatment along with the stuff that really works. This book contains a manifesto for improving things, and patients are at its heart. But it is an important book for doctors, medical students, and researchers too – all would benefit from its lessons. In an ideal world, it would be compulsory reading for every journalist, and available to every patient, because if doctors TT_text_press.indd 14 22/09/2011 10:02 xv FOREWORD TO THE FIRST EDITION are inadequate at weighing up scientific evidence, in general we, whose very mortality depends on this, are worse. One thing I promise: if this subject of testing treatments is new to you, once you have read this book you will never feel quite the same about your doctor’s advice again. Nick Ross TV and radio presenter and journalist 16 November 2005 TT_text_press.indd 15 22/09/2011 10:02 TT_text_press.indd 16 22/09/2011 10:02 xvii Preface The first edition of Testing Treatments, published in 2006, was inspired by a question: ‘How do you ensure that research into medical treatments best meets the needs of patients?’ Our collective experience – collective at that point meaning Imogen Evans, a medical doctor and former researcher and journalist, Hazel Thornton, a patient and independent lay advocate for quality in research and healthcare, and Iain Chalmers, a health services researcher – was that research often failed to address this key issue. Moreover, we were keenly aware that many medical treatments, both new and old, were not based on sound evidence. So we set out to write a book to promote more critical public assessment of the effects of treatments by encouraging patient- professional dialogue. We were heartened by the level of interest shown in Testing Treatments – both in the original British Library imprint and when we made the text freely available online at www.jameslindlibrary. org – and that it appealed to both lay and professional readers. The first edition of Testing Treatments has been used as a teaching aid in many countries, and several full translations are available for free download from www.testingtreatments.org. From the outset we thought of Testing Treatments as work in progress; there will almost always be uncertainties about the effects of treatments, whether new or old, and therefore a continuing need for all treatments to be tested properly. To do this it is essential to visit and re-visit the evidence; to review existing evidence critically and systematically before embarking on new research, and similarly to interpret new results in the light of up-to-date systematic reviews. Embarking on the second edition of Testing Treatments, we three became four, with the addition of Paul Glasziou, a general practitioner and researcher with a commitment to taking account TT_text_press.indd 17 22/09/2011 10:02 xviii TESTING TREATMENTS of high-quality research evidence in everyday clinical practice. We have a new publisher – Pinter & Martin, who reprinted the first edition in 2010 – and the new text is available free on line, as before, from www.testingtreatments.org. While our basic premise remains the same, the original text has been extensively revised and updated. For example, we have expanded coverage of the benefits and harms of screening in a separate chapter (Chapter 4) entitled Earlier is not necessarily better. And in Regulating tests of treatments: help or hindrance? (Chapter 9) we describe how research can become over-policed to the detriment of patients. In the penultimate chapter (Chapter 12) we ask: ‘So what makes for better healthcare?’ and show how the lines of evidence can be drawn together in ways that can make a real difference to all of us. We close with our blueprint for a better future and an action plan (Chapter 13). We hope our book will point the way to wider understanding of how treatments can and should be tested fairly and how everyone can play a part in making this happen. This is not a ‘best treatments guide’ to the effects of individual therapies. Rather, we highlight issues that are fundamental to ensuring that research is soundly based, properly done, able to distinguish harmful from helpful treatments, and designed to answer questions that matter to patients, the public, and health professionals. Imogen Evans, Hazel Thornton, Iain Chalmers, Paul Glasziou August 2011 TT_text_press.indd 18 22/09/2011 10:02 xix Introduction ‘There is no way to know when our observations about complex events in nature are complete. Our knowledge is finite, Karl Popper emphasised, but our ignorance is infinite. In medicine, we can never be certain about the consequences of our interventions, we can only narrow the area of uncertainty. This admission is not as pessimistic as it sounds: claims that resist repeated energetic challenges often turn out to be quite reliable. Such “working truths” are the building blocks for the reasonably solid structures that support our everyday actions at the bedside.’ William A. Silverman. Where’s the evidence? Oxford: Oxford University Press, 1998, p165 Modern medicine has been hugely successful. It is hard to imagine what life must have been like without antibiotics. The development of other effective drugs has revolutionized the treatment of heart attacks and high blood pressure and has transformed the lives of many people with schizophrenia. Childhood immunization has made polio and diphtheria distant memories in most countries, and artificial joints have helped countless people to be less troubled by pain and disability. Modern imaging techniques such as ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) have helped to ensure that patients are accurately diagnosed and receive the right treatment. The diagnosis of many types of cancer used to spell a death sentence, TT_text_press.indd 19 22/09/2011 10:02 xx TESTING TREATMENTS whereas today patients regularly live with their cancers instead of dying from them. And HIV/AIDS has largely changed from a swift killer into a chronic (long-lasting) disease. Of course many improvements in health have come about because of social and public health advances, such as piped clean water, sanitation, and better housing. But even sceptics would have difficulty dismissing the impressive impact of modern medical care. Over the past half century or so, better healthcare has made a major contribution to increased lifespan, and has improved the quality of life, especially for those with chronic conditions.1, 2 But the triumphs of modern medicine can easily lead us to overlook many of its ongoing problems. Even today, too much medical decision-making is based on poor evidence. There are still too many medical treatments that harm patients, some that are of little or no proven benefit, and others that are worthwhile but are not used enough. How can this be, when every year, studies into the effects of treatments generate a mountain of results? Sadly, the evidence is often unreliable and, moreover, much of the research that is done does not address the questions that patients need answered. Part of the problem is that treatment effects are very seldom overwhelmingly obvious or dramatic. Instead, there will usually be uncertainties about how well new treatments work, or indeed whether they do more good than harm. So carefully designed fair tests – tests that set out to reduce biases and take into account the play of chance (see Chapter 6) – are necessary to identify treatment effects reliably. The impossibility of predicting exactly what will happen when a person gets a disease or receives a treatment is sometimes called Franklin’s law, after the 18th-century US statesman Benjamin Franklin, who famously noted that ‘in this world nothing can be said to be certain, except death and taxes’.3 Yet Franklin’s law is hardly second nature in society. The inevitability of uncertainty is not emphasized enough in schools, nor are other fundamental concepts such as how to obtain and interpret evidence, or how to understand information about probabilities and risks. As one commentator put it: ‘At school you were taught about chemicals in test … University of South Florida Scholar Commons Textbooks Collection USF Tampa Library Open Access Collections 2012 Social Science Research: Principles, Methods, and Practices Anol Bhattacherjee University of South Florida, [email protected] Follow this and additional works at: http://scholarcommons.usf.edu/oa_textbooks Part of the American Studies Commons, Education Commons, Public Health Commons, and the Social and Behavioral Sciences Commons This Book is brought to you for free and open access by the USF Tampa Library Open Access Collections at Scholar Commons. It has been accepted for inclusion in Textbooks Collection by an authorized administrator of Scholar Commons. For more information, please contact [email protected] Recommended Citation Bhattacherjee, Anol, Social Science Research: Principles, Methods, and Practices (2012). Textbooks Collection. 3. http://scholarcommons.usf.edu/oa_textbooks/3 http://scholarcommons.usf.edu/?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu/?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu/oa_textbooks?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu/tloa?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu/oa_textbooks?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://network.bepress.com/hgg/discipline/439?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://network.bepress.com/hgg/discipline/784?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://network.bepress.com/hgg/discipline/738?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://network.bepress.com/hgg/discipline/316?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages http://scholarcommons.usf.edu/oa_textbooks/3?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages mailto:[email protected] SOCIAL SCIENCE RESEARCH: PRINCIPLES, METHODS, AND PRACTICES ANOL BHATTACHERJEE SOCIAL SCIENCE RESEARCH: PRINCIPLES, METHODS, AND PRACTICES Anol Bhattacherjee, Ph.D. University of South Florida Tampa, Florida, USA [email protected] Second Edition Copyright © 2012 by Anol Bhattacherjee Published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License mailto:[email protected] Social Science Research: Principles, Methods, and Practices, 2nd edition By Anol Bhattacherjee First published 2012 ISBN-13: 978-1475146127 ISBN-10: 1475146124 Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: Users are free to use, copy, share, distribute, display, and reference this book under the following conditions:  ATTRIBUTION: Whole or partial use of this book should be attributed (referenced or cited) according to standard academic practices.  NON-COMMERCIAL: This book may not be used for commercial purposes.  SHARE ALIKE: Users may alter, transform, or build upon this book, but must distribute the resulting work under the same or similar license as this one. For any reuse or distribution, the license terms of this work must be clearly specified. Your fair use and other rights are in no way affected by the above. Copyright © 2012 by Anol Bhattacherjee i Preface This book is designed to introduce doctoral and graduate students to the process of scientific research in the social sciences, business, education, public health, and related disciplines. This book is based on my lecture materials developed over a decade of teaching the doctoral-level class on Research Methods at the University of South Florida. The target audience for this book includes Ph.D. and graduate students, junior researchers, and professors teaching courses on research methods, although senior researchers can also use this book as a handy and compact reference. The first and most important question potential readers should have about this book is how is it different from other text books on the market? Well, there are four key differences. First, unlike other text books, this book is not just about “research methods” (empirical data collection and analysis) but about the entire “research process” from start to end. Research method is only one phase in that research process, and possibly the easiest and most structured one. Most text books cover research methods in depth, but leave out the more challenging, less structured, and probably more important issues such as theorizing and thinking like a researcher, which are often prerequisites of empirical research. In my experience, most doctoral students become fairly competent at research methods during their Ph.D. years, but struggle to generate interesting or useful research questions or build scientific theories. To address this deficit, I have devoted entire chapters to topics such as “Thinking Like a Researcher” and “Theories in Scientific Research”, which are essential skills for a junior researcher. Second, the book is succinct and compact by design. While writing the book, I decided to focus only on essential concepts, and not fill pages with clutter that can divert the students’ attention to less relevant or tangential issues. Most doctoral seminars include a fair complement of readings drawn from the respective discipline. This book is designed to complement those readings by summarizing all important concepts in one compact volume, rather than burden students with a voluminous text on top of their assigned readings. Third, this book is free in its download version. Not just the current edition but all future editions in perpetuity. The book will also be available in Kindle e-Book, Apple iBook, and on-demand paperback versions at a nominal cost. Many people have asked why I’m giving away something for free when I can make money selling it? Well, not just to stop my students from constantly complaining about the high price of text books, but also because I believe that scientific knowledge should not be constrained by access barriers such as price and availability. Scientific progress can occur only if students and academics around the world have affordable access to the best that science can offer, and this free book is my humble effort to that cause. However, free should not imply “lower quality.” Some of the best things in life such as air, water, and sunlight are free. Many of Google’s resources are free too, and one can well imagine where we would be in today’s Internet age without Google. Some of the most sophisticated software programs available today, like Linux and Apache, are also free, and so is this book. Fourth, I plan to make local-language versions of this book available in due course of time, and those translated versions will also be free. So far, I have had commitments to ii translate thus book into Chinese, French, Indonesian, Korean, Portuguese, Spanish versions (which will hopefully be available in 2012), and I’m looking for qualified researchers or professors to translate it into Arabic, German, and other languages where there is sufficient demand for a research text. If you are a prospective translator, please note that there will be no financial gains or royalty for your translation services, because the book must remain free, but I’ll gladly include you as a coauthor on the local-language version. The book is structured into 16 chapters for a 16-week semester. However, professors or instructors can add, drop, stretch, or condense topics to customize the book to the specific needs of their curriculum. For instance, I don’t cover Chapters 14 and 15 in my own class, because we have dedicated classes on statistics to cover those materials and more. Instead, I spend two weeks on theories (Chapter 3), one week to discussing and conducting reviews for academic journals (not in the book), and one week for a finals exam. Nevertheless, I felt it necessary to include Chapters 14 and 15 for academic programs that may not have a dedicated class on statistical analysis for research. A sample syllabus that I use for my own class in the business Ph.D. program is provided in the appendix. Lastly, I plan to continually update this book based on emerging trends in scientific research. If there are any new or interesting content that you wish to see in future editions, please drop me a note, and I will try my best to accommodate them. Comments, criticisms, or corrections to any of the existing content will also be gratefully appreciated. Anol Bhattacherjee E-mail: [email protected] mailto:[email protected] iii Table of Contents Introduction to Research 1. Science and Scientific Research.................................................................................................... 1 2. Thinking Like a Researcher ........................................................................................................... 9 3. The Research Process ................................................................................................................. 17 4. Theories in Scientific Research ................................................................................................... 25 Basics of Empirical Research 5. Research Design ......................................................................................................................... 35 6. Measurement of Constructs....................................................................................................... 43 7. Scale Reliability and Validity ....................................................................................................... 55 8. Sampling ..................................................................................................................................... 65 Data Collection 9. Survey Research ......................................................................................................................... 73 10. Experimental Research .............................................................................................................. 83 11. Case Research ............................................................................................................................ 93 12. Interpretive Research ............................................................................................................... 103 Data Analysis 13. Qualitative Analysis .................................................................................................................. 113 14. Quantitative Analysis: Descriptive Statistics ............................................................................ 119 15. Quantitative Analysis: Inferential Statistics ............................................................................. 129 Epilogue 16. Research Ethics ........................................................................................................................ 137 Appendix ............................................................................................................................................. 143 1 Chapter 1 Science and Scientific Research What is research? Depending on who you ask, you will likely get very different answers to this seemingly innocuous question. Some people will say that they routinely research different online websites to find the best place to buy goods or services they want. Television news channels supposedly conduct research in the form of viewer polls on topics of public interest such as forthcoming elections or government-funded projects. Undergraduate students research the Internet to find the information they need to complete assigned projects or term papers. Graduate students working on research projects for a professor may see research as collecting or analyzing data related to their project. Businesses and consultants research different potential solutions to remedy organizational problems such as a supply chain bottleneck or to identify customer purchase patterns. However, none of the above can be considered “scientific research” unless: (1) it contributes to a body of science, and (2) it follows the scientific method. This chapter will examine what these terms mean. Science What is science? To some, science refers to difficult high school or college-level courses such as physics, chemistry, and biology meant only for the brightest students. To others, science is a craft practiced by scientists in white coats using specialized equipment in their laboratories. Etymologically, the word “science” is derived from the Latin word scientia meaning knowledge. Science refers to a systematic and organized body of knowledge in any area of inquiry that is acquired using “the scientific method” (the scientific method is described further below). Science can be grouped into two broad categories: natural science and social science. Natural science is the science of naturally occurring objects or phenomena, such as light, objects, matter, earth, celestial bodies, or the human body. Natural sciences can be further classified into physical sciences, earth sciences, life sciences, and others. Physical sciences consist of disciplines such as physics (the science of physical objects), chemistry (the science of matter), and astronomy (the science of celestial objects). Earth sciences consist of disciplines such as geology (the science of the earth). Life sciences include disciplines such as biology (the science of human bodies) and botany (the science of plants). In contrast, social science is the science of people or collections of people, such as groups, firms, societies, or economies, and their individual or collective behaviors. Social sciences can be classified into disciplines such as psychology (the science of human behaviors), sociology (the science of social groups), and economics (the science of firms, markets, and economies). The natural sciences are different from the social sciences in several respects. The natural sciences are very precise, accurate, deterministic, and independent of the person 2 | S o c i a l S c i e n c e R e s e a r c h making the scientific observations. For instance, a scientific experiment in physics, such as measuring the speed of sound through a certain media or the refractive index of water, should always yield the exact same results, irrespective of the time or place of the experiment, or the person conducting the experiment. If two students conducting the same physics experiment obtain two different values of these physical properties, then it generally means that one or both of those students must be in error. However, the same cannot be said for the social sciences, which tend to be less accurate, deterministic, or unambiguous. For instance, if you measure a person’s happiness using a hypothetical instrument, you may find that the same person is more happy or less happy (or sad) on different days and sometimes, at different times on the same day. One’s happiness may vary depending on the news that person received that day or on the events that transpired earlier during that day. Furthermore, there is not a single instrument or metric that can accurately measure a person’s happiness. Hence, one instrument may calibrate a person as being “more happy” while a second instrument may find that the same person is “less happy” at the same instant in time. In other words, there is a high degree of measurement error in the social sciences and there is considerable uncertainty and little agreement on social science policy decisions. For instance, you will not find many disagreements among natural scientists on the speed of light or the speed of the earth around the sun, but you will find numerous disagreements among social scientists on how to solve a social problem such as reduce global terrorism or rescue an economy from a recession. Any student studying the social sciences must be cognizant of and comfortable with handling higher levels of ambiguity, uncertainty, and error that come with such sciences, which merely reflects the high variability of social objects. Sciences can also be classified based on their purpose. Basic sciences, also called pure sciences, are those that explain the most basic objects and forces, relationships between them, and laws governing them. Examples include physics, mathematics, and biology. Applied sciences, also called practical sciences, are sciences that apply scientific knowledge from basic sciences in a physical environment. For instance, engineering is an applied science that applies the laws of physics and chemistry for practical applications such as building stronger bridges or fuel efficient combustion engines, while medicine is an applied science that applies the laws of biology for solving human ailments. Both basic and applied sciences are required for human development. However, applied sciences cannot stand on their own right, but instead relies on basic sciences for its progress. Of course, the industry and private enterprises tend to focus more on applied sciences given their practical value, while universities study both basic and applied sciences. Scientific Knowledge The purpose of science is to create scientific knowledge. Scientific knowledge refers to a generalized body of laws and theories to explain a phenomenon or behavior of interest that are acquired using the scientific method. Laws are observed patterns of phenomena or behaviors, while theories are systematic explanations of the underlying phenomenon or behavior. For instance, in physics, the Newtonian Laws of Motion describe what happens when an object is in a state of rest or motion (Newton’s First Law), what force is needed to move a stationary object or stop a moving object (Newton’s Second Law), and what happens when two objects collide (Newton’s Third Law). Collectively, the three laws constitute the basis of classical mechanics – a theory of moving objects. Likewise, the theory of optics explains the properties of light and how it behaves in different media, electromagnetic theory explains the properties of electricity and how to generate it, quantum mechanics explains the properties of subatomic particles, and thermodynamics explains the properties of energy and mechanical S c i e n c e a n d S c i e n t i f i c R e s e a r c h | 3 work. An introductory college level text book in physics will likely contain separate chapters devoted to each of these theories. Similar theories are also available in social sciences. For instance, cognitive dissonance theory in psychology explains how people react when their observations of an event is different from what they expected of that event, general deterrence theory explains why some people engage in improper or criminal behaviors, such as illegally download music or commit software piracy, and the theory of planned behavior explains how people make conscious reasoned choices in their everyday lives. The goal of scientific research is to discover laws and postulate theories that can explain natural or social phenomena, or in other words, build scientific knowledge. It is important to understand that this knowledge may be imperfect or even quite far from the truth. Sometimes, there may not be a single universal truth, but rather an equilibrium of “multiple truths.” We must understand that the theories, upon which scientific knowledge is based, are only explanations of a particular phenomenon, as suggested by a scientist. As such, there may be good or poor explanations, depending on the extent to which those explanations fit well with reality, and consequently, there may be good or poor theories. The progress of science is marked by our progression over time from poorer theories to better theories, through better observations using more accurate instruments and more informed logical reasoning. We arrive at scientific laws or theories through a process of logic and evidence. Logic (theory) and evidence (observations) are the two, and only two, pillars upon which scientific knowledge is based. In science, theories and observations are interrelated and cannot exist without each other. Theories provide meaning and significance to what we observe, and observations help validate or refine existing theory or construct new theory. Any other means of knowledge acquisition, such as faith or authority cannot be considered science. Scientific Research Given that theories and observations are the two pillars of science, scientific research operates at two levels: a theoretical level and an empirical level. The theoretical level is concerned with developing abstract concepts about a natural or social phenomenon and relationships between those concepts (i.e., build “theories”), while the empirical level is concerned with testing the theoretical concepts and relationships to see how well they reflect our observations of reality, with the goal of ultimately building better theories. Over time, a theory becomes more and more refined (i.e., fits the observed reality better), and the science gains maturity. Scientific research involves continually moving back and forth between theory and observations. Both theory and observations are essential components of scientific research. For instance, relying solely on observations for making inferences and ignoring theory is not considered valid scientific research. Depending on a researcher’s training and interest, scientific inquiry may take one of two possible forms: inductive or deductive. In inductive research, the goal of a researcher is to infer theoretical concepts and patterns from observed data. In deductive research, the goal of the researcher is to test concepts and patterns known from theory using new empirical data. Hence, inductive research is also called theory-building research, and deductive research is theory-testing research. Note here that the goal of theory-testing is not just to test a theory, but possibly to refine, improve, and extend it. Figure 1.1 depicts the complementary nature of inductive and deductive research. Note that inductive and deductive research are two halves of the research cycle that constantly iterates between theory and observations. You cannot do inductive or deductive research if you are not familiar with both the theory and data 4 | S o c i a l S c i e n c e R e s e a r c h components of research. Naturally, a complete researcher is one who can traverse the entire research cycle and can handle both inductive and deductive research. It is important to understand that theory-building (inductive research) and theory- testing (deductive research) are both critical for the advancement of science. Elegant theories are not valuable if they do not match with reality. Likewise, mountains of data are also useless until they can contribute to the construction to meaningful theories. Rather than viewing these two processes in a circular relationship, as shown in Figure 1.1, perhaps they can be better viewed as a helix, with each iteration between theory and data contributing to better explanations of the phenomenon of interest and better theories. Though both inductive and deductive research are important for the advancement of science, it appears that inductive (theory-building) research is more valuable when there are few prior theories or explanations, while deductive (theory-testing) research is more productive when there are many competing theories of the same phenomenon and researchers are interested in knowing which theory works best and under what circumstances. Figure 1.1. The Cycle of Research Theory building and theory testing are particularly difficult in the social sciences, given the imprecise nature of the theoretical concepts, inadequate tools to measure them, and the presence of many unaccounted factors that can also influence the phenomenon of interest. It is also very difficult to refute theories that do not work. For instance, Karl Marx’s theory of communism as an effective means of economic production withstood for decades, before it was finally discredited as being inferior to capitalism in promoting economic growth and social welfare. Erstwhile communist economies like the Soviet Union and China eventually moved toward more capitalistic economies characterized by profit-maximizing private enterprises. However, the recent collapse of the mortgage and financial industries in the United States demonstrates that capitalism also has its flaws and is not as effective in fostering economic growth and social welfare as previously presumed. Unlike theories in the natural sciences, social science theories are rarely perfect, which provides numerous opportunities for researchers to improve those theories or build their own alternative theories. Conducting scientific research, therefore, requires two sets of skills – theoretical and methodological – needed to operate in the theoretical and empirical levels respectively. Methodological skills (know-how) are relatively standard, invariant across disciplines, and easily acquired through doctoral programs. However, theoretical skills (know-what) is considerably harder to master, requires years of observation and reflection, and are tacit skills that cannot be “taught” but rather learned though experience. All of the greatest scientists in the history of mankind, such as Galileo, Newton, Einstein, Neils Bohr, Adam Smith, Charles S c i e n c e a n d S c i e n t i f i c R e s e a r c h | 5 Darwin, and Herbert Simon, were master theoreticians, and they are remembered for the theories they postulated that transformed the course of science. Methodological skills are needed to be an ordinary researcher, but theoretical skills are needed to be an extraordinary researcher! Scientific Method In the preceding sections, we described science as knowledge acquired through a scientific method. So what exactly is the “scientific method”? Scientific method refers to a standardized set of techniques for building scientific knowledge, such as how to make valid observations, how to interpret results, and how to generalize those results. The scientific method allows researchers to independently and impartially test preexisting theories and prior findings, and subject them to open debate, modifications, or enhancements. The scientific method must satisfy four key characteristics:  Logical: Scientific inferences must be based on logical principles of reasoning.  Confirmable: Inferences derived must match with observed evidence.  Repeatable: Other scientists should be able to independently replicate or repeat a scientific study and obtain similar, if not identical, results.  Scrutinizable: The procedures used and the inferences derived must withstand critical scrutiny (peer review) by other scientists. Any branch of inquiry that does not allow the scientific method to test its basic laws or theories cannot be called “science.” For instance, theology (the study of religion) is not science because theological ideas (such as the presence of God) cannot be tested by independent observers using a logical, confirmable, repeatable, and scrutinizable. Similarly, arts, music, literature, humanities, and law are also not considered science, even though they are creative and worthwhile endeavors in their own right. The scientific method, as applied to social sciences, includes a variety of research approaches, tools, and techniques, for collecting and analyzing qualitative or quantitative data. These methods include laboratory experiments, field surveys, case research, ethnographic research, action research, and so forth. Much of this book is devoted to learning about these different methods. However, recognize that the scientific method operates primarily at the empirical level of research, i.e., how to make observations and analyze these observations. Very little of this method is directly pertinent to the theoretical level, which is really the more challenging part of scientific research. Types of Scientific Research Depending on the purpose of research, scientific research projects can be grouped into three types: exploratory, … The Field Guide to Human-Centered Design By IDEO.org 1st Edition © 2015 ISBN: 978-0-9914063-1-9 This work is licensed under the creative commons attribution, noncommercial, no derivatives 3.0 unported license. Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). Noncommerical — You may not use this work for commercial purposes. NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material. Printed in Canada THE FIELD GUIDE Contents Introduction Mindsets Creative Confidence Make It Learn from Failure Empathy Embrace Ambiguity Optimism Iterate, Iterate, Iterate Methods INSPIRATION Frame Your Design Challenge Create a Project Plan Build a Team Recruiting Tools Secondary Research Interview Group Interview Expert Interview Define Your Audience Conversation Starters Extremes and Mainstreams Immersion Analogous Inspiration Card Sort Peers Observing Peers 09 17 19 20 21 22 23 24 25 27 29 31 34 35 36 37 39 42 43 44 45 49 52 53 57 60 Collage Guided Tour Draw It Resource Flow Case Study: Vroom IDEATION Download Your Learnings Share Inspiring Stories Top Five Find Themes Create Insight Statements Explore Your Hunch How Might We Create Frameworks Brainstorm Brainstorm Rules Bundle Ideas Get Visual Mash-Ups Design Principles Create a Concept Co-Creation Session Gut Check Determine What to Prototype Storyboard Role Playing 61 64 65 67 71 75 77 78 79 80 81 84 85 89 94 95 97 101 104 105 108 109 110 111 113 118 119 123 126 127 129 133 135 136 137 140 141 144 145 146 147 148 149 152 153 157 159 163 189 Rapid Prototyping Business Model Canvas Get Feedback Integrate Feedback and Iterate Case Study: Asili IMPLEMENTATION Live Prototyping Roadmap Resource Assessment Build Partnerships Ways to Grow Framework Staff Your Project Funding Strategy Pilot Define Success Keep Iterating Create a Pitch Sustainable Revenue Monitor and Evaluate Keep Getting Feedback Case Study: Clean Team Resources Colophon 8 The Field Guide to Human-Centered Design 09 Introduction What Does It Mean to Be Embracing human-centered design means believing that all problems, even the seemingly intractable ones like poverty, gender equality, and clean water, are solvable. Moreover, it means believing that the people who face those problems every day are the ones who hold the key to their answer. Human-centered design offers problem solvers of any stripe a chance to design with communities, to deeply understand the people they’re looking to serve, to dream up scores of ideas, and to create innovative new solutions rooted in people’s actual needs. At IDEO.org and IDEO, we’ve used human-centered design for decades to create products, services, experiences, and social enterprises that have been adopted and embraced because we’ve kept people’s lives and desires at the core. The social sector is ripe for innovation, and we’ve seen time and again how our approach has the power to unlock real impact. Being a human-centered designer is about believing that as long as you stay grounded in what you’ve learned from people, your team can arrive at new solutions that the world needs. And with this Field Guide, you’re now armed with the tools needed to bring that belief to life. 10 The Field Guide to Human-Centered Design Human-centered designers are unlike other problem solvers—we tinker and test, we fail early and often, and we spend a surprising amount of time not knowing the answer to the challenge at hand. And yet, we forge ahead. We’re optimists and makers, experimenters and learners, we empathize and iterate, and we look for inspiration in unexpected places. We believe that a solution is out there and that by keeping focused on the people we’re designing for and asking the right questions, we’ll get there together. We dream up lots of ideas, some that work and some that don’t. We make our ideas tangible so that we can test them, and then we refine them. In the end, our approach amounts to wild creativity, to a ceaseless push to innovate, and a confidence that leads us to solutions we’d never dreamed of when we started. In the Field Guide, we share our philosophy of design and the seven mindsets that set us apart: Empathy, Optimism, Iteration, Creative Confidence, Making, Embracing Ambiguity, and Learning from Failure. Adopt the Mindsets 11 Introduction INSPIRATION In this phase, you’ll learn how to better understand people. You’ll observe their lives, hear their hopes and desires, and get smart on your challenge. IDEATION Here you’ll make sense of everything that you’ve heard, generate tons of ideas, identify opportunities for design, and test and refine your solutions. IMPLEMENTATION Now is your chance to bring your solution to life. You’ll figure out how to get your idea to market and how to maximize its impact in the world. Human-centered design isn’t a perfectly linear process, and each project invariably has its own contours and character. But no matter what kind of design challenge you’ve got, you’ll move through three main phases: Inspiration, Ideation, and Implementation. By taking these three phases in turn, you’ll build deep empathy with the communities and individuals you’re designing for; you’ll figure out how to turn what you’ve learned into a chance to design a new solution; and you’ll build and test your ideas before finally putting them out into the world. At IDEO.org and IDEO, we’ve used human-centered design to tackle a vast array of design challenges, and though our projects have ranged from social enterprises to communication campaigns to medical devices, this particular approach to creative problem solving has seen us through each time. Understand the Process 12 The Field Guide to Human-Centered Design Though no two human-centered design projects are alike, we draw from the same kit of tools for each of them. For example, to build deep empathy with the people we’re trying to serve, we always conduct interviews with them. To maintain creativity and energy, we always work in teams. To keep our thinking generative, sharp, and because it helps us work things through, we always make tangible prototypes of our ideas. And because we rarely get it right the first time, we always share what we’ve made, and iterate based on the feedback we get. The 57 methods in the Field Guide offer a comprehensive set of exercises and activities that will take you from framing up your design challenge to getting it to market. You’ll use some of these methods twice or three times and some not at all as you work through your challenge. But taken as a set, they’ll put you on the path to continuous innovation while keeping the community you’re designing for squarely at the center of your work. Use the Tools 13 Introduction Trust the Process Even if It Feels Uncomfortable D IV ER G E DI VE RG E C O N VERG E CONVERGE Human-centered design is a unique approach to problem solving, one that can occasionally feel more like madness than method—but you rarely get to new and innovative solutions if you always know precisely where you’re going. The process is designed to get you to learn directly from people, open yourself up to a breadth of creative possibilities, and then zero in on what’s most desirable, feasible, and viable for the people you’re designing for. You’ll find yourself frequently shifting gears through the process, and as you work through its three phases you’ll swiftly move from concrete observations to highly abstract thinking, and then right back again into the nuts and bolts of your prototype. We call it diverging and converging. By going really big and broad during the Ideation phase, we dream up all kinds of possible solutions. But because the goal is to have a big impact in the world, we have to then identif y what, among that constellation of ideas, has the best shot at really working. You’ll diverge and converge a few times, and with each new cycle you’ll come closer and closer to a market- ready solution. 14 The Field Guide to Human-Centered Design DESIRABLE Human Start here FEASIBLE Technology VIABLE Business Human-centered design is uniquely situated to arrive at solutions that are desirable, feasible, and viable. By starting with humans, their hopes, fears, and needs, we quickly uncover what’s most desirable. But that’s only one lens through which we look at our solutions. Once we’ve determined a range of solutions that could appeal to the community we’re looking to serve, we then start to home in on what is technically feasible to actually implement and how to make the solution financially viable. It’s a balancing act, but one that’s absolutely crucial to designing solutions that are successful and sustainable. Create Real Impact 15 Introduction 17 Introduction MINDSETS 18 The Field Guide to Human-Centered Design 19 Mindsets Creative Confidence —David Kelley, Founder, IDEO Anyone can approach the world like a designer. Often all it takes to unlock that potential as a dynamic problem solver is a bit of creative confidence. Creative confidence is the belief that everyone is creative, and that creativity isn’t the capacity to draw or compose or sculpt, but a way of understanding the world. Creative confidence is the quality that human- centered designers rely on when it comes to making leaps, trusting their intuition, and chasing solutions that they haven’t totally figured out yet. It’s the belief that you can and will come up with creative solutions to big problems and the confidence that all it takes is rolling up your sleeves and diving in. Creative confidence will drive you to make things, to test them out, to get it wrong, and to keep on rolling, secure in the knowledge that you’ll get where you need to go and that you’re bound to innovate along the way. It can take time to build creative confidence, and part of getting there is trusting that the human-centered design process will show you how to bring a creative approach to whatever problem is at hand. As you start with small successes and then build to bigger ones, you’ll see your creative confidence grow and before long you’ll find yourself in the mindset that you are a wildly creative person. 20 The Field Guide to Human-Centered Design As human-centered designers, we make because we believe in the power of tangibility. And we know that making an idea real reveals so much that mere theory cannot. When the goal is to get impactful solutions out into the world, you can’t live in abstractions. You have to make them real. Human-centered designers are doers, tinkerers, crafters, and builders. We make using anything at our disposal, from cardboard and scissors to sophisticated digital tools. We build our ideas so that we can test them, and because actually making something reveals opportunities and complexities that we’d never have guessed were there. Making is also a fantastic way to think, and it helps bring into focus the feasibility of our designs. Moreover, making an idea real is an incredibly effective way to share it. And without candid, actionable feedback from people, we won’t know how to push our ideas forward. As you move through the human-centered design process, it doesn’t matter what you make, the materials you use, or how beautiful the result is, the goal is always to convey an idea, share it, and learn how to make it better. Best of all, you can prototype anything at any stage of the process from a service model to a uniform, from a storyboard to the financial details of your solution. As human-centered designers, we have a bias toward action, and that means getting ideas out of our heads and into the hands of the people we’re looking to serve. Make It —Krista Donaldson, CEO, D-Rev 21 Mindsets Failure is an incredibly powerful tool for learning. Designing experiments, prototypes, and interactions and testing them is at the heart of human-centered design. So is an understanding that not all of them are going to work. As we seek to solve big problems, we’re bound to fail. But if we adopt the right mindset, we’ll inevitably learn something from that failure. Human-centered design starts from a place of not knowing what the solution to a given design challenge might be. Only by listening, thinking, building, and refining our way to an answer do we get something that will work for the people we’re trying to serve. “Fail early to succeed sooner” is a common refrain around IDEO, and part of its power is the permission it gives to get something wrong. By refusing to take risks, some problem solvers actually close themselves off from a real chance to innovate. Learn from Failure —Tim Brown, CEO, IDEO Thomas Edison put it well when he said, “I have not failed. I’ve just found 10,000 ways that won’t work.” And for human-centered designers, sorting out what won’t work is part of finding what will. Failure is an inherent part of human-centered design because we rarely get it right on our first try. In fact, getting it right on the first try isn’t the point at all. The point is to put something out into the world and then use it to keep learning, keep asking, and keep testing. When human-centered designers get it right, it’s because they got it wrong first. 22 The Field Guide to Human-Centered Design Empathy —Emi Kolawole, Editor-in-Residence, Stanford University d.school Empathy is the capacity to step into other people’s shoes, to understand their lives, and start to solve problems from their perspectives. Human- centered design is premised on empathy, on the idea that the people you’re designing for are your roadmap to innovative solutions. All you have to do is empathize, understand them, and bring them along with you in the design process. For too long, the international development community has designed solutions to the challenges of poverty without truly empathizing with and understanding the people it’s looking to serve. But by putting ourselves in the shoes of the person we’re designing for, human-centered designers can start to see the world, and all the opportunities to improve it, through a new and powerful lens. Immersing yourself in another world not only opens you up to new creative possibilities, but it allows you to leave behind preconceived ideas and outmoded ways of thinking. Empathizing with the people you’re designing for is the best route to truly grasping the context and complexities of their lives. But most importantly, it keeps the people you’re designing for squarely grounded in the center of your work. 23 Mindsets a generative process, and because we work so collaboratively, it’s easy to discard bad ideas, hold onto pieces of the so-so ones, and eventually arrive at the good ones. Though it may seem counterintuitive, the ambiguity of not knowing the answer actually sets up human-centered designers to innovate. If we knew the answer when we started, what could we possibly learn? How could we come up with creative solutions? Where would the people we’re designing for guide us? Embracing ambiguity actually frees us to pursue an answer that we can’t initially imagine, which puts us on the path to routine innovation and lasting impact. Human-centered designers always start from the place of not knowing the answer to the problem they’re looking to solve. And in a culture that can be too focused on being the first one to the right answer, that’s not a particularly comfortable place to be. But by starting at square one, we’re forced to get out into the world and talk to the people we’re looking to serve. We also get to open up creatively, to pursue lots of different ideas, and to arrive at unexpected solutions. By embracing that ambiguity, and by trusting that the human- centered design process will guide us toward an innovative answer, we actually give ourselves permission to be fantastically creative. One of the qualities that sets human-centered designers apart is the belief that there will always be more ideas. We don’t cling to ideas any longer than we have to because we know that we’ll have more. Because human-centered design is such Embrace Ambiguity —Patrice Martin, Co-Lead and Creative Director, IDEO.org 24 The Field Guide to Human-Centered Design Human-centered designers are persistently focused on what could be, not the countless obstacles that may get in the way. Constraints are inevitable, and often they push designers toward unexpected solutions. But it’s our core animating belief—that every problem is solvable—that shows just how deeply optimistic human-centered designers are. We believe that design is inherently optimistic. To take on a big challenge, especially one as large and intractable as poverty, we have to believe that progress is even an option. If we didn’t, we wouldn’t even try. Optimism is the embrace of possibility, the idea that even if we don’t know the answer, that it’s out there and that we can find it. In addition to driving us toward solutions, optimism makes us more creative, encourages us to push on when we hit dead ends, and helps all the stakeholders in a project gel. Approaching problems from the perspective that you’ll get to a solution infuses the entire process with the energy and drive that you need to navigate the thorniest problems. Optimism —John Bielenberg, Founder, Future Partners 25 Mindsets At base, we iterate because we know that we won’t get it right the first time. Or even the second. Iteration allows us the opportunity to explore, to get it wrong, to follow our hunches, but ultimately arrive at a solution that will be adopted and embraced. We iterate because it allows us to keep learning. Instead of hiding out in our workshops, betting that an idea, product, or service will be a hit, we quickly get out in the world and let the people we’re designing for be our guides. As human-centered designers, we adopt an iterative approach to solving problems because it makes feedback from the people we’re designing for a critical part of how a solution evolves. By continually iterating, refining, and improving our work, we put ourselves in a place where we’ll have more ideas, try a variety of approaches, unlock our creativity, and arrive more quickly at successful solutions. Iteration keeps us nimble, responsive, and trains our focus on getting the idea and, after a few passes, every detail just right. If you aimed for perfection each time you built a prototype or shared an idea, you’d spend ages refining something whose validity was still in doubt. But by building, testing, and iterating, you can advance your idea without investing hours and resources until you’re sure that it’s the one. Iterate, Iterate, Iterate —Gaby Brink, Founder, Tomorrow Partners 26 The Field Guide to Human-Centered Design 27 Introduction METHODS 2828 N 29 Methods: Inspiration Phase 29 The Inspiration phase is about learning on the fly, opening yourself up to creative possibilities, and trusting that as long as you remain grounded in desires of the communities you’re engaging, your ideas will evolve into the right solutions. You’ll build your team, get smart on your challenge, and talk to a staggering variety of people. INSPIRATION THIS PHASE WILL HELP YOU ANSWER How do I get star ted? How do I conduct an inter view? How do I keep people at the center of my research? What are other tools I can use to understand people? 30 The Field Guide to Human-Centered Design 31 Methods: Inspiration Phase STEPS 01 Start by taking a first stab at writing your design challenge. It should be short and easy to remember, a single sentence that conveys what you want to do. We often phrase these as questions which set you and your team up to be solution-oriented and to generate lots of ideas along the way. 02 Properly framed design challenges drive toward ultimate impact, allow for a variety of solutions, and take into account constraints and context. Now try articulating it again with those factors in mind. 03 Another common pitfall when scoping a design challenge is going either too narrow or too broad. A narrowly scoped challenge won’t offer enough room to explore creative solutions. And a broadly scoped challenge won’t give you any idea where to start. 04 Now that you’ve run your challenge through these filters, do it again. It may seem repetitive, but the right question is key to arriving at a good solution. A quick test we often run on a design challenge is to see if we can come up with five possible solutions in just a few minutes. If so, you’re likely on the right track. Getting the right frame on your design challenge will get you off on the right foot, organize how you think about your solution, and at moments of ambiguity, help clarif y where you should push your design. Framing your design challenge is more art than science, but there are a few key things to keep in mind. First, ask yourself: Does my challenge drive toward ultimate impact, allow for a variety of solutions, and take into account context? Dial those in, and then refine it until it’s the challenge you’re excited to tackle. TIME 90 minutes DIFFICULTY Hard WHAT YOU’LL NEED Pen, Frame Your Design Challenge worksheet p. 165 PARTICIPANTS Design team Properly framing your design challenge is critical to your success. Here’s how to do it just right. Frame Your Design Challenge 32 The Field Guide to Human-Centered Design Frame Your Design Challenge It’s rare that you’ll Frame Your Design Challenge just right on the first try; at IDEO.org we often go through a number of revisions and lots of debate as we figure out precisely how to hone the problem we’re looking to solve. For the second challenge in our Amplif y program, we knew that we wanted to focus on children’s education, but needed to narrow the scope so that it would drive real impact, allow for a variety of solutions, and still give us enough context to get started. Challenge manager Chioma Ume described how she and her team sharpened the challenge. “We knew we wanted to do something around kids, but of course we then have to ascertain which kids. Should it be all kids, just teens, young kids? Because of the tremendous importance of early childhood development, we settled on children, ages zero to five. But we certainly didn’t start knowing that we’d focus just on them.” Even then, the challenge needed refinement. By eventually landing not on children, but their parents, the team and its partners at the UK’s Department for International Development, crafted a brief that it thought would have the most impact. “We chose to focus on the people closest to children, their parents,” says Ume. But she stresses that though parents became the focus, the children remained the beneficiaries, a nuance that would keep the team from spinning off or focusing too heavily on improving parents’ lives. In the end, the team arrived at a well framed challenge, one that asks: How might parents in low-income communities ensure children thrive in their first five years? Use the Frame Your Design Challenge worksheet on p. 165 and take multiple passes to make sure that your question drives at impact, gives you a starting place, but still is broad enough to allow for a great variety of creative answers. METHOD IN ACTION 33 Methods: Inspiration Phase 1) Take a stab at framing it as a design question.  2) Now, state the ultimate impact you’re trying to have. 3) What are some possible solutions to your problem? Think broadly. It’s fine to start a project with a hunch or two, but make sure you allow for surprising outcomes. 4) Finally, write down some of the context and constraints that you’re facing. They could be geographic, technological, time-based, or have to do with the population you’re trying to reach. 5) Does your original question need a tweak? Try it again. What is the problem you’re trying to solve? Improving the lives of children. How might we improve the lives of children? We want very young children in low-income communities to thrive. Better nutrition, parents engaging with young kids to spur brain development, better education around parenting, early childhood education centers, better access to neonatal care and vaccines. Because children aren’t in control of their circumstances, we wanted to address our solution to their parents. We want a solution that could work across different regions. How might parents in low-income communities ensure children thrive in their first five years. Frame Your Design Challenge 34 The Field Guide to Human-Centered Design STEPS 01 A good place to start is with a calendar. Print out or make a large one and put it up in your workspace. Now mark key dates. They could be deadlines, important meetings, travel dates, or times when your team members are unavailable. 02 Now that you’ve got a sense of your timeline, look at your budget and staff. Do you have everything that you’ll need? If you foresee constraints, how can you get around them? 03 You’ll need to get smart on your topic before you head into the field. Who should you talk to now? What will you need to read to be up to speed on your challenge? 04 Answer questions like: When should my team head into the field? Will my team make one visit or two? Will our partners be visiting? Will we need to physically make something? How much time, money, and manpower will we need to produce it? 05 Your project plan will change as things evolve, and that’s perfectly OK. You can always amend things as you go but make sure that you’re really thinking through your project before you start. As you set out to solve your challenge, you’ll need to create a plan. This gives you a chance to think through all the logistics of your project, and even though they’re bound to change as things progress, you’ll be in much better shape if you can plan for what’s ahead. Reflect on your timeline, the space you’ll work in, your staff, your budget, what skills you’ll need, trips you’ll take, and what you’ll need to produce. Getting a good handle on all of this information can keep you on track. TIME 60-90 minutes DIFFICULTY Moderate WHAT YOU’LL NEED Pen, paper, Post-its, calendar PARTICIPANTS Design team Create a Project Plan Get organized, understand your strengths, and start identifying what your team will need to come up with innovative solutions. 35 Methods: Inspiration Phase STEPS 01 First, assess how many team members you’ll need, your staff’s availability, and when your project should start and end. 02 Look at the core members of your team and determine what they’re good at and what they’re not so good at. 03 Is there a clear technical capability that you’ll need but don’t currently have—maybe a mechanical engineer, a graphic designer, a skilled writer? Remember that you can always add a team member for a shorter period of time when their skills are most important. Human-centered design works best with cross-disciplinary teams. You could put three business designers to work on a new social enterprise, but if you throw a graphic designer, a journalist, or an industrial designer into the mix, you’re going to bring new modes of thinking to your team. It’s smart to have a hunch about what kind of talent your team will need—if you’re designing a social enterprise, a business designer is probably a good bet—but you won’t get unexpected solutions with an expected team. TIME 60 minutes DIFFICULTY Hard WHAT YOU’LL NEED Pen, paper PARTICIPANTS Project lead, partner organizations An interdisciplinary mix of thinkers, makers, and doers is just the right combination to tackle any design challenge. Build a Team 36 The Field Guide to Human-Centered Design Before you start talking to the people you’re designing for, it’s important to have a strategy around who you talk to, what you ask them, and what pieces of information you need to gather. By planning ahead, and tracking …
74014
CATEGORIES
Economics Nursing Applied Sciences Psychology Science Management Computer Science Human Resource Management Accounting Information Systems English Anatomy Operations Management Sociology Literature Education Business & Finance Marketing Engineering Statistics Biology Political Science Reading History Financial markets Philosophy Mathematics Law Criminal Architecture and Design Government Social Science World history Chemistry Humanities Business Finance Writing Programming Telecommunications Engineering Geography Physics Spanish ach e. Embedded Entrepreneurship f. Three Social Entrepreneurship Models g. Social-Founder Identity h. Micros-enterprise Development Outcomes Subset 2. Indigenous Entrepreneurship Approaches (Outside of Canada) a. Indigenous Australian Entrepreneurs Exami Calculus (people influence of  others) processes that you perceived occurs in this specific Institution Select one of the forms of stratification highlighted (focus on inter the intersectionalities  of these three) to reflect and analyze the potential ways these ( American history Pharmacology Ancient history . Also Numerical analysis Environmental science Electrical Engineering Precalculus Physiology Civil Engineering Electronic Engineering ness Horizons Algebra Geology Physical chemistry nt When considering both O lassrooms Civil Probability ions Identify a specific consumer product that you or your family have used for quite some time. This might be a branded smartphone (if you have used several versions over the years) or the court to consider in its deliberations. Locard’s exchange principle argues that during the commission of a crime Chemical Engineering Ecology aragraphs (meaning 25 sentences or more). Your assignment may be more than 5 paragraphs but not less. INSTRUCTIONS:  To access the FNU Online Library for journals and articles you can go the FNU library link here:  https://www.fnu.edu/library/ In order to n that draws upon the theoretical reading to explain and contextualize the design choices. Be sure to directly quote or paraphrase the reading ce to the vaccine. Your campaign must educate and inform the audience on the benefits but also create for safe and open dialogue. A key metric of your campaign will be the direct increase in numbers.  Key outcomes: The approach that you take must be clear Mechanical Engineering Organic chemistry Geometry nment Topic You will need to pick one topic for your project (5 pts) Literature search You will need to perform a literature search for your topic Geophysics you been involved with a company doing a redesign of business processes Communication on Customer Relations. Discuss how two-way communication on social media channels impacts businesses both positively and negatively. Provide any personal examples from your experience od pressure and hypertension via a community-wide intervention that targets the problem across the lifespan (i.e. includes all ages). Develop a community-wide intervention to reduce elevated blood pressure and hypertension in the State of Alabama that in in body of the report Conclusions References (8 References Minimum) *** Words count = 2000 words. *** In-Text Citations and References using Harvard style. *** In Task section I’ve chose (Economic issues in overseas contracting)" Electromagnetism w or quality improvement; it was just all part of good nursing care.  The goal for quality improvement is to monitor patient outcomes using statistics for comparison to standards of care for different diseases e a 1 to 2 slide Microsoft PowerPoint presentation on the different models of case management.  Include speaker notes... .....Describe three different models of case management. visual representations of information. They can include numbers SSAY ame workbook for all 3 milestones. You do not need to download a new copy for Milestones 2 or 3. When you submit Milestone 3 pages): Provide a description of an existing intervention in Canada making the appropriate buying decisions in an ethical and professional manner. Topic: Purchasing and Technology You read about blockchain ledger technology. Now do some additional research out on the Internet and share your URL with the rest of the class be aware of which features their competitors are opting to include so the product development teams can design similar or enhanced features to attract more of the market. The more unique low (The Top Health Industry Trends to Watch in 2015) to assist you with this discussion.         https://youtu.be/fRym_jyuBc0 Next year the $2.8 trillion U.S. healthcare industry will   finally begin to look and feel more like the rest of the business wo evidence-based primary care curriculum. Throughout your nurse practitioner program Vignette Understanding Gender Fluidity Providing Inclusive Quality Care Affirming Clinical Encounters Conclusion References Nurse Practitioner Knowledge Mechanics and word limit is unit as a guide only. The assessment may be re-attempted on two further occasions (maximum three attempts in total). All assessments must be resubmitted 3 days within receiving your unsatisfactory grade. You must clearly indicate “Re-su Trigonometry Article writing Other 5. June 29 After the components sending to the manufacturing house 1. In 1972 the Furman v. Georgia case resulted in a decision that would put action into motion. Furman was originally sentenced to death because of a murder he committed in Georgia but the court debated whether or not this was a violation of his 8th amend One of the first conflicts that would need to be investigated would be whether the human service professional followed the responsibility to client ethical standard.  While developing a relationship with client it is important to clarify that if danger or Ethical behavior is a critical topic in the workplace because the impact of it can make or break a business No matter which type of health care organization With a direct sale During the pandemic Computers are being used to monitor the spread of outbreaks in different areas of the world and with this record 3. Furman v. Georgia is a U.S Supreme Court case that resolves around the Eighth Amendments ban on cruel and unsual punishment in death penalty cases. The Furman v. Georgia case was based on Furman being convicted of murder in Georgia. Furman was caught i One major ethical conflict that may arise in my investigation is the Responsibility to Client in both Standard 3 and Standard 4 of the Ethical Standards for Human Service Professionals (2015).  Making sure we do not disclose information without consent ev 4. Identify two examples of real world problems that you have observed in your personal Summary & Evaluation: Reference & 188. Academic Search Ultimate Ethics We can mention at least one example of how the violation of ethical standards can be prevented. Many organizations promote ethical self-regulation by creating moral codes to help direct their business activities *DDB is used for the first three years For example The inbound logistics for William Instrument refer to purchase components from various electronic firms. During the purchase process William need to consider the quality and price of the components. In this case 4. A U.S. Supreme Court case known as Furman v. Georgia (1972) is a landmark case that involved Eighth Amendment’s ban of unusual and cruel punishment in death penalty cases (Furman v. Georgia (1972) With covid coming into place In my opinion with Not necessarily all home buyers are the same! When you choose to work with we buy ugly houses Baltimore & nationwide USA The ability to view ourselves from an unbiased perspective allows us to critically assess our personal strengths and weaknesses. This is an important step in the process of finding the right resources for our personal learning style. Ego and pride can be · By Day 1 of this week While you must form your answers to the questions below from our assigned reading material CliftonLarsonAllen LLP (2013) 5 The family dynamic is awkward at first since the most outgoing and straight forward person in the family in Linda Urien The most important benefit of my statistical analysis would be the accuracy with which I interpret the data. The greatest obstacle From a similar but larger point of view 4 In order to get the entire family to come back for another session I would suggest coming in on a day the restaurant is not open When seeking to identify a patient’s health condition After viewing the you tube videos on prayer Your paper must be at least two pages in length (not counting the title and reference pages) The word assimilate is negative to me. I believe everyone should learn about a country that they are going to live in. It doesnt mean that they have to believe that everything in America is better than where they came from. It means that they care enough Data collection Single Subject Chris is a social worker in a geriatric case management program located in a midsize Northeastern town. She has an MSW and is part of a team of case managers that likes to continuously improve on its practice. The team is currently using an I would start off with Linda on repeating her options for the child and going over what she is feeling with each option.  I would want to find out what she is afraid of.  I would avoid asking her any “why” questions because I want her to be in the here an Summarize the advantages and disadvantages of using an Internet site as means of collecting data for psychological research (Comp 2.1) 25.0\% Summarization of the advantages and disadvantages of using an Internet site as means of collecting data for psych Identify the type of research used in a chosen study Compose a 1 Optics effect relationship becomes more difficult—as the researcher cannot enact total control of another person even in an experimental environment. Social workers serve clients in highly complex real-world environments. Clients often implement recommended inte I think knowing more about you will allow you to be able to choose the right resources Be 4 pages in length soft MB-920 dumps review and documentation and high-quality listing pdf MB-920 braindumps also recommended and approved by Microsoft experts. The practical test g One thing you will need to do in college is learn how to find and use references. References support your ideas. College-level work must be supported by research. You are expected to do that for this paper. You will research Elaborate on any potential confounds or ethical concerns while participating in the psychological study 20.0\% Elaboration on any potential confounds or ethical concerns while participating in the psychological study is missing. Elaboration on any potenti 3 The first thing I would do in the family’s first session is develop a genogram of the family to get an idea of all the individuals who play a major role in Linda’s life. After establishing where each member is in relation to the family A Health in All Policies approach Note: The requirements outlined below correspond to the grading criteria in the scoring guide. At a minimum Chen Read Connecting Communities and Complexity: A Case Study in Creating the Conditions for Transformational Change Read Reflections on Cultural Humility Read A Basic Guide to ABCD Community Organizing Use the bolded black section and sub-section titles below to organize your paper. For each section Losinski forwarded the article on a priority basis to Mary Scott Losinksi wanted details on use of the ED at CGH. He asked the administrative resident