Exa 2 - Management
Answer Part 2) and Part 3) Only. No Outside References. Use the attached lectures and the readings pdf.
Due in 6 hours.
PH 30004
Exam 2
Questions on this exam come from class meetings, guest lectures, the visit to the DI Hub, provided videos and assigned chapter readings in Social Science Research, Testing Treatments, the IDEO book on human-centered design, and any other readings in Canvas.
The exam is due at the end of the day on Friday, October 22, 2021.
Respond to the questions on this page or use a separate document. If you do not use the test itself, please be certain to number your responses. I suggest you respond to every item;
Part 1 Multiple Choice – 10 questions; 2 points each
Choose the best response
1. In the book Testing Treatments, the authors recommend that in cases where the effect of a treatment is inconsistent or uncertain:
a. Practitioners express confidence so patients remain optimistic and trusting
b. Practitioners should discuss uncertainty with patients, and encourage further discussion about options, taking patient preferences into account
c. Practitioners should avoid any treatments that do not have immediate and clear benefits
d. Practitioners should recommend unproven treatments but not tell patients these are unproven as this knowledge might reduce the benefit or cause undue stress.
e. This is not actually a problem as uncertain treatments are never legally available to practitioners.
2. In the book Social Science Research, the author argues for use of a/an ____ number of values (choices) in a Likert-type scale because _________ :
a. Odd; this allows inclusion of a neutral choice since people may be neutral in their view on something
b. Even; this forces respondents to identify whether they agree or disagree
c. Larger; this allow participants to express more range in their responses
d. Smaller; this helps counter the ordinal nature of these items so improves analysis
e. Varying; this helps improve reliability of responses
3. According to information in the IDEO book (“Field Guide to Human-Centered Design”), design mindsets include or encompass:
a. Optimism and embracing ambiguity
b. Empathy, learning from failure, and creative confidence
c. Iterating
d. Making things
e. A and B
f. A, B, C
g. A, B, C, and D
4. In simple and explicit terms, p > 0.05 means (see Testing Treatments, chapter 7):
a. The researcher has achieved statistical significance
b. Given the data provided, the probability of something occurring is .05 \%
c. Given the data provided, the probability of something occurring is less than 1 in 20
d. Fail to reject the null hypothesis
5. According to information provided in the video about the Ottawa Charter, action areas to achieve health include:
a. Reorienting health services; creating supportive environments; providing national health care; developing personal skills
b. Strengthening community action; creating supportive environments; developing personal skills; reorienting health services
c. Reorienting health services; creating supportive environments; subsidizing costs for healthy food; subsidizing costs for access to health-promoting physical activity.
d. Reorienting health services; developing personal skills; providing national health care; subsidizing costs for healthy food; subsidizing costs for access to health-promoting physical activity.
e. None of these is correct
6. In the book Testing Treatments, the authors identify which potential issues that potentially undermine (interfere with) the credibility of a systematic review of prior research:
a. Conflicts of interests such as funding provided by a treatment manufacturer
b. Planning to publish the review in a high impact journal
c. Failure to plan a systematic review using a specific research protocol that specifies eligibility criteria, selection process, and other details
d. Failure to identify all of the relevant evidence, such as occurs when unsuccessful trials are not published
e. Use of a large research team to conduct a review
f. All of the above
g. A, B, C and D
h. A, C and D
7. Ms. Emily Nelson, doctoral candidate and guest lecturer, demonstrated use of Microsoft Excel to:
a. Track time/duration of excerpts of text/meaning units
b. Assign descriptions to meaning units
c. Categorize meaning units based on participant values or priorities
d. Calculate p values from qualitative data transformed into categories
e. All of the above
f. A, B, and D only
g. A, B, and C Only
8. According to information in the IDEO book (“Field Guide to Human-Centered Design”), the three phases of any design challenge include:
a. Inspiration, ideation, integration
b. Inspiration, implementation, evaluation
c. Inspiration, ideation, evaluation
d. Integration, evaluation, iteration
e. None of these
9. According to the guest lecture during class, what is the biggest weakness or limitation as respects Community Health Assessment (CHA) information for older adults, currently available to the Kent City Health Department:
a. Older adults are less likely to use the internet so might not refer to the Kent Health Dept Facebook page
b. The Health Dept’s new location at the top of the PARTA Gateway is less convenient than the old location on Main St.
c. CHA surveys track adult responses from ages 18 + as a single category
d. Surveys, focus group and key informant interviews are all used and it is difficult to take time to analyze the large amount of data
10. In the Youtube video about Needs Analysis, the presenter (Tim Slade) asserts:
a. The solution to most problems is to develop a learning or educational intervention
b. Results of a needs assessment might show that a learning intervention is not going to address the problem in a given context
c. The primary role of a needs assessment is to identify content for a learning or educational intervention
d. It is a mistake to build a training course to address an issue without first conducting a needs assessment
e. A and C
f. B and D
Part 2 Short Answer – 4 questions; 4 points each
Provide a brief, complete response as requested for each item. Partial credit may be given for partially correct responses.
1. List and describe two differences between the “scientific method” and human-centered design, as used in design innovation methods.
2. You created a survey with 32 items. In one subscale (section) of 16 items, you found respondents provided inconsistent answers (i.e., they agreed with something one time and disagreed with the same thing in a later question). You reduced the length of the subscale to 8 items and found you had more consistent responses. Which potential cause of methods bias was most likely responsible for the inconsistent answers? (refer to the list of items in the article by Podsakoff et al., in Table 2 on page 882).
3. Review the sample wording, following, in italic font, related to research methods. Within this wording, what quality criteria efforts were used or identified by the authors? You should identify at minimum 5 items. You do not have to define or describe what the quality criteria mean.
We gathered primary qualitative and quantitative data and integrated analysis of secondary sources. We conducted interviews with 16 purposively selected individuals who were experiencing depression. Interviewees’ mental health was verified through review of their scores on the Beck Depression Inventory (BDI). The BDI has been shown to be reliable and have construct validity in multiple contexts. We additionally administered a newly developed 45-item survey to 300 participants identified by a mental health counseling service provider. Exploratory factor analysis of the results indicated there were three distinct constructs measured in the survey. Item loadings ranged from .21 to .89.
Interview transcripts were created by one author and checked for accuracy by another. Surveys were administered in paper format; results were entered into Microsoft Excel by a research assistant. Following, a random sample of 25\% of data points was checked for accuracy by one of the authors.
4. Describe an example where a measurement, such as a survey item, might be reliable but not accurate. (Refer to Chapter 7 in Social Science Research for helpful information).
Part 3 Longer Response – 2 questions; 7 points each. Please read the instructions carefully.
Partial credit is available for partially correct responses.
1. . Use the design method “banned” – where you imagine “a world if a product, service system or experience did not exist and how people would possibly adapt” (Curedale, 2013, p. 52) and, in 3-4 sentences of bullet points, describe what the consequences might be if there was no government or private collection of garbage and recyclable items available.
2. Assume you have been asked to conduct a community needs assessment related to health needs of older adults living in Kent, Ohio. Provide a response to these items:
a. How would you define older adults? What criteria or sources would you use to develop this definition? What possible problems might arise from using a specific age or age range to define older adults?
b. How would you find out how many older adults are in Kent, Ohio?
c. If you wanted to recruit participants for interview or survey research, what are three methods or sources, other than social media, you might use to communicate with older adults in Kent, Ohio?
Assessing Community Health: Collecting Data, Sampling, and Getting Accurate Data
Mike Anguilano, Accreditation Coordiantor/Public Information Officer
Kent City Health Department
September 27, 2021
Collecting Data at the State Level
State health departments must do a State Health Assessment (SHA) and a corresponding State Health Improvement Plan (SHIP)
Involves surveys, focus groups, key informant interviews, and other community events to collect qualitative and quantitative data
From the results, the state determines focal points for local health departments (LHDs)
Local health departments must align their goals and programs to be in sync with the SHA findings
Collecting Data at the Local Level
LHDs planning to pursue accreditation (mandatory in Ohio) must have a community health assessment (CHA) completed every three years
In Portage County, the CHA steering committee and community partners gather to create a survey with core CDC-required questions and county-specific questions
Three surveys are created for each age group (0-11, 12-17, 18+)
Cross-sectional randomized sampling conducted in the county
Every school district participates in the survey
Kent State students are not included in the CHA – why?
Kent and the Community Health Assessment
Kent is the largest city in Portage County – for 10 out of 12 months
Very transient population makes it difficult to get representative results for the county
KSU students can skew the results
The Kent population turns over every year - this makes it tough to determine what the actual problems are
For the 2016 CHA, Kent was oversampled to get a more comprehensive overview of the city
Kent State University holds its own college health assessment, though it is not as robust as the county one
Other Forms of Data Collection
While CHAs are the biggest public health endeavor in the county, they are not the only time to collect data
In recent years KCHD has taken steps to collect more primary data specific to the City of Kent:
Held focus groups relating to LGBTQ health care access, tobacco control, COVID-19 (virtual), and vaccinations (pre-COVID)
Utilize surveys for program development, post-program assessment, and community perceptions
Key informant interviews – speak with key personnel or potential partners on projects and programs; specific questions for a select group of people
What are the Best Methods for Data Collection?
Depends on several factors
What is the goal/what will you do with the data?
How much data do you need to collect/what is a representative sample?
What kind of data will you need to collect? (qualitative, quantitative)
Is there a budget or limited number of resources to collect the data?
Who is the target audience? (ex. Older adults, youth, certain cultures or ethnicities)
Not really any right or wrong methods, but some are better than others and utilize resources better
Determining, and Reaching, Your Audience
Regardless of which method used, you need to consider the target audience and barriers in reaching them
Know your audience
Consider technological, cultural, age, ethnic, transportation, and socioeconomic barriers when choosing a sampling/data collection method
Plan for attrition in the sampling process – if you send out 30 surveys, plan on getting 15 back
Ethical considerations for research purposes must be followed
This is especially important for children, older adults, disadvantaged/disenfranchised, and disabled individuals
Review
Sampling is important – you may only get one opportunity
Data collection is an ongoing process and oftentimes slow - Rome was not built in a day and neither will your research project
There are a lot of things to consider with regards to barriers – don’t hamstring your project early on, as making changes can be difficult
Try not to lose sight of your goal – why are you sampling a certain population? What is the ultimate goal?
Consider your time, money, and other resources – they go quickly if not used properly
Questions?
Contact Info:
Mike Anguilano III, BSPH
Accreditation Coordinator/Public Information Officer
330-678-8109 ext. 5204
Useful information for assessing community needs
https://www.youtube.com/watch?v=G2quVLcJVBk
This video ties together individual health promotion, public health, and healthcare in an holistic approach to needs assessment
https://www.youtube.com/watch?v=-gIpE8kxFPk
This video presents a common sense discussion about how to determine whether needs will be met through informational or educational programming
PH 30004
Week 4 September 20, 2021
Update/schedule
Exams
Card
Data gathering/analysis assignment due 9/29 – no class meeting
Will review today
Secondary data – in class today
Qualtrics – variations in questions – in class Friday
Kent Health Dept speaker Monday 9/27
DI Hub Friday 10/1
Miro – invitations sent – explore as you like; class time eventually
MS Teams – plan to use in class at some point for screen sharing
Making questions into researchable questions
What is the relationship between sleep and mental health?
What is the relationship between online school and mental health?
What is the correlation between gut health and mental health?
What is the impact of COVID-19 on mental health?
Does marijuana usage have an affect on physical health?
Is working from home responsible for people’s weight gains during the COVID-19 pandemic?
Precision and measurability
Designing informational/interview questions
Finding data
Working through one example
How do people perceive their risk of injury due to motor vehicle accidents?
People – who – anyone, or just people who drive? If people who drive, how much or under what circumstances? Comparing groups leads to a different question but might be interesting
Risk, risk perception – defined how? Is there a risk perception instrument available for use?
Injury – any injury, any physical injury, or another definition?
Motor vehicle – cars, e bikes, others?
Accident – collision? Single car accident?
Figure out what terms need to be defined, described, or limited
Decide on specificity
Other research
Other resources (medical, government/census) – generally use scientific or academic, not informal language
Identify the areas of question
Share
Time permitting – revise the question
Update
Kent HD – Monday a.m.
No class 9/29 – assignment due
DI Hub Friday at 10
Today
Qualtrics activity
Introduce ICPRS extra credit
Review assignment details
Due 9/29/2021 – end of day. I will pre-review your draft and respond to your questions!
ICPSR
https://www.icpsr.umich.edu/web/pages/instructors/icsc/
Read (or skim) about the project/introduction/social capital/datasets
Go to exercises – search for the variable: CLUBMEET in the DDB and LEADGRP in the GSS
Screenshot the frequency distribution table for each
See Canvas for more
Qualtrics
Demo DI survey plus skip logic
Qualtrics
Go to Qualtrics
Create a survey – alone or in groups
Use one of these questions to guide your survey
What is the impact of COVID-19 on depression?
Does marijuana usage have an affect on back pain?
What is the relationship between sleep and anxiety?
Create 4 different types of questions – ex. Multiple choice is one type; text entry is another
(optional) use one instance of skip logic
Share question types with class – see how many unique types there are
PH 30004
Fall Semester 2021
Week of October 4
Sequence
Review schedule changes
Content and assignment due dates
Review, recap, and contextualization of content to date
Begin measurement and evaluation
Innovation in research design (Wed, next week if needed)
Schedule - important dates and changes
October 8, 2021 – guest speaker KHD
Wed, October 13, 2021 – new due date for next assignment
Review details on 10/06
October 15, 2021 – no class meeting Fall break
October 22, 2021 – Exam 2 (take home)
November 12, 2021 – Exam 3 (take home)
November 22, 2021 – new due date for assignment
December 10, 2021 – final proposal
December 17, 10:15 – 12:30 – final exam (take home)
Review, recap, contextualization
PH research is:
examination or study
Of hypotheses, premises, or theories,
Conducted by scientists, scholars and lay persons, through use of systematic processes or tests
to identify discoveries, develop solutions, and derive conclusions
That are credible and useful
For knowledge and benefit of communities, scientists, academics, professionals and other impacted populations
Review, recap, contextualization
Definition developed by a content analysis process
Process to identify relevant terms: assembling data
Clustering similar terms and naming clusters: data analysis via category development or thematic analysis, using provided/ a priori categories
Agreement via voting on best terms: interpretation via arriving at consensus based on individual expertise/preferences
Transformation of terms into narrative text: presentation of findings
Review, recap, contextualization
Review of common and useful research designs
Trial and experimental designs
Useful when comparing a treatment to an alternative
Applicable for human science and other interests (e.g., working with samples or non-humans)
Difficult to identify and remove other influences; limiting generalizability
Efficacy focused so tend to de-emphasize behavioral/individual aspects
Field surveys and focus groups
Useful to gather data from many responses
Can quickly key in on priorities or trends
Online survey programs do data analysis (surveys)
Difficult to obtain random sample so difficult to generalize
Results can be skewed – bots (surveys), groups that attempt to engineer responses (surveys or FGs), dominant individuals (FGs)
Review, recap, contextualization
Case research or ethnographic methods
Can capture in depth detail about best, worst, or middle performing systems or entities
Time and labor intensive
Generalizability limited to similar contexts
Action research
More likely to address participant-identified priorities
These may differ from researcher priorities
In conflict with scholarly or funding aims of research - interest in ongoing improvement rather than specific findings
Review, recap, contextualization
Secondary data analysis
Researcher or funder-driven
Incentives to work with and promote some datasets
Convenient sometimes instant way to conduct longitudinal/panel research
Large, random and/or population-based data
Review, recap, contextualization
Secondary data analysis
Not necessarily likely to address one’s specific interest, requiring revision of research questions or aims
Access may be difficult or limited
Data are rarely current; typically 18 months or older
Systematic reviews, meta-summaries, and meta-analyses are also types of secondary data analysis – data are results of prior research
Review, recap, contextualization
Priority on most frequent responses
Priority on treatments or solutions that are:
Convenient or
Profit-making or
Trendy or
Currently among government/agency funding priorities
Cost money to carry out
Major contributors of published research:
Students doing doctoral dissertations
Faculty seeking tenure
Review, recap, contextualization
Research in generaL
Short duration projects (10 – 30 weeks) are most appealing
Emphasis on problem identification rather than development of solutions
Design innovation
Aspects of action/community-based research
Human science methods
Adds innovative processes, i.e., variations on brainstorming
Prioritizes sustainability, even circular design
Emphasis on development of solutions
Review, recap, contextualization
Data – qualitative, quantitative, mixed/blended
Associated with some designs
Field survey – fixed responses
Focus group – interview recordings or transcripts
Trial – scores or efficacy measures
Transformed data
Quantitizing – words to categories
Mean centering; logarithmic transformations
Additional data types can be incorporated into most designs
Counter small sample size
Address ambiguous results
Explain unexpected findings
Review, recap, contextualization
Research design is the framework for answering your question or addressing your interest
Data are the bits of information, assembled within the framework, which undergo processes of compilation, combination, aggregation, transformation, and interpretation to provide an integrated and/or consensus response
Design should be focused and adequate to answer the question
Data should be sufficient and credible enough to answer the question
Data and design should be coherent
Schedule - important dates and changes
October 8, 2021 – guest speaker KHD
Wed, October 13, 2021 – new due date for next assignment
Review details on 10/06
No class meeting
October 15, 2021 – no class meeting Fall break
October 22, 2021 – Exam 2 (take home)
November 12, 2021 – Exam 3 (take home)
November 22, 2021 – new due date for assignment
Measurement
Biological/physiological
Blood pressure
Heart rate
Absence/presence of disease
Challenges?
Consistency
Other factors (“white coat” syndrome)
False positives, false negatives
Interpretation
Measurement
Knowledge
Symptoms
Health literacy
Challenges?
Interpretation, i.e., what constitutes irregular shape or change in appearance?
Language – what does “usually” or “often” mean? What is the difference between “agree,” “somewhat agree,” and “strongly agree”? ”Good” health compared to what?
Measurement
Traits, concepts, constructs and other subjectively defined things
Self-report
Diagnostic contributions (observation and results of interviews, e.g., intake survey for mental health treatment)
Challenges
No universally agreed upon definition for complex constructs
Many are context-specific
Many are individual-specific
Many are both context and individual-specific
Measurement Standards
Measurement
Mathematical
Reliability
Validity
Interpretive
Saturation
Validity
Consensus coding
Quality
Checking entry/checking transcripts
Assessment of normality
Model fit indices
Member checking
Interrater reliability
Quality efforts during scale development
Factor analysis – exploratory and confirmatory
Identify all potential items
Administer survey
Use mathematical/statistical methods to cluster scores
Look at clusters and assign a name
(similar to thematic analysis)
Delphi / expert panels
Identify items OR screen and comment on items
Add, delete, modify items
Rank and score items
Approve final instrument or measurement process
https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-018-5638-8
Examples of several methods used to develop a scale
Common methods bias
List of issues
Remedies
Multi-method and multi-trait
Comparative case studies
Triangulation
Multiple analysts
PLoS Medicine | www.plosmedicine.org 0696
Essay
Open access, freely available online
August 2005 | Volume 2 | Issue 8 | e124
Published research fi ndings are sometimes refuted by subsequent evidence, with ensuing confusion
and disappointment. Refutation and
controversy is seen across the range of
research designs, from clinical trials
and traditional epidemiological studies
[1–3] to the most modern molecular
research [4,5]. There is increasing
concern that in modern research, false
fi ndings may be the majority or even
the vast majority of published research
claims [6–8]. However, this should
not be surprising. It can be proven
that most claimed research fi ndings
are false. Here I will examine the key
factors that infl uence this problem and
some corollaries thereof.
Modeling the Framework for False
Positive Findings
Several methodologists have
pointed out [9–11] that the high
rate of nonreplication (lack of
confi rmation) of research discoveries
is a consequence of the convenient,
yet ill-founded strategy of claiming
conclusive research fi ndings solely on
the basis of a single study assessed by
formal statistical signifi cance, typically
for a p-value less than 0.05. Research
is not most appropriately represented
and summarized by p-values, but,
unfortunately, there is a widespread
notion that medical research articles
should be interpreted based only on
p-values. Research fi ndings are defi ned
here as any relationship reaching
formal statistical signifi cance, e.g.,
effective interventions, informative
predictors, risk factors, or associations.
“Negative” research is also very useful.
“Negative” is actually a misnomer, and
the misinterpretation is widespread.
However, here we will target
relationships that investigators claim
exist, rather than null fi ndings.
As has been shown previously, the
probability that a research fi nding
is indeed true depends on the prior
probability of it being true (before
doing the study), the statistical power
of the study, and the level of statistical
signifi cance [10,11]. Consider a 2 × 2
table in which research fi ndings are
compared against the gold standard
of true relationships in a scientifi c
fi eld. In a research fi eld both true and
false hypotheses can be made about
the presence of relationships. Let R
be the ratio of the number of “true
relationships” to “no relationships”
among those tested in the fi eld. R
is characteristic of the fi eld and can
vary a lot depending on whether the
fi eld targets highly likely relationships
or searches for only one or a few
true relationships among thousands
and millions of hypotheses that may
be postulated. Let us also consider,
for computational simplicity,
circumscribed fi elds where either there
is only one true relationship (among
many that can be hypothesized) or
the power is similar to fi nd any of the
several existing true relationships. The
pre-study probability of a relationship
being true is R⁄(R + 1). The probability
of a study fi nding a true relationship
refl ects the power 1 − β (one minus
the Type II error rate). The probability
of claiming a relationship when none
truly exists refl ects the Type I error
rate, α. Assuming that c relationships
are being probed in the fi eld, the
expected values of the 2 × 2 table are
given in Table 1. After a research
fi nding has been claimed based on
achieving formal statistical signifi cance,
the post-study probability that it is true
is the positive predictive value, PPV.
The PPV is also the complementary
probability of what Wacholder et al.
have called the false positive report
probability [10]. According to the 2
× 2 table, one gets PPV = (1 − β)R⁄(R
− βR + α). A research fi nding is thus
The Essay section contains opinion pieces on topics
of broad interest to a general medical audience.
Why Most Published Research Findings
Are False
John P. A. Ioannidis
Citation: Ioannidis JPA (2005) Why most published
research fi ndings are false. PLoS Med 2(8): e124.
Copyright: © 2005 John P. A. Ioannidis. This is an
open-access article distributed under the terms
of the Creative Commons Attribution License,
which permits unrestricted use, distribution, and
reproduction in any medium, provided the original
work is properly cited.
Abbreviation: PPV, positive predictive value
John P. A. Ioannidis is in the Department of Hygiene
and Epidemiology, University of Ioannina School of
Medicine, Ioannina, Greece, and Institute for Clinical
Research and Health Policy Studies, Department of
Medicine, Tufts-New England Medical Center, Tufts
University School of Medicine, Boston, Massachusetts,
United States of America. E-mail: [email protected]
Competing Interests: The author has declared that
no competing interests exist.
DOI: 10.1371/journal.pmed.0020124
Summary
There is increasing concern that most
current published research fi ndings are
false. The probability that a research claim
is true may depend on study power and
bias, the number of other studies on the
same question, and, importantly, the ratio
of true to no relationships among the
relationships probed in each scientifi c
fi eld. In this framework, a research fi nding
is less likely to be true when the studies
conducted in a fi eld are smaller; when
effect sizes are smaller; when there is a
greater number and lesser preselection
of tested relationships; where there is
greater fl exibility in designs, defi nitions,
outcomes, and analytical modes; when
there is greater fi nancial and other
interest and prejudice; and when more
teams are involved in a scientifi c fi eld
in chase of statistical signifi cance.
Simulations show that for most study
designs and settings, it is more likely for
a research claim to be false than true.
Moreover, for many current scientifi c
fi elds, claimed research fi ndings may
often be simply accurate measures of the
prevailing bias. In this essay, I discuss the
implications of these problems for the
conduct and interpretation of research.
It can be proven that
most claimed research
fi ndings are false.
PLoS Medicine | www.plosmedicine.org 0697
more likely true than false if (1 − β)R
> α. Since usually the vast majority of
investigators depend on α = 0.05, this
means that a research fi nding is more
likely true than false if (1 − β)R > 0.05.
What is less well appreciated is
that bias and the extent of repeated
independent testing by different teams
of investigators around the globe may
further distort this picture and may
lead to even smaller probabilities of the
research fi ndings being indeed true.
We will try to model these two factors in
the context of similar 2 × 2 tables.
Bias
First, let us defi ne bias as the
combination of various design, data,
analysis, and presentation factors that
tend to produce research fi ndings
when they should not be produced.
Let u be the proportion of probed
analyses that would not have been
“research fi ndings,” but nevertheless
end up presented and reported as
such, because of bias. Bias should not
be confused with chance variability
that causes some fi ndings to be false by
chance even though the study design,
data, analysis, and presentation are
perfect. Bias can entail manipulation
in the analysis or reporting of fi ndings.
Selective or distorted reporting is a
typical form of such bias. We may
assume that u does not depend on
whether a true relationship exists
or not. This is not an unreasonable
assumption, since typically it is
impossible to know which relationships
are indeed true. In the presence of bias
(Table 2), one gets PPV = ([1 − β]R +
uβR)⁄(R + α − βR + u − uα + uβR), and
PPV decreases with increasing u, unless
1 − β ≤ α, i.e., 1 − β ≤ 0.05 for most
situations. Thus, with increasing bias,
the chances that a research fi nding
is true diminish considerably. This is
shown for different levels of power and
for different pre-study odds in Figure 1.
Conversely, true research fi ndings
may occasionally be annulled because
of reverse bias. For example, with large
measurement errors relationships
are lost in noise [12], or investigators
use data ineffi ciently or fail to notice
statistically signifi cant relationships, or
there may be confl icts of interest that
tend to “bury” signifi cant fi ndings [13].
There is no good large-scale empirical
evidence on how frequently such
reverse bias may occur across diverse
research fi elds. However, it is probably
fair to say that reverse bias is not as
common. Moreover measurement
errors and ineffi cient use of data are
probably becoming less frequent
problems, since measurement error has
decreased with technological advances
in the molecular era and investigators
are becoming increasingly sophisticated
about their data. Regardless, reverse
bias may be modeled in the same way as
bias above. Also reverse bias should not
be confused with chance variability that
may lead to missing a true relationship
because of chance.
Testing by Several Independent
Teams
Several independent teams may be
addressing the same sets of research
questions. As research efforts are
globalized, it is practically the rule
that several research teams, often
dozens of them, may probe the same
or similar questions. Unfortunately, in
some areas, the prevailing mentality
until now has been to focus on
isolated discoveries by single teams
and interpret research experiments
in isolation. An increasing number
of questions have at least one study
claiming a research fi nding, and
this receives unilateral attention.
The probability that at least one
study, among several done on the
same question, claims a statistically
signifi cant research fi nding is easy to
estimate. For n independent studies of
equal power, the 2 × 2 table is shown in
Table 3: PPV = R(1 − βn)⁄(R + 1 − [1 −
α]n − Rβn) (not considering bias). With
increasing number of independent
studies, PPV tends to decrease, unless
1 − β < α, i.e., typically 1 − β < 0.05.
This is shown for different levels of
power and for different pre-study odds
in Figure 2. For n studies of different
power, the term βn is replaced by the
product of the terms β
i
for i = 1 to n,
but inferences are similar.
Corollaries
A practical example is shown in Box
1. Based on the above considerations,
one may deduce several interesting
corollaries about the probability that a
research fi nding is indeed true.
Corollary 1: The smaller the studies
conducted in a scientifi c fi eld, the less
likely the research fi ndings are to be
true. Small sample size means smaller
power and, for all functions above,
the PPV for a true research fi nding
decreases as power decreases towards
1 − β = 0.05. Thus, other factors being
equal, research fi ndings are more likely
true in scientifi c fi elds that undertake
large studies, such as randomized
controlled trials in cardiology (several
thousand subjects randomized) [14]
than in scientifi c fi elds with small
studies, such as most research of
molecular predictors (sample sizes 100-
fold smaller) [15].
Corollary 2: The smaller the effect
sizes in a scientifi c fi eld, the less likely
the research fi ndings are to be true.
Power is also related to the effect
size. Thus research fi ndings are more
likely true in scientifi c fi elds with large
effects, such as the impact of smoking
on cancer or cardiovascular disease
(relative risks 3–20), than in scientifi c
fi elds where postulated effects are
small, such as genetic risk factors for
multigenetic diseases (relative risks
1.1–1.5) [7]. Modern epidemiology is
increasingly obliged to target smaller
Table 1. Research Findings and True Relationships
Research
Finding
True Relationship
Yes No Total
Yes c(1 − β)R/(R + 1) cα/(R + 1) c(R + α − βR)/(R + 1)
No cβR/(R + 1) c(1 − α)/(R + 1) c(1 − α + βR)/(R + 1)
Total cR/(R + 1) c/(R + 1) c
DOI: 10.1371/journal.pmed.0020124.t001
Table 2. Research Findings and True Relationships in the Presence of Bias
Research
Finding
True Relationship
Yes No Total
Yes (c[1 − β]R + ucβR)/(R + 1) cα + uc(1 − α)/(R + 1) c(R + α − βR + u − uα + uβR)/(R + 1)
No (1 − u)cβR/(R + 1) (1 − u)c(1 − α)/(R + 1) c(1 − u)(1 − α + βR)/(R + 1)
Total cR/(R + 1) c/(R + 1) c
DOI: 10.1371/journal.pmed.0020124.t002
August 2005 | Volume 2 | Issue 8 | e124
PLoS Medicine | www.plosmedicine.org 0698
effect sizes [16]. Consequently, the
proportion of true research fi ndings
is expected to decrease. In the same
line of thinking, if the true effect sizes
are very small in a scientifi c fi eld,
this fi eld is likely to be plagued by
almost ubiquitous false positive claims.
For example, if the majority of true
genetic or nutritional determinants of
complex diseases confer relative risks
less than 1.05, genetic or nutritional
epidemiology would be largely utopian
endeavors.
Corollary 3: The greater the number
and the lesser the selection of tested
relationships in a scientifi c fi eld, the
less likely the research fi ndings are to
be true. As shown above, the post-study
probability that a fi nding is true (PPV)
depends a lot on the pre-study odds
(R). Thus, research fi ndings are more
likely true in confi rmatory designs,
such as large phase III randomized
controlled trials, or meta-analyses
thereof, than in hypothesis-generating
experiments. Fields considered highly
informative and creative given the
wealth of the assembled and tested
information, such as microarrays and
other high-throughput discovery-
oriented research [4,8,17], should have
extremely low PPV.
Corollary 4: The greater the
fl exibility in designs, defi nitions,
outcomes, and analytical modes in
a scientifi c fi eld, the less likely the
research fi ndings are to be true.
Flexibility increases the potential for
transforming what would be “negative”
results into “positive” results, i.e., bias,
u. For several research designs, e.g.,
randomized controlled trials [18–20]
or meta-analyses [21,22], there have
been efforts to standardize their
conduct and reporting. Adherence to
common standards is likely to increase
the proportion of true fi ndings. The
same applies to outcomes. True
fi ndings may be more common
when outcomes are unequivocal and
universally agreed (e.g., death) rather
than when multifarious outcomes are
devised (e.g., scales for schizophrenia
outcomes) [23]. Similarly, fi elds that
use commonly agreed, stereotyped
analytical methods (e.g., Kaplan-
Meier plots and the log-rank test)
[24] may yield a larger proportion
of true fi ndings than fi elds where
analytical methods are still under
experimentation (e.g., artifi cial
intelligence methods) and only “best”
results are reported. Regardless, even
in the most stringent research designs,
bias seems to be a major problem.
For example, there is strong evidence
that selective outcome reporting,
with manipulation of the outcomes
and analyses reported, is a common
problem even for randomized trails
[25]. Simply abolishing selective
publication would not make this
problem go away.
Corollary 5: The greater the fi nancial
and other interests and prejudices
in a scientifi c fi eld, the less likely
the research fi ndings are to be true.
Confl icts of interest and prejudice may
increase bias, u. Confl icts of interest
are very common in biomedical
research [26], and typically they are
inadequately and sparsely reported
[26,27]. Prejudice may not necessarily
have fi nancial roots. Scientists in a
given fi eld may be prejudiced purely
because of their belief in a scientifi c
theory or commitment to their own
fi ndings. Many otherwise seemingly
independent, university-based studies
may be conducted for no other reason
than to give physicians and researchers
qualifi cations for promotion or tenure.
Such nonfi nancial confl icts may also
lead to distorted reported results and
interpretations. Prestigious investigators
may suppress via the peer review process
the appearance and dissemination of
fi ndings that refute their fi ndings, thus
condemning their fi eld to perpetuate
false dogma. Empirical evidence
on expert opinion shows that it is
extremely unreliable [28].
Corollary 6: The hotter a
scientifi c fi eld (with more scientifi c
teams involved), the less likely the
research fi ndings are to be true.
This seemingly paradoxical corollary
follows because, as stated above, the
PPV of isolated fi ndings decreases
when many teams of investigators
are involved in the same fi eld. This
may explain why we occasionally see
major excitement followed rapidly
by severe disappointments in fi elds
that draw wide attention. With many
teams working on the same fi eld and
with massive experimental data being
produced, timing is of the essence
in beating competition. Thus, each
team may prioritize on pursuing and
disseminating its most impressive
“positive” results. “Negative” results may
become attractive for dissemination
only if some other team has found
a “positive” association on the same
question. In that case, it may be
attractive to refute a claim made in
some prestigious journal. The term
Proteus phenomenon has been coined
to describe this phenomenon of rapidly
Table 3. Research Findings and True Relationships in the Presence of Multiple Studies
Research
Finding
True Relationship
Yes No Total
Yes cR(1 − βn)/(R + 1) c(1 − [1 − α]n)/(R + 1) c(R + 1 − [1 − α]n − Rβn)/(R + 1)
No cRβn/(R + 1) c(1 − α)n/(R + 1) c([1 − α]n + Rβn)/(R + 1)
Total cR/(R + 1) c/(R + 1) c
DOI: 10.1371/journal.pmed.0020124.t003
DOI: 10.1371/journal.pmed.0020124.g001
Figure 1. PPV (Probability That a Research
Finding Is True) as a Function of the Pre-Study
Odds for Various Levels of Bias, u
Panels correspond to power of 0.20, 0.50,
and 0.80.
August 2005 | Volume 2 | Issue 8 | e124
PLoS Medicine | www.plosmedicine.org 0699
alternating extreme research claims
and extremely opposite refutations
[29]. Empirical evidence suggests that
this sequence of extreme opposites is
very common in molecular genetics
[29].
These corollaries consider each
factor separately, but these factors often
infl uence each other. For example,
investigators working in fi elds where
true effect sizes are perceived to be
small may be more likely to perform
large studies than investigators working
in fi elds where true effect sizes are
perceived to be large. Or prejudice
may prevail in a hot scientifi c fi eld,
further undermining the predictive
value of its research fi ndings. Highly
prejudiced stakeholders may even
create a barrier that aborts efforts at
obtaining and disseminating opposing
results. Conversely, the fact that a fi eld
is hot or has strong invested interests
may sometimes promote larger studies
and improved standards of research,
enhancing the predictive value of its
research fi ndings. Or massive discovery-
oriented testing may result in such a
large yield of signifi cant relationships
that investigators have enough to
report and search further and thus
refrain from data dredging and
manipulation.
Most Research Findings Are False
for Most Research Designs and for
Most Fields
In the described framework, a PPV
exceeding 50\% is quite diffi cult to
get. Table 4 provides the results
of simulations using the formulas
developed for the infl uence of power,
ratio of true to non-true relationships,
and bias, for various types of situations
that may be characteristic of specifi c
study designs and settings. A fi nding
from a well-conducted, adequately
powered randomized controlled trial
starting with a 50\% pre-study chance
that the intervention is effective is
eventually true about 85\% of the time.
A fairly similar performance is expected
of a confi rmatory meta-analysis of
good-quality randomized trials:
potential bias probably increases, but
power and pre-test chances are higher
compared to a single randomized trial.
Conversely, a meta-analytic fi nding
from inconclusive studies where
pooling is used to “correct” the low
power of single studies, is probably
false if R ≤ 1:3. Research fi ndings from
underpowered, early-phase clinical
trials would be true about one in four
times, or even less frequently if bias
is present. Epidemiological studies of
an exploratory nature perform even
worse, especially when underpowered,
but even well-powered epidemiological
studies may have only a one in
fi ve chance being true, if R = 1:10.
Finally, in discovery-oriented research
with massive testing, where tested
relationships exceed true ones 1,000-
fold (e.g., 30,000 genes tested, of which
30 may be the true culprits) [30,31],
PPV for each claimed relationship is
extremely low, even with considerable
Box 1. An Example: Science
at Low Pre-Study Odds
Let us assume that a team of
investigators performs a whole genome
association study to test whether
any of 100,000 gene polymorphisms
are associated with susceptibility to
schizophrenia. Based on what we
know about the extent of heritability
of the disease, it is reasonable to
expect that probably around ten
gene polymorphisms among those
tested would be truly associated with
schizophrenia, with relatively similar
odds ratios around 1.3 for the ten or so
polymorphisms and with a fairly similar
power to identify any of them. Then
R = 10/100,000 = 10−4, and the pre-study
probability for any polymorphism to be
associated with schizophrenia is also
R/(R + 1) = 10−4. Let us also suppose that
the study has 60\% power to fi nd an
association with an odds ratio of 1.3 at
α = 0.05. Then it can be estimated that
if a statistically signifi cant association is
found with the p-value barely crossing the
0.05 threshold, the post-study probability
that this is true increases about 12-fold
compared with the pre-study probability,
but it is still only 12 × 10−4.
Now let us suppose that the
investigators manipulate their design,
analyses, and reporting so as to make
more relationships cross the p = 0.05
threshold even though this would not
have been crossed with a perfectly
adhered to design and analysis and with
perfect comprehensive reporting of the
results, strictly according to the original
study plan. Such manipulation could be
done, for example, with serendipitous
inclusion or exclusion of certain patients
or controls, post hoc subgroup analyses,
investigation of genetic contrasts that
were not originally specifi ed, changes
in the disease or control defi nitions,
and various combinations of selective
or distorted reporting of the results.
Commercially available “data mining”
packages actually are proud of their
ability to yield statistically signifi cant
results through data dredging. In the
presence of bias with u = 0.10, the post-
study probability that a research fi nding
is true is only 4.4 × 10−4. Furthermore,
even in the absence of any bias, when
ten independent research teams perform
similar experiments around the world, if
one of them fi nds a formally statistically
signifi cant association, the probability
that the research fi nding is true is only
1.5 × 10−4, hardly any higher than the
probability we had before any of this
extensive research was undertaken!
DOI: 10.1371/journal.pmed.0020124.g002
Figure 2. PPV (Probability That a Research
Finding Is True) as a Function of the Pre-Study
Odds for Various Numbers of Conducted
Studies, n
Panels correspond to power of 0.20, 0.50,
and 0.80.
August 2005 | Volume 2 | Issue 8 | e124
PLoS Medicine | www.plosmedicine.org 0700
standardization of laboratory and
statistical methods, outcomes, and
reporting thereof to minimize bias.
Claimed Research Findings
May Often Be Simply Accurate
Measures of the Prevailing Bias
As shown, the majority of modern
biomedical research is operating in
areas with very low pre- and post-
study probability for true fi ndings.
Let us suppose that in a research fi eld
there are no true fi ndings at all to be
discovered. History of science teaches
us that scientifi c endeavor has often
in the past wasted effort in fi elds with
absolutely no yield of true scientifi c
information, at least based on our
current understanding. In such a “null
fi eld,” one would ideally expect all
observed effect sizes to vary by chance
around the null in the absence of bias.
The extent that observed fi ndings
deviate from what is expected by
chance alone would be simply a pure
measure of the prevailing bias.
For example, let us suppose that
no nutrients or dietary patterns are
actually important determinants for
the risk of developing a specifi c tumor.
Let us also suppose that the scientifi c
literature has examined 60 nutrients
and claims all of them to be related to
the risk of developing this tumor with
relative risks in the range of 1.2 to 1.4
for the comparison of the upper to
lower intake tertiles. Then the claimed
effect sizes are simply measuring
nothing else but the net bias that has
been involved in the generation of
this scientifi c literature. Claimed effect
sizes are in fact the most accurate
estimates of the net bias. It even follows
that between “null fi elds,” the fi elds
that claim stronger effects (often with
accompanying claims of medical or
public health importance) are simply
those that have sustained the worst
biases.
For fi elds with very low PPV, the few
true relationships would not distort
this overall picture much. Even if a
few relationships are true, the shape
of the distribution of the observed
effects would still yield a clear measure
of the biases involved in the fi eld. This
concept totally reverses the way we
view scientifi c results. Traditionally,
investigators have viewed large
and highly signifi cant effects with
excitement, as signs of important
discoveries. Too large and too highly
signifi cant effects may actually be more
likely to be signs of large bias in most
fi elds of modern research. They should
lead investigators to careful critical
thinking about what might have gone
wrong with their data, analyses, and
results.
Of course, investigators working in
any fi eld are likely to resist accepting
that the whole fi eld in which they have
spent their careers is a “null fi eld.”
However, other lines of evidence,
or advances in technology and
experimentation, may lead eventually
to the dismantling of a scientifi c fi eld.
Obtaining measures of the net bias
in one fi eld may also be useful for
obtaining insight into what might be
the range of bias operating in other
fi elds where similar analytical methods,
technologies, and confl icts may be
operating.
How Can We Improve
the Situation?
Is it unavoidable that most research
fi ndings are false, or can we improve
the situation? A major problem is that
it is impossible to know with 100\%
certainty what the truth is in any
research question. In this regard, the
pure “gold” standard is unattainable.
However, there are several approaches
to improve the post-study probability.
Better powered evidence, e.g., large
studies or low-bias meta-analyses,
may help, as it comes closer to the
unknown “gold” standard. However,
large studies may still have biases
and these should be acknowledged
and avoided. Moreover, large-scale
evidence is impossible to obtain for all
of the millions and trillions of research
questions posed in current research.
Large-scale evidence should be
targeted for research questions where
the pre-study probability is already
considerably high, so that a signifi cant
research fi nding will lead to a post-test
probability that would be considered
quite defi nitive. Large-scale evidence is
also particularly indicated when it can
test major concepts rather than narrow,
specifi c questions. A negative fi nding
can then refute not only a specifi c
proposed claim, but a whole fi eld or
considerable portion thereof. Selecting
the performance of large-scale studies
based on narrow-minded criteria,
such as the marketing promotion of a
specifi c drug, is largely wasted research.
Moreover, one should be cautious
that extremely large studies may be
more likely to fi nd a formally statistical
signifi cant difference for a trivial effect
that is not really meaningfully different
from the null [32–34].
Second, most research questions
are addressed by many teams, and
it is misleading to emphasize the
statistically signifi cant fi ndings of
any single team. What matters is the
Table 4. PPV of Research Findings for Various Combinations of Power (1 − β), Ratio
of True to Not-True Relationships (R), and Bias (u)
1 …
PH 30004
Fall 2021 Oct 11
Schedule reminders
No meeting on Oct 13 – assignment due end of day
No meeting Oct 15 – Fall break
Exam #2 Friday Oct 22 (study guide provided on Oct 18) – take home exam; no class meeting
Today – DI activity; brief lecture on human-centered design as compared to “scientific method”
Activity
Questions:
What is the relationship between sleep and mental health
What is the relationship between online school and mental health
What is the correlation between gut health and mental health
What is the impact of COVID-19 on mental health
Does marijuana usage have an affect on physical health
Is working form home responsible for people’s weight gains during the COVID-19 pandemic
Activity
Questions as problems:
What is the relationship between sleep and mental health
What is the relationship between online school and mental health
What is the correlation between gut health and mental health
What is the impact of COVID-19 on mental health
Does marijuana usage have an affect on physical health
Is working form home responsible for people’s weight gains during the COVID-19 pandemic
Scientific method v human centered
Emphasis on what is “known”
Consult expert sources
Bias reduction
Hierarchy of methods
Linear process
Key concern: is it correct?
Emphasis on people’s needs Consult impacted people
Prioritize preference and practicality
Creative, emergent methods
Non-linear process (iterative, circular, etc.)
Key concern: does it help?
Scientific method v human centered
Framed within postpositivism – there is/are, out there, correct answers or solutions, that can be identified through systematic exploration, and assessing and rejecting possibilities.* The end goal is identification of the answer/solution. While context specific modifications are acknowledged, revision is typically triggered by system failure and is a reactive process
*standards for rejection are clear while standards for acceptance are less clear
Framed within social constructivism and pragmatism – the best solutions are useful, will be used, and are resource effective and sustainable. The need to modify processes to adapt for changes in contexts and to accommodate new innovations is expected; the DI process does not end
University of South Florida
Scholar Commons
Textbooks Collection USF Tampa Library Open Access Collections
2012
Social Science Research: Principles, Methods, and
Practices
Anol Bhattacherjee
University of South Florida, [email protected]
Follow this and additional works at: http://scholarcommons.usf.edu/oa_textbooks
Part of the American Studies Commons, Education Commons, Public Health Commons, and
the Social and Behavioral Sciences Commons
This Book is brought to you for free and open access by the USF Tampa Library Open Access Collections at Scholar Commons. It has been accepted
for inclusion in Textbooks Collection by an authorized administrator of Scholar Commons. For more information, please contact
[email protected]
Recommended Citation
Bhattacherjee, Anol, Social Science Research: Principles, Methods, and Practices (2012). Textbooks Collection. 3.
http://scholarcommons.usf.edu/oa_textbooks/3
http://scholarcommons.usf.edu/?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu/?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu/oa_textbooks?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu/tloa?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu/oa_textbooks?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://network.bepress.com/hgg/discipline/439?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://network.bepress.com/hgg/discipline/784?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://network.bepress.com/hgg/discipline/738?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://network.bepress.com/hgg/discipline/316?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
http://scholarcommons.usf.edu/oa_textbooks/3?utm_source=scholarcommons.usf.edu\%2Foa_textbooks\%2F3&utm_medium=PDF&utm_campaign=PDFCoverPages
mailto:[email protected]
SOCIAL SCIENCE RESEARCH:
PRINCIPLES, METHODS, AND PRACTICES
ANOL BHATTACHERJEE
SOCIAL SCIENCE RESEARCH:
PRINCIPLES, METHODS, AND PRACTICES
Anol Bhattacherjee, Ph.D.
University of South Florida
Tampa, Florida, USA
[email protected]
Second Edition
Copyright © 2012 by Anol Bhattacherjee
Published under the
Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License
mailto:[email protected]
Social Science Research: Principles, Methods, and Practices, 2nd edition
By Anol Bhattacherjee
First published 2012
ISBN-13: 978-1475146127
ISBN-10: 1475146124
Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License:
Users are free to use, copy, share, distribute, display, and reference this book under the following
conditions:
ATTRIBUTION: Whole or partial use of this book should be attributed (referenced or cited)
according to standard academic practices.
NON-COMMERCIAL: This book may not be used for commercial purposes.
SHARE ALIKE: Users may alter, transform, or build upon this book, but must distribute the
resulting work under the same or similar license as this one.
For any reuse or distribution, the license terms of this work must be clearly specified. Your fair use
and other rights are in no way affected by the above.
Copyright © 2012 by Anol Bhattacherjee
i
Preface
This book is designed to introduce doctoral and graduate students to the process of
scientific research in the social sciences, business, education, public health, and related
disciplines. This book is based on my lecture materials developed over a decade of teaching the
doctoral-level class on Research Methods at the University of South Florida. The target
audience for this book includes Ph.D. and graduate students, junior researchers, and professors
teaching courses on research methods, although senior researchers can also use this book as a
handy and compact reference.
The first and most important question potential readers should have about this book is
how is it different from other text books on the market? Well, there are four key differences.
First, unlike other text books, this book is not just about “research methods” (empirical data
collection and analysis) but about the entire “research process” from start to end. Research
method is only one phase in that research process, and possibly the easiest and most structured
one. Most text books cover research methods in depth, but leave out the more challenging, less
structured, and probably more important issues such as theorizing and thinking like a
researcher, which are often prerequisites of empirical research. In my experience, most
doctoral students become fairly competent at research methods during their Ph.D. years, but
struggle to generate interesting or useful research questions or build scientific theories. To
address this deficit, I have devoted entire chapters to topics such as “Thinking Like a
Researcher” and “Theories in Scientific Research”, which are essential skills for a junior
researcher.
Second, the book is succinct and compact by design. While writing the book, I decided
to focus only on essential concepts, and not fill pages with clutter that can divert the students’
attention to less relevant or tangential issues. Most doctoral seminars include a fair
complement of readings drawn from the respective discipline. This book is designed to
complement those readings by summarizing all important concepts in one compact volume,
rather than burden students with a voluminous text on top of their assigned readings.
Third, this book is free in its download version. Not just the current edition but all
future editions in perpetuity. The book will also be available in Kindle e-Book, Apple iBook, and
on-demand paperback versions at a nominal cost. Many people have asked why I’m giving
away something for free when I can make money selling it? Well, not just to stop my students
from constantly complaining about the high price of text books, but also because I believe that
scientific knowledge should not be constrained by access barriers such as price and availability.
Scientific progress can occur only if students and academics around the world have affordable
access to the best that science can offer, and this free book is my humble effort to that cause.
However, free should not imply “lower quality.” Some of the best things in life such as air,
water, and sunlight are free. Many of Google’s resources are free too, and one can well imagine
where we would be in today’s Internet age without Google. Some of the most sophisticated
software programs available today, like Linux and Apache, are also free, and so is this book.
Fourth, I plan to make local-language versions of this book available in due course of
time, and those translated versions will also be free. So far, I have had commitments to
ii
translate thus book into Chinese, French, Indonesian, Korean, Portuguese, Spanish versions
(which will hopefully be available in 2012), and I’m looking for qualified researchers or
professors to translate it into Arabic, German, and other languages where there is sufficient
demand for a research text. If you are a prospective translator, please note that there will be no
financial gains or royalty for your translation services, because the book must remain free, but
I’ll gladly include you as a coauthor on the local-language version.
The book is structured into 16 chapters for a 16-week semester. However, professors
or instructors can add, drop, stretch, or condense topics to customize the book to the specific
needs of their curriculum. For instance, I don’t cover Chapters 14 and 15 in my own class,
because we have dedicated classes on statistics to cover those materials and more. Instead, I
spend two weeks on theories (Chapter 3), one week to discussing and conducting reviews for
academic journals (not in the book), and one week for a finals exam. Nevertheless, I felt it
necessary to include Chapters 14 and 15 for academic programs that may not have a dedicated
class on statistical analysis for research. A sample syllabus that I use for my own class in the
business Ph.D. program is provided in the appendix.
Lastly, I plan to continually update this book based on emerging trends in scientific
research. If there are any new or interesting content that you wish to see in future editions,
please drop me a note, and I will try my best to accommodate them. Comments, criticisms, or
corrections to any of the existing content will also be gratefully appreciated.
Anol Bhattacherjee
E-mail: [email protected]
mailto:[email protected]
iii
Table of Contents
Introduction to Research
1. Science and Scientific Research.................................................................................................... 1
2. Thinking Like a Researcher ........................................................................................................... 9
3. The Research Process ................................................................................................................. 17
4. Theories in Scientific Research ................................................................................................... 25
Basics of Empirical Research
5. Research Design ......................................................................................................................... 35
6. Measurement of Constructs....................................................................................................... 43
7. Scale Reliability and Validity ....................................................................................................... 55
8. Sampling ..................................................................................................................................... 65
Data Collection
9. Survey Research ......................................................................................................................... 73
10. Experimental Research .............................................................................................................. 83
11. Case Research ............................................................................................................................ 93
12. Interpretive Research ............................................................................................................... 103
Data Analysis
13. Qualitative Analysis .................................................................................................................. 113
14. Quantitative Analysis: Descriptive Statistics ............................................................................ 119
15. Quantitative Analysis: Inferential Statistics ............................................................................. 129
Epilogue
16. Research Ethics ........................................................................................................................ 137
Appendix ............................................................................................................................................. 143
1
Chapter 1
Science and Scientific Research
What is research? Depending on who you ask, you will likely get very different answers
to this seemingly innocuous question. Some people will say that they routinely research
different online websites to find the best place to buy goods or services they want. Television
news channels supposedly conduct research in the form of viewer polls on topics of public
interest such as forthcoming elections or government-funded projects. Undergraduate students
research the Internet to find the information they need to complete assigned projects or term
papers. Graduate students working on research projects for a professor may see research as
collecting or analyzing data related to their project. Businesses and consultants research
different potential solutions to remedy organizational problems such as a supply chain
bottleneck or to identify customer purchase patterns. However, none of the above can be
considered “scientific research” unless: (1) it contributes to a body of science, and (2) it follows
the scientific method. This chapter will examine what these terms mean.
Science
What is science? To some, science refers to difficult high school or college-level courses
such as physics, chemistry, and biology meant only for the brightest students. To others,
science is a craft practiced by scientists in white coats using specialized equipment in their
laboratories. Etymologically, the word “science” is derived from the Latin word scientia
meaning knowledge. Science refers to a systematic and organized body of knowledge in any
area of inquiry that is acquired using “the scientific method” (the scientific method is described
further below). Science can be grouped into two broad categories: natural science and social
science. Natural science is the science of naturally occurring objects or phenomena, such as
light, objects, matter, earth, celestial bodies, or the human body. Natural sciences can be further
classified into physical sciences, earth sciences, life sciences, and others. Physical sciences
consist of disciplines such as physics (the science of physical objects), chemistry (the science of
matter), and astronomy (the science of celestial objects). Earth sciences consist of disciplines
such as geology (the science of the earth). Life sciences include disciplines such as biology (the
science of human bodies) and botany (the science of plants). In contrast, social science is the
science of people or collections of people, such as groups, firms, societies, or economies, and
their individual or collective behaviors. Social sciences can be classified into disciplines such as
psychology (the science of human behaviors), sociology (the science of social groups), and
economics (the science of firms, markets, and economies).
The natural sciences are different from the social sciences in several respects. The
natural sciences are very precise, accurate, deterministic, and independent of the person
2 | S o c i a l S c i e n c e R e s e a r c h
making the scientific observations. For instance, a scientific experiment in physics, such as
measuring the speed of sound through a certain media or the refractive index of water, should
always yield the exact same results, irrespective of the time or place of the experiment, or the
person conducting the experiment. If two students conducting the same physics experiment
obtain two different values of these physical properties, then it generally means that one or
both of those students must be in error. However, the same cannot be said for the social
sciences, which tend to be less accurate, deterministic, or unambiguous. For instance, if you
measure a person’s happiness using a hypothetical instrument, you may find that the same
person is more happy or less happy (or sad) on different days and sometimes, at different times
on the same day. One’s happiness may vary depending on the news that person received that
day or on the events that transpired earlier during that day. Furthermore, there is not a single
instrument or metric that can accurately measure a person’s happiness. Hence, one instrument
may calibrate a person as being “more happy” while a second instrument may find that the
same person is “less happy” at the same instant in time. In other words, there is a high degree
of measurement error in the social sciences and there is considerable uncertainty and little
agreement on social science policy decisions. For instance, you will not find many
disagreements among natural scientists on the speed of light or the speed of the earth around
the sun, but you will find numerous disagreements among social scientists on how to solve a
social problem such as reduce global terrorism or rescue an economy from a recession. Any
student studying the social sciences must be cognizant of and comfortable with handling higher
levels of ambiguity, uncertainty, and error that come with such sciences, which merely reflects
the high variability of social objects.
Sciences can also be classified based on their purpose. Basic sciences, also called pure
sciences, are those that explain the most basic objects and forces, relationships between them,
and laws governing them. Examples include physics, mathematics, and biology. Applied
sciences, also called practical sciences, are sciences that apply scientific knowledge from basic
sciences in a physical environment. For instance, engineering is an applied science that applies
the laws of physics and chemistry for practical applications such as building stronger bridges or
fuel efficient combustion engines, while medicine is an applied science that applies the laws of
biology for solving human ailments. Both basic and applied sciences are required for human
development. However, applied sciences cannot stand on their own right, but instead relies on
basic sciences for its progress. Of course, the industry and private enterprises tend to focus
more on applied sciences given their practical value, while universities study both basic and
applied sciences.
Scientific Knowledge
The purpose of science is to create scientific knowledge. Scientific knowledge refers to
a generalized body of laws and theories to explain a phenomenon or behavior of interest that
are acquired using the scientific method. Laws are observed patterns of phenomena or
behaviors, while theories are systematic explanations of the underlying phenomenon or
behavior. For instance, in physics, the Newtonian Laws of Motion describe what happens when
an object is in a state of rest or motion (Newton’s First Law), what force is needed to move a
stationary object or stop a moving object (Newton’s Second Law), and what happens when two
objects collide (Newton’s Third Law). Collectively, the three laws constitute the basis of
classical mechanics – a theory of moving objects. Likewise, the theory of optics explains the
properties of light and how it behaves in different media, electromagnetic theory explains the
properties of electricity and how to generate it, quantum mechanics explains the properties of
subatomic particles, and thermodynamics explains the properties of energy and mechanical
S c i e n c e a n d S c i e n t i f i c R e s e a r c h | 3
work. An introductory college level text book in physics will likely contain separate chapters
devoted to each of these theories. Similar theories are also available in social sciences. For
instance, cognitive dissonance theory in psychology explains how people react when their
observations of an event is different from what they expected of that event, general deterrence
theory explains why some people engage in improper or criminal behaviors, such as illegally
download music or commit software piracy, and the theory of planned behavior explains how
people make conscious reasoned choices in their everyday lives.
The goal of scientific research is to discover laws and postulate theories that can explain
natural or social phenomena, or in other words, build scientific knowledge. It is important to
understand that this knowledge may be imperfect or even quite far from the truth. Sometimes,
there may not be a single universal truth, but rather an equilibrium of “multiple truths.” We
must understand that the theories, upon which scientific knowledge is based, are only
explanations of a particular phenomenon, as suggested by a scientist. As such, there may be
good or poor explanations, depending on the extent to which those explanations fit well with
reality, and consequently, there may be good or poor theories. The progress of science is
marked by our progression over time from poorer theories to better theories, through better
observations using more accurate instruments and more informed logical reasoning.
We arrive at scientific laws or theories through a process of logic and evidence. Logic
(theory) and evidence (observations) are the two, and only two, pillars upon which scientific
knowledge is based. In science, theories and observations are interrelated and cannot exist
without each other. Theories provide meaning and significance to what we observe, and
observations help validate or refine existing theory or construct new theory. Any other means
of knowledge acquisition, such as faith or authority cannot be considered science.
Scientific Research
Given that theories and observations are the two pillars of science, scientific research
operates at two levels: a theoretical level and an empirical level. The theoretical level is
concerned with developing abstract concepts about a natural or social phenomenon and
relationships between those concepts (i.e., build “theories”), while the empirical level is
concerned with testing the theoretical concepts and relationships to see how well they reflect
our observations of reality, with the goal of ultimately building better theories. Over time, a
theory becomes more and more refined (i.e., fits the observed reality better), and the science
gains maturity. Scientific research involves continually moving back and forth between theory
and observations. Both theory and observations are essential components of scientific
research. For instance, relying solely on observations for making inferences and ignoring
theory is not considered valid scientific research.
Depending on a researcher’s training and interest, scientific inquiry may take one of two
possible forms: inductive or deductive. In inductive research, the goal of a researcher is to
infer theoretical concepts and patterns from observed data. In deductive research, the goal of
the researcher is to test concepts and patterns known from theory using new empirical data.
Hence, inductive research is also called theory-building research, and deductive research is
theory-testing research. Note here that the goal of theory-testing is not just to test a theory, but
possibly to refine, improve, and extend it. Figure 1.1 depicts the complementary nature of
inductive and deductive research. Note that inductive and deductive research are two halves of
the research cycle that constantly iterates between theory and observations. You cannot do
inductive or deductive research if you are not familiar with both the theory and data
4 | S o c i a l S c i e n c e R e s e a r c h
components of research. Naturally, a complete researcher is one who can traverse the entire
research cycle and can handle both inductive and deductive research.
It is important to understand that theory-building (inductive research) and theory-
testing (deductive research) are both critical for the advancement of science. Elegant theories
are not valuable if they do not match with reality. Likewise, mountains of data are also useless
until they can contribute to the construction to meaningful theories. Rather than viewing these
two processes in a circular relationship, as shown in Figure 1.1, perhaps they can be better
viewed as a helix, with each iteration between theory and data contributing to better
explanations of the phenomenon of interest and better theories. Though both inductive and
deductive research are important for the advancement of science, it appears that inductive
(theory-building) research is more valuable when there are few prior theories or explanations,
while deductive (theory-testing) research is more productive when there are many competing
theories of the same phenomenon and researchers are interested in knowing which theory
works best and under what circumstances.
Figure 1.1. The Cycle of Research
Theory building and theory testing are particularly difficult in the social sciences, given
the imprecise nature of the theoretical concepts, inadequate tools to measure them, and the
presence of many unaccounted factors that can also influence the phenomenon of interest. It is
also very difficult to refute theories that do not work. For instance, Karl Marx’s theory of
communism as an effective means of economic production withstood for decades, before it was
finally discredited as being inferior to capitalism in promoting economic growth and social
welfare. Erstwhile communist economies like the Soviet Union and China eventually moved
toward more capitalistic economies characterized by profit-maximizing private enterprises.
However, the recent collapse of the mortgage and financial industries in the United States
demonstrates that capitalism also has its flaws and is not as effective in fostering economic
growth and social welfare as previously presumed. Unlike theories in the natural sciences,
social science theories are rarely perfect, which provides numerous opportunities for
researchers to improve those theories or build their own alternative theories.
Conducting scientific research, therefore, requires two sets of skills – theoretical and
methodological – needed to operate in the theoretical and empirical levels respectively.
Methodological skills (know-how) are relatively standard, invariant across disciplines, and
easily acquired through doctoral programs. However, theoretical skills (know-what) is
considerably harder to master, requires years of observation and reflection, and are tacit skills
that cannot be “taught” but rather learned though experience. All of the greatest scientists in
the history of mankind, such as Galileo, Newton, Einstein, Neils Bohr, Adam Smith, Charles
S c i e n c e a n d S c i e n t i f i c R e s e a r c h | 5
Darwin, and Herbert Simon, were master theoreticians, and they are remembered for the
theories they postulated that transformed the course of science. Methodological skills are
needed to be an ordinary researcher, but theoretical skills are needed to be an extraordinary
researcher!
Scientific Method
In the preceding sections, we described science as knowledge acquired through a
scientific method. So what exactly is the “scientific method”? Scientific method refers to a
standardized set of techniques for building scientific knowledge, such as how to make valid
observations, how to interpret results, and how to generalize those results. The scientific
method allows researchers to independently and impartially test preexisting theories and prior
findings, and subject them to open debate, modifications, or enhancements. The scientific
method must satisfy four key characteristics:
Logical: Scientific inferences must be based on logical principles of reasoning.
Confirmable: Inferences derived must match with observed evidence.
Repeatable: Other scientists should be able to independently replicate or repeat a
scientific study and obtain similar, if not identical, results.
Scrutinizable: The procedures used and the inferences derived must withstand critical
scrutiny (peer review) by other scientists.
Any branch of inquiry that does not allow the scientific method to test its basic laws or
theories cannot be called “science.” For instance, theology (the study of religion) is not science
because theological ideas (such as the presence of God) cannot be tested by independent
observers using a logical, confirmable, repeatable, and scrutinizable. Similarly, arts, music,
literature, humanities, and law are also not considered science, even though they are creative
and worthwhile endeavors in their own right.
The scientific method, as applied to social sciences, includes a variety of research
approaches, tools, and techniques, for collecting and analyzing qualitative or quantitative data.
These methods include laboratory experiments, field surveys, case research, ethnographic
research, action research, and so forth. Much of this book is devoted to learning about these
different methods. However, recognize that the scientific method operates primarily at the
empirical level of research, i.e., how to make observations and analyze these observations. Very
little of this method is directly pertinent to the theoretical level, which is really the more
challenging part of scientific research.
Types of Scientific Research
Depending on the purpose of research, scientific research projects can be grouped into
three types: exploratory, …
1774867819059
ISBN 978-1-905177-48-6
51795 >
ConneCt with
this book
HEALTH/MEDICINE
www.pinterandmartin.com
UK £9.99
US $17.95
Recommended retail price
Published by
Pinter & Martin Ltd
Cover design by
Klor
For more
great books visit
pinterandmartin.com
Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou
Better research for Better healthcare
foreword by Ben Goldacre — author of Bad Science
SECond EdITIon
SECond EdITIon
TESTIn
G
Tr
Ea
Tm
En
TS
Im
o
g
en Evans, H
azel Tho
rnto
n, Iain C
halm
ers and
P
aul G
laszio
u
How do we know whether a particular treatment
really works? How reliable is the evidence? And how
do we ensure that research into medical treatments
best meets the needs of patients? These are just
a few of the questions addressed in a lively and
informative way in Testing Treatments. Brimming with
vivid examples, Testing Treatments will inspire both
patients and professionals.
Building on the success of the first edition, Testing
Treatments has now been extensively revised and
updated. The second edition includes a thought-
provoking account of screening, explaining how early
diagnosis is not always better, and a new chapter
exploring how over-regulation of research can work
against the best interests of patients. Another new
chapter shows how robust evidence from research can
shape the practice of healthcare in ways that allow
treatment decisions to be reached jointly by patients
and clinicians.
Testing Treatments urges everyone to get involved in
improving current research and future treatment, and
outlines practical steps that patients and doctors can
take together.
Better research for Better healthcare
tt_cover_135x216x16_3.indd 1 07/09/2011 16:24
To buy the paperback edition of Testing Treatments,
please visit the Pinter & Martin website at
www.pinterandmartin.com
Enter the code TT25 at checkout
to get 25\% off and free UK p&p
on all Pinter & Martin titles
Imogen Evans, Hazel Thornton,
Iain Chalmers and Paul Glasziou
Foreword Ben Goldacre
We dedicate this book to William Silverman (1917–2004),
who encouraged us repeatedly to challenge authority.
Testing Treatments
Better Research for Better Healthcare
First published in 2006 by The British Library
This second edition first published 2011 by Pinter & Martin Ltd
Copyright © 2011 Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou
Foreword © 2011 Ben Goldacre
Foreword to the first edition © 2006 Nick Ross
The authors have asserted their moral right to be identified as the authors of
this work in accordance with the Copyright, Designs and Patents Act of 1988
All rights reserved
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
ISBN 978-1-905177-48-6
This book is sold subject to the condition that it shall not, by way of trade
and otherwise, be lent, resold, hired out, or otherwise circulated without the
publisher’s prior consent in any form or binding or cover other than that in
which it is published and without a similar condition being imposed on the
subsequent purchaser
Printed and bound in Great Britain by TJ International Ltd., Padstow,
Cornwall
This book has been printed on paper that is sourced and harvested from
sustainable forests and is FSC accredited
Pinter & Martin Ltd
6 Effra Parade
London SW2 1PS
www.pinterandmartin.com
Testing Treatments Interactive: www.testingtreatments.org
v
Contents
About the authors vi
Acknowledgements vii
Foreword by Ben Goldacre ix
Foreword to the first edition by Nick Ross xiii
Preface xvii
Introduction xix
1 New – but is it better? 1
2 Hoped-for effects that don’t materialize 13
3 More is not necessarily better 21
4 Earlier is not necessarily better 31
5 Dealing with uncertainty about the effects of treatments 50
6 Fair tests of treatments 64
7 Taking account of the play of chance 85
8 Assessing all the relevant, reliable evidence 92
9 Regulating tests of treatments: help or hindrance? 105
10 Research – good, bad, and unnecessary 115
11 Getting the right research done is everybody’s business 130
12 So what makes for better healthcare? 143
13 Research for the right reasons: blueprint for a better future 160
References 169
Additional resources 182
List of Vignettes 185
List of Key Points 190
Index 193
TT_text_press.indd 5 22/09/2011 10:02
vi
About the authors
Imogen Evans practised and lectured in medicine in Canada and
the UK before turning to medical journalism at The Lancet. From
1996 to 2005 she worked for the Medical Research Council, latterly
in research ethics, and has represented the UK government on
the Council of Europe Biomedical Ethics Committee.
Hazel Thornton, after undergoing routine mammography,
was invited to join a clinical trial, but the inadequate patient
information led to her refusal. However, it also encouraged her
advocacy for public involvement in research to achieve outcomes
relevant to patients. She has written and spoken extensively on
this topic.
Iain Chalmers practised medicine in the UK and Palestine
before becoming a health services researcher and directing
the National Perinatal Epidemiology Unit and then the UK
Cochrane Centre. Since 2003 he has coordinated the James Lind
Initiative, promoting better controlled trials for better healthcare,
particularly through greater public involvement.
Paul Glasziou is both a medical researcher and part-time
General Practitioner. As a consequence of observing the gap
between these, he has focused his work on identifying and
removing the barriers to using high-quality research in everyday
clinical practice. He was editor of the BMJ’s Journal of Evidence-
Based Medicine, and Director of the Centre for Evidence-Based
Medicine in Oxford from 2003 to 2010. He is the author of several
other books related to evidence-based practice. He is currently
the recipient of a National Health and Medical Research Council
Australia Fellowship which he commenced at Bond University in
July, 2010.
TT_text_press.indd 6 22/09/2011 10:02
vii
Acknowledgements
We thank the following people for their valuable comments and
other contributions that have helped us to develop the second
edition of Testing Treatments:
Claire Allen, Doug Altman, Patricia Atkinson, Alexandra
Barratt, Paul Barrow, Ben Bauer, Michael Baum, Sarah Boseley,
Joan Box, Anne Brice, Rebecca Brice, Amanda Burls, Hamish
Chalmers, Jan Chalmers, Yao-long Chen, Olivia Clarke, Catrin
Comeau, Rhiannon Comeau, Katherine Cowan, John Critchlow,
Sally Crowe, Philipp Dahm, Chris Del Mar, Jenny Doust, Mary
Dixon-Woods, Ben Djulbegovic, Iain Donaldson, George Ebers,
Diana Elbourne, Murray Enkin, Chrissy Erueti, Curt Furberg,
Mark Fenton, Lester Firkins, Peter Gøtzsche, Muir Gray,
Sally Green, Susan Green, Ben Goldacre, Metin Gülmezoğlu,
Andrew Herxheimer, Jini Hetherington, Julian Higgins, Jenny
Hirst, Jeremy Howick, Les Irwig, Ray Jobling, Bethan Jones,
Karsten Juhl Jørgensen, Bridget Kenner, Debbie Kennett, Gill
Lever, Alessandro Liberati, Howard Mann, Tom Marshall,
Robert Matthews, Margaret McCartney, Dominic McDonald,
Scott Metcalfe, Iain Milne, Martin McKee, Sarah Moore, Daniel
Nicolae, Andy Oxman, Kay Pattison, Angela Raffle, June Raine,
Jake Ranson, James Read, Kiley Richmond, Ian Roberts, Nick
Ross, Peter Rothwell, Karen Sandler, Emily Savage-Smith,
Marion Savage-Smith, John Scadding, Lisa Schwartz, Haleema
Shakur, Ruth Silverman, Ann Southwell, Pete Spain, Mark
Starr, Melissa Sweet, Tilli Tansey, Tom Treasure, Ulrich Tröhler,
Liz Trotman, Liz Wager, Renee Watson, James Watt, Hywel
Williams, Norman Williams, Steven Woloshin, Eleanor Woods,
and Ke-hu Yang.
Iain Chalmers and Paul Glasziou are grateful to the National
Institute for Health Research (UK) for support. Paul Glasziou
TT_text_press.indd 7 22/09/2011 10:02
viii
is also grateful to the National Health and Medical Research
Council (Australia).
And a special thank you to our publisher, Martin Wagner, of
Pinter & Martin for his forbearance, cheerful encouragement,
and cool head at all times.
TT_text_press.indd 8 22/09/2011 10:02
ix
Foreword
Medicine shouldn’t be about authority, and the most important
question anyone can ask on any claim is simple: ‘how do you
know?’ This book is about the answer to that question.
There has been a huge shift in the way that people who work in
medicine relate to patients. In the distant past, ‘communications
skills training’, such as it was, consisted of how not to tell your
patient they were dying of cancer. Today we teach students –
and this is a direct quote from the hand-outs – how to ‘work
collaboratively with the patient towards an optimum health
outcome’. Today, if they wish, at medicine’s best, patients are
involved in discussing and choosing their own treatments.
For this to happen, it’s vital that everyone understands how
we know if a treatment works, how we know if it has harms, and
how we weigh benefits against harms to determine the risk. Sadly
doctors can fall short on this, as much as anybody else. Even more
sadly, there is a vast army out there, queuing up to mislead us.
First and foremost in this gallery of rogues, we can mislead
ourselves. Most diseases have a natural history, getting better
and worse in cycles, or at random: because of this, anything you
do, if you act when symptoms are at their worst, might make a
treatment seem to be effective, because you were going to get
better anyway.
The placebo effect, similarly, can mislead us all: people really
can get better, in some cases, simply from taking a dummy pill
with no active ingredients, and by believing their treatments to be
effective. As Robert M Pirsig said, in Zen and the Art of Motorcycle
Maintenance: ‘the real purpose of the scientific method is to make
sure nature hasn’t misled you into thinking you know something
you actually don’t know’.
But then there are the people who brandish scientific studies.
If there is one key message from this book – and it is a phrase I
TT_text_press.indd 9 22/09/2011 10:02
x
TESTING TREATMENTS
have borrowed and used endlessly myself – it is the concept of
a ‘fair test’. Not all trials are born the same, because there are so
many ways that a piece of scientific research can be biased, and
erroneously give what someone, somewhere thinks should be the
‘right’ answer.
Sometimes evidence can be distorted through absent-
mindedness, or the purest of motives (for all that motive should
matter). Doctors, patients, professors, nurses, occupational
therapists, and managers can all become wedded to the idea that
one true treatment, in which they have invested so much personal
energy, is golden.
Sometimes evidence can be distorted for other reasons. It
would be wrong to fall into shallow conspiracy theories about
the pharmaceutical industry: they have brought huge, lifesaving
advances. But there is a lot of money at stake in some research,
and for reasons you will see in this book, 90\% of trials are
conducted by industry. This can be a problem, when studies
funded by industry are four times more likely to have a positive
result for the sponsor’s drug than independently funded trials. It
costs up to $800m to bring a new drug to market: most of that is
spent before the drug comes to market, and if the drug turns out
to be no good, the money is already spent. Where the stakes are
so high, sometimes the ideals of a fair test can fail.1
Equally, the way that evidence is communicated can be
distorted, and misleading. Sometimes this can be in the
presentation of facts and figures, telling only part of the story,
glossing over flaws, and ‘cherry picking’ the scientific evidence
which shows one treatment in a particular light.
But in popular culture, there can be more interesting processes
at play. We have an understandable desire for miracle cures,
even though research is frequently about modest improvements,
shavings of risk, and close judgement calls. In the media, all too
often this can be thrown aside in a barrage of words like ‘cure’,
‘miracle’, ‘hope’, ‘breakthrough’, and ‘victim’.2
At a time when so many are so keen to take control of
their own lives, and be involved in decisions about their own
healthcare, it is sad to see so much distorted information, as it
can only disempower. Sometimes these distortions are around a
TT_text_press.indd 10 22/09/2011 10:02
xi
FOREWORD
specific drug: the presentation in the UK media of Herceptin as
a miracle cure for breast cancer is perhaps the most compelling
recent example.3
Sometimes, though, in promoting their own treatments, and
challenging the evidence against them, zealots and their friends in
the media can do even greater damage, by actively undermining
the public’s very understanding of how we know if something is
good for us, or bad for us.
Homoeopathy sugar pills perform no better than dummy
sugar pills when compared by the most fair tests. But when
confronted with this evidence, homoeopaths argue that there
is something wrong with the whole notion of doing a trial, that
there is some complicated reason why their pills, uniquely among
pills, cannot be tested. Politicians, when confronted with evidence
that their favoured teaching programme for preventing teenage
pregnancy has failed, may fall into the same kind of special
pleading. In reality, as this book will show, any claim made about
an intervention having an effect can be subjected to a transparent
fair test.4
Sometimes these distortions can go even deeper into
undermining the public’s understanding. A recent ‘systematic
review’ of all the most fair and unbiased tests showed there was
no evidence that taking antioxidant vitamin pills can prolong life
(in fact, they may even shorten it). With this kind of summary –
as explained beautifully in this book – clear rules are followed,
describing where to look for evidence, what evidence can be
included, and how its quality should be assessed. But when
systematic reviews produce a result that challenges the claims
of antioxidant supplement pill companies, newspapers and
magazines are filled with false criticisms, arguing that individual
studies for the systematic review have been selectively ‘cherry
picked’, for reasons of political allegiance or frank corruption, that
favourable evidence has been deliberately ignored, and so on.5
This is unfortunate. The notion of systematic review – looking
at the totality of evidence – is quietly one of the most important
innovations in medicine over the past 30 years. In defending their
small corner of retail business, by undermining the public’s access
to these ideas, journalists and pill companies can do us all a great
disservice.
TT_text_press.indd 11 22/09/2011 10:02
xii
TESTING TREATMENTS
And that is the rub. There are many reasons to read this book.
At the simplest level, it will help you make your own decisions
about your own health in a much more informed way. If you work
in medicine, the chapters that follow will probably stand head
and shoulders above any teaching you had in evidence-based
medicine. At the population level, if more people understand
how to make fair comparisons, and see whether one intervention
is better than another, then as the authors argue, instead of
sometimes fearing research, the public might actively campaign to
be more involved in reducing uncertainties about the treatments
that matter to them.
But there is one final reason to read this book, to learn the tricks
of our trade, and that reason has nothing to do with practicality:
the plain fact is, this stuff is interesting, and beautiful, and clever.
And in this book it’s explained better than anywhere else I’ve ever
seen, because of the experience, knowledge, and empathy of the
people who wrote it.
Testing Treatments brings a human focus to real world
questions. Medicine is about human suffering, and death, but
also human frailty in decision makers and researchers: and this is
captured here, in the personal stories and doubts of researchers,
their motivations, concerns, and their shifts of opinion. It’s rare
for this side of science to be made accessible to the public, and
the authors move freely, from serious academic papers to the
more ephemeral corners of medical literature, finding unguarded
pearls from the discussion threads beneath academic papers,
commentaries, autobiographies, and casual asides.
This book should be in every school, and every medical
waiting room. Until then, it’s in your hands. Read on.
Ben Goldacre
August 2011
TT_text_press.indd 12 22/09/2011 10:02
xiii
Foreword to the first edition
This book is good for our health. It shines light on the mysteries
of how life and death decisions are made. It shows how those
judgements are often badly flawed and it sets a challenge for
doctors across the globe to mend their ways.
Yet it accomplishes this without unnecessary scares; and it
warmly admires much of what modern medicine has achieved. Its
ambitions are always to improve medical practice, not disparage it.
My own first insight into entrenched sloppiness in medicine
came in the 1980s when I was invited to be a lay member of a
consensus panel set up to judge best practice in the treatment of
breast cancer. I was shocked (and you may be too when you read
more about this issue in Chapter 2 [now Chapter 3]). We took
evidence from leading researchers and clinicians and discovered
that some of the most eminent consultants worked on hunch or
downright prejudice and that a woman’s chance of survival, and
of being surgically disfigured, greatly depended on who treated
her and what those prejudices were. One surgeon favoured heroic
mutilation, another preferred simple lump removal, a third opted
for aggressive radiotherapy, and so on. It was as though the age of
scientific appraisal had passed them by.
Indeed, it often had, and for many doctors it still does.
Although things have improved, many gifted, sincere and
skilful medical practitioners are surprisingly ignorant about
what constitutes good scientific evidence. They do what they
do because that is what they were taught in medical school, or
because it is what other doctors do, or because in their experience
it works. But personal experience, though beguiling, is often
terribly misleading – as this book shows, with brutal clarity.
Some doctors say it is naïve to apply scientific rigour to the
treatment of individual patients. Medicine, they assert, is both a
science and an art. But, noble as that sounds, it is a contradiction
TT_text_press.indd 13 22/09/2011 10:02
xiv
TESTING TREATMENTS
in terms. Of course medical knowledge is finite and with any
individual the complexities are almost infinite, so there is always
an element of uncertainty. In practice, good medicine routinely
requires good guesswork. But too often in the past many medical
professionals have blurred the distinction between guessing and
good evidence. Sometimes they even proclaim certainty when
there is really considerable doubt. They eschew reliable data
because they are not sure how to assess them.
This book explains the difference between personal experience
and more complex, but better ways of distinguishing what works
from what does not and what is safe from what is not. Insofar
as it can, it avoids technical terms, and promotes plain English
expressions like ‘fair tests’. It warns that science, like everything
else in human affairs, is prone to error and bias (through
mistakes, vanity or – especially pernicious in medicine – the
demands of commerce); but it reminds us that, even so, it is the
meticulous approach of science that has created almost all of the
most conspicuous advances in human knowledge. Doctors (and
media-types, like me) should stop disparaging clinical research
as ‘trials on human guinea-pigs’; on the contrary there is a moral
imperative for all practitioners to promote fair tests to their
patients and for patients to participate.
This is an important book for anyone concerned about their
own or their family’s health, or the politics of health. Patients are
often seen as the recipients of healthcare, rather than participants.
The task ahead is as much for us, the lay public in whose name
medicine is practised and from whose purse medical practitioners
are paid, as for doctors and medical researchers. If we are passive
consumers of medicine we will never drive up standards. If we
prefer simplistic answers we will get pseudoscience. If we do not
promote the rigorous testing of treatments we will get pointless
and sometimes dangerous treatment along with the stuff that
really works.
This book contains a manifesto for improving things, and
patients are at its heart. But it is an important book for doctors,
medical students, and researchers too – all would benefit from
its lessons. In an ideal world, it would be compulsory reading for
every journalist, and available to every patient, because if doctors
TT_text_press.indd 14 22/09/2011 10:02
xv
FOREWORD TO THE FIRST EDITION
are inadequate at weighing up scientific evidence, in general we,
whose very mortality depends on this, are worse.
One thing I promise: if this subject of testing treatments is
new to you, once you have read this book you will never feel quite
the same about your doctor’s advice again.
Nick Ross
TV and radio presenter and journalist
16 November 2005
TT_text_press.indd 15 22/09/2011 10:02
TT_text_press.indd 16 22/09/2011 10:02
xvii
Preface
The first edition of Testing Treatments, published in 2006, was
inspired by a question: ‘How do you ensure that research into
medical treatments best meets the needs of patients?’ Our
collective experience – collective at that point meaning Imogen
Evans, a medical doctor and former researcher and journalist,
Hazel Thornton, a patient and independent lay advocate for
quality in research and healthcare, and Iain Chalmers, a health
services researcher – was that research often failed to address this
key issue. Moreover, we were keenly aware that many medical
treatments, both new and old, were not based on sound evidence.
So we set out to write a book to promote more critical public
assessment of the effects of treatments by encouraging patient-
professional dialogue.
We were heartened by the level of interest shown in Testing
Treatments – both in the original British Library imprint and when
we made the text freely available online at www.jameslindlibrary.
org – and that it appealed to both lay and professional readers.
The first edition of Testing Treatments has been used as a teaching
aid in many countries, and several full translations are available
for free download from www.testingtreatments.org.
From the outset we thought of Testing Treatments as work
in progress; there will almost always be uncertainties about
the effects of treatments, whether new or old, and therefore a
continuing need for all treatments to be tested properly. To do
this it is essential to visit and re-visit the evidence; to review
existing evidence critically and systematically before embarking
on new research, and similarly to interpret new results in the light
of up-to-date systematic reviews.
Embarking on the second edition of Testing Treatments, we
three became four, with the addition of Paul Glasziou, a general
practitioner and researcher with a commitment to taking account
TT_text_press.indd 17 22/09/2011 10:02
xviii
TESTING TREATMENTS
of high-quality research evidence in everyday clinical practice.
We have a new publisher – Pinter & Martin, who reprinted the
first edition in 2010 – and the new text is available free on line, as
before, from www.testingtreatments.org.
While our basic premise remains the same, the original text
has been extensively revised and updated. For example, we have
expanded coverage of the benefits and harms of screening in a
separate chapter (Chapter 4) entitled Earlier is not necessarily
better. And in Regulating tests of treatments: help or hindrance?
(Chapter 9) we describe how research can become over-policed
to the detriment of patients. In the penultimate chapter (Chapter
12) we ask: ‘So what makes for better healthcare?’ and show how
the lines of evidence can be drawn together in ways that can make
a real difference to all of us. We close with our blueprint for a
better future and an action plan (Chapter 13).
We hope our book will point the way to wider understanding
of how treatments can and should be tested fairly and how
everyone can play a part in making this happen. This is not a ‘best
treatments guide’ to the effects of individual therapies. Rather, we
highlight issues that are fundamental to ensuring that research is
soundly based, properly done, able to distinguish harmful from
helpful treatments, and designed to answer questions that matter
to patients, the public, and health professionals.
Imogen Evans, Hazel Thornton,
Iain Chalmers, Paul Glasziou
August 2011
TT_text_press.indd 18 22/09/2011 10:02
xix
Introduction
‘There is no way to know when our observations about
complex events in nature are complete. Our knowledge
is finite, Karl Popper emphasised, but our ignorance is
infinite. In medicine, we can never be certain about the
consequences of our interventions, we can only narrow the
area of uncertainty. This admission is not as pessimistic as
it sounds: claims that resist repeated energetic challenges
often turn out to be quite reliable. Such “working truths” are
the building blocks for the reasonably solid structures that
support our everyday actions at the bedside.’
William A. Silverman. Where’s the evidence?
Oxford: Oxford University Press, 1998, p165
Modern medicine has been hugely successful. It is hard to imagine
what life must have been like without antibiotics. The development
of other effective drugs has revolutionized the treatment of heart
attacks and high blood pressure and has transformed the lives of
many people with schizophrenia. Childhood immunization has
made polio and diphtheria distant memories in most countries,
and artificial joints have helped countless people to be less
troubled by pain and disability. Modern imaging techniques
such as ultrasound, computed tomography (CT), and magnetic
resonance imaging (MRI) have helped to ensure that patients
are accurately diagnosed and receive the right treatment. The
diagnosis of many types of cancer used to spell a death sentence,
TT_text_press.indd 19 22/09/2011 10:02
xx
TESTING TREATMENTS
whereas today patients regularly live with their cancers instead
of dying from them. And HIV/AIDS has largely changed from a
swift killer into a chronic (long-lasting) disease.
Of course many improvements in health have come about
because of social and public health advances, such as piped clean
water, sanitation, and better housing. But even sceptics would have
difficulty dismissing the impressive impact of modern medical
care. Over the past half century or so, better healthcare has made
a major contribution to increased lifespan, and has improved the
quality of life, especially for those with chronic conditions.1, 2
But the triumphs of modern medicine can easily lead us to
overlook many of its ongoing problems. Even today, too much
medical decision-making is based on poor evidence. There are
still too many medical treatments that harm patients, some that
are of little or no proven benefit, and others that are worthwhile
but are not used enough. How can this be, when every year,
studies into the effects of treatments generate a mountain of
results? Sadly, the evidence is often unreliable and, moreover,
much of the research that is done does not address the questions
that patients need answered.
Part of the problem is that treatment effects are very seldom
overwhelmingly obvious or dramatic. Instead, there will usually
be uncertainties about how well new treatments work, or indeed
whether they do more good than harm. So carefully designed fair
tests – tests that set out to reduce biases and take into account
the play of chance (see Chapter 6) – are necessary to identify
treatment effects reliably.
The impossibility of predicting exactly what will happen when
a person gets a disease or receives a treatment is sometimes called
Franklin’s law, after the 18th-century US statesman Benjamin
Franklin, who famously noted that ‘in this world nothing can be
said to be certain, except death and taxes’.3 Yet Franklin’s law is
hardly second nature in society. The inevitability of uncertainty
is not emphasized enough in schools, nor are other fundamental
concepts such as how to obtain and interpret evidence, or how
to understand information about probabilities and risks. As one
commentator put it: ‘At school you were taught about chemicals
in test …
The Field Guide to Human-Centered Design
By IDEO.org
1st Edition © 2015
ISBN: 978-0-9914063-1-9
This work is licensed under the creative commons
attribution, noncommercial, no derivatives 3.0
unported license.
Attribution — You must attribute the work in the
manner specified by the author or licensor (but not
in any way that suggests that they endorse you or
your use of the work).
Noncommerical — You may not use this work for
commercial purposes.
NoDerivatives — If you remix, transform, or build
upon the material, you may not distribute the
modified material.
Printed in Canada
THE
FIELD GUIDE
Contents
Introduction
Mindsets
Creative Confidence
Make It
Learn from Failure
Empathy
Embrace Ambiguity
Optimism
Iterate, Iterate, Iterate
Methods
INSPIRATION
Frame Your Design Challenge
Create a Project Plan
Build a Team
Recruiting Tools
Secondary Research
Interview
Group Interview
Expert Interview
Define Your Audience
Conversation Starters
Extremes and Mainstreams
Immersion
Analogous Inspiration
Card Sort
Peers Observing Peers
09
17
19
20
21
22
23
24
25
27
29
31
34
35
36
37
39
42
43
44
45
49
52
53
57
60
Collage
Guided Tour
Draw It
Resource Flow
Case Study: Vroom
IDEATION
Download Your Learnings
Share Inspiring Stories
Top Five
Find Themes
Create Insight Statements
Explore Your Hunch
How Might We
Create Frameworks
Brainstorm
Brainstorm Rules
Bundle Ideas
Get Visual
Mash-Ups
Design Principles
Create a Concept
Co-Creation Session
Gut Check
Determine What to Prototype
Storyboard
Role Playing
61
64
65
67
71
75
77
78
79
80
81
84
85
89
94
95
97
101
104
105
108
109
110
111
113
118
119
123
126
127
129
133
135
136
137
140
141
144
145
146
147
148
149
152
153
157
159
163
189
Rapid Prototyping
Business Model Canvas
Get Feedback
Integrate Feedback and Iterate
Case Study: Asili
IMPLEMENTATION
Live Prototyping
Roadmap
Resource Assessment
Build Partnerships
Ways to Grow Framework
Staff Your Project
Funding Strategy
Pilot
Define Success
Keep Iterating
Create a Pitch
Sustainable Revenue
Monitor and Evaluate
Keep Getting Feedback
Case Study: Clean Team
Resources
Colophon
8
The Field Guide to Human-Centered Design
09
Introduction
What Does It
Mean to Be
Embracing human-centered design means believing that all problems,
even the seemingly intractable ones like poverty, gender equality,
and clean water, are solvable. Moreover, it means believing that the
people who face those problems every day are the ones who hold the
key to their answer. Human-centered design offers problem solvers of
any stripe a chance to design with communities, to deeply understand
the people they’re looking to serve, to dream up scores of ideas, and to
create innovative new solutions rooted in people’s actual needs.
At IDEO.org and IDEO, we’ve used human-centered design for decades
to create products, services, experiences, and social enterprises that
have been adopted and embraced because we’ve kept people’s lives and
desires at the core. The social sector is ripe for innovation, and we’ve
seen time and again how our approach has the power to unlock real
impact. Being a human-centered designer is about believing that as
long as you stay grounded in what you’ve learned from people, your
team can arrive at new solutions that the world needs. And with this
Field Guide, you’re now armed with the tools needed to bring that
belief to life.
10
The Field Guide to Human-Centered Design
Human-centered designers are unlike other
problem solvers—we tinker and test, we fail
early and often, and we spend a surprising
amount of time not knowing the answer to the
challenge at hand. And yet, we forge ahead.
We’re optimists and makers, experimenters
and learners, we empathize and iterate, and
we look for inspiration in unexpected places.
We believe that a solution is out there and
that by keeping focused on the people we’re
designing for and asking the right questions,
we’ll get there together. We dream up lots of
ideas, some that work and some that don’t. We
make our ideas tangible so that we can test
them, and then we refine them. In the end,
our approach amounts to wild creativity, to a
ceaseless push to innovate, and a confidence
that leads us to solutions we’d never dreamed
of when we started. In the Field Guide, we
share our philosophy of design and the
seven mindsets that set us apart: Empathy,
Optimism, Iteration, Creative Confidence,
Making, Embracing Ambiguity, and Learning
from Failure.
Adopt the Mindsets
11
Introduction
INSPIRATION
In this phase, you’ll learn how to better
understand people. You’ll observe their lives,
hear their hopes and desires, and get smart
on your challenge.
IDEATION
Here you’ll make sense of everything that
you’ve heard, generate tons of ideas, identify
opportunities for design, and test and refine
your solutions.
IMPLEMENTATION
Now is your chance to bring your solution to
life. You’ll figure out how to get your idea to
market and how to maximize its impact in
the world.
Human-centered design isn’t a perfectly
linear process, and each project invariably
has its own contours and character. But no
matter what kind of design challenge you’ve
got, you’ll move through three main phases:
Inspiration, Ideation, and Implementation.
By taking these three phases in turn, you’ll
build deep empathy with the communities
and individuals you’re designing for; you’ll
figure out how to turn what you’ve learned
into a chance to design a new solution; and
you’ll build and test your ideas before finally
putting them out into the world. At IDEO.org
and IDEO, we’ve used human-centered design
to tackle a vast array of design challenges, and
though our projects have ranged from social
enterprises to communication campaigns to
medical devices, this particular approach to
creative problem solving has seen us through
each time.
Understand the Process
12
The Field Guide to Human-Centered Design
Though no two human-centered design
projects are alike, we draw from the same
kit of tools for each of them. For example, to
build deep empathy with the people we’re
trying to serve, we always conduct interviews
with them. To maintain creativity and
energy, we always work in teams. To keep
our thinking generative, sharp, and because
it helps us work things through, we always
make tangible prototypes of our ideas. And
because we rarely get it right the first time,
we always share what we’ve made, and iterate
based on the feedback we get. The 57 methods
in the Field Guide offer a comprehensive set
of exercises and activities that will take you
from framing up your design challenge to
getting it to market. You’ll use some of these
methods twice or three times and some not
at all as you work through your challenge.
But taken as a set, they’ll put you on the path
to continuous innovation while keeping the
community you’re designing for squarely at
the center of your work.
Use the Tools
13
Introduction
Trust the Process Even if
It Feels Uncomfortable
D
IV
ER
G
E
DI
VE
RG
E
C
O
N
VERG
E
CONVERGE
Human-centered design is a unique approach
to problem solving, one that can occasionally
feel more like madness than method—but
you rarely get to new and innovative solutions
if you always know precisely where you’re
going. The process is designed to get you to
learn directly from people, open yourself up
to a breadth of creative possibilities, and then
zero in on what’s most desirable, feasible,
and viable for the people you’re designing
for. You’ll find yourself frequently shifting
gears through the process, and as you work
through its three phases you’ll swiftly move
from concrete observations to highly abstract
thinking, and then right back again into the
nuts and bolts of your prototype. We call it
diverging and converging. By going really
big and broad during the Ideation phase, we
dream up all kinds of possible solutions.
But because the goal is to have a big impact
in the world, we have to then identif y what,
among that constellation of ideas, has the
best shot at really working. You’ll diverge and
converge a few times, and with each new
cycle you’ll come closer and closer to a market-
ready solution.
14
The Field Guide to Human-Centered Design
DESIRABLE
Human
Start here
FEASIBLE
Technology
VIABLE
Business
Human-centered design is uniquely situated
to arrive at solutions that are desirable,
feasible, and viable. By starting with humans,
their hopes, fears, and needs, we quickly
uncover what’s most desirable. But that’s
only one lens through which we look at our
solutions. Once we’ve determined a range of
solutions that could appeal to the community
we’re looking to serve, we then start to home
in on what is technically feasible to actually
implement and how to make the solution
financially viable. It’s a balancing act, but one
that’s absolutely crucial to designing solutions
that are successful and sustainable.
Create Real Impact
15
Introduction
17
Introduction
MINDSETS
18
The Field Guide to Human-Centered Design
19
Mindsets
Creative Confidence
—David Kelley, Founder, IDEO
Anyone can approach the world like a designer.
Often all it takes to unlock that potential as
a dynamic problem solver is a bit of creative
confidence. Creative confidence is the belief that
everyone is creative, and that creativity isn’t the
capacity to draw or compose or sculpt, but a way
of understanding the world.
Creative confidence is the quality that human-
centered designers rely on when it comes to
making leaps, trusting their intuition, and chasing
solutions that they haven’t totally figured out
yet. It’s the belief that you can and will come
up with creative solutions to big problems and
the confidence that all it takes is rolling up your
sleeves and diving in. Creative confidence will
drive you to make things, to test them out, to get
it wrong, and to keep on rolling, secure in the
knowledge that you’ll get where you need to go and
that you’re bound to innovate along the way.
It can take time to build creative confidence,
and part of getting there is trusting that the
human-centered design process will show you how
to bring a creative approach to whatever problem
is at hand. As you start with small successes
and then build to bigger ones, you’ll see your
creative confidence grow and before long you’ll
find yourself in the mindset that you are a wildly
creative person.
20
The Field Guide to Human-Centered Design
As human-centered designers, we make because
we believe in the power of tangibility. And we know
that making an idea real reveals so much that mere
theory cannot. When the goal is to get impactful
solutions out into the world, you can’t live in
abstractions. You have to make them real.
Human-centered designers are doers, tinkerers,
crafters, and builders. We make using anything
at our disposal, from cardboard and scissors to
sophisticated digital tools. We build our ideas
so that we can test them, and because actually
making something reveals opportunities and
complexities that we’d never have guessed were
there. Making is also a fantastic way to think,
and it helps bring into focus the feasibility of
our designs. Moreover, making an idea real is an
incredibly effective way to share it. And without
candid, actionable feedback from people, we
won’t know how to push our ideas forward.
As you move through the human-centered design
process, it doesn’t matter what you make, the
materials you use, or how beautiful the result is,
the goal is always to convey an idea, share it, and
learn how to make it better.
Best of all, you can prototype anything at any stage
of the process from a service model to a uniform,
from a storyboard to the financial details of your
solution. As human-centered designers, we have
a bias toward action, and that means getting ideas
out of our heads and into the hands of the people
we’re looking to serve.
Make It
—Krista Donaldson, CEO, D-Rev
21
Mindsets
Failure is an incredibly powerful tool for
learning. Designing experiments, prototypes, and
interactions and testing them is at the heart of
human-centered design. So is an understanding
that not all of them are going to work. As we seek
to solve big problems, we’re bound to fail. But if
we adopt the right mindset, we’ll inevitably learn
something from that failure.
Human-centered design starts from a place of
not knowing what the solution to a given design
challenge might be. Only by listening, thinking,
building, and refining our way to an answer do we
get something that will work for the people we’re
trying to serve. “Fail early to succeed sooner” is
a common refrain around IDEO, and part of its
power is the permission it gives to get something
wrong. By refusing to take risks, some problem
solvers actually close themselves off from a real
chance to innovate.
Learn from Failure
—Tim Brown, CEO, IDEO
Thomas Edison put it well when he said, “I have
not failed. I’ve just found 10,000 ways that won’t
work.” And for human-centered designers, sorting
out what won’t work is part of finding what will.
Failure is an inherent part of human-centered
design because we rarely get it right on our first
try. In fact, getting it right on the first try isn’t the
point at all. The point is to put something out into
the world and then use it to keep learning, keep
asking, and keep testing. When human-centered
designers get it right, it’s because they got it
wrong first.
22
The Field Guide to Human-Centered Design
Empathy
—Emi Kolawole, Editor-in-Residence,
Stanford University d.school
Empathy is the capacity to step into other people’s
shoes, to understand their lives, and start to
solve problems from their perspectives. Human-
centered design is premised on empathy, on the
idea that the people you’re designing for are your
roadmap to innovative solutions. All you have to
do is empathize, understand them, and bring them
along with you in the design process.
For too long, the international development
community has designed solutions to the
challenges of poverty without truly empathizing
with and understanding the people it’s looking
to serve. But by putting ourselves in the shoes of
the person we’re designing for, human-centered
designers can start to see the world, and all the
opportunities to improve it, through a new and
powerful lens.
Immersing yourself in another world not only
opens you up to new creative possibilities, but
it allows you to leave behind preconceived ideas
and outmoded ways of thinking. Empathizing with
the people you’re designing for is the best route
to truly grasping the context and complexities
of their lives. But most importantly, it keeps the
people you’re designing for squarely grounded in
the center of your work.
23
Mindsets
a generative process, and because we work so
collaboratively, it’s easy to discard bad ideas, hold
onto pieces of the so-so ones, and eventually
arrive at the good ones.
Though it may seem counterintuitive, the
ambiguity of not knowing the answer actually
sets up human-centered designers to innovate.
If we knew the answer when we started, what
could we possibly learn? How could we come up
with creative solutions? Where would the people
we’re designing for guide us? Embracing ambiguity
actually frees us to pursue an answer that we
can’t initially imagine, which puts us on the path
to routine innovation and lasting impact.
Human-centered designers always start from the
place of not knowing the answer to the problem
they’re looking to solve. And in a culture that can
be too focused on being the first one to the right
answer, that’s not a particularly comfortable place
to be. But by starting at square one, we’re forced
to get out into the world and talk to the people
we’re looking to serve. We also get to open up
creatively, to pursue lots of different ideas, and
to arrive at unexpected solutions. By embracing
that ambiguity, and by trusting that the human-
centered design process will guide us toward
an innovative answer, we actually give ourselves
permission to be fantastically creative.
One of the qualities that sets human-centered
designers apart is the belief that there will always
be more ideas. We don’t cling to ideas any longer
than we have to because we know that we’ll have
more. Because human-centered design is such
Embrace Ambiguity
—Patrice Martin, Co-Lead and Creative Director,
IDEO.org
24
The Field Guide to Human-Centered Design
Human-centered designers are persistently
focused on what could be, not the countless
obstacles that may get in the way. Constraints are
inevitable, and often they push designers toward
unexpected solutions. But it’s our core animating
belief—that every problem is solvable—that shows
just how deeply optimistic human-centered
designers are.
We believe that design is inherently optimistic.
To take on a big challenge, especially one as large
and intractable as poverty, we have to believe
that progress is even an option. If we didn’t, we
wouldn’t even try. Optimism is the embrace of
possibility, the idea that even if we don’t know the
answer, that it’s out there and that we can find it.
In addition to driving us toward solutions,
optimism makes us more creative, encourages
us to push on when we hit dead ends, and helps
all the stakeholders in a project gel. Approaching
problems from the perspective that you’ll get to
a solution infuses the entire process with the
energy and drive that you need to navigate the
thorniest problems.
Optimism
—John Bielenberg, Founder, Future Partners
25
Mindsets
At base, we iterate because we know that we won’t
get it right the first time. Or even the second.
Iteration allows us the opportunity to explore, to
get it wrong, to follow our hunches, but ultimately
arrive at a solution that will be adopted and
embraced. We iterate because it allows us to keep
learning. Instead of hiding out in our workshops,
betting that an idea, product, or service will be
a hit, we quickly get out in the world and let the
people we’re designing for be our guides.
As human-centered designers, we adopt an
iterative approach to solving problems because it
makes feedback from the people we’re designing
for a critical part of how a solution evolves. By
continually iterating, refining, and improving our
work, we put ourselves in a place where we’ll have
more ideas, try a variety of approaches, unlock
our creativity, and arrive more quickly at
successful solutions.
Iteration keeps us nimble, responsive, and trains
our focus on getting the idea and, after
a few passes, every detail just right. If you aimed
for perfection each time you built a prototype
or shared an idea, you’d spend ages refining
something whose validity was still in doubt. But
by building, testing, and iterating, you can advance
your idea without investing hours and resources
until you’re sure that it’s the one.
Iterate, Iterate, Iterate
—Gaby Brink, Founder, Tomorrow Partners
26
The Field Guide to Human-Centered Design
27
Introduction
METHODS
2828
N
29
Methods: Inspiration Phase
29
The Inspiration phase is about learning on the fly,
opening yourself up to creative possibilities,
and trusting that as long as you remain grounded in
desires of the communities you’re engaging, your
ideas will evolve into the right solutions. You’ll build
your team, get smart on your challenge, and talk to
a staggering variety of people.
INSPIRATION
THIS PHASE WILL HELP YOU ANSWER
How do I get star ted?
How do I conduct an inter view?
How do I keep people at the center of my research?
What are other tools I can use to understand people?
30
The Field Guide to Human-Centered Design
31
Methods: Inspiration Phase
STEPS
01 Start by taking a first stab at writing your design challenge. It should be
short and easy to remember, a single sentence that conveys what you want
to do. We often phrase these as questions which set you and your team up
to be solution-oriented and to generate lots of ideas along the way.
02 Properly framed design challenges drive toward ultimate impact, allow for
a variety of solutions, and take into account constraints and context. Now
try articulating it again with those factors in mind.
03 Another common pitfall when scoping a design challenge is going either
too narrow or too broad. A narrowly scoped challenge won’t offer enough
room to explore creative solutions. And a broadly scoped challenge won’t
give you any idea where to start.
04 Now that you’ve run your challenge through these filters, do it again. It
may seem repetitive, but the right question is key to arriving at a good
solution. A quick test we often run on a design challenge is to see if we
can come up with five possible solutions in just a few minutes. If so, you’re
likely on the right track.
Getting the right frame on your design challenge will get you off on the
right foot, organize how you think about your solution, and at moments of
ambiguity, help clarif y where you should push your design. Framing your
design challenge is more art than science, but there are a few key things to
keep in mind. First, ask yourself: Does my challenge drive toward ultimate
impact, allow for a variety of solutions, and take into account context? Dial
those in, and then refine it until it’s the challenge you’re excited to tackle.
TIME
90 minutes
DIFFICULTY
Hard
WHAT YOU’LL NEED
Pen, Frame Your Design Challenge
worksheet p. 165
PARTICIPANTS
Design team
Properly framing your design challenge is critical to
your success. Here’s how to do it just right.
Frame Your Design Challenge
32
The Field Guide to Human-Centered Design
Frame Your Design Challenge
It’s rare that you’ll Frame Your Design Challenge
just right on the first try; at IDEO.org we often go
through a number of revisions and lots of debate
as we figure out precisely how to hone the problem
we’re looking to solve.
For the second challenge in our Amplif y program,
we knew that we wanted to focus on children’s
education, but needed to narrow the scope so that
it would drive real impact, allow for a variety of
solutions, and still give us enough context to get
started. Challenge manager Chioma Ume described
how she and her team sharpened the challenge.
“We knew we wanted to do something around
kids, but of course we then have to ascertain which
kids. Should it be all kids, just teens, young kids?
Because of the tremendous importance of early
childhood development, we settled on children,
ages zero to five. But we certainly didn’t start
knowing that we’d focus just on them.”
Even then, the challenge needed refinement.
By eventually landing not on children, but their
parents, the team and its partners at the UK’s
Department for International Development,
crafted a brief that it thought would have the
most impact.
“We chose to focus on the people closest to
children, their parents,” says Ume. But she stresses
that though parents became the focus, the children
remained the beneficiaries, a nuance that would
keep the team from spinning off or focusing too
heavily on improving parents’ lives. In the end,
the team arrived at a well framed challenge, one
that asks: How might parents in low-income
communities ensure children thrive in their first
five years?
Use the Frame Your Design Challenge worksheet on
p. 165 and take multiple passes to make sure that
your question drives at impact, gives you a starting
place, but still is broad enough to allow for a great
variety of creative answers.
METHOD IN ACTION
33
Methods: Inspiration Phase
1) Take a stab at framing it as a design question.
2) Now, state the ultimate impact you’re trying to have.
3) What are some possible solutions to your problem?
Think broadly. It’s fine to start a project with a hunch or two, but make sure you allow for surprising outcomes.
4) Finally, write down some of the context and constraints that you’re facing.
They could be geographic, technological, time-based, or have to do with the population you’re trying to reach.
5) Does your original question need a tweak? Try it again.
What is the problem you’re trying to solve?
Improving the lives of children.
How might we improve the lives of children?
We want very young children in low-income communities to thrive.
Better nutrition, parents engaging with young kids to spur brain development, better education around parenting, early childhood
education centers, better access to neonatal care and vaccines.
Because children aren’t in control of their circumstances, we wanted to address our solution to their parents.
We want a solution that could work across different regions.
How might parents in low-income communities ensure children thrive in their first five years.
Frame Your Design Challenge
34
The Field Guide to Human-Centered Design
STEPS
01 A good place to start is with a calendar. Print out or make a large one
and put it up in your workspace. Now mark key dates. They could be
deadlines, important meetings, travel dates, or times when your team
members are unavailable.
02 Now that you’ve got a sense of your timeline, look at your budget
and staff. Do you have everything that you’ll need? If you foresee
constraints, how can you get around them?
03 You’ll need to get smart on your topic before you head into the field.
Who should you talk to now? What will you need to read to be up to
speed on your challenge?
04 Answer questions like: When should my team head into the field?
Will my team make one visit or two? Will our partners be visiting? Will
we need to physically make something? How much time, money, and
manpower will we need to produce it?
05 Your project plan will change as things evolve, and that’s perfectly OK.
You can always amend things as you go but make sure that you’re really
thinking through your project before you start.
As you set out to solve your challenge, you’ll need to create a plan. This gives
you a chance to think through all the logistics of your project, and even
though they’re bound to change as things progress, you’ll be in much better
shape if you can plan for what’s ahead. Reflect on your timeline, the space
you’ll work in, your staff, your budget, what skills you’ll need, trips you’ll
take, and what you’ll need to produce. Getting a good handle on all of this
information can keep you on track.
TIME
60-90 minutes
DIFFICULTY
Moderate
WHAT YOU’LL NEED
Pen, paper, Post-its, calendar
PARTICIPANTS
Design team
Create a Project Plan
Get organized, understand your strengths, and start
identifying what your team will need to come up with
innovative solutions.
35
Methods: Inspiration Phase
STEPS
01 First, assess how many team members you’ll need, your staff’s
availability, and when your project should start and end.
02 Look at the core members of your team and determine what they’re
good at and what they’re not so good at.
03 Is there a clear technical capability that you’ll need but don’t
currently have—maybe a mechanical engineer, a graphic designer, a
skilled writer? Remember that you can always add a team member for
a shorter period of time when their skills are most important.
Human-centered design works best with cross-disciplinary teams. You could
put three business designers to work on a new social enterprise, but if you
throw a graphic designer, a journalist, or an industrial designer into the mix,
you’re going to bring new modes of thinking to your team. It’s smart to have
a hunch about what kind of talent your team will need—if you’re designing a
social enterprise, a business designer is probably a good bet—but you won’t get
unexpected solutions with an expected team.
TIME
60 minutes
DIFFICULTY
Hard
WHAT YOU’LL NEED
Pen, paper
PARTICIPANTS
Project lead, partner
organizations
An interdisciplinary mix of thinkers, makers, and
doers is just the right combination to tackle any
design challenge.
Build a Team
36
The Field Guide to Human-Centered Design
Before you start talking to the people you’re designing for, it’s important to
have a strategy around who you talk to, what you ask them, and what pieces of
information you need to gather. By planning ahead, and tracking …
CATEGORIES
Economics
Nursing
Applied Sciences
Psychology
Science
Management
Computer Science
Human Resource Management
Accounting
Information Systems
English
Anatomy
Operations Management
Sociology
Literature
Education
Business & Finance
Marketing
Engineering
Statistics
Biology
Political Science
Reading
History
Financial markets
Philosophy
Mathematics
Law
Criminal
Architecture and Design
Government
Social Science
World history
Chemistry
Humanities
Business Finance
Writing
Programming
Telecommunications Engineering
Geography
Physics
Spanish
ach
e. Embedded Entrepreneurship
f. Three Social Entrepreneurship Models
g. Social-Founder Identity
h. Micros-enterprise Development
Outcomes
Subset 2. Indigenous Entrepreneurship Approaches (Outside of Canada)
a. Indigenous Australian Entrepreneurs Exami
Calculus
(people influence of
others) processes that you perceived occurs in this specific Institution Select one of the forms of stratification highlighted (focus on inter the intersectionalities
of these three) to reflect and analyze the potential ways these (
American history
Pharmacology
Ancient history
. Also
Numerical analysis
Environmental science
Electrical Engineering
Precalculus
Physiology
Civil Engineering
Electronic Engineering
ness Horizons
Algebra
Geology
Physical chemistry
nt
When considering both O
lassrooms
Civil
Probability
ions
Identify a specific consumer product that you or your family have used for quite some time. This might be a branded smartphone (if you have used several versions over the years)
or the court to consider in its deliberations. Locard’s exchange principle argues that during the commission of a crime
Chemical Engineering
Ecology
aragraphs (meaning 25 sentences or more). Your assignment may be more than 5 paragraphs but not less.
INSTRUCTIONS:
To access the FNU Online Library for journals and articles you can go the FNU library link here:
https://www.fnu.edu/library/
In order to
n that draws upon the theoretical reading to explain and contextualize the design choices. Be sure to directly quote or paraphrase the reading
ce to the vaccine. Your campaign must educate and inform the audience on the benefits but also create for safe and open dialogue. A key metric of your campaign will be the direct increase in numbers.
Key outcomes: The approach that you take must be clear
Mechanical Engineering
Organic chemistry
Geometry
nment
Topic
You will need to pick one topic for your project (5 pts)
Literature search
You will need to perform a literature search for your topic
Geophysics
you been involved with a company doing a redesign of business processes
Communication on Customer Relations. Discuss how two-way communication on social media channels impacts businesses both positively and negatively. Provide any personal examples from your experience
od pressure and hypertension via a community-wide intervention that targets the problem across the lifespan (i.e. includes all ages).
Develop a community-wide intervention to reduce elevated blood pressure and hypertension in the State of Alabama that in
in body of the report
Conclusions
References (8 References Minimum)
*** Words count = 2000 words.
*** In-Text Citations and References using Harvard style.
*** In Task section I’ve chose (Economic issues in overseas contracting)"
Electromagnetism
w or quality improvement; it was just all part of good nursing care. The goal for quality improvement is to monitor patient outcomes using statistics for comparison to standards of care for different diseases
e a 1 to 2 slide Microsoft PowerPoint presentation on the different models of case management. Include speaker notes... .....Describe three different models of case management.
visual representations of information. They can include numbers
SSAY
ame workbook for all 3 milestones. You do not need to download a new copy for Milestones 2 or 3. When you submit Milestone 3
pages):
Provide a description of an existing intervention in Canada
making the appropriate buying decisions in an ethical and professional manner.
Topic: Purchasing and Technology
You read about blockchain ledger technology. Now do some additional research out on the Internet and share your URL with the rest of the class
be aware of which features their competitors are opting to include so the product development teams can design similar or enhanced features to attract more of the market. The more unique
low (The Top Health Industry Trends to Watch in 2015) to assist you with this discussion.
https://youtu.be/fRym_jyuBc0
Next year the $2.8 trillion U.S. healthcare industry will finally begin to look and feel more like the rest of the business wo
evidence-based primary care curriculum. Throughout your nurse practitioner program
Vignette
Understanding Gender Fluidity
Providing Inclusive Quality Care
Affirming Clinical Encounters
Conclusion
References
Nurse Practitioner Knowledge
Mechanics
and word limit is unit as a guide only.
The assessment may be re-attempted on two further occasions (maximum three attempts in total). All assessments must be resubmitted 3 days within receiving your unsatisfactory grade. You must clearly indicate “Re-su
Trigonometry
Article writing
Other
5. June 29
After the components sending to the manufacturing house
1. In 1972 the Furman v. Georgia case resulted in a decision that would put action into motion. Furman was originally sentenced to death because of a murder he committed in Georgia but the court debated whether or not this was a violation of his 8th amend
One of the first conflicts that would need to be investigated would be whether the human service professional followed the responsibility to client ethical standard. While developing a relationship with client it is important to clarify that if danger or
Ethical behavior is a critical topic in the workplace because the impact of it can make or break a business
No matter which type of health care organization
With a direct sale
During the pandemic
Computers are being used to monitor the spread of outbreaks in different areas of the world and with this record
3. Furman v. Georgia is a U.S Supreme Court case that resolves around the Eighth Amendments ban on cruel and unsual punishment in death penalty cases. The Furman v. Georgia case was based on Furman being convicted of murder in Georgia. Furman was caught i
One major ethical conflict that may arise in my investigation is the Responsibility to Client in both Standard 3 and Standard 4 of the Ethical Standards for Human Service Professionals (2015). Making sure we do not disclose information without consent ev
4. Identify two examples of real world problems that you have observed in your personal
Summary & Evaluation: Reference & 188. Academic Search Ultimate
Ethics
We can mention at least one example of how the violation of ethical standards can be prevented. Many organizations promote ethical self-regulation by creating moral codes to help direct their business activities
*DDB is used for the first three years
For example
The inbound logistics for William Instrument refer to purchase components from various electronic firms. During the purchase process William need to consider the quality and price of the components. In this case
4. A U.S. Supreme Court case known as Furman v. Georgia (1972) is a landmark case that involved Eighth Amendment’s ban of unusual and cruel punishment in death penalty cases (Furman v. Georgia (1972)
With covid coming into place
In my opinion
with
Not necessarily all home buyers are the same! When you choose to work with we buy ugly houses Baltimore & nationwide USA
The ability to view ourselves from an unbiased perspective allows us to critically assess our personal strengths and weaknesses. This is an important step in the process of finding the right resources for our personal learning style. Ego and pride can be
· By Day 1 of this week
While you must form your answers to the questions below from our assigned reading material
CliftonLarsonAllen LLP (2013)
5 The family dynamic is awkward at first since the most outgoing and straight forward person in the family in Linda
Urien
The most important benefit of my statistical analysis would be the accuracy with which I interpret the data. The greatest obstacle
From a similar but larger point of view
4 In order to get the entire family to come back for another session I would suggest coming in on a day the restaurant is not open
When seeking to identify a patient’s health condition
After viewing the you tube videos on prayer
Your paper must be at least two pages in length (not counting the title and reference pages)
The word assimilate is negative to me. I believe everyone should learn about a country that they are going to live in. It doesnt mean that they have to believe that everything in America is better than where they came from. It means that they care enough
Data collection
Single Subject Chris is a social worker in a geriatric case management program located in a midsize Northeastern town. She has an MSW and is part of a team of case managers that likes to continuously improve on its practice. The team is currently using an
I would start off with Linda on repeating her options for the child and going over what she is feeling with each option. I would want to find out what she is afraid of. I would avoid asking her any “why” questions because I want her to be in the here an
Summarize the advantages and disadvantages of using an Internet site as means of collecting data for psychological research (Comp 2.1) 25.0\% Summarization of the advantages and disadvantages of using an Internet site as means of collecting data for psych
Identify the type of research used in a chosen study
Compose a 1
Optics
effect relationship becomes more difficult—as the researcher cannot enact total control of another person even in an experimental environment. Social workers serve clients in highly complex real-world environments. Clients often implement recommended inte
I think knowing more about you will allow you to be able to choose the right resources
Be 4 pages in length
soft MB-920 dumps review and documentation and high-quality listing pdf MB-920 braindumps also recommended and approved by Microsoft experts. The practical test
g
One thing you will need to do in college is learn how to find and use references. References support your ideas. College-level work must be supported by research. You are expected to do that for this paper. You will research
Elaborate on any potential confounds or ethical concerns while participating in the psychological study 20.0\% Elaboration on any potential confounds or ethical concerns while participating in the psychological study is missing. Elaboration on any potenti
3 The first thing I would do in the family’s first session is develop a genogram of the family to get an idea of all the individuals who play a major role in Linda’s life. After establishing where each member is in relation to the family
A Health in All Policies approach
Note: The requirements outlined below correspond to the grading criteria in the scoring guide. At a minimum
Chen
Read Connecting Communities and Complexity: A Case Study in Creating the Conditions for Transformational Change
Read Reflections on Cultural Humility
Read A Basic Guide to ABCD Community Organizing
Use the bolded black section and sub-section titles below to organize your paper. For each section
Losinski forwarded the article on a priority basis to Mary Scott
Losinksi wanted details on use of the ED at CGH. He asked the administrative resident