Fluent Essays

PLEASE SEE ARTICLES AND CASE STUDY ATTACHED!! Check out articles by Skidmore (2008) and Henrich, Heine, & Norenzayan (2010) attached. Carefully review the PSY635 Week Two Discussion Scenario. Apply the scientific method to the information included within the scenario and develop a null and a research hypothesis based on it. Using the hypotheses you have developed, compare the characteristics of the different experimental research designs discussed in the Skidmore (2008) article and choose the one that is most appropriate to adequately test your hypotheses. Identify potential internal threats to validity and explain how you might mitigate these threats. Apply ethical principles to the proposed research and describe the implications of this type of research in terms of the population(s) and cultural consideration(s) represented in the sample(s) within the scenario. PSY635 Week Two Scenario Three instructors teach the same online course and have devised an experimental intervention to improve student motivation to actively participate in discussions. The course is a core requirement for all psychology students, and students are assigned to particular sections at random rather than by instructor choice. The average class size for this particular course is 45 students. To get a large enough sample for adequate analysis, the instructors have decided to include two sections for each instructor in the experiment. The first section will serve as the control group (no experimental intervention), and the second section will receive the intervention. Anonymous data about the dependent variable will be pooled for the three sections comprising the control group and the three sections that receive the intervention. The independent variable is the intervention, which may be an incentive such as digital badges or an instructional intervention involving changing the instructions for the guided response. The dependent variable will be the number of response (not initial) posts per student that exceed two lines of text. The researchers have decided to use the Week Four discussion for data collection, reasoning that it may take some time for the intervention to become effective. Experimental Design 1 Running Head: EXPERIMENTAL DESIGN Experimental Design and Some Threats to Experimental Validity: A Primer Susan Skidmore Texas A&M University Paper presented at the annual meeting of the Southwest Educational Research Association, New Orleans, Louisiana, February 6, 2008. Experimental Design 2 Abstract Experimental designs are distinguished as the best method to respond to questions involving causality. The purpose of the present paper is to explicate the logic of experimental design and why it is so vital to questions that demand causal conclusions. In addition, types of internal and external validity threats are discussed. To emphasize the current interest in experimental designs, Evidence- Based Practices (EBP) in medicine, psychology and education are highlighted. Finally, cautionary statements regarding experimental designs are elucidated with examples from the literature. Experimental Design 3 The No Child Left Behind Act (NCLB) demands “scientifically based research” as the basis for awarding many grants in education (2001). Specifically, the 107th Congress (2001) delineated scientifically-based research as that which “is evaluated using experimental or quasi-experimental designs”. Recognizing the increased interest and demand for scientifically-based research in education policy and practice, the National Research Council released the publication, Scientific Research in Education (Shavelson & Towne, 2002) a year after the implementation of NCLB. Almost $5 billion have been channeled to programs that provide scientifically-based evidence of effective instruction, such as the Reading First Program (U. S. Department of Education, 2007). With multiple methods available to education researchers, why does the U. S. government show partiality to one particular method? The purpose of the present paper is to explicate the logic of experimental design and why it is so vital to questions that demand causal conclusions. In addition, types of internal and external validity threats are discussed. To emphasize the current interest in experimental designs, Evidence-Based Practices (EBP) in medicine, psychology and education are highlighted. Finally, cautionary statements regarding experimental designs are elucidated with examples from the literature. Experimental Design An experiment is “that portion of research in which variables are manipulated and their effects upon other variables observed” (Campbell & Stanley, 1963, p. 171). Or stated another way, experiments are concerned with an independent variable (IV) that causes or predicts the outcome of the Experimental Design 4 dependent variable (DV). Ideally, all other variables are eliminated, controlled or distributed in such a way that a conclusion that the IV caused the DV is validly justified. Figure 1. Diagram of an experiment. In Figure 1 above you can see that there are two groups. One group receives some sort of manipulation that is thought (theoretically or from previous research) to have an impact on the DV. This is known as the experimental group because participants in this group receive some type of treatment that is presumed to impact the DV. The other group, which does not receive a treatment or instead receives some type of alternative treatment, provides the result of what would have happened without experimental intervention (manipulation of the IV). So how do you determine whether participants will be in the control group or the experimental group? The answer to this question is one of the characteristics that underlie the strength of true experimental designs. True experiments must have three essential characteristics: random assignment to Outcome measured as DV No manipulation or alternate manipulation of IV (treatment or intervention) Control Group Manipulation of IV (treatment or intervention) Experimental Group Experimental Design 5 groups, an intervention given to at least one group and an alternate or no intervention for at least one other group, and a comparison of group performances on some post-intervention measurement (Gall, Gall, & Borg, 2005). Participants in a true experimental design are randomly allocated to either the control group or the experimental group. A caution is necessary here. Random assignment is not equivalent to random sampling. Random sampling determines who will be in the study, while random assignment determines in which groups participants will be. Random assignment makes “samples randomly similar to each other, whereas random sampling makes a sample similar to a population” (Shadish, Cook, & Campbell, 2002, p. 248, emphasis in original). Nonetheless, random assignment is extremely important. By randomly assigning participants (or groups of participants) to either the experimental or control group, each participant (or groups of participants) is as likely to be assigned to one group as to the other (Gall et al., 2005). In other words, by giving each participant an equal probability of being a member of each group, random assignment equates the groups on all other factors, except for the intervention that is being implemented, thereby ensuring that the experiment will produce “unbiased estimates of the average treatment effect” (Rosenbaum, 1995, p. 37). To be clear, the term “unbiased estimates” describes the fact that any observed effect differences between the study results and the “true” population are due to chance (Shadish et al., 2002). Experimental Design 6 This equality of groups assertion is based on the construction of infinite number of random assignments of participants (or groups of participants) to treatment groups in the study and not to the single random assignment in the particular study (Shadish et al., 2002). Thankfully, researchers do not have to conduct an infinite number of random assignments in an infinite number of studies for this assumption to hold. The equality of groups‟ assumption is supported in studies with large sample sizes, but not in studies with very small sample sizes. This is true due to the law of large numbers. As Boger (2005) explained, “If larger and larger samples are successively drawn from a population and a running average calculated after each sample has been drawn, the sequence of averages will converge to the mean, µ, of the population” (p. 175). If the reader is interested in exploring this concept further, the reader is directed to George Boger‟s article that details how to create a spreadsheet simulation of the law of large numbers. In addition, a medical example of this is found in Observational Studies (Rosenbaum, 1995, pp. 13-15). To consider the case of small sample size, let us suppose that I have a sample of 10 graduate students that I am going to randomly assign to one of two treatment groups. The experimental group will have regularly scheduled graduate advisor meetings to monitor students‟ educational progress. The control group will not have regularly scheduled graduate advisor meetings. Just to see what happens, I choose to do several iterations of this random assignment process. Of course, I discover that the identity of the members in the groups across iterations is wildly different. Experimental Design 7 Recognizing that most people are outliers on at least some variables (Thompson, 2006), there may be some observed differences that are due simply to the variable characteristics of the members of the treatment groups. For example, let‟s say that six of the ten graduate students are chronic procrastinators, and might benefit greatly from regular scheduled visits with a graduate advisor, while four of the ten graduate students are intrinsically motivated and tend to experience increased anxiety with frequent graduate advisor inquiries. If the random assignment process distributes these six procrastinator graduate students equally among the two groups, a bias due to this characteristic will not evidence itself in the results. If instead, due to chance all four intrinsically motivated students end up in the experimental group, the results of the study may not be the same had the groups been more evenly distributed. Ridiculously small sample sizes, therefore would result in more pronounced differences between the groups that are not due to treatment effects, but instead are due to the variable characteristics of the members in the groups. If instead I have a sample of 10,000 graduate students that that I am going to randomly assign to one of two treatment groups, the law of large numbers works for me. As explained by Thompson et al. (2005), “The beauty of true experiments is that the law of large numbers creates preintervention group equivalencies on all variables, even variables that we do not realize are essential to control” (p. 183). While there is still not identical membership across treatment groups, and I still expect that the observed differences between the control group and the experimental group are going to be due to any possible treatment effects Experimental Design 8 and to the error associated with the random assignment process, the expectation of equality of groups is nevertheless reasonably approximated. In other words, I expect the ratio of procrastinators to intrinsically motivated students to be approximately the same across the two treatment groups. In fact, I expect proportions of variables I am not even aware of to be the same, on average, across treatment groups! The larger sample size has greatly decreased the error due to chance associated with the random assignment process. As you can see in Figure 2, even if both of the sample studies produce identical treatment effects, the results are not equally valid. The majority of the effect observed in the small sample size study is actually due to error associated with the random assignment process and not a result of the treatment. This effect due to error is greatly reduced in the large sample size study. Figure 2. Observed treatment effects in two studies with different sample sizes. The white area represents the amount of the observed effect due to the error associated with the random assignment process. The grey area represents the “true” treatment effect. Three Experimental Designs When well-conducted, a randomized experiment is considered the “gold standard” in causal research (Campbell, 1957; Campbell & Stanley, 1963; “True” treatment effects n=10 error “True” treatment effects n=10,000 e r r o r Experimental Design 9 Sackett, Strauss, Richardson, Rosenberg, & Haynes, 2000; Thompson, 2006). In fact, “No other type of quantitative research (descriptive, correlational, or causal- comparative) is as powerful in demonstrating the existence of cause-and-effect relationships among variables as experimental research” (Gall et al., 2005, p. 249). There are three designs that meet the characteristics of true experimental designs, first described by Campbell (1957) and revisited in several research design texts. While other designs have the potential to produce causal effects (see Odom et al., 2005; Rosenbaum, 1995; Thompson et al., 2005) only the three classic true experimental designs are discussed in the present paper. For a more extensive description of other experimental designs, the reader is directed to research design works such as Campbell (1957); Campbell and Stanley (1963); Creswell (2003); Gall et al. (2005); Shadish et al. (2002); and Thompson (2006). The first true experimental design is known as the Pretest-Posttest Control-Group Design. This research design meets the characteristics of a true experiment because participants are randomly assigned (denoted by an R) to either the experimental or control group. There is an intervention or treatment (denoted by an X) given to one group, the experimental group, and no intervention (or alternate intervention) given to the other group, the control group. Finally, there is some form of post-intervention measurement (denoted by an O). This is also known as a posttest, because this measurement occurs after the intervention. In addition, in this particular design, there is also a pretest, denoted by an O prior to the intervention. The pretest allows the researcher to test for Experimental Design 10 equality of groups on the variable of interest prior to the intervention. These designs are “read” left to right to correspond to the passage of time (i.e., what happens first, second). Experimental Group R O X O Control Group R O O The second true experiment is the Posttest-Only Control Group Design. This design varies from the first in that it controls for possible confounding effects of a pretest because it does not use a pre-intervention measurement. All three characteristics of a true experimental design are present as in the previous design: random assignment, intervention implemented with experimental group only, and post-intervention measurement. Experimental Group R X O Control Group R O The third and final design is the Solomon Four-Group Design. This design is the strongest of the three. It not only corrects for the possible confounding effects of a pretest, but allows you to compare these results, to an experimental and control group that did receive a pretest. The major drawback to this design compared to the others is the obvious increase in sample size needed to meet the needs of four treatment groups as opposed to two treatment groups. Experimental Group (with pre-test) R O X O Control Group (with pre-test) R O O Experimental Group (without pre-test) R X O Control Group (without pre-test) R O In addition to detailing these designs in their seminal work, Campbell and Stanley (1963) firmly established their explicit commitment to experiments “as Experimental Design 11 the only means for settling disputes regarding educational practice, as the only way of verifying educational improvements, and as the only way of establishing a cumulative tradition in which improvements can be introduced without the danger of a faddish discard of old wisdom in favor of inferior novelties”(Campbell & Stanley, 1963, p. 172). Validity Threats Even when these designs are used, there are differences in how rigidly they are followed as well as to what extent the researcher addresses the multiple threats to validity (see Figure 3 below). Threats to validity are important not only to research designer but also to consumers of research. An informed consumer of research wants to rule out all competing hypothesis and be firmly convinced that the evidence supports the claim that the IV caused the DV. To merit this conclusion, an evaluation of the study is necessary to determine whether threats to experimental validity were recognized and mitigated. Figure 3. Example of a research experiment and the questions you should ask yourself about internal and external validity. Adapted from (Sani & Todman, 2006). hypothesized effects internal validity Are we really observing these effects or the effects of other variables on the DV (procrastination vs. intrinsically motivated)? external validity Are these effects to be found in other contexts and people, or are they specific to our experimental setting and participants? Independent Variable Graduate Advisor Meetings Dependent Variable Procrastination/ Motivation scale Experimental Design 12 Internal Validity Creswell defines internal validity threats as those “experimental procedures, treatments, or experiences of the participants that threaten the researchers‟ ability to draw correct inferences from the data in an experiment” (2003, p. 171). In their classic text, Campbell and Stanley (1963) identified eight threats to internal validity. In a more recent text, Shadish, Cook and Campbell (2002) addressed nine threats to validity which are described below. For an extensive list of threats to internal and external validity, the reader is directed to Onwuegbuzie‟s work that cogently expresses the need to evaluate “all quantitative research studies” (2000, p. 7), not just experimental design studies, for threats to internal and external validity. 1. Ambiguous temporal precedence: uncertainty about which occurred first (IV or DV) which would lead to questions about which variable is the cause and which is the effect. 2. Selection bias: a systematic bias resulting in non-random selection of participants to groups. By definition random assignment prevents selection bias, if and only if the law of large numbers can be invoked. 3. History: an event that may occur between measurements that is not part of the intervention that could impact the posttest measurement. For example, let us return to the ten fictional graduate students described previously in the study. Let‟s say they were all living in the same dorm and the fire alarm kept going off the night before they were to take the motivation/ procrastination measurement instrument. Due to lack of sleep, participants may perform differently on the Experimental Design 13 motivation/ procrastination scale than they would have had they gotten enough sleep. 4. Maturation: an observed change that is naturally occurring (such as aging, fatigue, hair length, number of graduate hours completed) that may be confused with the intervention effects but is really a function of the passage of time. 5. Statistical regression: the phenomenon that occurs when participant selection is based on extreme scores whereby the scores become less extreme, which may appear to be the intervention effect. If in our study of graduate students we purposively select students based on pretest scores of extreme procrastination, the extreme procrastinator graduate students will on the posttest not be as extreme in their procrastination tendencies. Regression toward the mean was first documented by Sir Francis Galton in the late 1800s. Galton (1886) measured the heights of fathers and sons at a World Exposition. Galton found that very tall fathers tended to have sons who were not quite as tall, and that very short fathers tended to have sons who were not quite as short. Clearly, this phenomenon is not a function of the exercise of will (i.e., fathers did not say to their wives, “Let‟s make a shorter son” or “Let‟s make a taller son”)! 6. Experimental mortality or attrition: a concern about a differential loss of participants, or of different types of participants from the experimental or control group that may produce an effect that appears to be due to the intervention. For example, if half of the students in the experimental group drop out of the study, but none of the control group members drop, we would likely question the results. Experimental Design 14 Were those students that left somehow different from the ones that remained? If so, would that difference have produced differential results than the ones we observed with the remaining participants? 7. Testing: the concern that a testing event will impact scores of a subsequent testing event. For example, if we give the graduate students the procrastination/ motivation scale prior to any graduate advisor meetings (the intervention), and then after the intervention we give them the procrastination/ motivation scale again, we may observe difference in the pre- and posttest that are due partly to familiarity with the test or the influence of the testing itself. 8. Instrumentation: the change in either the measurement instrument itself or the manner in which the instrument is implemented or scored that may cause changes that appear to be due to the intervention, or the failure to detect changes that actually did occur. For example, if between the first and second time that the procrastination/motivation test is given, the developers of the exam decide to remove ten of the questions, we do not know if the exclusion of those questions is responsible for differential scores or if the differences are due to treatment effects. 9. Additive and interactive effect of threats to internal validity: the concern that the impact of the threats may be additive or that presence of one threat may impact another. A selection-history additive effect occurs when nonequivalent groups are selected. For example, groups may be selected from two different locations, such as, rural and urban areas. The participants in the groups are nonequivalent by selection and they also have unique local histories. The Experimental Design 15 resulting net bias is dependent on both the direction and magnitude of each individual bias and how the biases combine. Selection-maturation, and selection- instrumentation are other versions of this type of effect. External Validity External validity threats are threats of “incorrect inferences from the sample data to other persons, other settings, and past or future situations” (Creswell, 2003, p. 171). Researchers must always remember the context from which their sample comes from, and take caution not to overgeneralize beyond that. Campbell and Stanley (1963) included four threats to external validity. Shadish (2002) listed five external validity threats, as detailed below. 1. Interaction of the causal relationship with participants: an effect with certain kinds of participants that may not be present (or present to the same extent) with other kinds of participants. For example, reduction of salt intake in hypertensive patients is more beneficial to certain populations than others (American Heart Association Nutrition Committee, 2006). 2. Interaction of the causal relationship over treatment variations: the permanence of the causal relationship is dependent on fidelity to the specific treatment, thus possibly producing differential effects when treatments are varied. If a particular instructional intervention includes 5 components, the causal relationship may not hold if only 2 or 3 of the components are utilized. 3. Interaction of the causal relationships with outcomes: an effect that is present with one type of outcome measurement that may not be present (or present to Experimental Design 16 the same extent) if other outcome measurements were used. For example, if a person scores highest on a test for physical strength they may not necessarily score highest on a flexibility test. 4. Interactions of the causal relationship with settings: an effect that is present in a particular setting may not be present (or present to the same extent) in a different setting. For example, a particular after school character development program involving community project work may not work equally well in rural versus urban areas. 5. Context-dependent mediation: an explanatory mediator of a causal relationship in one context may not have the same impact in another context. For example, a study might find that a reduction in federal funding has no impact on student achievement because schools were able to turn to education foundation grants to provide them with additional resources. In another school district where schools did not have access to education foundation resources, the same causal mechanism may not be available. In addition to internal and external validity threats, there are other threats that we need to be aware of in the design and evaluation of studies. Interested readers may refer to such texts as Experimental and Quasi-Experimental Designs (Shadish et al., 2002) or Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (Creswell, 2003) for information about statistical conclusion validity and construct validity concerns. Experimental Design 17 EBP in Medicine, Psychology and Education While the origins of EBP may date back to the origin of scientific reasoning, the Evidence-Based Medicine Working Group (EBMWG) brought the discussion of EBP to the forefront of medicine (1992). In 1996, Evidence-Based Medicine (EBM) was defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996, p. 71). While EBP has many supporters in medicine, EBP has caused some concerns among practitioners. Researchers have addressed concerns regarding the perception of EBM as a top down approach that results in ivory tower researchers dictating how practitioners should practice (Sackett et al., 1996) or similarly that evidence from randomized controlled trials may be valued more highly than practitioner expertise (Kübler, 2000). Yet, it is difficult to deny that there is great support for EBP considering the number of periodicals that have emerged since the years after EBMWG convened. A keyword search for “evidence-based” returns 100 serials on WorldCAT. A keyword search for “evidence-based” returns 96 serials in Ulrich’s Periodical Directory. At least 32 active periodicals, either in print form, electronic form, or both contain “evidence-based” within the title of the periodical. At least 26 of these periodicals are available electronically. See Table 1. Experimental Design 18 From the titles you can see that the majority of these periodicals are from a health-related field. It is important to note that while EBP do not only include randomized, experimental trials, the purpose of the table is to demonstrate the popularity of EBP that began in the mid 1990s and continues today. Table 1 “Evidence-Based” periodicals Start Year Title of Periodical 1994 Bandolier: Evidence-Based Healthcare 1995 Evidence-Based Medicine 1996 Focus on Alternative and Complementary Therapies: An Evidence- Based Approach 1997 Evidence-Based Cardiovascular Medicine 1997 Evidence-Based Medicine in Practice 1997 (1998) Evidence-Based Mental Health 1997 (1998) Evidence-Based Nursing 1997 Evidence-Based Obstetrics and Gynecology 1998 EBN Online 1998 Evidence-Based Dentistry 1998 Evidence-Based Practice 1998 Evidence-Based Practice: Patient Oriented Evidence That Matters 1999 Evidence-Based Dental Practice 1999 (2002) Trends in Evidence-Based Neuropsychiatry: T.E.N. Experimental Design 19 Table 1 (continued). Start Year Title of Periodical 2000 Evidence-Based Gastroenterology 2000 Evidence-Based Oncology 2000 Trauma Reports: Evidence-Based Medicine for the ED 2001 Journal of Evidence-Based Dental Practice 2003 Evidence-Based Integrative Medicine 2003 Evidence-Based Midwifery 2003 Evidence-Based Preventive Medicine 2003 Evidence-Based Surgery 2003 (2005) International Journal of Evidence-Based Healthcare 2004 Evidence-Based Complementary and Alternative Medicine: eCAM 2004 Journal of Evidence-Based Social Work 2004 Worldviews on Evidence-Based Nursing 2005 Advances in Psychotherapy: Evidence-Based Practice 2005 Evidence-Based Ophthalmology 2005 Journal of Evidence-Based Practices for Schools 2006 Evidence-Based Child Health 2006 Evidence-Based Library and Information Practice 2007 Evidence-Based Communication Assessment and Intervention Periodicals available electronically are shown in bold. Parenthetical dates indicate different start year date in WorldCAT. Experimental Design 20 The popularity of EBP is evident in psychology as well. The American Psychological Association‟s Presidential Task Force on Evidence-Based Practice specifically defined Evidence-Based Practice in Psychology (EBPP) as “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (2006, p. 273). In addition to advocating evidence-based practices, this task force also established the two necessary components for evaluation of psychological interventions: treatment efficacy and clinical utility. Treatment efficacy specifically addresses questions such as how well a particular treatment works. This type of question lends itself to experimental investigation to draw valid causal conclusions about the effect of a particular intervention (or lack thereof) on a particular disorder (American Psychological Association, … Much research on human behaviour and psychology assumes that everyone shares most fundamental cognitive and affective processes, and that findings from one population apply across the board. A growing body of evidence suggests that this is not the case. Experimental findings from several disci- plines indicate considerable variation among human populations in diverse domains, such as visual perception, analytic reasoning, fairness, cooperation, memory and the herit- ability of IQ1,2. This is in line with what anthropologists have long suggested: that people from West- ern, educated, industrialized, rich and democratic (WEIRD) socie- ties — and particularly American undergraduates — are some of the most psychologically unusual peo- ple on Earth1. So the fact that the vast majority of studies use WEIRD participants presents a challenge to the under- standing of human psychology and behaviour. A 2008 survey of the top psychology journals found that 96\% of subjects were from Western industrialized countries — which house just 12\% of the world’s population3. Strange, then, that research articles routinely assume that their results are broadly representative, rarely adding even a cautionary footnote on how far their findings can be generalized. The evidence that basic cognitive and motivational processes vary across populations has become increasingly difficult to ignore. For example, many studies have shown that Ameri- cans, Canadians and western Europeans rely on analytical reasoning strategies — which separate objects from their contexts and rely on rules to explain and predict behaviour — substantially more than non-Westerners. Research also indi- cates that Americans use analytical thinking more than, say, Europeans. By contrast, Asians tend to reason holistically, for example by con- sidering people’s behaviour in terms of their situation1. Yet many long-standing theories of how humans perceive, categorize and remember emphasize the centrality of analytical thought. It is a similar story with social behaviour related to fairness and equality. Here, research- ers often use one-shot economic experiments such as the ultimatum game, in which a player decides how much of a fixed amount to offer a second player, who can then accept or reject this proposal. If the second player rejects it, neither player gets anything. Participants from industrialized societies tend to divide the money equally, and reject low offers. Peo- ple from non-industrialized societies behave differently, especially in the smallest-scale non- market societies such as foragers in Africa and horticulturalists in South America, where peo- ple are neither inclined to make equal offers nor to punish those who make low offers4. Recent developments in evolutionar y biology, neuroscience and related fields sug- gest that these differences stem from the way in which populations have adapted to diverse culturally constructed environments. Ama- zonian groups, such as the Piraha, whose languages do not include numerals above three, are worse at distinguishing large quan- tities digitally than groups using extensive counting systems, but are similar in their abil- ity to approximate quantities. This suggests the kind of counting system people grow up with influences how they think about integers1. Costly generalizations Using study participants from one unusual population could have important practical consequences. For example, economists have been developing theories of decision-making incorporating insights from psychology and social science — such as how to set wages — and examining how these might translate into policy5. Researchers and policy-makers should recognize that populations vary con- siderably in the extent to which they display certain biases, patterns and preferences in economic decisions, such as those related to optimism1. Such differences can, for example, affect the way that experienced investors make decisions about the stock market6. We offer four suggestions to help put theories of human behaviour and psychology on a firmer empirical footing. First, editors and reviewers should push researchers to support any generalizations with evidence. Second, granting agencies, reviewers and editors should give researchers credit for comparing diverse and inconvenient subject pools. Third, granting agencies should prioritize cross-disciplinary, cross-cultural research. Fourth, researchers must strive to evaluate how their findings apply to other populations. There are several low-cost ways to approach this in the short term: one is to select a few judiciously chosen populations that provide a ‘tough test’ of universality in some domain, such as societies with limited count- ing systems for testing theories about numerical cognition1,2. A crucial longer-term goal is to establish a set of principles that researchers can use to distinguish variable from universal aspects of psychology. Establishing such principles will remain difficult until behavioural scientists develop interdisciplinary, international research networks for long-term studies on diverse populations using an array of methods, from experimental techniques and ethnography to brain-imaging and biomarkers. Recognizing the full extent of human diver- sity does not mean giving up on the quest to understand human nature. To the contrary, this recognition illuminates a journey into human nature that is more exciting, more complex, and ultimately more consequential than has previously been suspected ■ Joseph Henrich, Steven J. Heine and Ara Norenzayan are in the Department of Psychology, University of British Columbia, Vancouver, British Columbia V6T 1Z4, Canada. Joseph Henrich is also in the Department of Economics. e-mail: [email protected] 1. Henrich, J., Heine, S. J. & Norenzayan, A. Behav. Brain Sci. doi:10.1017/S0140525X0999152X (2010). 2. Henrich, J., Heine, S. J. & Norenzayan, A. Behav. Brain Sci. doi:10.1017/S0140525X10000725 (2010). 3. Arnett, J. Am. Psychol. 63, 602–614 (2008). 4. Henrich, J. et al. Science 327, 1480–1484 (2010). 5. Foote, C. L., Goette, L. & Meier, S. Policymaking Insights from Behavioral Economics (Federal Reserve Bank of Boston, 2009). 6. Ji, L. J., Zhang, Z. Y. & Guo, T. Y. J. Behav. Decis. Making 21, 399–413 (2008). Most people are not WEIRD To understand human psychology, behavioural scientists must stop doing most of their experiments on Westerners, argue Joseph Henrich, Steven J. Heine and Ara Norenzayan. G RA C IA L A M 29 Vol 466|1 July 2010 OPINION © 20 Macmillan Publishers Limited. All rights reserved10 Copyright of Nature is the property of Nature Publishing Group and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holders express written permission. However, users may print, download, or email articles for individual use.