Fluent Essays

This week, you will submit the annotation of a qualitative research article on a topic of your interest. Narrative, ethnographic, grounded theory, case study, and phenomenology are examples of types of research designs or approaches used in qualitative research. RESEARCH ETHICS APPROVAL CHECKLIST (PLANNING WORKSHEET) This worksheet is designed to help a student researcher anticipate and manage possible ethical concerns that are relevant to planning and executing a study. This worksheet contains the same 40 ethics questions that you will find in Form C (the most critical component of an IRB application to collect data). These 40 ethical standards will be evaluated for your study by the IRB (Institutional Review Board), once your proposal is approved. INSTRUCTIONS: To ensure a smooth ethics review, build your proposal and your research design so that the answer to each question below is “yes.” You should be as objective and self-critical as possible during this self-evaluation in order to spot and resolve any potential ethical challenges in advance. Researchers proposing to complete research in the following specialized areas are encouraged to review the relevant research ethics FAQs on the IRB website: Clinical or Intervention Research Educational Research International Research Research in One’s Own Workplace Research about Bullying or Other Potential Issues Related to Safety If you don’t know how to address one of the ethical standards below, just email [email protected] for support or join the IRB office hours at one of the posted times . Footnotes containing tips, samples, and definitions can be viewed by hovering your mouse over the underlined phrase. SECTION I: RESEARCHER’S CONFIRMATION OF ETHICAL STANDARDS COMPLIANCE Answer each question below with yes, no, or N/A. If you cannot easily answer “yes” or “N/A” to each of the ethical standards below, then you probably need to build extra protections into your research procedures. Sample: Will data be stored securely? Sample response: Yes. Supporting details: Paper surveys will be stored in a locked file cabinet at the researcher’s home. Electronic files will be stored on the researcher’s password-protected computer and backed up on a password-protected cloud drive. 1. Has each recruitment , consent, and data collection step been articulated such that the responsibilities of the researcher and any partner organization(s) are clearly documented? (Provide a numbered list of the data collection steps that includes details for each step, in sequential order. Here are samples . Describe steps first if you are doing a pilot or road test.) [1: HOW = Write this like a recipe, including enough details so that a person could replicate your study. Submit copies of any of the following that apply: flyer, invitation email, ad/posting. WHO = Which parties are involved in each step? In particular, we need details about any partners who might be assisting the researcher in identifying or contacting participants. Note that doctoral students may not delegate the tasks of obtaining consent or collecting data to anyone else. WHERE = Specify whether the interactions will take place via phone, email, online, or in-person at a specific location.] [2: It is fine to road test an interview or survey with friends or family prior to IRB approval and that data may not be used in the study’s analysis. However, any piloting done outside of friends/family requires prior IRB approval, regardless of whether the data would be included in the final analysis or not.] 2. Will the research procedures ensure during data collection? Describe how. [3: Privacy risks might include unintended breach of confidential information (such as educational or medical records); being observed/overheard by others while providing data; or intrusion on the privacy of others who not involved in study (e.g. participant’s family).] 3. Will data be stored ? Describe how. [4: Secure data storage requires password protection on electronic files and locks for physical data.] 4. Will the data be stored for at least 5 years? Describe how data disposal will occur. 5. If participants’ names or contact info will be recorded in the research records, are they absolutely ? Describe why or clarify that data collection is 100\% (which is preferable). [5: Retaining identifiers and/or contact information might be necessary if the researcher needs to follow up for a memberchecking step. Note that consent forms do not require signatures if the participant can indicate consent by some action such as clicking on a link, returning a completed survey, etc. Whenever possible, data should be collected without names or other identifiers.] [6: “Anonymous” means that no one (not even the researcher) knows who volunteered or declined. If a researcher documents consent in a manner that tracks their names, then the data is “confidential” rather than “anonymous.”] 6. The research procedures and analysis/writeup plans must include all possible measures to ensure that participant identities are not directly or indirectly disclosed. a. Only for research topics that possibly involve some stigma (i.e., workplace bullying): Can you confirm that the volunteering and data collection process will not result in others learning of your volunteers’ participation in the study? b. For all topics: If participant demographic details (i.e., age, ethnicity, number of years in a position) are going to be shared in the final results, will they be shared in a manner that will not render certain participants [footnoteRef:7]? [7: Participant identities might be indirectly and unintentionally disclosed if a researcher’s final research report fails to withhold demographic details or site descriptions that might permit a reader to deduce the identity of a participant. So the researcher needs to think about which demographic descriptors are most important to collect and report, while ensuring that the identity of individual participants is protected. For example, readers may be able to deduce a participant’s identity if a qualitative analysis stated, “One African-American vice-principal with 14 years of administrative experience described her professional development experience as…” A general rule of thumb is to only include a particular demographic descriptor combination if at least 3 people have that combination of demographic details. So if a district had 4 African-American vice-principals with 10+ years experience but only 2 were female, then an appropriate demographic description would be: “One African-American vice-principal with 10+ years of administrative experience described the professional development experience as…”] c. For all topics: The standard for Walden capstones is to not name the partner organization in published reports, including ProQuest. Will you mask the identity of any partner organizations that are playing a role in data collection and/or recruitment of potential participants? Exceptions to the organization-masking practice must be granted by the Program Director and approved by the IRB. Place an X here if you were granted an exception as part of your prospectus approval. 7. Will be signed by who may view data that that contains identifiers? (e.g., transcriber, translator) If so, submit a blank copy. If only the researcher and Walden faculty/staff will view the raw data, then enter NA. [8: A sample confidentiality agreement can be found here.] [9: Confidentiality agreements are required for transcribers or translators but not for the researcher or Walden faculty/staff (who are all automatically bound to confidentiality). Some professional transcribers/statisticians/etc address confidentiality in their work agreement and this is acceptable.] 8. The ProQuest publication at the end of the doctoral program is for the academic community. Is a specific in place for sharing results with the participants and community stakeholders? Describe. [10: Sharing the study’s results with participants is required. It is important that the format is audience-appropriate. Stakeholders may lack the time or inclination to digest a full dissertation. Typically, a 1 to 2 page summary or verbal presentation is most appropriate. Note, member checking and transcript review are not applicable to this section. If you plan to do transcript review or memberchecking, these steps need to be described as part of your data collection procedures in #1 above. Privacy risks are present if there is any possibility of data being linked to participants’ identities or if anyone could possibly see or hear responses during data collection. ] 9. Social science research typically involves [footnoteRef:11] risks and sometimes involves [footnoteRef:12] risks. Those risks must be acknowledged and described in order to mitigate them and document that they are outweighed by the study’s benefits (in #12 below). [11: Minimal risk is defined as follows in U.S. federal regulations: “that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves THAN THOSE ORDINARILY ENCOUNTERED IN DAILY LIFE or during the performance of routine physical or psychological examinations or tests.” ] [12: Substantial risk is anything beyond the “minimal risks” of daily life. Substantial risks are acceptable when justified by the research problem and research design, as long as adequate preventive protections are in place.] a. Are potential [footnoteRef:13] risks acknowledged and described here? [13: Psychological risks are present if materials or questions could be considered embarrassing, stressful, sensitive, offensive, threatening, judgmental, triggering, etc. Psychological risks are typically present if any of the other risks below are also present. SAMPLE DESCRIPTION: Some participants might disclose things that they later regret sharing, causing mild distress.] Mark one: NA Minimal psychological risks include: Substantial psychological risks include: b. Are potential [footnoteRef:14] risks acknowledged and described here? [14: Relationship risks are present if the recruitment or data collection process could impact the existing dynamics between the researcher and participant (who may be coworkers or have some professional relationship), dynamics among participants (if they know one another), or dynamics between the participant and the participant’s friends, coworkers, or family members. SAMPLE DESCRIPTION: Since I am recruiting people I know professionally, they might have concerns about how volunteering or declining could impact our professional relationship.] Mark one: NA Minimal relationship risks include: Substantial relationship risks include: c. Are potential [footnoteRef:15] risks acknowledged and described here? [15: Legal risks are present if data collection might result in a participant’s disclosure of violation of laws (by the participant or others). SAMPLE DESCRIPTION: Participants could inadvertently disclose a legal violation during the course of the interview.] Mark one: NA Minimal legal risks include: Substantial legal psychological risks include: d. Are potential [footnoteRef:16] risks acknowledged and described here? [16: Economic/professional risks are present if data collection could result in the participant disclosing violation of workplace policies, disagreement with leadership decisions, poor work performance, or anything else that could be damaging to the participant’s position, professional reputation, promotability, or employability. SAMPLE DESCRIPTION: Since the questionnaire asks participants to rate their agreement with leadership decisions in their workplace, there could be an impact on a participant’s promotion opportunities if there were a confidentiality breach.] Mark one: NA Minimal professional risks include: Substantial professional risks include: e. If there are any other potential risks, have they been acknowledged and described here? Mark one: NA Other minimal risks include: Other substantial risks include: 10. Have the above risks been minimized as much as possible? In other words, are measures in place to provide participants with reasonable protection from loss of privacy, psychological distress, relationship harm, legal risks, economic loss, and damage to professional reputation? In the brown column, explain how each risk identified in #9 above will be minimized. 11. Will the researcher be proactively managing any potential conflicts of (particularly when researcher is known to the participants in some professional role)? Describe how. [17: A conflict of interest is caused when the researcher has some sort of dual role in the research context, such as being a teacher, therapist, investor, business-owner, manager, etc. Conflict of interest must be managed to ensure that the research reveals “truth,” not just the outcome that the researcher might desire to see due to their other role. The simplest way to ensure this impartiality is to conduct research OUTSIDE of one’s own context but other methods are possible (e.g., using anonymous data collection to encourage honest responses).] 12. Are the research risks and reasonable, in consideration of the new that this research design can offer? Describe why. [18: All research activities place some degree of burden on the participants by asking the participants to share personal information, volunteer time, and assume risks.] [19: Examples of “new knowledge” include: effectively addressing a gap in the literature, generating new theory, enhancing understanding of a phenomenon, assessing effectiveness of a particular professional practice, addressing a local practical problem via data analysis.] 13. the research partner organization(s) grant for all relevant access, access to participants, facility use, and/or use of personnel time for research purposes? IRB staff will advise which type of partner agreement is needed, if any. State whether you will be obtaining written partner organization approval before or after Walden IRB approval. [20: If a partner organization requires the researcher to obtain Walden’s IRB approval before they can provide their written approval, that’s fine. (Walden can issue a “conditional IRB approval” letter to the researcher and then Walden’s IRB approval will then be finalized once the Walden IRB receives the partner organization’s letter of cooperation.)] [21: No Letter of Cooperation is required (a) if the researcher will simply be asking organizations to distribute research invitations on the researcher’s behalf, or (b) if the researcher is using only public means to identify/contact participants. ] [22: Note that when medical, educational, or any type of operational records would be analyzed or used to identify potential research participants, the partner organization needs to explicitly approve access to data for research purposes (even if the researcher normally has access to that data to perform their job).] 14. Is participant recruitment coordinated in a manner that is ? Describe. Coercive elements include: leveraging an existing relationship to “encourage” participation, recruiting in a setting, extravagant compensation, recruiting individuals in a setting, involving a service in the recruitment process, etc. A researcher must disclose here whether/how the researcher may already be known to the participants and explain how perceptions of coerced research participation will be . [23: For example, anonymous surveys and/or low-pressure communications such as email invitations permit potential participants to opt out with minimal fear of retaliation or other negative consequences.] [24: It is not ethically acceptable to invite a “captive audience” to participate in research on the spot (i.e., to ask an entire class or a group of meeting attendees to complete a survey during their session). Such a dynamic would not provide sufficient privacy or respect for their right to decline research participation. However, a researcher may use the last few minutes of a class session or meeting to introduce a study and distribute materials, such that the potential participants can then take their time to decide later about participation. ] [25: Generally, data collection cannot be approved during work hours or school hours unless a “free period” has been identified (e.g., break, study hall) so the research activities can be separated from the participants’ regular activities. It is important to maintain an “opt in” dynamic rather than implying that employees/students/group members are expected to participate.] [26: A researcher can ask a service provider (nurse, physician, therapist, etc.) or an aid provider (shelter staff) to give research invitations to clients who meet the inclusion criteria. However, we cannot approve for the service/aid provider to answer questions about the study, obtain consent, or collect data (unless the data is being collected by the organization itself for purposes other than the study).] [27: Doctoral research directly benefits the student (allowing him or her to obtain a degree), and so the researcher should minimize the potential for either (a) conflict of interest or (b) perceived coercion to participate. Researchers who are in positions of authority or familiarity must take extra precautions to ensure that potential participants are not pressured to take part in their study. EXAMPLES: A professor researcher may recruit her students AFTER grades have been assigned. A psychologist researcher may recruit clients from ANOTHER psychologist’s practice. A manager researcher may conduct ANONYMOUS data collection so that subordinates do not perceive their responses or [non]participation as being associated with their job standing.] 15. If you were directed to complete Form D in order to specifically recruit certain as participants, answer the following question: Is targeting this population by a research design that will specifically benefit that vulnerable group at large? Describe why. [28: For this purpose, vulnerable individuals include children, prisoners, people with cognitive impairments, on-duty military personnel, people living in an institutional setting such as a prison, inpatient care, rehab center, or shelter.] [29: Convenience sampling is not approvable. Targeted recruitment of children as participants can only be approved when a majority of the IRB votes that the study’s benefits justify its risks/costs (such as disruption to instructional time). For recruitment of adult vulnerable populations, IRB staff will determine on a case-by-case basis whether approval must be issued via the full board’s vote (as opposed to expedited ethics review).] If you were not directed to complete Form D, enter NA. 16. All samples could potentially include adults who are (without the researcher’s awareness) and it is important to include their perspectives. Would the benefits of including these individuals outweigh the risks? [30: It is ethically appropriate to include certain vulnerable adult populations if screening for that particular status would be overly invasive, given the research topic. For example, a researcher might unknowingly have participants who happen to be pregnant, residents of a facility, low-income, mentally/emotionally disabled, victims of a crisis, or elderly. We don’t expect researchers to screen for these statuses routinely for minimal risk research. However, minors may never be unknowingly recruited; adult recruitment procedures must deliberately avoid recruiting minors and/or include a reliable way of discerning that participants are 18 or older.] 17. If anyone would be excluded from participating, is their exclusion justified? Is their exclusion handled respectfully and without ? Describe. [31: When a study has exclusion criteria, they should be listed upfront in the recruitment material (flyer, invitation email,etc.) or consent form to prevent situations in which the researcher rejects volunteers in a stigmatizing manner.] 18. If the research procedures might reveal criminal activity or child/elder abuse that reporting, are there suitable procedures in place for managing this? Describe. [32: Typically, researchers only break confidentiality when they are LEGALLY REQUIRED to report certain information to authorities. Mandated reporting requirements in the USA vary by state so researchers will need to make themselves aware of state requirements. (Typically, mandated reporting only applies to researchers in professions that involve a sworn oath such as police officers or professions with licensure requirements, such as teachers and some other care providers.) Outside of mandated reporting, researchers are expected to maintain confidentiality. Any professional limits to confidentiality (i.e., duty to report) must be mentioned in the consent form. ] 19. If the research procedures might reveal or create an acute psychological state that necessitates referral, are there suitable in place to manage this? Describe. [33: At minimum, the consent form should describe a free or low-cost referral to a support resource when it is possible that the study activities may cause distress.] 20. If the research design has multiple groups, are measures in place to ensure that all participants can potentially benefit from the research? Describe how. [34: Control groups must be eligible to partake in the intervention after the study, if results show the intervention to be beneficial. If the design does not involve a control group, then the researcher only needs to ensure that all participants have equal access to the study results.] 21. Applicable for all student researchers: Will this researcher be appropriately and in all data collection procedures? Describe . [35: Researchers must be able to document their training in the data collection techniques and the IRB might require the researcher to obtain additional training prior to ethics approval. For most student researchers, the research course sequence is sufficient but some research procedures (such as interviewing people with mental disabilities) may require additional training. For psychological assessments, the manual indicates specific qualifications required. Data collection from children requires a background check/clearance through a local agency.] [36: Remote supervision is suitable for most studies but onsite supervision may be required for certain types of sensitive data collection (e.g., interviews or assessment regarding emotional topics).] [37: If your study is targeting a vulnerable population or involves a sensitive topic, describe any additional training or experience beyond the research courses and ethics training you have completed.] 22. If an existing survey or other data collection tool will be used, has the researcher appropriately complied with the for legal usage? Describe how and submit relevant documentation. [38: IF YOU ARE USING A PUBLISHED INSTRUMENT: Many assessment instruments published in journals can be used in research as long as commercial gain is not sought and proper credit is given to the original source (United States Code, 17USC107). However, publication of an assessment tool’s results in a journal does not necessarily indicate that the tool is in the public domain. The copyright holder of each assessment determines whether permission and payment are necessary for use of that assessment tool. Note that the copyright holder could be either the publisher or the author or another entity (such as the Myers and Briggs Foundation, which holds the copyright to the popular Myers-Briggs personality assessment). The researcher is responsible for identifying and contacting the copyright holder to determine which of the following are required for legal usage of the instrument: purchasing legal copies, purchasing a manual, purchasing scoring tools, obtaining written permission, obtaining explicit permission to reproduce the instrument in the dissertation, or simply confirming that the tool is public domain and providing credit by citation. Even for public domain instruments, Walden University encourages students to provide the professional courtesy of notifying the primary author of the student’s plan to use that tool in their own research. Sometimes this is not possible or there is no response. We recommend that the student make at least three attempts to contact the author at his or her most recently listed institution across a reasonable time period (such as 2 weeks). The author often provides helpful updates or usage tips and asks to receive a copy of the results. This type of communication with the author is not necessary when a website or publisher clearly states that the tool is public domain or can be used for academic/research purposes. Some psychological assessments are restricted for use only by suitably qualified individuals and these requirements are typically covered in the assessment’s manual. When in doubt, researchers must check with the assessment’s publisher to make sure that the student (or their faculty supervisor) is qualified to administer and interpret any particular assessments that they wish to use. IF YOU ARE CREATING YOUR OWN INSTRUMENT OR MODIFYING AN EXISTING INSTRUMENT: It is only acceptable to modify data collection tools if one explicitly cites the original work and details the precise nature of the revisions. Note that even slight modifications to items or instructions threaten the reliability and validity of the tool and make comparisons to other research findings difficult, if not impossible. ] Questions 23-40 pertain to the process of ensuring that potential participants make an informed decision about the study, in accordance with the ethical principle of “respect for persons.” 23. Do the procedures provide adequate time to review the study information and ask questions before giving consent? [39: Informed consent is not just a form; it is a process of explaining the study to the participant and encouraging questions before the participant makes a decision about participation. The IRB website provides an informed consent form template that researchers are invited (but not required) to use.] 24. Will informed consent be documented? [40: While documenting consent via signature is common, note that anonymous surveys can obtain “implied consent” by informing the participant, “To protect your privacy, no consent signature is requested. Instead, you may indicate your consent by clicking here/returning this survey in the enclosed envelope.”) See posted sample consent form for email options.] 25. Has the Consent Form Template been tailored using language that will be to the potential participants? [41: Walden encourages tailoring the language to the readers as long as a professional tone is maintained. For the general population in the USA, aim for grade 5 to 7 reading level.] 26. Does the consent form explain the sample’s inclusion criteria in such a way that the participants can understand why THEY are being asked to participate? 27. Does the consent form include an understandable explanation of the research purpose? 28. Does the consent form include an understandable description of the data collection procedures? 29. Does the consent form include an estimate of the time for participation? [42: Provide an estimate (in minutes or hours) of each component of data collection (e.g., survey, interview, memberchecking. etc. )] 30. Does the consent form clearly state that participation is voluntary and that the participant has the right to decline or stop participation at any time? When recruitment occurs through the researcher’s network or through an organization, the consent form must include written assurance that declining or stopping will not negatively impact the participant’s relationship with the researcher or access to services provided by the organization, as applicable). 31. Does the consent form state how many participants the researcher is seeking? 32. Does the consent form include a description of reasonably foreseeable or discomforts? [43: The consent form should state that the minimal risks are similar to those encountered in daily life, in addition to describing any risks that are substantial.] 33. Does the consent form include a description of anticipated benefits to and/or society? [44: For most social science studies, it is appropriate to state that there are no particular direct benefits to the individual. In this case, just present the benefits to society and/or benefits to the group to which the participant belongs (i.e., teachers).] 34. Does the consent form describe any , compensation, or reimbursement (for travel costs, etc.) or lack thereof? [45: Offering a modest gift card (between $5 and $20) is common for interview studies and questionnaires that are not anonymous. Note that raffles are not permitted.] 35. Does the consent form describe how privacy will be and state that that the data will not be used for any purposes other than research? For this item, we strongly recommend using the language … 2 Assignment: Annotation of a Qualitative Research Article Submit: Annotation of a Qualitative Research Article This week, you will submit the annotation of a qualitative research article on a topic of your interest. Narrative, ethnographic, grounded theory, case study, and phenomenology are examples of types of research designs or approaches used in qualitative research. An annotation consists of three separate paragraphs that cover three respective components: summary, analysis, and application. These three components convey the relevance and value of the source. As such, an annotation demonstrates your critical thinking about, and authority on, the source topic. This week’s annotation is a precursor to the annotated bibliography assignment due in Week 10. An annotated bibliography is a document containing selected sources accompanied by a respective annotation of each source. In preparation for your own future research, an annotated bibliography provides a background for understanding a portion of the existing literature on a particular topic. It is also a useful first step in gathering sources in preparation for writing a subsequent literature review as part of a dissertation. Please review the assignment instructions below and click on the underlined words for information about how to craft each component of an annotation. Please use the document Annotated Bibliography Template with Example for additional guidance. It is recommended that you use the grading rubric as a self-evaluation tool before submitting your assignment.    By Day 7 · Use the Walden library database to search for a  qualitative  research article from a  peer-reviewed  journal on a topic of your interest. · Before you read the full article and begin your annotation, locate the methodology section in the article to be sure that the article describes a qualitative study. Confirm that one of the types of qualitative research designs or approaches, such as narrative, ethnographic, grounded theory, case study, or phenomenology, was used in the study. · Annotate one qualitative research article from a peer-reviewed journal on a topic of your interest. · Provide the  reference list entry  for this article in APA Style followed by a three-paragraph annotation that includes: · A  summary · An  analysis · An  application  as illustrated in this  example · Format your annotation in Times New Roman, 12-point font, double-spaced. A separate References list page is not needed for this assignment. References http://nsuworks.nova.edu/tqr/vol8/iss4/6 Burkholder, G. J., Cox, K. A., Crawford, L. M., & Hitchcock, J. H.  (Eds.). (2020). Research designs and methods: An applied guide for the scholar-practitioner. Thousand Oaks, CA: Sage. · Chapter 12, “Quality Considerations” · Chapter 13, Ethical Considerations https://class.content.laureate.net/57306a76331448d58d043563925ab95a.pdf http://academicguides.waldenu.edu/researchcenter/resources/planning Research Theory, Design, and Methods Walden University © 2016 Laureate Education, Inc. Page 1 of 3 Trustworthiness (Lincoln & Guba, 1985) Trustworthiness is 1. The extent to which one can have confidence in the study’s findings 2. Parallel of reliability, validity, and objectivity in traditional “quantitative” research Trustworthiness Criteria Credibility Findings and interpretations are plausible to the “researched” (the participants) Do findings accurately reflect reality as seen by participants? Transferability Applicability of findings based on comparability of contexts Are conditions similar enough to make findings applicable? Dependability Account for factors of instability and change within the natural context Document naturally occurring phenomena (stability and change) Confirmability Capacity to authenticate the internal coherence of data, findings, interpretations, and recommendations Document “researcher as instrument” and potential sources of bias Research Theory, Design, and Methods Walden University © 2016 Laureate Education, Inc. Page 2 of 3 Insuring Trustworthiness Action Description Insures Prolonged engagement Investing sufficient time to learn the culture, build trust with stakeholders, understand the scope of target phenomena, and test for misinformation/misinterpretation due to distortion by the researcher or informant Credibility (internal validity) Persistent observation Continuing data collection process to permit identification and assessment of salient factors, and investigation in sufficient detail to separate relevant (typical) from irrelevant (atypical) Credibility (internal validity) Triangulation Data collection and analysis interpretation based on multiple sources, methods, investigators, and theories Credibility (internal validity) Peer debriefing Engage in analytic discussions with neutral peer (e.g., colleague not involved in the project) Credibility (internal validity) Member checks Test veracity of the data, analytic categories (e.g., codes), interpretations, and conclusions with stakeholders to ensure accurate representation of emic perspectives Credibility (internal validity) Thick description Describe procedures, context, and participants in sufficient detail to permit judgment by others of the similarity to potential application sites; specify minimum elements necessary to “recreate” findings Transferability (external validity) Audit trail Records that include raw data; documentation of process and products of data reduction, analysis, and synthesis; methodological process notes; reflexive notes; and instrument development/piloting techniques Dependability Confirmability (reliability and objectivity) Negative case analysis Investigate “disconfirming” instance or outlier; continue investigation until all known cases are accounted for so that data reflects range of variation (vs. normative portrayal) Credibility (internal validity) Research Theory, Design, and Methods Walden University © 2016 Laureate Education, Inc. Page 3 of 3 Action Description Insures Reflexive journal Researcher’s personal notes; documentation of researcher’s thinking throughout the research process Credibility (internal validity) Transferability (external validity) Dependability Confirmability (reliability and objectivity) Referential adequacy Archiving of a portion of the raw data for subsequent analysis and interpretation, for verification of initial findings and conclusions Credibility (internal validity) References Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. Trustworthiness Trustworthiness is Trustworthiness Criteria Insuring Trustworthiness Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. Educational Evaluation and Policy Analysis Winter, Vol. 6, No. 4, 1984, pp. 379-391 The Problem of Criteria for Judging Interpretive Inquiry John K. Smith University of Northern Iowa Over the years various interpretive (or qualitative, naturalistic, etc.) investiga- tors have been concerned with develop- ing criteria for judging, and defining pro- cedures for doing, this type of investiga- tion. Apparently the aim is to establish some set of foundational criteria to sepa- rate the good interpretive study from the not so good. These criteria and procedures would help advocates of this approach refute charges of subjectivity and allow them to lay claim to that certitude that is available, at least as it is generally pre- sumed, to their quantitatively oriented counterparts. Given the assumptions con- ventionally made to distinguish interpre- tive from quantitative inquiry, this paper takes the position that the quest for foun- dations may be a misguided one. If the assumptions and their implications are taken seriously, arguments that attempt to establish such nonarbitrary criteria and/or procedures will, perhaps inevita- bly, be marked by confusion and incon- sistency. To defend this contention, this paper will begin with a brief historical overview of the issue. This discussion will then be followed by a specific example of the problem and an examination of why inconsistency mars our attempts to de- velop criteria given certain assumptions. T h e a u t h o r wishes to t h a n k Lous Heshusius, Jack Yates, Joe Blase, and three a n o n y m o u s reviewers for their insightful c o m m e n t s of earlier versions of this paper. Historical Context In that this issue of assumptions and criteria (and thus the problem of certi- tude) is not a new one for this area of interpretive inquiry, a brief historical note will help to clarify the nature of the problem. One might easily argue that the tenor of our present discussions was es- tablished around the turn of this century by, among others, Dilthey and Weber. Dil- they deserves mention in that he pre- sented one of the first serious and sus- tained challenges to positivism and, in doing so, elaborated a different approach to social inquiry based on assumptions quite different from those of positivism (Hodges, 1944, 1952; Smith, 1983). Weber merits attention in this instance not for his numerous important achievements, but for the legacy left by his substantial, but nonetheless unsuccessful, attempt to develop a social science that combined a focus on subjective meaning with the de- sire for universally valid judgments (Aron, 1967, pp. 185-216). Dilthey argued that social inquiry should not be conducted with the meth- ods of the physical sciences because there was a fundamental difference in subject matter. Whereas the physical sciences dealt with a series of inanimate objects that could be seen as existing independ- ently of us (a world of independent but objectively knowable facts), the cultural sciences focused on the products of hu- man minds. Those products were inevit- 379 380 John K. Smith ably and intimately connected to human minds—including those of the investiga- tors—with all of their emotions, values, subjectivity, and so on. Accordingly, since society was the result of conscious human intention, the interrelationships of what was investigated and the investigator were impossible to separate. For Dilthey, there simply was no objective social real- ity as such, that is, divorced from the people who participate with others in con- structing or interpreting reality. Bergner (1981), in his discussion of Dilthey, has put the situation as follows: We cannot meaningfully stand outside of the events and occurrences of life and say with cer- tainty what they signify. What we can know, however, is that historical and so- cial events do signify things to those who enact them and to those who endure them (p. 64). Dilthey clearly believed that positivism erred by making the discovery of social laws, analogous to physical laws, the prin- ciple goal of social inquiry. He said that, if nothing else, the complexity of the so- cial makes it impossible to find such laws. More important, he also argued that to find laws should not even be the point of the cultural studies. The primary concern in this case was understanding (verste- hen). This point was the basis for his dis- cussion of the ideas of lived experience and that understanding required a her- meneutical perspective. In his discussion of the latter idea he noted that, at least at one level, to understand the experiences of another required a constant movement or interplay between the individual expression and the context (hermeneuti- cal circle)—a process with no absolute beginning or absolute ending points. It is here, however, that Dilthey faced a problem that he was unable to resolve. If meaning must be taken in context and understanding must be hermeneutical, clearly the interpretation of expressions could vary. Given this position, could there be any such thing as a correct inter- pretation? Dilthey could find no founda- tional standards to apply for sorting out conflicting possibilities and, at the same time, he rejected an appeal to transcen- dental values or to a metaphysical solu- tion. Whereas positivism had developed standards against which to judge true from false or correct from incorrect, Dil- they was caught between assumptions that are epistemologically antifounda- tional and a desire for criteria that are foundational. In other words, his idealist tendencies led to a relativism which he found unacceptable, but from which he could find no escape (Hughes, 1958, p. 199). Even though Weber obviously disa- greed with Dilthey in various ways (e.g., Weber felt the two sciences differed not because of an inherent subject matter dif- ference, but rather because of a different interest taken in the subject matter), he was influenced by Diltheys ideas of un- derstanding and interpretation (Roche, 1973; Wrong, 1970). However, Weber re- alized that this idealist-oriented position, in contrast to a realist-oriented positivist position, brought with it a problem of certitude. He therefore attempted to find a middle ground, in the sense of a synthe- sis, between the two approaches. In his conception the problem was that a Dil- thean approach paid too little attention to existent social reality, whereas the other perspective devoted too little attention to the ideas of subjective meaning and social reality as an interpreted reality. Aron (1967) has concisely summarized Webers project in this area: How can there be an objective science—one not distorted by our value judgments—of the value- charged productions of man? This is the central question Weber asked himself and to which he tried to provide an answer (p. 193). The answer Weber provided cannot be discussed in all of its detail and richness. Suffice it to say that one of the central tenets of his argument (along with the concept of ideal type) was the separa- tion of the idea of value relevance from that of value judgment. Weber accepted that social reality is meaningful only to the extent that it is based on values. More- over, he said that without the investiga- tors evaluative ideas, there would be no principle of selection and no meaningful knowledge of the concrete reality (1949, p. 82). Yet, at the same time he argued that universally valid knowledge is pos- sible because all research in the cultural sciences . . . once it is oriented to a given subject matter . . . and has established its Judging Interpretive Inquiry 381 methodological principles, will consider t h e analysis of data as a n end in i t s e l f (1949, p. 112). In other words, Weber held that even though values are t h e basis for m e a n i n g a n d are crucial to t h e content of inquiry, this does not m e a n that t h e in- quirer must evaluate or m a k e value judg- m e n t s about w h a t is investigated. That Webers arguments for a distinc- tion b e t w e e n value relevance and value judgment, and m o r e generally for t h e sep- aration of facts and values, cannot be sus- tained (and even ends in contradiction), is not unrecognized. Various scholars, ar- guing from a r a t h e r wide variety of differ- ent positions, h a v e m a d e substantial cri- tiques of Webers distinctions and thereby, at least indirectly, of his attempt to synthesize the two perspectives (see, e.g., Habermas, 1968; Hindess, 1977; Lu- kacs, 1971; Rorty, 1982; Strauss, 1953). Simey (1969), however, has put the situ- ation as clearly as anyone: Weber, t h e r e - fore, was attempting to find a via media between, on the one hand, realism (in its positivistic form), and, on the other, phil- osophical idealism. He accepted both al- ternatives, in t h e sense that h e sought to establish a synthesis b e t w e e n t h e m . But t h e synthesis in fact eluded h i m (p. 83). T h e question of importance for our pur- poses is, w i t h w h a t consequences for the development of social inquiry? Two things, very m u c h related, follow from Webers failure of synthesis. First, his inability to find a via media m a y well indicate that such a position cannot be found. Certainly since his time social in- quiry has b e e n characterized far less by synthesis and far more by an e n d e m i c conflict b e t w e e n competing frameworks (Outhwaite, 1983, p. 2). T h e two ap- proaches have moved along separate lines, differing not only in t e c h n i q u e , b u t also in terms of their philosophical tem- p e r a m e n t s or logics of justification. What w e refer to as t h e quantitative ap proach is based on an external perspec- tive—facts separate from values, t r u t h as a m a t t e r of correspondence to an inde- p e n d e n t l y existing reality, and t h e idea that w e may adopt a Gods eye point of view. T h e qualitative perspective is based on an internalist position—truth as so- cially and historically conditioned agree- ment, social reality as m i n d constructed, and the idea that t h e only point of view w e can take is that of various people based on their various interests, purposes, and so on (Putnam, 1981, pp. 49-74). Second, in one way Webers failure is an expression, albeit in different form, of the Dilthean problem of combining as- sumptions that are epistemologically an- tifoundational (meaning as subjective, etc.) with the attempt to find epistemolog- ically foundational criteria (certitude). Webers via media was really a matter of bringing together a frame of reference based on subjective choice and proce- dures that would yield universally valid judgments. In other words, how can one maintain or have the kind of certitude claimed by and generally granted to the realist perspective in the face of a concern over meaning and intention? Diltheys problem became, even if in a different form, Webers problem and, more impor- tant for our purposes, it is the problem that continues to be at the center of any discussion of interpretive approaches to social inquiry. Finally, one additional c o m m e n t is needed about this historical context. Over the years n u m e r o u s schools of thought h a v e b e e n put forth as t h e basis for t h e interpretive forms of inquiry. T h e n a m e s of these schools are quite well-known (phenomenological, ethnomethodologi- cal, h e r m e n e u t i c a l , etc.). Clearly it would be a mistake not to recognize that t h e r e are differences among these schools. However, it would also be a mistake, pos- sibly even a more serious one, not to rec- ognize their similarities. A good case can be m a d e that the i n s i g h t s . . . from [these] schools of t h o u g h t . . . stand close to phil- osophical idealism (Giddens, 1976, p. 155). They all, in at least some important aspects such as the idea that meaning must be grasped hermeneutically, are part of a broad m o v e m e n t away from empiri- cism along the lines of an idealist orien- tation. As Benton (1977) s u m m a r i z e s the situation, The philosophical sources of h u m a n i s t approaches in the social sci- ences at first glance appear to be very heterogeneous—phenomenological, eth- nomethodological, existentialist. . . But it t u r n s out that most of these philosophical traditions are related, through either his- torical origin, or conceptual affinities or 382 John K. Smith both, to the work of. . . Kant (p. 101; see also, Bernstein, 1976; Bleicher, 1982; Hughes, 1980; Levy, 1981; O u t h w a i t e , 1975, 1983; Psathas, 1973; Roche, 1973). A Contemporary Example To clarify, in more tangible terms t h a n have b e e n employed u p to this point, the n a t u r e of this problem of criteria, an ex- ample of some length and detail is neces- sary. While m a n y people have discussed, in different forms and with different de- grees of aggressiveness, the topic of crite- ria a n d / o r procedures for interpretive in- quiry (e.g., Bogdan & Biklen, 1982; Bruyn, 1966; Glaser & Strauss, 1967; LeCompte & Goetz, 1982; Johnson, 1975; Wolcott, 1975), an article by Guba (1981) merits special attention (see also, Guba, 1978; Guba & Lincoln, 1981, 1982a, 1982b, 1982c). Unlike some authors, Guba dis- cusses some of the specific assumptions underlying a naturalistic approach and t h e n elaborates the criteria to be used and the procedures to be employed for this mode of inquiry. His article is one of the more precise and concise t r e a t m e n t s of this topic to be found, especially in the educational research literature. Hence, Gubas work will serve as an excellent vehicle with w h i c h to demonstrate the problems that can occur w h e n one ac- cepts certain assumptions and t h e n at- tempts to elaborate certain criteria for judging naturalistic research. Guba (1981) begins by noting that the rationalist and naturalist paradigms are based on different assumptions (pp. 7 6 - 79). Rationalism assumes that a single reality exists, w h e r e a s naturalism as- sumes there are multiple realities. T h e relationship of the investigator to w h a t is investigated is a n o t h e r area of contrast. In rationalism the object of study is seen as existing i n d e p e n d e n t l y of the investiga- tor. Reality is assumed to be self-subsis- tent, a separate entity w h e t h e r or not it is an object of interest or study. In natural- ism, the investigator and w h a t is investi- gated are seen as i n t e r d e p e n d e n t . Accord- ing to this view, w h a t we treat as real is in large m e a s u r e m i n d - d e p e n d e n t as op- posed to m i n d - i n d e p e n d e n t . As Guba puts it, The rationalistic paradigm rests on t h e assumption that the inquirer can main- tain a discrete (and discreet) distance from the objects of t h e inquiry . . . the relation- ship . . . is essentially one of independ- ence (p. 77), w h e r e a s for naturalism, the inquirer a n d t h e r e s p o n d e n t . . . are inter- related, with each influencing the other (p. 77). Another important difference b e t w e e n the two paradigms concerns t h e n a t u r e of truth statements. In this case the ration- alist paradigm assumes that generaliza- tions—enduring t r u t h statements that are context free (p. 77)—are possible. Naturalism, however, holds that such generalizations are not possible and that at best w h a t one can hope for are working hypotheses that relate to a particular con- text (p. 77). T h e implication of this dif- ference in position is that while a n e u t r a l (in the sense of nonevaluative) posture is possible from a rationalist perspective, es- pecially in regard to physical science, from a naturalist perspective neutrality is impossible w h e n the objects of investi- gation are people (p. 78). T h e reasons for this impossibility are that h u m a n behav- ior is never context free and that both respondent and investigator exert a recip- rocal influence on one another. Guba has found the two paradigms dif- fer significantly in terms of their basic assumptions. He s u m m a r i z e s the under- lying assumptions of the naturalistic ap- proach quite well w h e n he says, Social/ behavioral p h e n o m e n a exist chiefly in the minds of people, and t h e r e are as m a n y realities as persons (p. 77). Social reality is m i n d - d e p e n d e n t in the sense that people (including investigators) shape or construct reality and people (in- cluding investigators) may construct their realities in their own different ways at different times and places. Given such a world of multiply con- structed realities, how is it possible to find some versions of reality trustworthy and others not? Although he does not specifically define the term trustworthy, Guba does discuss four major aspects of it (pp. 79-82). First, the findings of a trust- worthy study exhibit isomorphism to the respondents perceptions (p. 80). In other words, the investigators statements ac- curately reflect the respondents percep- tions. Second, the findings are neutral in that they are a function solely of subjects (respondents) and conditions of t h e in- Judging Interpretive Inquiry 383 quiry and not of the biases, motivations, interests, perceptions, and so on of the inquirer (p. 80). Third, if the inquiry is repeated with the same or similar sub- jects, the findings should be consistent with those of the first inquiry. Fourth, the results must be transferable to other sim- ilar situations. Taken together these points seem to represent an attempt to separate inquiry that tells us how people really perceive their own situations from inquiry that distorts reality by the in- volvement of the investigators percep- tions, biases, and interests. Using this interpretation of trustworthiness, Guba then describes a series of procedures, such as member checks, structural corrobora- tion, and dependability audits, which are used in the naturalist mode to deal with this problem. Given his assumptions, what Guba is attempting to do by specifying criteria is not possible. The assumptions of multiple realities and reality as mind-involved se- riously undermine the notion of applying foundational criteria to distinguish trust- worthy results from untrustworthy ones. These assumptions and foundational cri- teria are, in a word, incompatible. To ac- cept that social reality is mind-con- structed and that there are multiple real- ities is to deny that there are any givens upon which to found knowledge. If one accepts these assumptions, different claims about reality result not from incor- rect procedures but may simply be a case of one investigators interpretation of real- ity versus anothers. In a world of multi- ply constructed, mind-dependent reali- ties, there may be no court of last resort to appeal to to sort out trustworthy inter- pretations from untrustworthy ones. Some examples of the inconsistencies that result from accepting antifounda- tional assumptions while attempting to pose foundational criteria can now be pre- sented. Gubas problems with consistency are clearly evident in his varied use and definition of the term criterion. By formal definition criterion can mean slightly dif- ferent things. Among other possibilities, the term may refer to a characterizing trait. In this usage it has, at best, mild implications as a prescription for inquirer behavior and does not necessarily refer to something that is held to be foundational. Criterion may also, however, mean a standard against which to make a judg- ment. Here the term brings with it a very strong implication of something founda- tional—it is that touchstone that can be employed to sort out the good from the bad, the correct from the incorrect. In some instances Guba seems to be employing the former definition and seems aware of the antifoundational im- plications of his assumptions. Toward the end of the article he says his criteria are not prescriptions and that he does not want them to become an orthodoxy (p. 90): But—most emphatically—the [cri- teria] are not prescriptions of how inquiry must be done (p. 90). If this is the case, then he is discussing simply the charac- terizing traits of a particular approach to inquiry and saying that these criteria are merely the way inquirers seem to do it. The elements he discusses cannot there- fore be thought of in such terms as mini- mums, absolutes, and essentials. Throughout the paper, however, there is considerable evidence that Guba de- sires criteria to carry more weight than as simply characterizing traits. His first sen- tence demonstrates that he is concerned with criteria for judging the trustworthi- ness of inquiries conducted with the na- turalistic inquiry paradigm (p. 75). In fact, the entire paper is directed at dem- onstrating that naturalistic researchers also have standards. For example, Guba poses his criteria as equivalent to or par- allel to those of a rationalist paradigm. The tone is one of we are equally well protected from the problems of subjectiv- ity. More specifically, he uses terms such as newer rules (p. 90), implies that his efforts are an attempt to codify the safe- guards (p. 76), and says that he has elab- orated criteria that are to be generally applied by editors, referees, and research- ers to sort out the trustworthy from the untrustworthy: When a naturalistic study is to be judged, it is these criteria that ought to be brought to bear (p. 88). This is all rather strongly put and defi- nitely goes beyond describing simply how inquirers proceed through this type of study. Even though at times his use of language makes it difficult to discern his exact meaning or intention, it is clear that he desires standards that are neutral and 384 John K. Smith nonarbitrary. In the end, his antifounda- tionalist assumptions push his use of cri- teria in one direction, while his desire for something to hang on to pushes it in the other direction. A second major area of inconsistency involves the status of t h e procedures Guba lists. At one point he seems to rec- ognize the epistemological consequences of his assumptions, but at other points he does not. He contends that procedures such as triangulation, peer debriefing, and prolonged engagement will not guarantee that a naturalistic study will be trustwor- thy. These procedures do not constitute, as Guba feels is the case for their ration- alist counterparts, a theoretically com- plete w h o l e and will therefore not in- sure, but only increase, the probability of a studys trustworthiness (p. 88). How- ever, shortly after discussing these pro- cedures in terms of ambiguity and in- d e t e r m i n a n c y (p. 88), he gives some of the m e m b e r s on the list the status of sine qua nons (p. 88). As he notes, It is incon- ceivable that one would be persuaded of t h e trustworthiness of a study that in- volved no triangulation and no m e m b e r checks (p. 88). This is quite clearly an attempt to pose an absolute (a necessary if not sufficient condition) on w h i c h to found naturalistic inquiry. T h e establish- m e n t of absolute conditions for trustwor- thiness negates the fact that only indeter- minate procedures can follow from the relativism of multiply constructed reali- ties. T h e assumptions on w h i c h natural- istic research is based are violated by pos- iting procedures that are indispensable. T h e r e is a great deal of difference b e t w e e n describing w h a t one might do and man- dating w h a t one must do. Inherent in Gubas assumptions and specifically stated by him (p. 78), is the significant idea that the investigator can- not be neutral. This idea immediately pro- vokes problems since earlier h e argues for neutrality as one of the key elements for trustworthiness. This point aside, this is- sue of neutrality appears w h e n Guba at- tempts to preserve the idea of inquirer distance from the subject matter and even claims there is such a thing as op- timal distance (p. 77). Once it is granted that reality is constructed by people (among w h o m are investigators), this prospect of maintaining distance is very difficult to defend. To do so, the object of study must exist i n d e p e n d e n t l y of the in- quirer, and the inquirer must have inde- p e n d e n t access to that object. Once the investigator is seen as mind-involved with the construction of reality that he or she investigates, there is no way for the m i n d to stand apart (that Gods eye view) and measure distance. The mind, after all, cannot be distanced from itself. Once one grants that reality is m i n d - d e p e n d e n t and that there are as m a n y realities as t h e r e are persons, it follows that it may be im- possible to sort out w h o has maintained proper distance and w h o has not. Gubas work therefore provides an ex- cellent example in tangible terms of w h a t happens w h e n antifoundational assump- tions are combined with the attempt to pose foundational criteria. T h e inevitable result is inconsistency of argument. It is now time to discuss w h y this result is inevitable. The Issue of Assumptions and Criteria Given the assumptions of naturalistic or interpretive inquiry, w h y is it impos- sible to develop and apply foundational criteria to this type of investigation? To answer this question, it is necessary to review briefly how the dominant mode of inquiry—quantitative—has dealt with this issue of assumptions and criteria. One reason for this digression is that interpre- tive inquiry, at least since the time of the challenge initiated by Dilthey and others to positivism, has very often been para- sitic (i.e., taken its strength from and de- veloped through critique of the quantita- tive approach). As Rorty (1979) notes, Hermeneutics is always parasitic upon the possibility of epistemology (p. 365). Therefore, to u n d e r s t a n d the claim that there can be no criteria to reconcile dis- course (to sort out the trustworthy from the u n t r u s t w o r t h y results) for interpre- tive inquiry, we must first e x a m i n e the opposing claim that t h e r e are indeed foundations on w h i c h to base epistemo- logical claims. A n u m b e r of assumptions, some of w h i c h are often implicit, are crucial to the assertion that t h e r e are nonarbitrary criteria for judging and nonarbitrary pro- cedures for doing quantitative inquiry. Judging Interpretive Inquiry 385 Even though other terms, such as exter- nalism, objectivism, and so on, could be used here (Putnam, 1981, pp. 49-50; Ru- binstein, 1981, pp. 24-25), for our pur- poses the assumptions of this approach can be located within the general philo- sophical perspective or temperament known as realism. A most critical feature of a realist perspective is the idea that what exists does so whether men con- ceive of it or not (Trigg, 1980, p. vii). In other words, reality is self-subsistent; it exists out there independently of us and of our interest in it. Following the lead of Durkheim (1938), social and educational facts are considered external to us, exist- ing prior to an investigators study of them, awaiting discovery. While this po- sition grants that minds are necessary to conceptualize reality (by definition only minds can do so), it does not accept the idea that reality is mind-dependent in the sense of being created or constructed. As Trigg notes, The realist must hold that the mind and the world are separate items (p. 93). Given this separation of the knower from what is or can be known, the basic epistemological thrust of realism is that to know reality is to be able to describe or reflect it accurately. According to most realists . . . a judgment is true when it corresponds to an independent reality and false when it does not (Ewing, 1974, p. 195). Not surprisingly, this position is la- beled as the correspondence theory of truth. True statements are judged to have accurately reflected the qualities and characteristics of what is out there, whereas false ones are those judged to have in some way distorted the nature of that independently existing reality. For example, if one says the cat is on the mat and the cat is observed to be there, we accept that one has produced a true state- ment. Similarly, if one says that school performance and self-concept are posi- tively related in first graders and on ob- servation this turns out to be the case, one has made a statement we judge to be true. The significant point is that truth has its source in an independently existing real- ity—a reality that can be known for what it really is. As these examples demonstrate, the process of …