Fluent Essays

1. What is an independent variable? Provide examples. 2. Explain the difference between quasi-experimental and experimental research? 3. Identify the 3 elements to consider when deciding whether to use the experimental research design.   4. Epidemiological models of causation can be used as a logical framework when examining concepts within health informatics. Provide one example of this. Health Informatics Research Methods: Principles and Practice, Second Edition Chapter 5: Epidemiological Research © 2017 American Health Information Management Association © 2017 American Health Information Management Association 1 Learning Objectives Explain the purpose of epidemiology and its importance in health informatics. Apply epidemiological principles and models to examine health informatics topics. Describe the different types of epidemiological study designs and how they can be used in health informatics research. Assess the impact of confounding, recall bias, and other types of bias in epidemiological studies. Determine which statistical tests should be used for each study design, such as the odds ratio (OR) for the retrospective study and the relative risk (RR) for the prospective study. List and explain the rules of evidence when considering whether an association is causal. © 2017 American Health Information Management Association Introduction Epidemiology examines patterns of disease occurrence in human populations, and the factors that influence these patterns in relation to time, place, and persons. Essential tool when developing specific research methodologies in health informatics. This chapter provides examples of epidemiological principles to study disease and health informatics. © 2017 American Health Information Management Association John Snow Map of the Outbreak of Cholera How might this map look today using modern HIT? © 2017 American Health Information Management Association Types of Epidemiology Epidemics—What caused them and how they could be controlled and prevented Expanded rapidly beyond the study of infectious diseases into the study of all types of illnesses Cancer epidemiology, pharmaco-epidemiology, environmental epidemiology, nutritional epidemiology, chronic disease epidemiology, health services epidemiology © 2017 American Health Information Management Association Epidemiology and Health Informatics Epidemiological principles can be used to study any type of behavior, outcome, occurrence, community, or healthcare system. The key is to know which epidemiological study design to use to inspect a particular problem. Epidemiological principles and study designs are used to examine many of the health informatics systems and structures that sustain the healthcare system today. © 2017 American Health Information Management Association Example Bell and colleagues (2003) used a cross-sectional study to determine whether physician offices located in high-minority and low-income neighborhoods in southern California have different levels of access to information technology than offices located in lower-minority and higher-income areas. © 2017 American Health Information Management Association Example (cont.) Used epidemiological principles similar to those that Snow developed Researched physician offices in targeted geographic areas and neighborhoods to determine the use of different types of health information technology Did not establish the cause of any particular disease, but determined whether or not socioeconomic demographics play a part in the use of information technology © 2017 American Health Information Management Association 8 Infectious Disease Model Host Agent Environment Age, gender, race religion, marital status, ethnicity, genomics, social behaviors, anatomy and physiology, prior illness or disease Nutritional, chemical, physical, infectious Physical environment Tornado, flood, hurricane, war Occupational environment Infectious Disease Source: Gordis 2004, 16; Lilienfeld 1994, 37–38. © 2017 American Health Information Management Association Chronic Disease Model © 2017 American Health Information Management Association Chronic Disease Model: Example of Lung Disease © 2017 American Health Information Management Association Using the Epidemiological Models of Causation in Health Informatics Host Agent Environment Experience Training Understanding of CAC system Computer problems Coding errors in system User friendly Encoder issues Documentation in EHR incomplete Structured text Free text Artificial Intelligence Epidemiological model: Health informatics example— Computer-assisted coding (CAC) © 2017 American Health Information Management Association Chronic Disease Model: Example of Reluctance to Use PHR © 2017 American Health Information Management Association Epidemiological Study Designs Descriptive study Cross-sectional or prevalence study Analytic Studies Retrospective (case-control) study Prospective study Experimental study Clinical and community trial © 2017 American Health Information Management Association Progression of Epidemiological Study Designs © 2017 American Health Information Management Association Analytic Study Design: Retrospective case-control Prospective study Historical-prospective study Experimental Study: Clinical trial Community trial Descriptive Study Design: Cross-sectional Prevalence Components of the Cross-sectional or Prevalence Study Describes health characteristics at one point or period in time Generates hypotheses Determines whether the disease or health characteristic exists now Generates new ideas Performed when very little is known about a topic Excellent design when studying new concepts in health informatics Leads to analytic studies © 2017 American Health Information Management Association Prevalence Rate Example of how a prevalence rate is determined: Where 10 n is usually 1 or 100 for common attributes. The value of 10 n might be 1000 if expressing the rate per 1,000 facilities, 10,000 if expressing the rate per 10,000 facilities, and so forth (Source: CDC 2012) © 2017 American Health Information Management Association Sensitivity and Specificity Sensitivity and specificity rates can be used in prevalence studies when assessing correct measurement or correct labeling. True positives (TP): Correctly categorize true cases as cases (cases are individuals with the disease or outcome) = valid labeling False negatives (FN): Incorrectly label true cases as non-cases (non-cases are those individuals without the disease or outcome = invalid labeling © 2017 American Health Information Management Association Sensitivity and Specificity (cont.) True negatives (TN): Correctly label non-cases as non-cases = valid labeling False positives (FP): Incorrectly label non-cases as cases = invalid labeling Sensitivity = Percentage of all true cases correctly identified where TP/(TP+FN) Specificity = Percentage of all true non-cases correctly identified where TN/(TN+FP) (Source: Lilienfeld and Stolley 1994) © 2017 American Health Information Management Association Example of Prevalence Study The American Hospital Association (AHA) conducted a prevalence study by surveying AHA member hospitals to determine their use of health information technology. Survey instruments were sent to hospital CEOs from all types of hospitals and from different geographic areas across the country. © 2017 American Health Information Management Association Analytic Study Designs: Case-Control (Retrospective) Steps to follow when conducting a case-control (retrospective) study Step 7 Design the instrument used to collect the exposure or risk factor data. Collect it through phone or in-person interviews, self-report questionnaires, abstracts from existing sources such as the EHR, cancer registry, birth certificates, death certificates, financial records and so forth. Step 8 Analyze the data to include the appropriate statistics. Step 9 Summarize the results and determine if they support or refute the hypothesis Step 10 Publish the results Step 1 Determine the hypothesis and decide whether to use prevalence cases (existing cases of disease) or incidence cases (new cases of disease) Step 2 If using prevalence cases, seek out cases from the state or hospital-based cancer registry. If using incidence cases, have healthcare facilities provide new cases as they are treated Step 3 Decide who will be part of the study by using inclusion criteria to validate the disease under study, such as ICD-10-CM codes, laboratory reports, radiology reports, and health records. Step 4 Randomly select the cases by obtaining a list of possible cases (either from the state or hospital-based cancer registry or from a list of ICD-10-CM codes, and so forth) and using a systematic sample, such as choosing every fifth case. © 2017 American Health Information Management Association Analytic Study Designs: Case-Control (Retrospective) (cont.) Step 5 Choose controls from siblings or friends who are the same gender and of similar age and socioeconomic status, or from similar patients at the same hospital. Controls and cases should share all characteristics except the disease under study. For example, if studying melanoma, controls could be chosen from the same hospital-affiliated cancer registry as the cases, but the controls would have another type of cancer, such as colon cancer or lung cancer. Select these controls from a list of cancer cases identified by their ICD-10-CM code and validate the diagnosis through pathology reports and health records. Also, choose control participants from this list who are similar in age by at least five years. Step 6 Decide whether to match the cases and controls for certain variables. Matching on variables such as age, gender, race and so forth should only be used when the researcher is certain that there is a relationship between a given variable and the dependent variable. For example, age is always related to cancer because the likelihood of developing cancer increases as people age. Therefore, age is a confounding variable in case-control studies of cancer because it may be the underlying factor that leads to the development of the cancer. Matching patients by age will reduce the chance that age confounds the study of the specific cancer risk factor being studied. © 2017 American Health Information Management Association Analytic Study Designs: Case-Control (Retrospective) cont’d Step 7 Design the instrument used to collect the exposure or risk factor data. Collect data through phone or in-person interviews, self-report questionnaires, or abstracts from existing sources such as the EHR, cancer registry, birth certificates, death certificates, and financial records. Some researchers, such as Watzlaf, design a research instrument to collect both interview-related data and health record data (Watzlaf 1989). The full research instrument is provided in online appendix 5A. It is extremely important that the researcher choose appropriate data sources so that information related to both the cases and controls can be found. Step 8 Analyze the data to include the appropriate statistics. Step 9 Summarize the results and determine if they support or refute the hypothesis. Step 10 Publish the results. © 2017 American Health Information Management Association Example—Odds Ratio (OR) The OR for this example is: The value of 4 means that individuals who use tanning lamps are 4 times more likely to develop melanoma than individuals who do not use tanning lamps. If the OR for this example equaled 1, the risk for melanoma would be equal for the cases and controls and use of tanning lamps is not a risk factor for melanoma. If an OR is less than 1, the factor (the use of tanning lamps) decreases the risk of disease and provides a protective effect. © 2017 American Health Information Management Association 24 Example: Case-Control Study in Health Informatics Vinogradova, Coupland, and Hippisley-Cox (2014; 2015) used the case-control design to examine the relationship between oral contraceptives and risk of venous thromboembolism (VTE). The authors of this study used two research databases that have been tested and validated in the United Kingdom called QRESEARCH and Clinical Practice Research Data link (CPRD) to examine this relationship. Data within this database were tested and analyzed, and found to be valid in more than 90 percent of the cases when compared with paper-based records. Cases included those individuals with VTE and they were individually matched with up to five female controls with the same age and physician practice. Odds ratios were computed. Combined oral contraceptives was associated with an increased risk of VTE (OR = 2.97). Confounding variables, or other variables that can also play a part in the development of the disease, VTE were also collected and controlled for, and included smoking status, alcohol consumption, ethnic group, body mass index, comorbidities, and other contraceptive medications. © 2017 American Health Information Management Association 25 Cohort (Prospective) Study Design This study design has two groups of study participants: One with the exposure (independent variable) One without the exposure (dependent variable) Both groups are then followed forward in time to determine if and when they develop the disease or outcome variable under study. © 2017 American Health Information Management Association Calculating the Relative Risk and Incidence Rate The calculation for relative risk (RR) is: The calculation for incidence rate is: © 2017 American Health Information Management Association 27 Calculating the Relative Risk (cont.) To calculate relative risk, perform the following: Incidence rate of exposed = Incidence rate of unexposed = Relative risk: © 2017 American Health Information Management Association RR Example for Risk of Migraines in Children Who Play Video Games The RR for this example is: Incidence rate of exposed = Incidence rate of unexposed = Relative risk: In this example, the RR of 9.4 is very high for the association between use of video games and migraine headaches, and those children who play video games are almost nine times more likely to develop migraine headaches than those who do not play video games. © 2017 American Health Information Management Association 29 JBlock (JB) - Reviewer: The result should be explanted as children who play video games would be 9 times as likely as those who do not play video games to develop migraine headaches. Prospective Study Example in Health Informatics A prospective study (Tierney et al. 2015) in which researchers restricted access to providers from an urban health system’s EHR information, based on patient preferences, was conducted in one clinic setting. Patients could choose to allow or restrict providers’ access to medications, diagnoses, results and reports or only sensitive data such as STDs, HIV, drugs, alcohol use, behavioral health information, and so on. Providers were followed over time to study what occurred when these restrictions to patient information were made. It was found that providers did have to “break the glass” over 100 times to gain access to EHR information in order to provide patient care. © 2017 American Health Information Management Association Experimental Study Designs in Epidemiology   Experimental research studies expose participants to different interventions (independent variables) to compare the result of these interventions with the outcome (dependent variables). Two examples of experimental research studies in epidemiology include the clinical and community trial. © 2017 American Health Information Management Association Clinical Trials Clinical trials are designed to help healthcare professionals test new approaches to the diagnosis, treatment, or prevention of different diseases. Patients who are at high risk for developing these diseases are often the ones who participate in the clinical trial. The clinical trial is designed to test new medications (most common) and surgical procedures, as well as new treatments or combinations of treatments to prevent disease. © 2017 American Health Information Management Association Community Trials Very similar to clinical trials but take place in a particular community and have less control over the intervention than one would have with the clinical trial. The community trial’s goal is to produce changes in a specific population within a community, organization, or association. Participation includes all members of the community and the intervention tends to be provided throughout the population. (Friis and Sellers 2014, 322–323; UPMC 2015). © 2017 American Health Information Management Association Clinical and Community Trial Protocol Rationale and background Specific aims Randomization Blinding or masking Types and duration of treatment Number of subjects Criteria for including and excluding participants Outline of treatment procedures Procedures for observing and recording side effects Informed consent Analysis of data Dissemination of results © 2017 American Health Information Management Association Types of Clinical Trials Treatment trials test experimental treatments, new combinations of medicines, different types of surgery, radiation or chemotherapy. Prevention trials aim to prevent disease in a person who has never had the disease or to prevent it from advancing or reoccurring. Diagnostic trials are conducted to find better tests, procedures, or screenings to detect a disease or condition. © 2017 American Health Information Management Association Types of Clinical Trials (cont.) Screening trials examine the best method to detect diseases or health conditions. Quality of life trials explore methods used to improve comfort and the quality of life for individuals with a chronic disease © 2017 American Health Information Management Association Phases of Clinical Trials Phase I, II, III, or IV based on the size of the population and the intervention being tested. The FDA provides guidelines for the different types of clinical trials. Phase I: Usually test a new drug or treatment in a small group of people (20–100) Phase II: Study the intervention in a larger group of people (100–300) Phase III: The study drug or treatment is given to even larger groups of people (300–3,000) Phase IV: Include studies that collect additional information after the drug has been marketed, such as the drug’s risks, benefits, and optimal use © 2017 American Health Information Management Association Rules of Evidence for Causality Strength of association The strength of the association is measured by the RR. A strong RR is important, and those >2 are effective to show causality. Repeated findings of weak RRs may be of equal importance if it is found in studies with reliable methodology. Consistency of the observed association Confirmation of results in many different types of epidemiological studies in different populations and different settings. © 2017 American Health Information Management Association Rules of Evidence for Causality (cont.) Specificity A one-to-one relationship between an independent variable and a dependent variable, or between the exposure and the disease is necessary to add weight to causality. Because some exposures may lead to many different adverse outcomes, if specificity is not found this does not mean an association is not causal. Temporality The independent variable must precede the dependent variable, not follow it. For example, in order to state that decision support systems decrease medical errors, the use of the decision support system must precede the development of the medical error. Sometimes this is not easy to determine. A prospective study design can help support this rule. © 2017 American Health Information Management Association Rules of Evidence for Causality (cont.) Dose-response relationship As the dose of the independent variable is increased, it strengthens the relationship with the dependent variable. In epidemiology, this can be demonstrated for smoking, in which dose and duration increase risk of disease. In health informatics, if clinical reminder systems for colonoscopy reduce the likelihood of developing colon cancer, increasing the use of the clinical reminder systems for other types of cancer screening can be assumed to also reduce the development of cancer. © 2017 American Health Information Management Association Rules of Evidence for Causality (cont.) Biological plausibility The relationship must make sense in relation to what is known about it in the sciences, animal experiments, and so forth. Experimental evidence A well-conducted RCT may confirm the causal relationship between an independent variable and a dependent variable. © 2017 American Health Information Management Association Rules of Evidence for Causality Coherence: Association should be in accordance with other factors known about the disease. Analogy: If similar associations have demonstrated causality, then the more likely this association is probably causal. © 2017 American Health Information Management Association Summary Epidemiology and its principles can be used effectively when studying health informatics. Researchers can use infectious disease or chronic disease models of causation to do this. Many different types of epidemiological study designs can also be used to examine health informatics. © 2017 American Health Information Management Association Summary cont’d These include the descriptive (prevalence or cross-sectional), case-control (retrospective), prospective, and the experimental (clinical and community trial) study designs. Most epidemiologists conduct research by beginning with the cross-sectional or prevalence study, and then move forward to the case-control, prospective, and experimental study designs. © 2017 American Health Information Management Association Header 45 © 2017 American Health Information Management Association Number of U.S.ambulatory healthcare facilities that use digital radiology systems Number of ambulatory care facilities in the U.S.× 10 𝑛 Health Informatics Research Methods: Principles and Practice, Second Edition Chapter 4: Experimental and Quasi-Experimental Research © 2017 American Health Information Management Association © 2017 American Health Information Management Association Learning Objectives Differentiate between experimental and quasi-experimental research types and methodologies and decide when each should be used in health informatics. Distinguish between the pretest-posttest control group method and the Solomon four-group method and how they are used in health informatics Explain when the posttest-only control group method should be used. Provide examples of when to use the one-shot case study. Demonstrate when to use the one-group pretest-posttest method and when to use the static group comparison method. © 2017 American Health Information Management Association Experiment Begin with a hypothesis Test it Refine hypothesis Test again Reach conclusions Try to establish cause and effect © 2017 American Health Information Management Association Experimental Research Most powerful when trying to establish cause and effect Expose participants to different interventions In order to compare the result of these interventions with the outcome © 2017 American Health Information Management Association Independent Variable The intervention or factor you wish to measure in order to determine if it will have an effect on the outcome or disease under study. Examples: Medications, diet, exercise, education, health information system © 2017 American Health Information Management Association Dependent Variable The outcome, end point, or disease under study Examples include: Survival time for patients with cancer, reduction of pressure sores in patients using specific type of wheelchair, decrease in the number of adverse events in health care facilities using a CPOE © 2017 American Health Information Management Association Dose-Response Relationship Experimental research also tries to determine a dose-response relationship. If a new medication has slowed the progression of cancer, will a higher dose slow the progression even faster? Or if a specific factor is removed from the environment it may also decrease the progression of a certain disease. © 2017 American Health Information Management Association 7 Use Experimental Research Consider the following Eligibility of appropriate participants Randomization Ethical Issues © 2017 American Health Information Management Association Quasi-Experimental Research Similar to experimental research but does not include randomization of participants Independent variable may not be manipulated by the researcher, and there may be no control group It may be used over time with something other than individual participants © 2017 American Health Information Management Association Example of Quasi-Experimental Research Study the effects of automated coding system to determine if there is an increase in hospital reimbursement before and after the system is implemented Study the cost/benefits of using an EHR before and after its implementation. Cost/benefits of a paper-based system is compared to the cost of an EHR in all HIM functions. © 2017 American Health Information Management Association Study Design Characteristics Diagram Pretest-posttest control group method Randomly assigned to intervention or non-intervention (control) group Pretests given to both groups Posttests given to both groups after intervention Treatment: R---O---X---O Control: R---O--- ---O Solomon four group method Two intervention groups Two control groups Randomization used to assign to all four groups Pretest for one pair of intervention and control groups Same intervention used in both groups Posttest used in all four groups Treatment: R---O---X---O Control: R---O--- ---O Treatment: R--- ---X---O Control: R--- --- ---O Posttest only control group method Randomization used for assignment into intervention and control groups No pretest given Intervention given to one group only Posttest given to both groups Treatment: R--- ---X---O Control: R--- --- ---O R = randomization O = observation X = intervention Overview of Experimental Research Designs © 2017 American Health Information Management Association Study Design Characteristics Diagram One-shot case study Simple design One group Intervention Posttest Treatment: ---X---O One group pretest-posttest method One group Pretest Intervention Posttest Treatment: O---X---O Static group comparison method Two groups Intervention No intervention Posttest for both groups Treatment: ---X---O Control: --- ---O Overview of Quasi-Experimental Study Designs © 2017 American Health Information Management Association Elements Randomization When study participants are randomly chosen to be in the experimental, control, or comparison group using a random method, such as probability sampling, so that each participant has an equal chance of being selected for one of the groups. Intervention = experimental group No intervention = control group Different intervention = comparison group © 2017 American Health Information Management Association Example of Randomization Researcher may be interested in determining whether individuals retain more if they do higher levels of exercise before they learn how to use the PHR Three different groups of participants will be established. First group will run for 30 minutes before sitting down in front of the computer to learn to use their PHR Second group will walk for 30 minutes before learning to use the PHR Third group will not do any type of exercise before learning to use the PHR Therefore, the first group is called the experimental group, the second is called the comparison group and the third is called the control group and randomization will be used © 2017 American Health Information Management Association Example of Randomization (cont.) Develop a list of all the study participants and number them Pick out each number and allocate to a particular group. For example, the first number drawn will go into the experimental group, the second number to the control group and the third number to the comparison group and so forth until all the numbers are drawn and participants allocated The goal is to have the experimental, control, or comparison groups as similar as possible except for the intervention under study Randomization techniques can be performed using statistical software programs © 2017 American Health Information Management Association Comparison Group May be unethical to withhold a certain intervention from one group of participants Some experimental research studies do not contain a control group but instead use two comparison groups. The intervention under study is still used but all members of the study are receiving some type of intervention For example, if researchers are assessing the effect of using HIEs to decrease the incidence of hospital-acquired infections in four nursing facilities in a particular region, then two of the nursing facilities will use the HIE based data and two of the other nursing facilities will need to utilize some other type of database in order to minimize the unethical consequences of not providing any type of data © 2017 American Health Information Management Association Crossover Design A crossover design can be used to minimize the unethical effects of not providing certain types of interventions Includes using one group of participants as both the experimental group and the control group. A group of participants start out by being assigned to the experimental group and receive this intervention for a certain period of time, such as 6 months or a year. After they receive the intervention, they cross over to receiving no intervention or another comparison intervention for another 6 months to a year © 2017 American Health Information Management Association Example of Crossover Design May be used when studying whether certain types of telerehabilitation will improve the outcomes of patients with multiple sclerosis Patients may start out using sensors and body monitoring and then cross-over to using a PDA or the traditional in-house therapy monitoring in order to monitor their functional levels after treatment The same group is used as the control (or comparison group) and the experimental group © 2017 American Health Information Management Association Observation Pretest—Observing the experimental and control or comparison groups before the intervention Posttest—Observing the experimental and control or comparison groups after the intervention © 2017 American Health Information Management Association Examples of Observation Blood pressure taken before and after the administration of medication, diet, or exercise Questionnaire given to determine levels of depression before and after a medication intervention. Observing a group of individuals before the administration of a policy or procedure change and then observing them again after the change has been in place for one month. © 2017 American Health Information Management Association Examples of Observation (cont.) Midtests—Observations administered neither before or after the intervention but during the middle of the particular study Other observations may be conducted several months or years after the intervention ends to determine its long term impact. Time-series tests—Conducted throughout the study period as a new policy or law is implemented Might be used to examine the number of breaches of confidentiality after the implementation of HIPAA. These rates could be compared to rates before HIPAA was implemented © 2017 American Health Information Management Association Control Group Use of the control group allows the researcher to determine if the effect seen is really due to the intervention and not other extraneous factors or confounding variables In clinical trials when medication is being tested as the intervention, the control group is given a placebo so that they are as similar as possible to the intervention group but not receiving the medication under study © 2017 American Health Information Management Association Treatment Treatments or interventions are also the independent variable Use of experimental medications, changes in an individuals’ behavior such as smoking or alcohol cessation, or changes in a particular assistive device, technology, software or system. Should be administered in the same way for all participants in the experimental group. © 2017 American Health Information Management Association Treatment (cont.) Example: If physical therapists are educated and trained online in using a new rehabilitation EHR system, the level (hours of training), quality (content of the online education and training), and hands-on application (amount of time using the EHR system) should be the same for all physical therapists in the experimental group Control group may consist of those physical therapists that will receive the traditional in-class education and training. Hypothesis: Those physical therapists trained on-line or with distance education will be the same or better than those trained using the in-class method. 24 © 2017 American Health Information Management Association Experimental Studies Pretest/Posttest Control Group Method Similar to the randomized controlled trial (RCT) or clinical trial Pretest-posttest control group method provides an intervention that may include a specific program or system change than a medication or treatment Participants are randomly assigned to either the intervention (experimental) or a non-intervention (control) group Control/Comparison group may receive a different intervention other than the one under study Pretests are given to both groups at the same time to assess their similarities and differences Posttests are given to both groups to determine the effect of the intervention © 2017 American Health Information Management Association Example in Health Informatics An experimental pretest-posttest control group design was used to assess a smartphone mobile application that provided personalized, real-time sun protection advice to adults aged 18 and older who owned an Android smartphone (Buller et al. 2015) The mobile app, called Solar Cell, provided sun protection advice The authors hypothesized that providing personalized information to adults through a mobile app when they are in the sun may help reduce sun exposure Participants were randomly assigned to either the intervention or control group. Those in the intervention group completed a pre-test survey, used the mobile app and received information when in the sun and then completed a post-test survey. Those in the control group received no intervention Results of the study indicate that the Solar Cell app provided some assistance in sun protection but that it was not as strong as the researchers anticipated However, it was found that the Solar Cell app may be beneficial to those with high risk skin types or those who spend a great deal of time outdoors to make appropriate prevention decisions that reduce their exposure to the sun © 2017 American Health Information Management Association Solomon Four-Group Method Two experimental groups which both receive the intervention One group receives a pretest and posttest while the other experimental group receives a posttest only Two control groups are also used in this design One control group receives a pre and posttest while the other group receives the posttest only All participants are randomly assigned to the groups This method controls for pretest exposure but also requires more time, effort, and cost due to the additional groups © 2017 American Health Information Management Association Example of Solomon Four Group Adaptation of the Solomon four-group design was used by researchers evaluating the effectiveness of a multi-media tutorial in the preparation of dental students to recognize and respond to domestic violence (Danley et al. 2004) First experimental group of dental students was randomly assigned to take the pretest, the tutorial (intervention), and then a posttest The second experimental group first took the tutorial and then the posttest The third group (control group) took the pretest and then the posttest © 2017 American Health Information Management Association Posttest-Only Control Group Method Participants are randomly assigned to an experimental group or a control group and posttests are the only means of observation No pretests are used This is done to reduce the effect of familiarity with exposure to a pretest Not using a pretest eliminates the ability to assess an improvement in scores from before the intervention to after the intervention © 2017 American Health Information Management Association Example of Posttest-Only Control Group Method The experimental posttest only control group method was used by researchers assessing the effect of community nursing support on clients with schizophrenia (Beebe 2001) 24 participants randomly assigned to control group (routine follow-up care and informational telephone contact at 6 and 12 weeks) Experimental group(weekly telephone intervention plus routine follow-up care for 3 months) All were followed for 3 months after hospital discharge to determine the length of survival as well as frequency and length of stay for re-hospitalizations © 2017 American Health Information Management Association Quasi-Experimental Studies One-Shot Case Study This study is a simple design in which an intervention is provided to one group which is followed forward in time after intervention to assess the outcome (posttest) No randomization, no control group, and no pretest is included No baseline measurement to provide a comparison to the intervention outcome © 2017 American Health Information Management Association Example of One-Shot Case Study Researchers conducted a quasi-experimental one shot case study to determine if an automated two-way messaging system will help HIV-positive patients comply with complex medication treatments (Dunbar et al. 2003) 19 HIV-positive patients enrolled and received two-way pagers that included reminders to take all medication doses and follow any dietary requirements No control group Outcome measures consisted of the number of times participants reported missing one or more medication doses, medication side effects, and participant’s satisfaction level in using the messaging system © 2017 American Health Information Management Association One-Group Pretest-Posttest Method   Similar to one-shot case study except that the pretest is used before the intervention No control group and no randomization Used when it is unethical or inappropriate to withhold the intervention from a group of participants © 2017 American Health Information Management Association Example of One-Group Pretest-Posttest Method Researchers assessed the timeliness and access to healthcare services using telemedicine in individuals aged 18 and younger in state correctional facilities (Fox et al. 2007) Data were collected one year before implementation of the telemedicine program and two years after implementation The telemedicine intervention consisted primarily of remote delivery of behavioral health care services Timeliness of care and use of healthcare services before and after telemedicine implementation was examined The data was collected primarily from medical records and other claims and information assessment logs © 2017 American Health Information Management Association Static Group Comparison   Two groups are examined One with the intervention One without the intervention Posttest is given to assess the result of the intervention There are no pretests and no randomization, but a control group is used   © 2017 American Health Information Management Association Example Static Group Comparison Researchers assessed the use of alcohol in patients after a traumatic brain injury (TBI) based on patients’ and relatives’ reports (Sander et al. 1997) This study examined the validity of patients’ reports by comparing them to relatives’ descriptions of post-injury alcohol use In this design, researchers use the brain injury as the intervention and then assess via a post-injury questionnaire whether drinking habits as perceived by the patient with the TBI and the close relative are similar or different © 2017 American Health Information Management Association Internal and External Validity Internal validity demonstrates that the dependent variable (outcome measure) is only caused by the independent variable (intervention) rather than other confounding variables External validity is concerned with being able to generalize the results to other populations (Campbell and Stanley 1963). © 2017 American Health Information Management Association Factors Affecting Internal Validity History History or the events happening in the course of the experiment that could impact the results. Researcher collects level of functioning data on hip replacement patients before and after the use of a new physical therapy device. During the time that this device is being used, the developer becomes ill and unable to fully train all physical therapists in its proper use. Therefore, the study may be affected by inadequate time in training rather than the device itself. © 2017 American Health Information Management Association Factors Affecting Internal Validity Maturation Maturation and refers to the natural changes of research subjects over time due to the length of time that they are in the study. For example, older individuals may become very fatigued after completing a training session on using a computer to manage their finances. Their fatigue could then affect their responses on the posttest. © 2017 American Health Information Management Association Factors Affecting Internal Validity Testing Testing is the effect created once exposed to questions that may be on the posttest Example: Participants of a study that is assessing whether a course module on the use of privacy and security within the electronic health record (EHR) improves their knowledge content of this subject, use a pretest and posttest to assess whether there is improvement due to the course module. Since the students are already exposed to the pretest and are able to think of some of the test questions, they may change their answers on the posttest and do better by learning from the pretest. Therefore, the use of the pretest is what may be causing the improvement in test scores more so than the course module on privacy and security of the EHR © 2017 American Health Information Management Association Factors Affecting Internal Validity Instrumentation Instrumentation—Changes in instruments, interviewers, or observers may all cause changes in the results. Example: Interviewers may probe for answers more from one individual they are interviewing more so than others, if training is not performed consistently across all interviewers. © 2017 American Health Information Management Association Factors Affecting Internal Validity Statistical Regression Statistical regression (regression toward the mean)—When extreme scores of measurement tend to move toward the mean because they have extreme scores, not because of the intervention under study. Example: Coders who performed poorly on the ICD-10-CM coding exam are selected to receive training. The mean of their posttest scores will be higher than their pretest scores because of statistical regression not necessarily because of the ICD-10-CM training session. © 2017 American Health Information Management Association Factors Affecting Internal Validity Selection Selection—When there are systematic differences in the selection and composition of subjects in the experimental and control groups based on knowledge or ability. Example: One group of subjects who have viewed an instructional video on how to give themselves insulin injections is compared to another group which has not watched this video. No randomization is used. © 2017 American Health Information Management Association Factors Affecting Internal Validity Attrition Attrition—The withdrawal of subjects from the study. Those individuals who leave a study can be very different than those who remain in the study and the characteristics of these individuals can affect the results. Example: A study which focuses on trying to reduce the number of incomplete medical records due to incomplete nursing documentation have 15 nurses leave the experimental group and 2 nurses leave the control group. The 15 nurses who leave the group may be very different than those who remain in the experimental group. Also, the difference in the numbers of nurses who leave each group may be a problem. © 2017 American Health Information Management Association Factors Affecting Internal Validity Interaction An interaction of factors or a combination of the factors discussed previously may lead to bias in the final results The researcher needs to be aware of the effect of a combination of some of the factors discussed above and their impact on internal validity (Shadish and Cook, 1998, Key, 1997, Shi, 1997). © 2017 American Health Information Management Association Factors that Affect External Validity Testing Selection bias Participants are chosen who are frequently under medical care Volunteers Participants who receive compensation All may be different than the general population © 2017 American Health Information Management Association Control for Internal and External Validity Randomization—Most powerful to control for selection, regression to the mean, interaction of factors, improves external validity because subjects are not pre-selected but uses random assignment Use of control or comparison groups—help control for effects of history, maturation, instrumentation, interaction of factors (Key, 1997, Shi, 1997) © 2017 American Health Information Management Association Poor Experimental Procedures Control group exposed to part of the intervention Multiple treatment interference Length of time of treatment intervention Loss of participants © 2017 American Health Information Management Association Summary Experimental study designs are one of the most powerful designs to use when trying to prove cause and effect. Quasi-experimental study designs are also very effective but tend to have many more problems with external validity since most do not include randomization of subjects Researchers in health informatics choose to use the quasi-experimental design for many reasons such as ethical considerations, the difficulty in randomization of subjects and small sample size (Harris et al. 2006). Several examples of experimental and quasi-experimental studies and the methodology used in the health informatics and healthcare setting demonstrate that this study design is a viable option for health informatics research. © 2017 American Health Information Management Association