Fluent Essays

you need to write two discussion question:1.Choose one presentation from these two attachment and write a brief summary (in your own words) about this paper. 2.Lesson 1 Presentation (slide 15) - You are the advisor to the minister – what do you tell her to do? there is no minimum words for these two discussion assignment ,and this is the discussion question, you do not need to write so complicate , but you should write out the main point for the question. and i recommand you write 350-400 words.(200 words for each question) lesson_1.pptx lesson_2.pptx Unformatted Attachment Preview INTRODUCTION TO IMPACT EVALUATION WHY EVALUATE?  The purpose of evaluation is not always clear, particularly for those who have watched surveys conducted, data entered, and then the ensuing reports filed away only to collect dust.  This is most common when evaluations are imposed by others.  If, on the other hand, those responsible for the day-to-day operations of a program have critical questions, evaluations can help find answers. WHY EVALUATE?  As an example, the NGO responsible for distributing chlorine pills may speak with their local field staff and hear stories of households diligently using the pills, and occasionally see improvements in their health. But each time it rains heavily, the clinics fill up with people suffering from diarrheal diseases.  The NGO might wonder, “if people are using chlorine to treat their water, why are they getting sick when it rains? Even if the water is more contaminated, the chlorine should kill all the bacteria.”  The NGO may wonder whether the chlorine pills are indeed effective at killing bacteria.  Are people using it in the right proportion?  Maybe our field staff is not telling us the truth.  Perhaps the intended beneficiaries are not using the pills.  Perhaps they aren’t even receiving them.  And then when confronted with this fact, the field staff claims that during the rains it is difficult to reach households and distribute pills.  Households, on the other hand, will reply that they most diligently use pills during the rains, and that the pills have helped them substantially. WHY EVALUATE?  Speaking to individuals at different levels of the organization as well as to stakeholders can uncover many stories of what is going on. These stories can be the basis for theories. But plausible explanations are not the same thing as answers.  Evaluations involve developing hypotheses of what’s going on, and then testing those hypotheses. WHAT IS EVALUATION? WHAT IS EVALUATION?  The word “evaluation” can be interpreted quite broadly.  It means different things to different people and organizations.  Engineers, for example, might evaluate or test the quality of a product design, the durability of a material, efficiency of a production process, or the safety of a bridge.  Critics evaluate or review the quality of a restaurant, movie or book.  A child psychologist may evaluate or assess the decision-making process of toddlers.  Put simply, a program evaluation is meant to answer the question, “how is our program or policy doing?” This can have different implications depending on who is asking the question, and to whom they are talking. WHAT IS EVALUATION?  For example, if a donor asks the NGO director “how is our program doing?” she may imply, “have you been wasting our money?” This can feel interrogatory.  Alternatively, if a politician asks her constituents, “how is our program doing?” she could imply, “is our program meeting your needs? How can we make it better for you?”  Program evaluation, therefore, can be associated with positive or negative sentiments, depending on whether it is motivated by a demand for accountability versus a desire to learn.  At a very basic level, randomized evaluation can answer the question: was the program effective? But if thoughtfully designed and implemented, it can also answer the questions, how effective was it? Were there unintended side-effects? Who benefited most? Who was harmed? Why did it work or not work? What lessons can be applied to other contexts, or if the program was scaled up? How cost-effective was the program? How does it compare to other programs designed to accomplish similar goals? NEEDS ASSESSMENT  Programs and policies are introduced to address a specific need.  For example, we may observe that the incidence of diarrhea in a community is particularly high. This might be due to contaminated food or water, poor hygiene, or any number of plausible explanations.  A needs assessment can help us identify the source of the problem and those most harmed.  Needs assessment is a systematic approach to identifying the nature and scope of a social problem, defining the target population to be served, and determining the service needed to meet the problem.  A needs assessment is essential because programs will be ineffective if the services are not properly designed to meet the need or if the need does not actually exist.  So, for example, if the source of pollution contaminating drinking water is agricultural, investment in sanitation infrastructure such as toilets and sewage systems may not solve the problem.  Needs assessments may be conducted using publicly available social indicators, surveys and censuses, interviews, etc. PROGRAM THEORY ASSESSMENT  Social programs or policies are introduced to meet a social need.  Meeting that need usually requires more thought than finding and pressing a single magic button, or taking a pill.  For policymakers, it requires identifying the reasons that are causing undesirable outcomes (see Needs Assessment), and choosing a strategy from a large set of options to try to bring about different outcomes.  For example, if people are drinking unclean water, one program might be designed to prevent water from becoming contaminated—by improving sanitation infrastructure—while another may be designed to treat contaminated water using chlorine.  One proposed intervention might target those responsible for the pollution. Another might target those who drink the water.  One strategy may rest on the assumption that people don’t know their water is dirty, another, that they are aware but have no access to chlorine, and even another, that despite awareness and access, people choose not to chlorinate their water for other reasons (e.g. misinformation, taste, cost, etc.). THEORY OF CHANGE  A Program Theory Assessment models the theory behind the program, presenting a plausible and feasible plan for improving the target social condition.  If the goals and assumptions are unreasonable, then there is little prospect that the program will be effective.  Program theory assessment involves first articulating the program theory and then assessing how well the theory meets the targeted needs of the population. IMPACT EVALUATION IMPACT EVALUATION  Programs and policies are designed to achieve a certain goal (or set of goals).  For example, a chlorine distribution program may be implemented specifically to combat high-incidence of waterborne illness in a region. We may want to know whether this program is succeeding in its goal.  Impact evaluations gauge the success of a program—where success can be broadly or narrowly defined.  They help us weed out less effective interventions from successful ones and also help us improve existing programs.  The primary purpose of impact evaluation is to determine whether a program has an impact (on a few key outcomes), and more specifically, to quantify how large that impact is. What is impact?  In the chlorine example, impact is how much healthier people are because of the program than they would have been without the program. Or more specifically, how much lower the incidence of diarrhea is than it would have been otherwise. THE FUNDAMENTAL CHALLENGE – COUNTERFACTUAL?  Getting this number correct is more difficult than it sounds.  It is possible to measure the incidence of diarrhea in a population that received the program. But “how they would have been otherwise” is impossible to measure directly—just as it is impossible to measure the United States economy today had the Nazis won World War II, or to determine today’s most deadly disease if penicillin was not discovered in Alexander Fleming’s dirty laboratory in 1928 in London.  It is possible that Germany would have become the dominant economy in the world, or alternatively, the Nazis may have fallen just a few years later.  It is possible that minor wounds would still be one of the largest killers, or alternatively, some close relative of penicillin could have been discovered in another laboratory in a different part of the world.  In our chlorine example, it is possible that without chlorine, people would have remained just as sick as they were before. Or it is possible that they would have started boiling their water instead, and the only thing chlorine did was substitute one technology for another—suggesting that people are not really any healthier because of the chlorine.  To address this problem, we need a “counterfactual”. THE FUNDAMENTAL CHALLENGE – COUNTERFACTUAL?  Impact evaluations estimate program effectiveness usually by comparing outcomes of those (individuals, communities, schools, etc.) who participated in the program against those who did not participate.  The key challenge in impact evaluation is finding a group of people who did not participate, but closely resemble the participants had those participants not received the program.  Measuring outcomes in this comparison group is as close as we can get to measuring “how participants would have been otherwise.” EXAMPLE - IMPROVING EDUCATION – FLIP CHARTS  An officer in the Kenyan education ministry comes to the minister and suggests that children learn better with visual aids and pictures than with text books – also much more cost effective because only need one per classroom.  He has found a producer who can make large flip charts with pre-printed pictures that are mounted on an easel and can be used in a ‘show and tell’ fashion.  He strongly recommends that the Ministry of Education procure flip charts and distribute them to all government-run schools as a cheap way of improving quality of education.  You are the advisor to the minister – what do you tell her to do? IMPACT EVALUATION - FLIP CHARTS  The relationship you want to estimate can be formally written as:  Outcome (for e.g. test score) = fn (School quality, Child characteristics, Household characteristics, Peers)  “School Quality” is a composite measure that can be affected in several ways and providing additional inputs to schools is one way of doing so.  The flip-charts are an example of this:  Test Score = a + b*(Flip Chart) + e  Outcome variable = Test Score  a = constant  b = coefficient of interest, in this case will tell us the impact of the school input, flip chart  What is the problem in this kind of estimation? OMITTED VARIABLE BIAS  The previous result is subject to several omitted variable problems.  For e.g., test score can be high because:  Students are of high ability  Teachers are of high ability  Maybe, community, political leaders who care about education, therefore, this could both improve infrastructure and reduce teacher absence  Therefore, correlation does not mean causation! WHAT IS RANDOMIZATION?  In its most simple sense, randomization is what happens when a coin is flipped, a die is cast, or a name on a piece of paper is drawn blindly from a basket, and the outcome of that flip, cast, or draw determines what happens next.  Perhaps, the outcome of the coin flip determines who has to do some chore; the role of the die determines who gets a pile of money; the draw of a name determines who gets to participate in some activity, or a survey.  When these tools (the coin, the die, the lottery) are used to make decisions, the outcome can said to be left to chance, or, randomized.  Why do people let chance determine their fate? Sometimes, because they perceive it as fair. Other times, because uncertainty adds an element of excitement.  Statisticians use randomization because, when enough people are randomly chosen to participate in a survey, conveniently, the attributes of those chosen individuals are representative of the entire group from which they were chosen.  In other words, what is discovered about them is probably true about the larger group. Using a lottery to get a representative sample is known as random sampling or random selection. WHY IS RANDOM ASSIGNMENT IMPORTANT  When two groups are randomly selected from the same population, they both represent the larger group.  They are not only statistically equivalent to the larger group; they are also statistically equivalent to each other.  The same logic carries forward if more than two groups are randomly selected.  When two or more groups are selected in this way, we can say that individuals have been randomly assigned to groups. This is called random assignment. (Random assignment is also the appropriate term when all individuals from the larger group divided randomly into different groups. As before, all groups represent the larger group and are statistically equivalent to each other.)  Random assignment is the key element of randomized evaluation. WHAT IS RANDOMIZATION?  What happens next in a simple randomized evaluation (with two groups) is that one group receives the program that is being evaluated and the other does not.  If we were to evaluate a water purification program using this method, we would randomly assign individuals to two groups.  At the beginning, the two groups would be statistically equivalent (and are expected to have equivalent trajectories going forward). But then we introduce something that makes them different.  One group would receive the water purification program and the other would not.  Then, after some time, we could measure the relative health of individuals in the two groups.  Because they were statistically equivalent at the beginning, any differences seen later on can be attributed to one having been given the water purification program, and the other not. OTHER NAMES  Randomized Evaluations go by many names:  Randomized Controlled Trials  Social Experiments  Random Assignment Studies  Randomized Field Trials  Randomized Controlled Experiments BENEFITS OF RANDOMIZATION  Randomized evaluations are often deemed the gold standard of impact evaluation, because they consistently produce the most accurate results.  Like all impact evaluations, the primary purpose of randomized evaluations is to determine whether a program has an impact, and more specifically, to quantify how large that impact is.  Impact evaluations measure program effectiveness typically by comparing outcomes of those (individuals, communities, schools, etc.) who participated in the program against those who did not participate.  Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, any difference that subsequently arises between them can be attributed to the treatment rather than to other factors. BENEFITS OF RANDOMIZATION  What distinguishes randomized evaluations from other non-randomized impact evaluations is that participation (and non-participation) is determined randomly—before the program begins.  This random assignment is the method used in clinical trials to determine who gets a drug versus who gets a placebo when testing the effectiveness (and side-effects) of new drugs.  As with clinical trials, those in the impact evaluation who were randomly assigned to the “treatment group” are eligible to receive the treatment (i.e. the program). And they are compared to those who were randomly assigned to the “control group” –those who do not receive the program.  Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, any difference that subsequently arises between them can be attributed to the treatment rather than to other factors.  Relative to results from non-randomized evaluations, results from randomized evaluations are:  Less subject to methodological debates  Easier to convey  More likely to be convincing to program funders and/or policymakers  Beyond quantifying the intended outcomes caused by a program, randomized evaluations can also quantify the occurrence of unintended side-effects (good or bad).  And like other methods of impact evaluation, randomized evaluations can also shed light on why the program has or fails to have the desired impact. CONSTRUCTING A “COMPARISON” GROUP  What is impact? In the chlorine example, impact is how much healthier people are because of the program than they would have been without the program. Or more specifically, it is how much lower the incidence of diarrhea is than it would have been otherwise.  Getting this number correct is more difficult than it sounds. It is possible to measure the incidence of diarrhea in a population that received the program.  But “how they would have been otherwise” (termed, the counterfactual) is impossible to measure directly, it can only be inferred.  Getting the right “counterfactual” or “control group” is the key to any impact evaluation design. CONSTRUCTING A “COMPARISON” GROUP  Impact evaluations estimate program effectiveness usually by comparing outcomes of those (individuals, communities, schools, etc.) who participated in the program against those who did not participate.  The key challenge in impact evaluation is finding a group of people who did not participate, but closely resemble the participants, and in particular, the participants if they had not received the program.  Measuring outcomes in this comparison group is as close as we can get to measuring “how participants would have been otherwise.”  Therefore, our estimate of impact is only as good as our comparison group is equivalent. CONSTRUCTING A “COMPARISON” GROUP  There are many methods of creating a comparison group. Some methods do a better job than others.  All else equal, randomized evaluations do the best job. They generate a statistically identical comparison group, and therefore produce the most accurate (unbiased) results. Or stated more strongly: other methods often produce misleading results—results that would lead policymakers to make exactly the opposite decision relative to where the truth would have directed them.  These other methods don’t always give us the wrong answer, but they rely on more assumptions.  When the assumptions hold, the answer is unbiased. But it is usually impossible, and always difficult, to ensure that the assumptions are true.  In fact, it is likely that most debates about the validity of an evaluation are fueled by disagreements over whether these assumptions are reasonable. A TABLE COMPARING COMMON METHODS OF EVALUATION A TABLE COMPARING COMMON METHODS OF EVALUATION WHEN TO CONDUCT A RANDOMIZED EVALUATION?  The value added by rigorously evaluating a program or policy changes depending on when in the program or policy life cycle the evaluation is conducted.  The evaluation should not come too soon: when the program is still taking shape, and kinks are being ironed out. And the evaluation should not come too late: after money has been allocated, and the program, rolled out, so that there is no longer space for a control group.  An ideal time is during the pilot phase of a program or before scaling up.  During these phases there are often important questions that an evaluator would like to answer: How effective is the program? Is it effective among different populations? Are certain aspects are working better than others, and can “the others” be improved? Is it effective when it reaches a larger population?  During the pilot phase, the effects of a program on a particular population are unknown. The program itself may be new or it ... Purchase answer to see full attachment