Fluent Essays

Program Implementation & Evaluation Ok, public programs are up and running and the issue is—How well are they doing? This is a very broad question and an open-ended question but one which requires various approaches and methods of evaluation. From the readings that you will go through, some readings address what we might call “methodology” where you are trying to lay out a way to actual evaluate a program. Evaluation comes in many shapes and sizes. Some of the readings look at different public programs and give you an idea, or an insight, into how each is evaluated. Feedback is important, we assume that nothing works perfectly (if you took the “Public administration: Principles, Applications, & Ethics” course, you did an assignment on the “Law of Unintended Consequences”). We need feedback to look at how to “tinker” with a program or make major overhauls. Evaluation helps us determine what to do to make those either small or major changes. In this assignment address evaluation. Do discuss the different methods or approaches. can then be a combination of you discussing these different methods and approaches and how evaluation has been addressed regarding different public programs. 1) minimum of five typed pages. 2) I do want you to discuss the readings. It might be a good idea if you think of the readings as having these two categories (one on methods and approaches, the other on case studies where you see different programs being evaluated). The readings do not easily fall into either category, but from going through them, you can begin to figure out how to break what you read into the two categories. 78 E D U C A T I O N N E X T / F A L L 2 0 1 2 educationnext.org Im p a c t o n m a th t e s t s c o re s ( st a n d a rd d e v ia ti o n s) TES evaluation year Years after evaluation Years before evaluation 0.25 0.2 0.15 0.1 0.05 0 -0.05 -0.1 Evaluation year 6 65 54 43 31 102 2 IL L U S T R A T E D F IG U R E / B R U C E S A N D E R S D E S IG N / G E T T Y I M A G E S Different from scores in year before evalua- tion at 90\% confidence level Not statistically different from scores in year before evaluation Improvement through Evaluation (Figure 1) Veteran teachers in Cincinnati became more effective in raising student math test scores the year they participated in the district’s evaluation system (TES), and even more effective in the years after evaluation. Note: Chart shows teachers’ estimated impact on student math test scores in the years before, during, and after their participation in the TES evaluation system. Estimates with solid markers are statistically significant at the 90\% confidence level. These estimates do not control for teacher experience, while the main results discussed in the text do include experience controls. SOURCE: Authors’ calculations educationnext.org F A L L 2 0 1 2 / E D U C A T I O N N E X T 79 r e s e a r c h The modernization of teacher evaluation systems, an increasingly common component of school reform efforts, promises to reveal new, systematic information about the performance of individual classroom teachers. Yet while states and districts race to design new systems, most discussion of how the information might be used has focused on traditional human resource–management tasks, namely, hiring, firing, and compensa- tion. By contrast, very little is known about how the availability of new information, or the experience of being evaluated, might change teacher effort and effectiveness. Evidence of systematic growth in the effectiveness of midcareer teachers by ERIC S. TAYLOR and JOHN H. TYLER Can Teacher Evaluation Improve Teaching? In the research reported here, we study one approach to teacher evaluation: practice-based assessment that relies on multiple, highly struc- tured classroom observations conducted by experi- enced peer teachers and administrators. While this approach contrasts starkly with status quo “prin- cipal walk-through” styles of class observation, its use is on the rise in new and proposed evaluation systems in which rigorous classroom observation is often combined with other measures, such as teacher value-added based on student test scores. Proponents of evaluation systems that include high-quality classroom observations point to their potential value for improving instruction (see “Capturing the Dimensions of Effective Teach- ing,” features, page 34). Individualized, specific information about performance is especially scarce in the teaching profession, suggesting that a lack of information on how to improve could be a sub- stantial barrier to individual improvement among teachers. Well-designed evaluation might fill that knowledge gap in several ways. First, teachers could gain information through the formal scor- ing and feedback routines of an evaluation pro- gram. Second, evaluation could encourage teach- ers to be generally more self-reflective, regardless of the evaluative criteria. Third, the evaluation process could create more opportunities for con- versations with other teachers and administrators about effective practices. In short, there are good reasons to expect that well-designed teacher-evaluation programs could have a direct and lasting effect on individual teacher performance. To our knowledge, however, 80 E D U C A T I O N N E X T / F A L L 2 0 1 2 educationnext.org ours is the first study to test this hypothesis directly. We study a sample of midcareer elementary and middle school teachers in the Cincinnati Public Schools, all of whom were evaluated in a yearlong program, based largely on classroom observa- tion, sometime between the 2003–04 and 2009–10 school years. The specific school year of each teacher’s evaluation was determined years earlier by a district planning process. This policy-based assignment of when evaluation occurred permits a quasi-experimental analysis. We compare the achievement of individual teachers students before, during, and after the teachers evaluation year. We find that teachers are more effective at raising student achievement during the school year when they are being evaluated than they were previously, and even more effec- tive in the years after evaluation. A student instructed by a teacher after that teacher has been through the Cincinnati evaluation will score about 11 percent of a standard deviation (4.5 percentile points for a median student) higher in math than a similar student taught by the same teacher before the teacher was evaluated. Our data do not allow us to identify the exact mecha- nisms driving these improvements. Nevertheless, the results contrast sharply with the view that the effectiveness of indi- vidual teachers is essentially fixed after the first few years on the job. Indeed, we find that postevaluation improvements in performance were largest for teachers whose performance was weakest prior to evaluation, suggesting that rigorous teacher evaluation may offer a new way to think about teacher professional development. Evaluation in Cincinnati The data for our analysis come from the Cincinnati Public Schools. In the 2000–01 school year, Cincinnati launched the Teacher Evaluation System (TES) in which teachers’ perfor- mance in and out of the classroom is assessed through class- room observations and a review of work products. During the yearlong TES process, teachers are typically observed in the classroom and scored four times: three times by an assigned peer evaluator—a high-performing, experienced teacher who previously taught in a different school in the district—and once by the principal or another school administrator. Teach- ers are informed of the week during which the first observation will occur, with all other observations unannounced. Owing mostly to cost, tenured teachers are typically evaluated only once every five years. The evaluation measures dozens of specific skills and practices covering classroom management, instruction, con- tent knowledge, and planning, among other topics. Evalu- ators use a scoring rubric based on Charlotte Danielson’s Enhancing Professional Practice: A Framework for Teach- ing, which describes performance of each skill and practice at four levels: “Distinguished,” “Proficient,” “Basic,” and “Unsatisfactory.” (See Table 1 for a sample standard.) Both the peer evaluators and administrators complete an intensive TES training course and must accurately score videotaped teaching examples. After each classroom obser- vation, peer evaluators and administrators provide writ- ten feedback to the teacher and meet with the teacher at least once to discuss the results. At the end of the evalua- tion school year, a final summative score in each of four domains of practice is calculated and presented to the evalu- ated teacher. Only these final scores carry explicit conse- quences. For beginning teachers (those evaluated in their first and fourth years), a poor evaluation could result in nonrenewal of their contract, while a successful evaluation is required before receiving tenure. For tenured teachers, evaluation scores determine eligibility for some promotions or additional tenure protection, or, in the case of very low scores, placement in a peer assistance program with a small risk of termination. Despite the training and detailed rubric provided to evalu- ators, the TES program experiences some of the leniency bias typical of other teacher-evaluation programs. More than 90 percent of teachers receive final overall TES scores in the highest two categories. Leniency is much less frequent in the individual rubric items and individual observations. We hypothesize that this microlevel evaluation feedback is We find that teachers are more effective at raising student achievement during the school year when they are being evaluated than they were previously, and even more effective in the years after evaluation. educationnext.org F A L L 2 0 1 2 / E D U C A T I O N N E X T 81 r e s e a r c h TEACHER EVALUATION SYSTEM TAYLOR & TYLER more important to lasting performance improvements than the final, overall TES scores. Previous research has found that the scores produced by TES predict student achievement gains (see “Evaluating Teacher Effectiveness,” research, Summer 2011). Student math achieve- ment was 0.09 standard deviations higher for teachers whose overall evaluation score was 1 standard deviation higher (the estimate for reading was 0.08). This relationship suggests that Cincinnati’s evaluation program provides feedback on teaching skills that are associated with larger gains in student achievement. As mentioned above, teachers only undergo comprehensive evaluation periodically. All teachers newly hired by the district, regardless of experience, are evaluated during their first year working in Cincinnati schools. Teachers are also evaluated just prior to receiving tenure, typically their fourth year after being hired, and every fifth year after achieving tenure. Teachers hired before the TES program began in 2000–01 were not initially evaluated until some years into the life of the program. Our analysis only includes these pre-TES hires: specifically, teachers hired by the district in the school years from 1993–94 through 1999–2000. We further focus, given available data, on those who were teaching 4th through 8th grade in the years 2003–04 through 2009–10. We limit our analysis to this sample of midcareer teachers for three rea- sons. First, for teachers hired before the new TES program began in 2000–01, the timing of their first TES review was determined largely by a “phase-in” schedule devised during the program’s planning stages. This schedule set the year How Teachers Are Evaluated in Cincinnati: A Sample (Table 1) SOURCE: Cincinnati Public Schools Teacher Evaluation System 2005. The complete rubric is available at http://www.cps-k12.org/employment/tchreval/stndsrubrics.pdf. Teacher frequently asks questions that are inappropriate to objectives of the lesson. Teacher frequently does not ask follow-up questions. Teacher answers own questions. Teacher frequently does not provide appropriate wait time. Teacher structures and facilitates discourse at the evaluative, synthesis, and/or analysis levels between teacher and students and among students to explore and extend content knowledge. Distinguished Proficient Basic Unsatisfactory D is c o u rs e T h o u g h t- P ro v o k in g Q u e s ti o n s Teacher initiates and leads discourse at the evaluative, synthesis, and/or analysis levels to explore and extend the content knowledge. Teacher frames content- related discussion that is limited to a question and answer session. Teacher permits off-topic discussions, or does not elicit student responses. Teacher routinely asks thought -provoking questions at the evaluative, synthesis, and/ or analysis levels that focus on the objectives of the lesson. Teacher seeks clarification and elaboration through additional questions. Teacher provides appropriate wait time. Teacher asks thought- provoking questions at the evaluative, synthesis, and/or analysis levels that focus on the objectives of the lesson. Teacher seeks clarification through additional questions. Teacher provides appropriate wait time. Teacher asks questions that are relevant to the objectives of the lesson. Teacher asks follow-up questions. Teacher is inconsistent in providing appropriate wait time. Standard 3.4: The teacher engages students in discourse and uses thought-provoking questions aligned with the lesson objectives to explore and extend content knowledge. 82 E D U C A T I O N N E X T / F A L L 2 0 1 2 educationnext.org of first evaluation based on a teacher’s year of hire, thus reducing the potential for bias that would arise if the tim- ing of evaluation coincided with, for example, a favorable class assignment. Second, because the timing of evaluation was determined by year of hire, and not experience level, teachers in our sample were evaluated at different points in their careers. This allows us to measure the effect of evalu- ation on performance separate from any gains that come from increased experience. Third, the delay in first evalu- ation allows us to observe the achievement gains of these teachers’ students in classes the teachers taught before the TES assessment so that we can make before-and-after com- parisons of the same teacher. Additionally, our study focuses on math test scores in grades 4–8. For most other subjects and grades, student achievement measures are simply not available. Students are tested in reading, but empirical research frequently finds less teacher-driven variation in reading achievement than in math, and ultimately this is the case for the present analysis as well. While not the focus of our research, we briefly discuss read- ing results below. Data provided by the Cincinnati Public Schools identify the year(s) in which a teacher was evaluated by TES, the dates when each observation occurred, and the scores. We combine these TES data with additional administrative data provided by the district that allow us to match teachers to students and student test scores. As we would expect, the 105 teachers in our analysis sample are a highly experienced group: 66.5 percent have 10 to 19 years of experience, compared to 29.3 percent for the rest of the district. Teachers in our analysis are also more likely to have a graduate degree and be certified by the National Board for Professional Teaching Standards, two characteristics correlated with experience. Methodology Our objective is to measure the impact of practice-based per- formance evaluation on teacher effectiveness. Simply compar- ing the test scores of students whose teachers are evaluated in a given year to the scores of other teachers’ students would produce misleading results because, among other method- ological issues, less-experienced teachers are more likely to be evaluated than more-experienced teachers. Instead, we compare the achievement of a teacher’s stu- dents during the year that she is evaluated to the achieve- ment of the same teacher’s students in the years before and after the evaluation year. As a result, we effectively control for any characteristics of the teacher that do not change over time. In addition, we control for determinants of student achievement that may change over time, such as a teacher’s experience level, as well as for student characteristics, such as prior-year test scores, gender, racial/ethnic subgroup, special education classification, gifted classification, Eng- lish proficiency classification, and whether the student was retained in the same grade. Our approach will correctly measure the effect of evalua- tion on teacher effectiveness as long as the timing of a teach- er’s evaluation is unrelated to any student characteristics that we have not controlled for in the analysis but that affect achievement growth. This key condition would be violated, for example, if during an evaluation year or in the years after, teachers were systematically assigned students who were bet- ter (or worse) in ways we cannot determine and control for using the available data. It would also be violated if evaluation coincided with a change in individual teacher performance unrelated to evaluation per se. Below, we discuss evidence that our results are not affected by these kinds of issues. We also find no evidence that teachers are systematically assigned students with better (or worse) observable characteristics in their evaluation year compared to prior and subsequent years. Results We find suggestive evidence that the effectiveness of indi- vidual teachers improves during the school year when they are evaluated. Specifically, the average teacher’s students score 0.05 standard deviations higher on end-of-year math tests during the evaluation year than in previous years, although this result is not consistently statistically significant across our different specifications. The effects [of going through the TES evaluation] were largest for teachers who received more critical feedback and for those with the most room for improvement. educationnext.org F A L L 2 0 1 2 / E D U C A T I O N N E X T 83 r e s e a r c h TEACHER EVALUATION SYSTEM TAYLOR & TYLER These improvements persist and, in fact, increase in the years after evaluation (see Figure 1). We estimate that the aver- age teacher’s students score 0.11 standard deviations higher in years after the teacher has undergone an evaluation compared to how her students scored in the years before her evaluation. To get a sense of the magnitude of this impact, consider two students taught by the same teacher in different years who both begin the year at the 50th percentile of math achieve- ment. The student taught after the teacher went through the TES process would score about 4.5 percentile points higher at the end of the year than the student taught before the teacher went through the evaluation. We also find evidence that the effects of going through evaluation in the TES system are not the same for all teach- ers. The improvement in teacher performance from before to after evaluation is larger for teachers who received rela- tively low TES scores, teachers whose TES scores improved the most during the TES year, and especially for teachers who were relatively ineffective in raising student test scores prior to TES. The fact that the effects were largest for teach- ers who, presumably, received more critical feedback and for those with the most room for improvement strengthens our confidence in the causal interpretation of the overall results. Our findings remain similar when we make changes to our methodological choices, such as varying the way we control for teacher experience, not controlling for teacher experi- ence, and not controlling for student characteristics. We also examine whether our results could be biased by a preexist- ing upward trend in each teacher’s performance unrelated to experience or evaluation, and find no evidence of such a trend. Finally, we find no evidence that our results reflect teacher turnover from school to school or from grade to grade that causes them not to appear in our data in later years (for example, by moving to a nontested grade or leaving the Cin- cinnati Public Schools). In contrast to the results for math achievement, we do not find any evidence that being evaluated increases the impact that teachers have on their students’ reading achieve- ment. Many studies find less variation in teachers’ effect on reading achievement compared to teachers’ effect on math achievement, a pattern that is also evident in our data from Cincinnati. Some have hypothesized that the smaller differences in effectiveness among reading teachers could arise because students learn reading in many in- and out- of-school settings (e.g., reading with family at home) that are outside of a formal reading class. If teachers have less influence on reading achievement, then even if evaluation induces changes in teacher practices, those changes would have smaller effects on achievement growth. Discussion The results presented here—greater teacher performance as measured by student achievement gains in years following TES review—strongly suggest that teachers develop skills or otherwise change their behavior in a lasting manner as a result of undergoing subjective performance evaluation in the TES process. A potential explanation for these results is that teach- ers learn new information about their own performance dur- ing the evaluation and subsequently develop new skills. New information is potentially created by the formal scoring and feedback routines of TES, as well as increased opportunities for self-reflection and for conversations regarding effective teaching practice in the TES environment. Moreover, two features of this study—the analysis sample of experienced teachers and Cincinnati’s use of peer eval- uators—may increase the saliency of these hypothesized mechanisms. First, the teachers we study experienced their first rigorous evaluation after 8 to 17 years on the job. Thus they may have been particularly receptive to and in need of Greater teacher performance as measured by student achievement gains strongly suggest that teachers develop skills or otherwise change their behavior in a lasting manner as a result of undergoing Greater teacher performance as measured by student achievement gains strongly suggest that teachers develop skills or otherwise change their behavior in a lasting manner as a result of undergoing performance evaluation. information on their performance. If, by contrast, teachers were evaluated every school year (as they are in a new but similar program in Washington, D.C.), the effect result- ing from each subsequent year’s evaluation might well be smaller. Second, Cincinnati’s use of peer evaluators may result in teachers being more receptive to feedback from their subjective evaluation than they would be were the feedback to come solely from their supervising principals. Teachers also appear to generate higher test-score gains during the year they are being evaluated, though these esti- mates, while consistently positive, are smaller. These improve- ments during the evaluation could represent the beginning of the changes seen in years following the review, or they could be the result of simple incentives to try harder during the year of evaluation, or some combination of the two. A remaining question is whether the effects we find are small or large. A natural comparison would be to the esti- mated effects of different teacher professional-development programs (in-service training often delivered in formal class- room settings). Unfortunately, despite the substantial budgets allocated to such programs, there is little rigorous evidence on their effects. There are, however, other results from research on teacher effectiveness that can be used for comparison. First, the largest gains in teacher effectiveness appear to occur as teachers gain on-the-job experience in the first three to five years. Jonah Rockoff reports gains of about 0.10 student standard deviations over the first two years of teaching when effectiveness is measured by improvements in math computa- tion skills; when using an alternative student math test mea- suring conceptual understanding, the gains are about half as large. Second, Kirabo Jackson and Elias Bruegmann find that having more effective teacher peers improves a teacher’s own performance; a 1-standard-deviation increase in teacher-peer quality is associated with a 0.04-standard-deviation increase in student math achievement. Compared to these two find- ings, the sustained effect of TES assessment is large. But are these benefits worth the costs? The direct expendi- tures for the TES program are substantial, which is not sur- prising given its atypically intensive approach. From 2004–05 to 2009–10, the Cincinnati district budget directly allocated between $1.8 and $2.1 million per year to the TES program, or about $7,500 per teacher evaluated. More than 90 percent of this cost is associated with evaluator salaries. A second, potentially larger “cost” of the program is the departure from the classroom of the experienced and presumably highly effective teachers selected to be peer evaluators. The students who would otherwise have been taught by the peer evaluators will likely be taught by less- effective, less-experienced teachers; in those classrooms, the students’ achievement gains will be smaller on aver- age. (The peer evaluator may in practice be replaced by an equally effective or more effective teacher, but that teacher must herself be replaced in the classroom she left.) While this second cost is more difficult to calculate, it is certainly offset by the larger gains made by students in the evaluated teachers’ classrooms. Those students are scoring, on average, 10 percent of a standard deviation better than they would have otherwise, and since each peer evaluator evalu- ates 10 to 15 teachers each year, those gains are occurring in multiple teachers’ classrooms for a number of years. The results of our study provide evidence that subjective evaluation can improve employee performance, even after the evaluation period ends. This is particularly encouraging for the education sector. In recent years, the consensus among policymakers and researchers has been that after the first few years on the job, teacher performance, at least as measured by student test-score growth, cannot be improved. In contrast, we demonstrate that, at least in this setting, experienced teachers provided with unusually detailed information on their perfor- mance improved substantially. American public schools have been under new pressure from regulators and constituents to improve teacher per- formance. To date, the discussion has focused primarily on evaluation systems as sorting mechanisms, a means to identify the lowest-performing teachers for selective ter- mination. Our work suggests optimism that, while costly, well-structured evaluation systems can not only serve this sorting purpose but can also enhance education through improvements in teacher effectiveness. In other words, if done well, performance evaluation can be an effective form of teacher professional development. Eric S. Taylor is a doctoral student at Stanford University. John H. Tyler is professor of education, economics, and public policy at Brown University. This article is based in part on a forthcoming study in the American Economic Review. 84 E D U C A T I O N N E X T / F A L L 2 0 1 2 educationnext.org The consensus has been that after the first few years on the job, teacher performance cannot be improved. In contrast, we demonstrate that experienced teachers provided with detailed information improved substantially. 1 Evaluating Welfare Reform in the United States Rebecca M. Blank University of Michigan and NBER May 2002 (revised) Contact information: Gerald R. Ford School of Public Policy 440 Lorch Hall 611 Tappan Street University of Michigan Ann Arbor, MI 48109-1220 734-763-2258 734-763-9181 FAX [email protected] This paper was commissioned by the Journal of Economic Literature. Thanks are due to Lucie Schmidt and to Elizabeth Scott for excellent research assistance, and to Jeffrey Grogger, Charles Michalopoulos, Robert Moffitt and an anonymous referee for comments and advice. 2 Abstract This paper reviews the economics literature on welfare reform over the 1990s. A brief summary of the policy changes over this period is followed by a discussion of the methodological techniques utilized to analyze the effects of these changes on outcomes. The paper them critically reviews the econometric and experimental literature on caseload changes, labor force changes, poverty and income changes, and family formation changes. A growing body of evidence suggests that the recent policy changes have influenced economic behavior and well-being in a variety of ways. One particular set of “new-style” welfare programs seems to show especially promising results, with significantly increased work and earnings and reduced poverty. 3 Over the 1990s the United States fundamentally changed the structure of its public assistance programs to low-income families. These policy changes have, in turn, generated a growing body of economic research that has evaluated their effects. This article reviews the major changes in U.S. welfare programs over the 1990s and critiques some of the key methodological approaches and results in areas where a substantial economic research literature has accumulated. I particularly focus on areas where the new research contributes to long-standing debates. It is worth noting that the U.S. policy changes have been much discussed in other countries and the evaluation literature from the U.S. may be increasingly relevant to policy debates elsewhere. For instance, in 1996, Canada gave provinces greater discretion over their social assistance programs, similar to changes the U.S. As we shall discuss below, Canada enacted a very interesting demonstration program in the 1990s (the Self Sufficiency Project), designed to move women on welfare into work. In 1999, Great Britain enacted the Working Families Tax Credit, a generous tax credit for low- income working families, similar to the U.S. Earned Income Tax Credit program. Some communities in Germany are imposing time limits on the receipt of public assistance (Feist and Schöb, 1998). In contrast to earlier decades, when the different design and lower generosity of U.S. social welfare programs led U.S. policies to be dismissed as irrelevant or aberrant by other westernized nations, during the 1990s many of these countries watched the U.S. welfare experiments with great interest.1 1 Not discussed here are social insurance programs such as Social Security or Unemployment Insurance, around which there has also been a great deal of trans-Atlantic conversation. 4 I. Federal Changes in U.S. Welfare Programs Over the 1990s The U.S. enacted major welfare reform legislation in August, 1996. The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) passed with a relatively high degree of bipartisan support. President Bill Clinton had, however, vetoed two earlier versions of this bill and it remained controversial. Several of his senior advisors resigned in protest when he signed PRWORA into law.2 The major provisions of PRWORA included: • Devolution of greater program authority to the states. PRWORA replaced the federal Aid to Families with Dependent Children Program (AFDC) – the primary cash assistance program for low-income families -- with the Temporary Assistance for Needy Families (TANF) block grant. This essentially removed almost all federal eligibility and payment rules, giving states much greater discretion in designing their own cash public assistance programs. This also eliminated a federal entitlement to cash assistance. States could choose which families they supported. • Changes in financing. TANF replaced a matching fund arrangement under AFDC, in which federal funding moved up or down with state funding. The TANF block grant was fixed and the contribution for each state was determined by the federal AFDC matching grant contribution in the years prior to PRWORA. States were required to maintain at least 75 percent of their previous state spending levels on AFDC in order to receive the full block grant.3 • Ongoing work requirements. By 2002, at least 50 percent of all recipient families and 90 percent of two-parent families were required to be working or in work preparation programs, although states were given great discretion to design and implement these programs. The law treated caseload reductions as similar to work, however. Thus, a state which reduced its caseload by 50 percent would meet its work requirement, regardless of how many current or former recipients were actually employed. • Incentives to reduce non-marital births. There was more rhetoric than program in the legislation in this area, but three of the four stated goals of PRWORA involved reducing non-marital births and encouraging marriage. States that reduced out-of- wedlock child bearing without raising abortion rates qualified for special bonuses. 2 For a detailed description of the events leading up to this legislation, see Weaver (2000). For further discussion about the provisions of PRWORA see Blank (1997b) or Blank and Ellwood (2002). Moffitt (1999b) discusses the factors behind PRWORA’s passage. Moffitt (forthcoming) provides a more detailed summary of the changes from AFDC to TANF. 3 Not included in this paper is any discussion of the public finance literature that investigates the potential impact of block grants on welfare funding. For a good overview of these issues, see Chernick (1998). 5 • Five year maximum time limit. PRWORA set a lifetime limit of 60 months on the receipt of TANF-funded aid. States could exempt up to 20 percent of their caseload from this limit, could set shorter time limits if they chose, or could continue funding assistance to families entirely out of state funds after 60 months. Although this paper will focus less on these issues, PRWORA also imposed additional limits on eligibility for Food Stamps and Supplemental Security Income (SSI, the cash assistance program to low-income aged and disabled individuals) among certain populations. Legal immigrants who arrived after August 1996 were largely denied access to TANF and to these other programs; the impact of this policy change will grow over time as an increasing share of U.S. immigrants will have arrived post-PRWORA. Finally, PRWORA made changes designed to encourage greater paternity establishment and more payment of child support by absent parents. While the 1996 legislation has received the most public attention, it was preceded by a variety of earlier and significant changes. Growing dissatisfaction with AFDC had led an increasing number of states to seek waivers from the AFDC rules. These waivers were mostly designed to allow states to more stringently enforce work requirements for welfare recipients. Such waivers had started under President Ronald Reagan, but the Clinton Administration actively encouraged more expansive state-wide waiver programs. As a result, by the time PRWORA passed, 27 states had major state-wide waivers in place. Most of these states designed new TANF-funded welfare programs that were closely based on their waiver experiments, although virtually all waiver states used their new discretion under PRWORA to make additional program changes. All of these waiver programs had to be seriously evaluated by the states that implemented them. The Department of Health and Human Services (HHS), which 6 approved and administered the waivers, typically required some form of random- assignment evaluation. Over time, this generated a body of literature about welfare-to- work programs that was crucial in convincing people that such programs could have positive effects on earnings and labor supply and negative effects on welfare spending. Along with reform of traditional cash welfare programs, there were also major changes in federal legislation affecting low wage jobs and workers over the 1990s. The minimum wage rose from $3.35 at the end of 1989 to $5.15 in 1997. By 2000, this left real minimum wages 10.8 percent above their levels in 1989. Even more important, one of the first legislative proposals from the Clinton administration to receive Congressional approval in 1993 was a major expansion of the Earned Income Tax Credit (EITC). The EITC operates as a refundable tax credit through the federal tax system to subsidize low wage workers in low income families. Figure 1 describes the EITC subsidy as of 2000. Nonworkers receive no subsidy. Low income low wage workers with one child (two or more children) are initially subsidized at a rate of 34 percent (40 percent). Over some income range they receive a flat subsidy of $2353 ($3888), and as their income increases further this is taxed away at a rate of 15.98 percent (21.06 percent). This subsidy offsets federal income tax obligations (including taxes that fund the Social Security and Medicaid programs) and results in subsidies (checks from the government) for workers whose EITC subsidy is greater than their tax obligations.4 The combination of increased minimum wages and increased EITC subsidies meant that the real earnings plus wage subsidy (in 2000 dollars) received by a woman with one child working full time at the minimum wage rose from $10,568 in 1989 to 4 More detail on the EITC is available in U.S. House of Representatives (2000, p808-813). For a history of the EITC, see Ventry (2000). 7 $12,653 in 2000, a 19.7 percent increase. For a similar woman with two or more children, real earnings and subsidies rose from $10,568 in 1989 to $14,188 in 2000, a 34.3 percent increase. These changes should have greatly increased the work incentives for low-wage single mothers with children. Two other federal legislative changes also deserve mention. First, from the mid- 1980s on, access to public health insurance became increasingly delinked from participation in cash public assistance programs. By 1999, all children in families whose income was below 100 percent of the poverty line were eligible for Medicaid, the publicly-funded health insurance program for low-income persons.5 In addition, women who left welfare for work were eligible for one year of transitional Medicaid coverage.6 Because many eligible children did not appear to be accessing Medicaid, in 1997 Congress funded a $24 billion, 5-year program known as the Children’s Health Insurance Program (CHIP), providing incentives and funding to states to expand health care usage and health insurance access among low-income children. There were also substantial changes in subsidies for child care assistance. PRWORA abolished a plethora of older programs and created a single Child Care and Development Block Grant. States were also allowed to use a certain share of their TANF funds for child care. In addition, there were expansions in the Child Care Tax Credit for lower middle income families.7 5 Since 1983, all pregnant women and children age 5 or less in families with incomes below 133 percent of the Federal poverty line have access to Medicaid; 23 states use a higher cutoff point. Older children, born after September 1983, in families below 100 percent of the poverty line are also covered by Medicaid; 26 states set a higher cutoff point for eligibility for these older children (Ku, Ullman and Almeida, 1999). For more detail on Medicaid and how it operates, see Gruber (forthcoming). 6 Thirteen states extend this for more than one year. 7 Loprest, Schmidt and Witte (2000) discuss these changes in more detail. For a summary of the research on the impacts of child care subsidies, see Blau (forthcoming) and Anderson and Levine (2000). 8 Combined, these changes constitute a revolution in public assistance programs within the United States over this past decade. Federal dollars available to support working low income families increased from $11.0 billion in 1988 to $66.7 billion in 1999.8 Dollars paid in cash welfare support to (largely nonworking) families headed by non-elderly, non-disabled adults rose from $24 billion in 1988 to $27 billion in 1992, then fell to $13 billion by 1999 (all numbers in 2000 dollars). This suggests that the work incentives imbedded in the public assistance system should have increased markedly over this period: cash assistance became far less available, welfare recipients were pushed much harder to find employment and leave the rolls, the returns to low wage work rose, and the availability of work supports (child care and health insurance) increased to low income families. Not unimportant, these changes took place at the same time as a major economic boom. The U. S. unemployment rate fell to 5 percent in April, 1997, and remained at or below this level until October 2001. Most places experienced worker shortages in the years following the passage of the 1996 legislation, making employers more willing to hire ex-welfare recipients. Wages among less skilled workers started to rise in 1995, for the first time since the late 1970s (Blank and Schmidt, 2001). This meant that the macroeconomy reinforced and supported the direction of legislative change over the 1990s. In many ways, the late 1990s were the best time imaginable to enact and implement work-oriented welfare reform. 8 Blank and Ellwood (2002, Figure 1). This includes dollars spent on the EITC, child care assistance to poor and near-poor families, and Medicaid and CHIP expenditures on low-income children and adults who are not receiving cash assistance. It does not include money spent on job training or job placement assistance, or cash benefits paid to working families. 9 II. The State Response Describing the Federal changes provides only half of the picture. After the passage of PRWORA each state began to design and enact its own TANF-funded program.9 Historically, analysis of public assistance programs has focused on two parameters: benefit levels and benefit reduction rates (BRRs). Figure 2 shows the income available to a low-wage family under a typical welfare program. A maximum benefit level, G, is available to non-workers. Workers earn an hourly wage rate w. As hours of work (and earnings) increase, benefits are taxed away at a rate t (the BRR). Ongoing historical discussion has focused on the trade-offs of higher benefits (raising G provides a stronger safety net but discourages work and raises program costs) and higher BRRs (raising t reduces the return to low levels of work but leads to lower program costs). The Negative Income Tax experiments of the 1970s were largely experiments involving different levels of G and t (Burtless, 1986; Ashenfelter and Plant, 1990). Frustration with the work disincentives imbedded in traditional welfare programs led President Reagan to promote welfare-to-work programs in the early 1980s. Mandatory job search or job placement programs would replace the endless effort to tinker with the contradictory incentives imbedded in a given level of G and t, by forcing welfare recipients to work regardless of the resulting loss in benefit income. A strong version of work requirements is a so-called “workfare” program, which mandates a certain level of work in a publicly provided job as a condition of ongoing welfare receipt. A less extreme requirement might mandate participation in a job preparation or job 9 For a description of the structure of means-tested programs prior to 1996, see Blank (1997a). 10 search program. A wide variety of states have experimented with different versions of work requirements over the past 20 years. Initially using waivers and later using their authority under PRWORA, states have transformed the nature of public assistance programs. While benefit levels and BRRs remain important parameters, states are increasingly using a wide variety of additional program design components to promote work and to reduce caseloads. What follows is a brief description of these changes. What will be clear is that both the number of possible program parameters available to states has increased markedly, and that different states are choosing very different combinations of these parameters. Hence, the variance across states in their TANF-funded programs is enormous and still growing. Benefit Levels. States have always been able to choose their own maximum benefit levels for non-workers. This part of the system has changed little. Most states over the 1990s made only small legislative changes in their benefit levels, even after the passage of PRWORA. In fact, the overwhelming trend in benefit levels in the 1990s has been inflation erosion in benefits (a trend visible since the early 1970s). Table 1 indicates that the median benefit level (in $2000) fell from $480/month for a family of three to $379 between 1990 and 2000. Most of this decline was due to inflation erosion. Similar changes occurred across the distribution of benefit levels, as Table 1 indicates.10 As cash assistance becomes less broadly available, benefit levels are of decreasing importance. The steady decline in benefit levels, however, should increase work incentives over this period. Benefit Reduction Rates. Under AFDC, BRRs were set by federal law (although a few states received waivers to experiment with alternative BRRs in the early 1990s). 11 BRRs had been raised significantly in the early 1980s, and many AFDC recipients faced almost 100 percent tax rates on their earnings. A major change post-PRWORA is that many states have chosen lower BRRs, in order to encourage work (and to a lesser extent, as a way of supplementing income among low-wage workers). Free to set their own rules, many states have also chosen to have BRRs rise at some point after a woman goes to work, so her public assistance subsidy is reduced over time even if her earnings do not increase. Table 2 is based on calculations of the cumulative cash welfare benefits available over the first 24 months of work by a welfare recipient with two children whose earnings are $6/hour (slightly above the minimum wage of $5.15 per hour) and who works part- time (30 hours/week) or full-time (40 hours/week). The first two columns show the cumulative cash benefits that a welfare recipient family would have expected to receive in each state in January 1996 (all numbers adjusted to 2000 dollars) if the mother went to work under the old AFDC program. The second two columns show the cumulative cash benefits that a family would have expected to receive in 2000 if the mother went to work under each state’s TANF program. The AFDC program provided little cash support to workers. Almost half the states in 1995 would have paid no cash benefits to a part-time worker.11 Only 13 states would have provided any support to a full-time worker. By 2000, BRRs had fallen in almost all states, dramatically changing these results. Almost all states provide some support to the mother who enters part-time work in 2000; in 28 states this support exceeds $1000 over the first 24 months of work. Half 10 The standard deviation in benefits across states changed little over these 10 years. 12 the states also provide some cash supplement to the woman who enters full-time work,12 with the median state paying $299 in cumulative cash benefits over the first 24 months. Sixteen states pay more than $1000 in benefits over these first 24 months.13 A change in the BRR is equivalent to a change in the effective wage rate (see Figure 2). Because this imbeds both income and substitution effects, it is theoretically ambiguous whether work incentives should rise or fall. Most labor economists assume that substitution effects dominate income effects for low wage workers. This suggests that lower BRRs should increase work incentives. Moffitt (1992) notes the remarkable historical inelasticity of responsiveness among welfare recipients to changes in BRRs, however, which suggests that the work incentive effect of lowering BRRs in the mid- 1990s might not be large. On the other hand, the changes in BRRs implemented in the 1990s were often made in conjunction with strong work requirements. As the discussion of financial incentive programs in Section VIII below indicates, the combination of lower BRRs and work mandates may have a quite powerful combined effect. Note that lower BRRs may have other effects as well. In the presence of time limits, lower BRRs keep welfare recipients on welfare longer and encourage families to “use up” their time. If clients are aware of time limits and worried about using up their public assistance eligibility, this will further increase their incentives to work and may lead them to leave welfare even while still eligible for some benefits in order to preserve future months’ welfare eligibility. 11 Differences across the states in columns 1 and 2 of Table 2 are entirely due to differences in AFDC benefit maximums across states; all states are subject to identical (federally determined) BRRs. 12 Cumulative benefits equal 0 for a woman earning $6/hour if BRRs are very high (and in some states they are 100 percent, so benefits are reduced $1 for $1 of earnings) and/or if benefit levels are very low (so that one “works one’s way off welfare” more quickly). 13 More detailed information on these earnings disregard calculations are available from the author upon request. 13 Welfare-to-Work Programs. Virtually all states have tried to expand their welfare-to-work programs starting in the early 1980s. Since the passage of PRWORA, states are mandating participation in job search assistance and work preparation among a much higher share of their caseload. By 1999 states reported that 38.3 percent of their caseload was engaged in work or job activities, up from 20.4 percent in 1994.14 The exact meaning of “welfare-to-work” varies substantially across states. In the early 1990s, many states ran both job placement and job training programs. By the late 1990s, the focus of most state programs was “work first”, aimed at getting recipients into a job as soon as possible. Hence, most programs focus on narrow job preparation skills (interviewing, getting along on the job, organizing child care) and job search assistance. Relatively little money is currently being spent on longer-term training, a somewhat controversial fact in many states.15 These work programs should increase work incentives, both by improving employment-related skills and by establishing job search as an expected activity for welfare recipients. Indeed, a number of states have focused on changing the “culture” of their public assistance offices, retraining and reorganizing staff so that their primary goal is to encourage work rather than to provide monthly assistance (Gais, et. al., 2001). Sanctions. To enforce job search and work requirements, states have implemented a variety of sanction policies aimed primarily at penalizing individuals who do not respond to work requirements (most commonly, these are individuals who miss 14 The 1994 data is from U.S. House of Representatives (2000, Table 7-25). The 1999 data is from U.S. DHHS (2000, Table 3:1). 15 See discussion of this issue by Strawn, Greenberg, and Savner (2001). Job training or education among adults can be counted as a work activity, but cannot count toward the first 20 hours/week of required work participation. An exception is teen mothers under age 18, who are required to participate in education activities unless they hold a high school degree. 14 required job preparation or job search sessions). Sanctions involve a reduction in welfare benefits, but states vary in how much they reduce benefits and for how long. Pavetti and Bloom (2001) classify 25 states as “strict”, including a number of states that impose permanent full benefit losses on the families of noncompliant individuals. They classify 13 states as “lenient,” imposing only temporary and partial reductions. If low BRRs are the “carrot” for participating in welfare-to-work programs (providing ongoing subsidies to those who can only find low wage jobs), then sanctions are the “stick.” All states have some form of sanctioning policy, which is to say that no state relies only on positive work incentives to get people employed. Time Limits. While all states are subject to the 60 month federal time limit for individuals using TANF-related funds, they can also set shorter time limits, or can provide state funding beyond 60 months.16 Seventeen states have time limits of less than 60 months for some families, 26 states use the 60 month federal time limit, and 8 states have not imposed time limits which mandatorily end all benefits.17 For instance, several of these states impose time limits on adult recipients but continue benefits for children (Pavetti and Bloom, 2001). Time limits should have two work-inducing effects. First, they should provide incentives for recipients who might need welfare in the future to leave welfare as rapidly as possible, in order to preserve future eligibility.18 This requires a thorough understanding of the fact that “the clock is ticking” and some states have been better at reminding recipients of this. There is some evidence that many recipients misunderstand 16 States are allowed to exempt up to 20 percent of their caseload from the 60-month time limit. 17 As in Tables 1 and 2, Washington, D.C.c is included as the 51st “state”. 18 For example, Swann (2000) develops a model indicating that time limits will have larger effects when welfare recipients are forward looking. 15 where they are on their time clock (Bloom, 1999). Second, once time limits are imposed, ex-recipients can no longer use cash assistance as a back-up to work. Time limits have not yet been widely imposed; the first recipients did not begin to hit the 60 month limit in most states until late 2001 or early 2002. As noted before, there are somewhat perverse interactions between time limits and lower BRRs. In addition, there is also evidence that time limits and sanctions interact in interesting ways. Sanctions tend to affect the same less responsive and often more disadvantaged population that is likely to hit time limits. This suggests that time limits may not have a very large effect if many individuals will have already been removed from eligibility through sanctions (Pavetti and Bloom, 2001). Diversion. With no national entitlement to public assistance, states can deny assistance to individuals. Many states have implemented eligibility determination processes that encourage some applicants to be diverted from cash public assistance. Ten states impose work search requirements on applicants prior to eligibility (i.e., applicants must show that they’ve applied for a certain number of jobs as a condition of eligibility). Twelve states provide short-term cash payments as an alternative to public assistance eligibility, designed to meet some immediate need of the applicant which will then allow her to return to work. Nine states use both techniques in order to divert applicants from welfare, while 20 states make no effort at diversion.19 Work Support Subsidies. With more attention to moving welfare recipients into work, states have also recognized the need to help families with work-related expenses. States have greatly increased their expenditures on work support programs, primarily child care subsidies. Between 1993 and 2000, federal funds available to the states for 16 child … s we approach the 21st century, the public seems increasingly disen- chanted with the record of govern- ment, and less and less inclined to believe in the value of empirical analysis as a guide to action. Evidence of the loss of confidence in the public sector’s ability to operate effectively and efficiently is found in opinion polls, falling rates of electoral participation, and the rising influence of “anti-govern- ment” politicians. In such an environment, it is useful to reflect on the historical role that applied social science has played in the public sector and the role it might play in the future. The Past Influence of Social Policy Research Social scientific research can dramatically shape public policy, both directly and indirect- ly. Directly, research can prove pivotal in situ- ations where policymakers have reached con- sensus about goals but have inadequate infor- mation about alternative means of reaching them. Perhaps the clearest illustrations have been the impacts of demonstration research on guiding choices among different types of housing subsidies, personnel retraining pro- grams, and law enforcement initiatives. Indirectly, research can provide a cogni- tive backdrop to decisionmaking, sensitizing policymakers to new issues and crystallizing opinion about which social problems have reached intolerable proportions. Social scien- tific analysis has often played a larger role in shaping public perceptionof problems than in directly influencing policy responses to them. In 1968, for example, President Johnson appointed former Illinois Governor Otto Kerner to chair a commission of social scien- tists to report on the causes of urban civil dis- orders. Though the Kerner Com- mission’s findings resulted in a lim- ited number of policy responses, its work fundamentally changed public opinion and played an educational role that, even today, shapes pol- icymakers’ conceptions about urban racial issues. Another example is the reformulation of urban pover- ty provided by William Julius Wilson’s 1987 book, The Truly Disadvantaged.Through quantitative and qualitative analyses, Wilson riveted atten- tion on deindustrialization and the spatial con- centration and social isolation of its victims. His views laid the intellectual groundwork for the comprehensive, place-based empower- ment zone policy that materialized seven years later. Future Challenges Despite the influence on policy and pub- lic perception that research has at times wielded, a number of forces threaten to weak- en its future impact. These include external challenges—such as many Americans’ loss of confidence in the public sector, the trend toward devolution without evaluation, and the emergence of morality politics—and internal trends including what I call the “eso- terica fetish” and advocacy research. Loss of Public Confidence Much of the public’s loss of confidence in government can be traced to factors having only a tangential link to policymaking itself. The Challenges for Policy Research in a Changing Environment George Galster THE URBAN INSTITUTE A series on the long-term forces affecting U.S. social policy TH E F U TU R E O F TH E F U TU R E O F TH E PU BL IC S EC T TH E PU BL IC S EC T O R O R No. 7, November 1996 A Research can prove pivotal in situa- tions where policymakers have reached consensus about goals but have inade- quate information about alternative means of reaching them. These factors include corruption among public figures, negative cam- paign tactics, and the media’s empha- sis on the personal foibles of candi- dates for political office. However, another force—money and special interests—is directly tied to the policy process and, as such, can have a sig- nificant impact on policy research. If in the foreseeable future campaign financing undergoes only rhetorical rather than actual reform, and the influence of monied special interests continues unabated, policy research will be confronted by two growing obstacles. First, special interests will further corrupt social science by sponsoring more “advocacy research,” an exercise in creating political ammunition by selectively culling or distorting infor- mation to support pre-ordained conclu- sions (more on this below). Second, the public’s increasing skepticism toward government should provide fertile elec- toral ground for across-the- board budget cuts and disman- tling of social programs, rather than targeted budget actions based on the efficacy of such programs. In such an environ- ment of defensive public policy, it is likely that financial or polit- ical support for the public spon- sorship of policy research will diminish, and less notice will be paid to evidence produced by social scientists concerning the effects of public programs. Devolution without Evaluation Disillusionment with the ability of public policy to successfully confront complex social issues and to operate in a fiscally responsible manner has led to a loss of confidence in federal poli- cy in particular. The federal govern- ment is widely regarded as being para- lyzed by large political egos that put personal power ahead of the public good; by an inept, bloated bureaucra- cy; and, more recently, by gridlock between Congress and the president. This view, which has been building up for a number of years, is one source of initiatives to devolve power from the federal government to state and local governments. Underlying this devolu- tion are incentives for states to experi- ment with a variety of social program reforms, backed up by a series of rev- enue block grants allowing expanded latitude for state use of federal funds. Experimentation at the state level is a practical way to test ideas for reform and potentially can prove valu- able to the nation as a whole. Creating 50 different laboratories, however, will make some kinds of policy research more difficult, unless there is a con- comitant increase in commitment to evaluation. Evaluating a multitude of variations on a particular program theme and applying findings from individual states to other states are truly formidable tasks. It has been expensive and time-consuming for program analysts to evaluate and reach consensus even about major national programs such as CETA employment training and Head Start. It seems unlikely that government will be able to support a similar level of effort to replicate and refine assessments of 50 program variants. At the same time, it may be unre- alistic to expect states themselves to be able to evaluate their experiments in a way that will be meaningful across states. Many state agencies either lack policy evaluation and research divisions altogether, or use standards for program evaluation that are not comparable to those set by their federal counterparts. The quanti- ty and quality of many state-initiated evaluations of state-sponsored pro- grams may thus prove problematic. The inability to adequately test these state experiments will surely inhibit development of program enhance- ments and may ultimately erode polit- ical support for them. Morality Politics The alienation and secularization of large segments of our society have played a role in the growth of what I call “morality politics,” the emer- gence of single-issue constituencies whose policy positions are primarily grounded on unwavering moral prin- ciples. Efforts to reinvigorate faith- based values and reinforce virtuous behavior have numerous laudable consequences. I worry, however, that in the face of complex contemporary social problems and the increasingly complicated solutions they require, morality politics will lead more peo- ple to “vote with their hearts, not with their heads.” Research findings can be antithetical to cherished values and, as Isabel Sawhill has recently observed, “when research and values collide, values will always win.” What is more disturbing is that the worth of any research findings may eventually be ignored as legislators resort to a politics of morality and opt out of policy debate based on objec- tive information. The Esoterica Fetish Beyond the external chal- lenges to policy research lie internal weaknesses that threat- en to erode the credibility and effectiveness of the research community. One of these is social scientists’ proclivity toward technical jargon that results in their communicating with one another rather than with policymakers and the public. There are few incentives for social scientists, especially those in universities, to popularize their find- ings for a lay audience. The profes- sion typically rewards research that is published in highly technical forums rather than in easily communicated language designed to reach wider audiences. It seems that only social scientists who have already “proven” themselves can afford to write in a more popular vein. Because of the dearth of experts who are also effec- tive communicators, even educated citizens do not always understand the issues, and increasing numbers of the public and of politicians turn to sim- plistic nostrums for “easy” solutions. Advocacy Research Legitimate social science can and should provide higher standards of objectivity and methodological sound- T H E F U T U R E O F T H E P U B L IC S E C T O R N o . 7 2 The inability to adequately test state experiments will surely inhibit development of program enhancements and may ultimately erode political support for them. ness than those of advocacy research. Advocacy research attempts to mimic certain features of social science, such as use of theory and statistical tech- niques, but fails to be genuinely objec- tive. This is primarily because, whether based on liberal or conserva- tive ideology or simply arising out of a particular cause not linked to any ideology, advocacy research starts out with a firmly held position rather than with a potentially rejectable hypothe- sis. Such research looks for those facts and theories that bolster its position, finding reasons to reject those that fail to support it. A substantial amount of applied social science has for some time been commis- sioned and funded by govern- ments and special interests who often keenly hope for a particular outcome. Even sup- posedly nonpartisan think tanks and universities have not remained untainted by spe- cial interest groups that may supply research funds. Under such condi- tions, the temptations for bias may be enormous. Seasoned social scientists can typically identify research find- ings that may have been influenced by the sponsor and interpret them accord- ingly. Other users of research often cannot. The news media, for example, may not have the time or ability to judge the soundness of scientific evi- dence on all sides of an issue. For the general public, the task can be even harder. Politicians’ continued reliance on partisan research as artillery for political battles will, I fear, sully the reputation of objective social science. Maintaining Standards for Social Science Research The weakening of the role of pol- icy research comes at a time when a strong voice has never been more important. Evaluation research is cru- cial for policymakers who wish to scrap ineffective programs while expanding successful ones. Deciding what to cut and what to keep based on objective, scientifically sound evalua- tions rather than on the potentially biased assessments of special inter- ests will help restore the public’s faith in the public sector. Even in a regime of morality politics, research remains vital as a means of demonstrating alternative strategies and ascertaining which might best help us reach highly valued goals. I offer three practical suggestions for helping social science policy research effectively meet the chal- lenges of the future outlined above. First, legislatures should more fre- quently employ blue-ribbon panels of respected social scientists to produce concise, readable, state-of-the-art briefs on topics of forthcoming legis- lation. Such briefs could also form the core of legislative hearings. Second, new programs should be legislated with an accompanying requirement and appropriation for eval- uation after a point at which program- matic benefits might be expected to accrue. This procedure has been used in the past. Examples include the Fair Housing Initiatives Program, autho- rized under the Housing and Com- munity Development Act of 1987, and the Youth Employment Demonstration Project, authorized under the Youth Act of 1977. Federal waivers recently granted to states to experiment with wel- fare reforms have carried similar evaluation requirements. In the future, however, every state will need to evaluate its welfare initia- tives under the 1996 block grant legislation. Third, proposals solicited from social scientists to conduct such evaluations should be reviewed by government-convened panels of independent experts, so that to the greatest extent possible, applied social science can be subjected to a peer review process similar to that of the basic sciences. To avoid becoming marginalized or, worse, a tool of scientific nihilism, policy research must be characterized by open inquiry into all aspects of social problems, a willingness to test conventional conceptions and conclu- sions, the most rigorous and sophisti- cated methods available, accessible reporting of the results, and open dis- cussion of the potential uses and mis- uses of research by future American policymakers. N o . 7 T H E F U T U R E O F T H E P U B L IC S E C T O R 3 Deciding what [programs] to cut and what to keep based on objective, scientifically sound evaluations rather than on the potentially biased assessments of special interests will help restore the public’s faith in the public sector. George Galster is Hilberry Professor of Urban Affairs at Wayne State University. He was director of housing research at the Urban Institute from 1994 to 1996. RELATED READING Galster, George, ed. 1996. Reality and Research: Social Science and U.S. Urban Policy Since 1960. Washington, D.C.: Urban Institute Press. Nathan, Richard. 1988. Social Science and Government.New York: Basic Books. Robinson, William, and Clay Wellborn. 1991. Knowledge, Power and the Congress.Washington, D.C.: Congressional Quarterly, Inc. Sawhill, Isabel. 1995. “The Economist vs. Madmen in Authority.” Journal of Economic Perspectives, vol. 9, no. 3. No. 1. Declining Economic Opportunity in America, Isabel V. Sawhill and Daniel P. McMurrer No. 2. Whither Federalism?, Martha Derthick No. 3. Growing Income Inequality: Roots and Remedies, Edward M. Gramlich and Mark Long No. 4. Reforming Employment and Training Policy, Paul Osterman No. 5. The Public Can Make Hard Choices, Susan Tanaka No. 6. When to Devolve, Paul E. Peterson n The Importance of Process in Domestic Reforms n Antistatism and Government Downsizing T H E F U T U R E O F T H E P U B L IC S E C T O R N o . 7 THE URBAN INSTITUTE 2100 M Street, N.W. Washington, D.C. 20037 This series, funded in part by the Ford Foundation, focuses on chal- lenges for policymaking in the 21st century. Advisory Board C. Eugene Steuerle Christopher Edley, Jr. Edward M. Gramlich Hugh Heclo Pamela Loprest Demetra S. Nightingale Isabel V. Sawhill William Gorham Published by The Urban Institute 2100 M Street, N.W. Washington, D.C. 20037 Copyright © 1996 The views expressed are those of the author and do not necessarily reflect those of the Urban Institute, its board, its sponsors, or other authors in the series. Extra copies may be requested by calling (202) 857-8687. Designed byRobin Martell and Barbara Willis Address Correction Requested Nonprofit Org. U.S. Postage PAID Permit No. 8098 Washington, D.C. Telephone: (202) 833-7200 n Fax: (202) 429-0687 n E-Mail: [email protected] n Web Site: http://www.urban.org Other Briefs in the Series Among Future Topics FROM: The Center for Civic Partnerships SOURCE: http://www.civicpartnerships.org/docs/tools_resources/Quan_Qual\%20Methods\%209.07.htm Quantitative and Qualitative Evaluation Methods Top Tips Evaluation methods and the data they produce are grouped into two basic categories – quantitative and qualitative. In general, quantitative methods produce “hard numbers” while qualitative methods capture more descriptive data. The method(s) you choose are determined by the purpose(s) of your evaluation and the resources you have to design and conduct it. In practice, most researchers and evaluators agree that combining quantitative and qualitative techniques (sometimes called “mixed method” evaluations) produces a richer and more comprehensive understanding of a project’s accomplishments and learnings. How are Quantitative and Qualitative Data Different? At the most basic level, data are considered quantitative if they are numbers and qualitative if they are words. Qualitative data may also include photos, videos, audio recordings and other non-text data. Those who favor quantitative data claim that their data are hard, rigorous, credible and scientific. Those in the qualitative camp counter that their data are sensitive, detailed, nuanced and contextual. Quantitative data best explain the why and how of your program, while qualitative data best explain the what, who and when. Different techniques are used to collect and analyze quantitative and qualitative data: Quantitative Techniques Qualitative Techniques Surveys/Questionnaires Observations Pre/post Tests Interviews Existing Databases Focus Groups Statistical Analysis Non-statistical (methods vary) In general, evaluators agree that qualitative and quantitative data and methods have different strengths, weaknesses, and requirements that affect decisions about which methodologies are appropriate for which purposes. Quantitative Data and Evaluation Methods What are Quantitative Data?  Pieces of information that can be counted mathematically  Usually gathered by surveys from large numbers of respondents selected randomly  Secondary data such as census data, government statistics, etc. often included in quantitative evaluations  Analyzed using statistical methods  Best used to answer what, when and who questions  Not well suited to how and why questions Strengths Limitations Findings can be generalized, if selection process well-designed and sample is representative of study population Related secondary data sometimes not available, or accessing available data is difficult/impossible Relatively easy to analyze Difficult to understand context of program activities Data can be very consistent, precise, reliable Data may not be robust enough to explain complex issues Data collection is usually cost efficient http://www.civicpartnerships.org/docs/tools_resources/Quan_Qual\%20Methods\%209.07.htm http://www.civicpartnerships.org/docs/tools_resources/Evaluation\%209.07.htm How do you collect Quantitative Data? Surveys o Most common method o Self-administered or by someone else o Face-to-face, telephone, mail, web-based Secondary Data o Often used in conjunction with survey data o Includes census data, knowledge/attitude/behavior (KABB) studies, criminal justice statistics, performance data, non-confidential client information, agency progress reports, etc. Qualitative Data and Evaluation Methods What are Qualitative Data?  Usually gathered by observations, interviews or focus groups  May also be gathered from written documents and through case studies  Less emphasis on counting numbers of people who think or behave in certain ways and more emphasis on explaining why people think and behave in certain ways  Involves smaller numbers of respondents  Utilizes open-ended questionnaires or protocols  Best used to answer how and why questions  Not well suited to what, when and who questions Strengths Limitations Complement and refine quantitative data Findings usually can not be generalized to the study population or community Provide more detailed information to explain complex issues More difficult to analyze; don’t fit neatly in standard categories Multiple methods for gathering data on sensitive subjects Data collection is usually time consuming and costly How do you collect Qualitative Data? Observations o Looking at what is happening rather than directly questioning participants o Used to better understand behaviors, their social context and meanings attached to them o Useful for certain populations - children, infants o Can identify unanticipated outcomes Interviews (in-depth, individual) o Usually provide rich data, details, insights from program participants and stakeholders about their experiences, behaviors and opinions o Particularly useful for complex or sensitive subjects o Use open-ended questions Focus Groups o 8-12 people selected by non-random method, share some characteristics or experience relevant to the evaluation, ideally do not know each other, respond to questions from group facilitator o Use group dynamics to generate data and insights o Useful for generating ideas and strategies, defining problems in project implementation, assist with interpreting quantitative findings o Open-ended questions or topics designed to stimulate discussion; topics usually broader than interview questions These Top Tips” were adapted primarily from: Evaluation for Learning: Basic Concepts and Practical Tools, LaFrance Associates, and User-Friendly Handbook for Mixed Method Evaluations, Division of Research, Evaluation and Communication of the National Science Foundation. FROM: US Department of Transportation, Federal Highway Administration SOURCE: http://www.fhwa.dot.gov/tcsp/evalguide2.html Evaluation Guidance for the Preparation of TCSP Evaluation Plans - June 2001 Table of Contents 1. Importance of Evaluation 2. Grantee Roles and Responsibilities o Evaluation Plan o Assistatnce with Evaluation 3. General Approach to Evaluation o Steps in Developing an Evaluation Plan o What Should be Evaluated o Evaluation Reports 4. Detailed Evaluation Guidance o Process Evaluation o Product Evaluation o Outcome Evaluation  Approaches to Measuring Outcomes  General Measurement Issures  Available Evaluation Methods and Data Sources 5. Evaluation References o Transportation-Related Data Collection, Evaluation, and Experimental Design o Qualitative Assessement Techniques o Planning Processes List of Figures 1. Steps in Developing an Evaluation Plan 2. Use of Control Group in Before-and-After Data Collection List of Tables 1. Examples of TCSP Goals and Objectives 2. TCSP Process Evaluation 3. TCSP Product Evaluation: Planning Grant 4. TCSP Product Evaluation: Implementation Grant 5. TCSP Outcome Evaluation 6. Potential Existing Data Sources for Evaluation 7. Methods for Collecting New Data http://www.fhwa.dot.gov/tcsp/evalguide2.html http://www.fhwa.dot.gov/tcsp/evalguide2.html#s1 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s2 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s21 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s22 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s3 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s31 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s32 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s33 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s4 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s41 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s42 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s43 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s44 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s45 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s46 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s5 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s51 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s51 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s52 http://www.fhwa.dot.gov/tcsp/evalguide2.html#s53 http://www.fhwa.dot.gov/tcsp/evalguide2.html#fig1 http://www.fhwa.dot.gov/tcsp/evalguide2.html#fig2 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab1 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab2 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab3 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab4 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab5 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab6 http://www.fhwa.dot.gov/tcsp/evalguide2.html#tab7 http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html http://www.fhwa.dot.gov/tcsp/docs.html http://www.fhwa.dot.gov/tcsp/projects.html http://www.fhwa.dot.gov/tcsp/resource.html http://www.fhwa.dot.gov/tcsp/links.html http://www.fhwa.dot.gov/tcsp/sitemap.htm http://www.fhwa.dot.gov/tcsp/index.html 1. Importance of Evaluation The purpose of the Transportation and Community and System Preservation (TCSP) Pilot Program is to fund innovative projects that will increase the knowledge of the costs and benefits of different approaches to integrating transportation investments and strategies, community preservation, land development patterns, and environmental quality. Planning and implementation projects may be undertaken at the neighborhood, local, metropolitan, State, and regional levels by States, local governments, and metropolitan planning organizations (MPOs) working in cooperation with non-traditional partners. The TCSP is a pilot program explicitly designed to encourage innovative strategies and techniques, the results of which can then be used by other public and private organizations throughout the country. While TCSP funding is not sufficient to implement projects on a nationwide basis, all organizations nonetheless will benefit by being able to easily tap into the experience of others to learn what might be applicable for their own situations and how these new transportation strategies and techniques can be most effectively implemented. To accomplish this learning and the desired resultant transfer of experience, the evaluation of individual projects is a key component of the TCSP program. Evaluation of projects which are new or experimental in character will indicate the success of various activities at achieving the desired transportation, community, and system preservation objectives. The lessons learned from this process will be used in evaluating the overall TCSP program and will help develop more effective TCSP projects in the future. As a result, the TCSP program will provide an important nationwide learning experience. In keeping with the TCSP programs emphasis on learning and evaluation, grant applicants should bear in mind the following points:  An evaluation plan for each grant is to be included as part of each proposal.  The evaluation plan should describe roles, responsibilities, project objectives, performance measures, evaluation methodologies, data sources, schedule milestones, and budgets.  Efforts to reflect an effective approach to evaluation will receive a higher priority in the evaluation of proposals.  The TCSP is a pilot program to develop and evaluate new strategies. Falling short of expected goals is acceptable, as long as the evaluations identify barriers encountered and efforts undertaken to overcome them. Documentation of both successes and problems encountered in carrying out TCSP activities will help other areas develop more effective approaches in the future. In presenting this guidance for the preparation of evaluation plans, it is recognized that, one size does not fit all. This guidance, therefore, provides ideas for evaluation plans rather than a mandated approach. [Top of Section, Table of Contents, Top of Document] 2. Grantee Roles and Responsibilites http://www.fhwa.dot.gov/tcsp/evalguide2.html#s1 http://www.fhwa.dot.gov/tcsp/evalguide2.html#toc http://www.fhwa.dot.gov/tcsp/evalguide2.html#top Evaluation Plan TCSP grantees are responsible for conducting a systematic evaluation of their TCSP project. Each grant application should include an evaluation plan which describes how the grantee proposes to evaluate the project. This will assist in demonstrating the applicants commitment to the evaluation component. The strength of the evaluation component, including identification of resources required, will be an important factor in the selection of final grant awards. FHWA will use the results from individual evaluations, in conjunction with other overall program evaluation criteria and methods, in assessing the overall effectiveness of the TCSP program. As results and lessons learned from individual TCSP grant awards become available and the overall program can be assessed, the FHWA will coordinate and disseminate results, tools, and information developed through the program. In the evaluation plan submitted, grantees should identify program goals and objectives, performance measures, measurement techniques, potential data sources, and schedule milestones. Proposals should identify existing sources of information which will be utilized (either qualitative or quantitative), and should also identify any new data collection efforts which may be required or useful for evaluating the effectiveness of the program. The evaluation plan also should contain clear roles, responsibilities, commitments by participants, and a budget estimate. The resources required for evaluation activities should be included in the overall grant budget proposed for the project. As a component of the TCSP program evaluation, a grant workshop is planned for the Spring of 1999, at which grantees will share experiences and initial results from their projects. Budgets for grant applications should include travel for the key investigator to this workshop, as well as a second such conference, as part of the evaluation component. Assistance with Evaluation The remainder of this document provides guidance relating to the development of an evaluation plan. The purpose of this guidance is to provide ideas rather than a mandated approach, and agencies should not be discouraged from applying for TCSP program funding simply because they lack expertise in particular evaluation methods. It is more important that grant applicants commit to undertaking a systematic evaluation, including the designation of project resources, than they demonstrate proficiency in any particular evaluation method. Grant applicants not already having the desired level of in-house evaluation expertise may want to consider working in cooperation with another agency or a university. [Top of Section, Table of Contents, Top of Document] General Approach to Evaluation This section provides an overview of how to develop an evaluation plan, with http://www.fhwa.dot.gov/tcsp/evalguide2.html#s2 http://www.fhwa.dot.gov/tcsp/evalguide2.html#toc http://www.fhwa.dot.gov/tcsp/evalguide2.html#top more detail on how to structure and conduct the evaluation described in Section 4.0. In particular, Section 4.0 identifies specific techniques that may be used, issues to consider, and key questions to ask in evaluating a TCSP project. Steps in Developing an Evaluation Plan Grant applicants are encouraged to take the following steps in developing an approach to project evaluation, as illustrated in Figure 1: Figure 1. Steps in Developing an Evaluation Plan [List of Figures] 1. Define Project goals and objectives. What is the motivation for undertaking the project? What is the project intended to accomplish? Table 1 shows examples of general goals and objectives for the overall TCSP program. Goals and objectives for individual TCSP projects may be a subset of these program goals and objectives. In addition, grant applicants may have additional goals and objectives which are important for the project to achieve locally. 2. Identify performance measures. Performance measures are either quantitative or qualitative measures which indicate the success of the project at achieving its stated goals and objectives, e.g., total emissions per capita or land consumed per unit of development. Examples of performance measures for the identified TCSP program goals and objectives are shown in Section 4.0. Applicants, however, should resist the temptation to establish a laundry list of performance measures, but instead should identify a few key measures which best reflect the impacts of the program. It is also important to select performance measures which are simple to understand, are as objective as possible, and can be constructed from available data sources. 3. Identify data and information sources and evaluation methods. Grant applicants should identify data and information sources to support each performance measure. In the case of quantitative data, applicants should identify both existing sources and potential new data collection efforts. In the case of qualitative data, proponents should identify key sources of information (people, agencies, committees, etc.), along with appropriate techniques for obtaining and evaluating information (interviews, direct observation, etc.) Some potential data sources and evaluation techniques are identified in Section 4.0. Consideration also should be given to identifying the baseline condition from which changes will be assessed. Table 1. Examples of TCSP Project Goals and Objectives http://www.fhwa.dot.gov/tcsp/evalguide2.html#lof  Improve efficiency of transportation system  Maximize use of existing infrastructure  Reduce impacts on environment  Reduce costs of infrastructure investment  Ensure efficient access to jobs, services, centers of trade  Encourage private sector land development patterns to achieve above goals  Involve non-traditional partners  Integrate transportation, community preservation, and environmental activities [List of Tables] Once potential performance measures, data sources, and evaluation methods have been identified, an overall evaluation plan should be developed for collecting and analyzing the required information. This includes identifying the individual work tasks required to carry out the evaluation and establishment of the associated budget and timeline for these tasks. What Should Be Evaluated? A TCSP evaluation should focus on identifying both the magnitude and the distribution of the costs and benefits of a project, and on those aspects of the planning and implementation process that will be useful to other organizations in deciding whether or not to implement similar strategies. Thus, evaluations can focus on three different aspects of a TCSP project: process, products, and outcomes. Appropriate goals and objectives, performance measures, and evaluation methods will differ for each, as will the timeframe over which the evaluation is conducted.  Process evaluation focuses on the approach through which a project is developed and implemented. A process evaluation can focus on questions such as the number and types of both traditional and non- traditional groups or persons involved, the manner in which these groups have been involved, the degree to which stakeholder commitment and buy-in were achieved, and the nature of the issues which emerged as being important in the deliberations.  Product evaluation focuses on what was produced by the process or activity. For example, a description of the plan that was developed or project that was implemented, and how it compares to what was originally implemented. How many miles of sidewalk were built connecting residential neighborhoods with employment and activity centers, public transportation systems, or recreational areas?  Outcome evaluation focuses on determining the effectiveness of the project at achieving the defined transportation, community, and system preservation objectives. How much are emissions reduced? What is the reduction in infrastructure cost per unit of person-travel? These three aspects of a project are interrelated and important to the evaluation http://www.fhwa.dot.gov/tcsp/evalguide2.html#lot of a TCSP project. Outcome goals are of ultimate interest to society, but achievement of process and product goals can indicate the likelihood of success at achieving the desired outcomes. Process and product goals are also desirable for their own sake. For example, an open and participatory process is important for ensuring that all viewpoints and potential impacts are considered. The involvement of non-traditional partners will help to identify strategies that encourage private sector development patterns that are consistent with the goals of the TCSP program. Examining the linkages among process, product, and outcome also can be useful. For example, desirable outcomes can be facilitated by the relationships developed during a planning process. Conversely, difficulties encountered during implementation may be traceable to the unintentional omission of an important factor during the planning stage. Finally, evaluation of all aspects of a project serves as an important learning tool, helping to identify both successful and unsuccessful approaches to a problem. Evaluation Reports An initial evaluation plan is to be included by an applicant as part of the application for a TCSP grant. This initial plan then may be refined in negotiating the terms of a grant awarded to the applicant. While the evaluation plan is expected to cover the basic approach proposed for evaluating a TCSP planning or implementation grant, the details of an evaluation plan, such as the statistical basis for a stratified sampling plan, may not be fully developed until after a project is actually underway. The evaluation activities associated with a TCSP grant should result in one or more reports. The purpose of these reports is to provide the information needed by other organizations throughout the country to decide whether similar projects would be beneficial within their jurisdictions, and how they should go about planning or implementing this particular kind of action. The initial evaluation report should document the process by which the TCSP grant project was developed or implemented, as well as the final product of the grant. This report can be produced shortly after completion of the project. Initial information on the results or outcomes of the project may also be available soon after completion and can be documented in this initial evaluation report. It is possible, however, that the full impacts of a project will not occur immediately, and that additional documentation of project outcomes will be appropriate in the future as data on longer-term impacts become available. The proposed approach to reporting should be explained in the evaluation plan portion of the grant application. [Top of Section, Table of Contents, Top of Document] 4. Detailed Evaluation Guidance This section provides more detailed guidance on evaluating the process, product, and outcomes of TCSP projects. For process and product evaluations, key questions for obtaining information as background to the evaluation are identified. For outcome evaluations, specific techniques and issues to consider in either estimating or measuring the impacts of the TCSP project are identified. For http://www.fhwa.dot.gov/tcsp/evalguide2.html#s3 http://www.fhwa.dot.gov/tcsp/evalguide2.html#toc http://www.fhwa.dot.gov/tcsp/evalguide2.html#top all three types of evaluations, examples of goals and objectives, performance measures, and evaluation methods relevant to TCSP projects are provided. It is important to identify in each case a baseline from which a change is being determined. For a process evaluation, this can be simply a comparison of the new or TCSP planning process with the existing or traditional approach. For a product evaluation, this can include an assessment of how the final project differs from what was initially proposed. Baseline considerations in estimating project outcomes include issues of time scale and differentiating project impacts from parallel changes in other significant factors, as discussed in Section 4.3. Process Evaluation Evaluation of the process by which the TCSP plan or project was produced or implemented can serve a number of useful functions. Process evaluation can identify reasons for success or failure of the plan or project as well as specific strategies and tactics which were most effective. Evaluation of specific aspects of the process, such as who participated and their respective roles, also can help indicate how likely the product is to achieve success. For example, extensive participation of a variety of affected parties or groups may mean that the project is more likely to be successful, since potential obstacles and stumbling blocks can be resolved. A number of techniques can be used to gather information for evaluating the process, including:  Direct observations of process activities;  Interviews or discussions with facilitators of the process and process participants;  Reviews of documents, including process schedules, timelines, and workplans; participation and attendance lists; meeting agendas and minutes; plans and reports produced; and letters of support. Questions that can be asked as a basis for evaluating the process include:  Who participated (organizations, titles, level of authority to act on behalf of organization, etc.);  Who did not participate; whether they (a) opted out or (b) were not invited; and why;  What the participants roles were (e.g., attend meetings, read and critique materials, produce data/reports, partners in planning, partners in decision- making, etc.);  What the process for planning was: o Who established the agenda and how it was done; o Schedule and organization of meetings and other actions; o When and how were goals established; o Development of background information and supporting analysis (what was performed; how was it used in supporting plan development or project selection); o Process for reaching decisions (discussion and vote, discussion to agreement, recommended options and a decision by others, consultation with others followed by decision, etc.); o Support in documentation of process for goals and decisions; o What factors influenced the decision;  Relationship of process to existing planning processes and activities, including the metropolitan and statewide transportation planning process;  Substantive issues covered;  Timeframe of substantive issues (current focus, future - short-term, future - long-term);  Actions taken;  Legitimacy to implement plan or project: o Legal authority; o Political legitimacy; o Financial resources identified. Documenting answers to the above questions can determine the degree to which the process met its defined goals and objectives. Some process-related goals and objectives for the TCSP program, as well as associated performance measures, are shown in Table 2. Local agencies may also hold other goals and objectives for activities carried out under the TCSP program. Documenting the answers to these questions also will help in identifying circumstances or actions that influenced the level of success of the final product. Table 2. TCSP Process Evaluation Sample Goals/Objectives and Performance Measures Goal/Objective Performance Measures Involvement of non- traditional partners Number/type of groups involved:  Public utility operators Social services agencies  Community groups  Environmental organizations  Non-profit organizations  Public health agencies  Economic development agencies  Private land development organizations  Home builder associations  Real estate investors  Zoning commissions  Other public or private groups Contribution (policies, actions, ideas) and commitment (financial and other resources) of each group Consistent with Statewide and MPO planning process Construction projects are ultimately included in approved State or MPO Transportation Improvement Program Project included in air quality conformity analysis if required Changes to State or MPO plans are coordinated with other affected jurisdictions Other demonstrated linkages to planning process Broadens scope and impact of planning process to integrate transportation, community preservation, environmental activities Number/type of interests involved:  Public sector  Community/interest groups  Private sector Elements of process/plan/project that affect or consider:  Land development planning  Community preservation  Environmental impacts  Economic development  Social equity  Private sector activities New ways of doing business Evidence of common goals Achieves stakeholder commitment and buy-in Endorsement of results by:  Participants  Other affected parties Participation of stakeholders in plan development:  Attendance/participation at meetings  Other participation/communication Individuals/organizations/groups not supporting plan Commitment to implementation (through responsibility, funding, etc.) Process led to learning and innovation New approaches taken Innovative ideas generated New relationships formed (formal or informal) for implementation Process is directed at achieving desired TCSP outcomes Development of background information and analysis to support plan development or project selection:  Empirical evidence based on implementation of other, similar plans or activities  Modeling/forecasting  Surveys Other qualitative assessment of potential impacts Evidence of consideration of this information in planning process Development and implementation of evaluation plan and activities [List of Tables] Evaluation of improved linkages to metropolitan or statewide planning process, as encouraged by TEA-21, is of particular importance, although this may not be relevant to all TCSP grants. As applicable, grantees might evaluate their ability to improve connections through the funded project with the broad metropolitan or statewide transportation planning processes at the center of TEA-21. Linkages to the planning process can be flexible, and could be demonstrated, for example, by:  Contributing to alleviation of priority area transportation and related problems identified in the 20-year plan and any visioning  Applying performance indicators, possibly including those in transportation management systems;  General or specific support from a public involvement process;  Development through collaborative partnerships, for example, involving the MPO, state transportation and environmental agencies, city planning agencies, transit, or non-traditional partners; or  Projecting life-cycle costs developed through financially constrained planning. With respect to the public involvement process for transportation planning in particular, federal guidelines suggest the following desirable outcomes of public involvement (Reference: A Guide to Metropolitan Transportation Planning Under ISTEA: How the Pieces Fit Together. U.S. Department of Transportation, http://www.fhwa.dot.gov/tcsp/evalguide2.html#lot Washington, D.C., 1995.):  Informed and involved citizens with access to public records and the decision-making process;  A planning approach that is proactive and open to early participation by all;  A process that not only encourages broad public participation but also considers and responds to public input;  Appropriate and early interagency consultation in air quality non-attainment areas;  Ample opportunity for public comment when the final plan or TIP differs from the draft. Product Evaluation Product evaluation focuses on what was produced by the planning or implementation activity. A description of the project as it was actually produced or implemented can serve as an interim step in identifying the likely outcomes or impacts of the project. Some general questions that can be asked about the product include:  What was the product of the activity, and how does it compare to what was originally planned?  What did the product accomplish?  Why does it matter - what impact did the product make, with respect to both the defined project objectives and the overall objectives of the TCSP program?  To whom does it matter - who is impacted?  Is there anything innovative in the project? What was done that had not been done before?  What was learned that wasnt already known? What was the added knowledge and how important is it?  How can the lessons learned from this project be generalized to other situations? Evaluation of the product of a TCSP activity will differ significantly depending on whether the activity is a planning or implementation grant. In the case of an implementation activity, product evaluation can focus on describing what was actually built, or what service was developed, and why it is significant. In the case of a planning activity, product evaluation will focus on the content of the plan, agreement, etc. (e.g., what will be achieved if the plan is implemented or the agreement carried …