Fluent Essays

The four readings issues associated with how to study the implementation of public policies or programs:  The relationship of politics to public policy?  Whether and how to address cooperation usually between or among different government agencies or departments? Uncertainty (what we know and don’t know)? Bias associated with evaluating public policies or programs.  discuss these four parts that make up the study of implementation?  show how they are inter-related. Help discuss the readings as part minimum of six pages When Does Cooperation Improve Public Policy Implementation? Martin Lundin Interorganizational cooperation is often considered valuable in the public sector. However, in this article it is suggested that the impact of cooperation on public policy implementation is dependent on the type of policy being carried out. It is argued that complex policies are more effectively put into practice if agencies cooperate a lot, whereas less difficult tasks are handled just as well without interorganizational cooperation. Thus, two policies within the Swedish active labor market policy are examined. The empirical test focuses on the cooperation between Public Employment Service offices and municipal labor market administrations. In agreement with the hypothesis, the findings suggest that policy matters. The implementation of one of the policies—the complex policy—is enhanced if cooperation between agencies increase. On the other hand, cooperation does not improve implementa- tion of the less complex task. The study is based on quantitative data. KEY WORDS: implementation, interorganizational cooperation, policy, active labor market policy, Sweden Relationships between authorities impinge on policy implementation (O’Toole, 2003) and it is often emphasized that public policy can be carried out better if cooperation increases among agencies. Some scholars claim that interorganizational cooperation is inherently good (e.g., Hudson, Hardy, Henwood, & Wistow, 1999; Jones, Thomas, & Rudd, 2004), although empirical evidence suggests that coopera- tion only sometimes enhances performance (Jennings & Ewalt, 1998). The purpose of this article is to enrich our understanding of policy implementation by examining when a cooperative strategy actually makes implementation output better. More precisely, it is argued that the effects of cooperation vary with the complexity of the policy carried out. An agency that cooperates with others can make use of additional resources, such as expertise and information. Activities can hopefully also be better co- ordinated. This suggests that interorganizational cooperation improves an agency’s ability to put policy into practice. On the other hand, it is difficult to work across organizational boundaries. For example, the collaborating authorities have to devote a lot of time and other resources to establishing and maintaining a productive relationship. Thus, we cannot be sure that cooperation improves implementation in The Policy Studies Journal, Vol. 35, No. 4, 2007 629 0190-292X © 2007 The Policy Studies Journal Published by Blackwell Publishing. Inc., 350 Main Street, Malden, MA 02148, USA, and 9600 Garsington Road, Oxford, OX4 2DQ. every situation. In this article, it is suggested that task complexity is a key factor that explains why the effect of cooperation on implementation output will be greater in certain situations. Basically, the benefits of cooperation are likely to more than offset the costs if the task is complex. In contrast, when the policy is not complex co- operation adds very little value, but still involves some costs. To my knowledge, this idea has previously not been thoroughly discussed and empirically evaluated in implementation research. The empirical focus is relations between different units of government. More precisely, the analysis concerns Public Employment Service (PES) offices and their relationship to local governments (municipalities) in the Swedish active labor market policy (ALMP). The PES is the main local labor market actor. But in recent years, municipalities have become an increasingly important factor (Lundin & Skedinger, 2006; Salonen & Ulmestig, 2004). The Swedish government encourages cooperation between these actors. But will implementation of labor market activities actually be improved if the PES cooperates a lot with the municipality? Is it possible to anticipate a positive effect on a broad range of policies, or is it only certain labor market activities that are affected positively? Two policies are examined. One of the policies—activities for unemployed youth—is not that complex. The other—activities for individuals with especially long spells of unemployment—is more intricate. Thus, I expect cooperation to be a more efficient strategy in the latter case. The quantitative analysis is based on recent data consisting of information from several sources. The findings indicate that coopera- tion improves the implementation of activities for individuals unemployed for an especially long time. On the other hand, cooperation does not enhance the imple- mentation of the youth policy. This indicates that complex tasks can be carried out better if a cooperative approach is employed, but that it is not reasonable to assume that interorganizational cooperation will always have a positive impact on how policies are implemented. The rest of the article is structured as follows. First, the theoretical discourse is outlined. An introduction of the research setting comes next, followed by a section that discusses methodological issues and measures. Empirical findings are reported. I conclude by summing up the results and discussing their possible implications. Implementation and Interorganizational Cooperation Local practices are not always the same as the intentions stated in official docu- ments endorsed by politicians. In addition, performance frequently varies from one local context to another. As a result, it is wise not to assume that the study of statutes, government bills, and regulations will be enough to understand what political deci- sions imply “in the real world.”1 An implementation problem occurs when a political decision is not carried out in accordance with what the decision maker wants. We assume that local practice should be in line with the elected officials’ intentions, that is, agents should follow the principal’s instructions. This perspective is easy to endorse, based on normative democratic theory. In a modern democracy, citizens freely elect representatives who 630 Policy Studies Journal, 35:4 can be held to account at the ballot box. Politicians cannot implement a policy all by themselves; they have to rely on a civil service to do this. But the citizens cannot replace the bureaucracy by casting their vote. Consequently, a prerequisite for satis- factory democracy is that politicians control and govern a civil service that respects their decisions (Sannerstedt, 2001). There are arguments in favor of another point of view, which argues that the local civil service is more receptive to local desires and needs. Discrepancies between a decision and its implementation might therefore in practice mean a “better” policy and a greater responsiveness to citizens’ wishes. Depending on which normative starting point is assumed, variation in—or insufficient—implementation perfor- mance is thus not always necessarily a disadvantage (deLeon, 1999). Nonetheless, most scholars would concur that it is troublesome if there is a gap between the law and practice (Keiser & Soss, 1998). In this article, I focus on the top-down aspect of implementation, that is, implementation as compliance. It is, however, necessary to acknowledge that other angles of approach are just as relevant in implementation research. Certain political decisions are quite easy to carry out and can be managed almost exclusively by a unitary public administration. A change in a tax rate or the level of a general welfare benefit is virtually self-implemented. But implementation is normally more complicated and involves several participants. Hence, an important component in almost every contemporary framework explaining implementation success and failure has to do with how interorganizational relationships are managed (e.g., Bardach, 1998; Goggin, Bowman, Lester, & O’Toole, 1990; Hjern & Porter, 1981; O’Toole, 2003; O’Toole & Montjoy, 1984; Pressman & Wildavsky, 1984; Winter, 2003a). An agency assigned the task of carrying out a political decision may use different techniques to implement a policy. For instance, the agency may try to cooperate as much as possible with other organizations. Cooperation (collaboration)—that is, all the interactions among organizations aimed at solving public problems by working together (cf. Smith, Carroll, & Ashford, 1995)—is one of the golden words in public- sector management. Through partnerships and other collaborative endeavors, public- sector performance is often considered to be improved. All else being equal, one would expect a public agency with access to significant resources to carry out a policy or program better than an agency which lacks resources (e.g., Keiser & Soss, 1998; Meier & McFarlane, 1996). Organizations possess resources. An agency carrying out a political decision may thus enhance its own capacity by collaborating with other organizations (e.g., Jennings, 1994). The surrounding organizations may contribute with information, and they could have staff, knowledge, money, and premises, making the business of putting ideas into practice easier. Cooperation can also mean that resource-consuming and conflicting activities that may result in socially perverse outcomes are avoided (Hooghe & Marks, 2003). Thus, it is easy to see why inter- organizational cooperation is often assumed to improve implementation. But there are a number of prerequisites. First, the potential partner must have additional resources that can be of use to the focal agency. Second, the partner must be willing to share resources. Third, there are always costs associated with Lundin: Cooperation and Public Policy Implementation 631 collaboration (e.g., Schermerhorn, 1975; Van de Ven, 1976; Weiss, 1987). Although a potential partner is willing to share valuable resources with the focal agency, coop- eration may be a complicated process. To at least some extent, all organizations have different agendas and various routines to handle things. As a consequence, interor- ganizational cooperation may be a complicated process involving bargaining, and therefore call for considerable time and other resources on the part of the agents concerned. Another cost to be considered is that the principal’s objectives—for example, central government intentions—may be put on one side when the local agents concerned are trying to reach their best joint solution. In other words, instead of pursuing central government directives, local actors may work for other goals they can agree on. In sum, the costs of cooperating might well outweigh the benefits and we should not assume that more cooperation implies that political intentions are always realized to a greater extent. Cooperation can, of course, have important implications for other aspects than implementation output. For instance, a public program might become more effective if authorities cooperate, even if implementation is not enhanced.2 Cooperation might also imply that a decision becomes more legitimate in the eyes of the target group for a certain policy. But this study is limited to the question of how to make a political decision come true in accordance with officials’ intentions. Jennings and Ewalt (1998) note that there is only anecdotal evidence that coordination—which in practice is measured as the level of cooperation—actually improves public services. Jennings and Ewalt examine the accomplishment of policy goals in employment and training services in the United States. In a prior article, Jennings (1994) indicated positive effects of coordination on the administrators’ subjective perceptions of performance. But in the study from 1998, objective outcome measures were employed as dependent variables. The analysis shows that coordination has a limited positive effect; most of the indicators were unaffected by the level of coordination. The findings suggest that although interorganizational cooperation may sometimes be a good strategy, we cannot expect it always to improve performance. There is some indication that there is a positive effect in the long run, but the evidence is not decisive. On the other hand, Hudson et al. (1999, p. 238) note that “while recognizing that there are other positions, this article takes the normative position that collaboration is generally a ‘good thing’—a stance which is consistent with the rather long history of collaboration in organization theory and public administration.” In many ways this is certainly true, but if we want to under- stand public administration and public policy implementation it must be better to improve theory on what the balance of benefits and costs would look like in various situations. This can tell us when cooperation actually is a good strategy. There seems to be a lack of research on these matters. Pressman and Wildavsky (1984) suggest that the number of actors involved in an interorganizational setting determines whether implementation will be successful; more actors mean greater probability of failure. O’Toole and Montjoy (1984) refine this argument and hold that a large number of actors makes implementation worse in cases of reciprocal inter- dependence (when actors poses contingencies for each other) and in sequential interdependence (when the output of one actor is the input of another). Contrarily, 632 Policy Studies Journal, 35:4 in cases of pooled interdependence (when actors contribute to a task without dealing with each other) and in sequential competitive interdependence (when there are actors competing to do the same thing in the implementation process) a larger number of actors improves policy output. These scholars focus on the effects of the organizational setting per se, not the extent to which actors work together, which is the focus in this article. The question here is rather whether a lot of cooperation generates a better payoff, given certain tasks. More specifically, is a high degree of cooperation especially rewarding for certain policy types? The discussion is restricted to one policy dimension that deserves particular attention: task complexity. Task complexity is defined as the product of the scope of actions and the intensity required to carry out a policy. By scope I mean the extent to which many areas of expertise are required to fulfill the goals or, put another way, “the degree to which tasks are variable and require a multidisciplinary or multidi- mensional approach” (Alter & Hage, 1993, p. 117). Intensity is a matter of how much work has to be done in order to realize ambitions; a task that requires many man- hours on the part of an agency is considered intense. A possible objection is that complexity is only about the scope. However, I think that we should consider intensity as well because the problems that a policy which has a great scope implies are multiplied if intense actions also are required. Thus, in this article a policy that has large scope and involves intense actions is considered complex. In research on interfirm partnerships, it is quite common to talk about joint task complexity as a function of the scope and depth (intensity) of interactions between firms (White, 2005). This definition parallels the one used in the present study. Most organization theory scholars define complexity in a related vein, although there are alternatives on the same theme (see, e.g., Alter & Hage, 1993, pp. 116–22). Task complexity is an incentive for cooperation (cf. Alter & Hage, 1993). A need for external resources boosts interorganizational cooperation (e.g., Van de Ven & Walker, 1984; Weiss, 1987) and complex tasks may demand resources that cannot be found within a single agency. In this perspective, actors cooperate when cooperation is needed, otherwise they do not. But the extent of cooperation between two agen- cies, A and B, is certainly not only a direct function of the task complexity of policy X that A implements. Cooperation is a consequence of the agencies’ perceived total value of working together, which may be something rather different from the actual value that cooperation adds to the task of realizing policy X in accordance with official intentions. First, the agencies do not have perfect information and cannot take in all benefits and costs of cooperation: A does not know exactly what greater value cooperation actually implies. Second, A cannot set cooperation precisely at the optimal level on its own. If B does not want cooperation to increase, A cannot do much about this. Third, the incentives and disincentives for A to cooperate with B are numerous.3 External pressure for cooperation is one example. Political representa- tives or other bureaucratic levels may exhort A to collaborate with B. In some cases, a legal mandate may require A to cooperate with B. Norms and values in both organizations are other things that determine cooperation. Moreover, working together with B can make it easier for A to accomplish other goals than those related to policy X. Accordingly, although the net benefit of cooperation for A can be Lundin: Cooperation and Public Policy Implementation 633 positive, this does not mean that policy X becomes better implemented. In sum, it is too restrictive to assume that task complexity impinges only on cooperation levels. It can also tell us something about the effect on policy output. Whereas the other factors mentioned above affect cooperation, it is only the cost–benefit analysis related to the policy being implemented that determines the effect on implementation in the specific case. So the question is now why we should assume that the effect of cooperation would be greater as policy complexity increases. In a less complex policy, it is likely that the benefits of cooperation would be quite low. In these situations, the agency’s capacity is not going to increase that much by using other organizations’ resources because the most important resources are already available within the agency. Organizations are formed to perform certain tasks and in a less complex policy the most important resources are already available. This means that increasing levels of cooperation do not add that much value. As complexity rises, cooperation provides more benefits. The agency needs more expertise, information, money, premises, and so on. Through cooperation these resources can be attained. The costs must also be considered. As complexity increases, the costs of coop- eration are also likely to increase. For instance, the risk of disputes and impediments becomes greater. But while the benefits of cooperation are very low given little complexity, there will always be costs. In fact, many costs are independent of the level of complexity. Cooperation is a dynamic process and decisions and activities within a policy are not independent of each other. Given that actors agree on some basic things, the additional problems that arise with increasing complexity will not be that high. In another context—interfirm alliances—White (2005) considers the costs asso- ciated with cooperation. Even though he claims that costs are higher given a complex task, he repeatedly notes that the benefits are often likely to be even greater. He claims that “additional costs of a more complex interface may be more than offset by the benefits possible from more extensive interaction” (p. 1388) and if the partners have relatively similar objectives and values, “the additional marginal cooperation costs resulting from greater scope or depth-related coordination costs should be more than offset by increased benefits” (p. 1395). In sum, it seems likely that in a situation characterized by low task complexity there will be very little (or no) benefit from cooperation, but some costs. These costs increase somewhat as complexity increases, but the increase will be lower than the increase in benefits. If complexity is high, the large benefits will prevail over the costs. Cooperation is therefore a more value-adding activity given complex tasks. Accordingly, we should expect the effect of cooperation on policy output to be greater given complex tasks. These arguments are perhaps not brand new, but to my knowledge they have not been discussed and evaluated empirically in the context of local policy implementation. Intergovernmental Cooperation in Swedish Active Labor Market Policy To discern whether complexity is an important variable, we need to know the effects of cooperation on the implementation of at least two policies: one complex 634 Policy Studies Journal, 35:4 task and another less complex task. Swedish ALMP provides rather good opportu- nities to examine this. Unemployment is a huge problem in most OECD countries (Martin & Grubb, 2001). Job brokering activities and labor market programs are examples of ALMPs a government can use to reduce these problems. In Sweden, the National Labor Market Administration (Arbetsmarknadsverket) implements ALMPs. Local PES offices (Arbetsförmedlingar) carry out most of the programs in practice. There is a PES in nearly all cities, and in larger cities there is usually more than one office. Although the National Labor Market Administration is clearly the most impor- tant authority managing ALMPs in Sweden, other actors are involved. One feature in recent years has been more involvement on the part of local governments in the implementation of ALMPs (e.g., Lundin, 2007; Lundin & Skedinger, 2006; Salonen & Ulmestig, 2004). The municipalities take an active part by, for example, organizing labor market programs. About 40 percent of the clients involved in programs admin- istered by the National Labor Market Administration are participating in activities in which the municipalities are involved (Lundin & Skedinger, 2006). A lot of the municipal activities are targeted at social welfare benefit recipients (Salonen & Ulmestig, 2004) and at unemployed youth (Carling & Larsson, 2005). The content of the activities varies, but job-search assistance and work practice are quite common. The municipalities have a premier position within the Swedish political system. For example, they have the constitutional right of self-government; they are comparably large; they can decide on their own organization; and their incomes come mostly from a proportional income tax they can set freely (see, e.g., Bäck, 2003; Gustafsson, 1999). The municipalities provide most of the services of the Swedish welfare state. For instance, they supply day care, care of the elderly, social welfare services, introduction of immigrants into Swedish society, and primary education. Several of these policy areas—the most obvious example being social welfare services—clearly intertwine with ALMPs. This suggests that collaboration with the municipality could be beneficial for the PES. In the analysis, effects of cooperation between the PES and the municipality on implementation of ALMPs are studied. The study concerns two policies. One of the policies is more complex. The positive effect of cooperation is anticipated to be greater in this policy. In August 2000, the Swedish government launched a new labor market program called the Activity Guarantee (Aktivitetsgarantin). The target group was individuals who had been unemployed for a considerable time period. By means of intense job-search assistance and close monitoring, persons who were, or who risked being, long-term unemployed were to be given a place in the program. All traditional ALMPs could be used within the Activity Guarantee. Accordingly, the content of the activities varies to a large extent. The political ambition was that jobseekers should be enrolled in the program after 27 months of unbroken PES registration, at the latest. What distinguishes the Activity Guarantee from other labor market programs is its intensity. All activities are assumed to be full-time activities and the participants should meet their personal PES supervisor on a regular basis, and more frequently than before they entered the program. Since the persons participating in the Lundin: Cooperation and Public Policy Implementation 635 guarantee are “hard cases” and often have multiple problems, the government calls for intense treatment and encourages PES offices to collaborate with other local actors. The local governments are, in practice, the PES offices’ most common coop- eration partners (Forslund, Fröberg, & Lindqvist, 2004). In 1994, the Swedish government declared that no young person should remain unemployed for more than 100 days. The government regarded long-term unem- ployment as devastating for future labor market prospects. Young people are defined as long-term unemployed if they are unemployed for more than 100 days. Thus, the ambition was that every individual of below 25 years of age should be offered a labor market program if they were unable to find a job within three months of registration at the local PES. The central government encouraged the municipalities to take part in actions to reduce long-term unemployment among young people in different ways. For instance, in 1995 the government introduced the Municipal Youth Program (Kommunala ungdomsprogrammet) and three years later the UVG-guarantee (Ung- domsgarantin). The PES offices are expected to collaborate with local governments in both programs (Carling & Larsson, 2005). Recall that task complexity is defined in terms of the scope and intensity of a policy. The Activity Guarantee is more intense and has greater scope than the youth policy. Thus, it is a good representative of a complex policy (and this applies even if complexity is defined only in terms of the scope). First, consider the client groups. In the Activity Guarantee, the clients are comparatively old and they are obviously not particularly attractive on the labor market because they have been looking for a job for such a long time. It is also reasonable to assume that a relatively large part of the clients have multiple problems because the program is directed at those with the weakest position on the labor market. In comparison, youth clients have not been unemployed for such a long time. Consequently, they are probably more motivated, more homogenous, and easier to handle. Accordingly, the treatment of clients in the Activity Guarantee needs more areas of expertise and more intense activities. Second, although various youth activities are probably not so simple to imple- ment, the Activity Guarantee is more complex. The activities are supposed to be rigorous and carried out in small groups. Contacts between clients and personal supervisor are also expected to be more frequent than is generally the case. In short, these activities are more intense. Moreover, the government’s intention was that the program should be flexible and contain various activities suited for the individual client. This means that the Activity Guarantee is intended to have greater scope than other ALMPs. On balance, it is safe to conclude that the Activity Guarantee is much more complex than the youth policy, even though some youth activities can also be said to contain complexities. Thus, I predict a greater effect of cooperation on imple- mentation output in the Activity Guarantee. The selected research setting is suitable for several reasons. First, unemployment is a large societal problem and a lot of resources are allocated to diminishing the unemployment rate all around world. Studies that can bring some clarity to what is going on when ALMPs are put into practice are therefore important. Second, this study is focused on the relationships between the same types of actors who imple- ment the same policies around Sweden. This implies that features of the agencies and 636 Policy Studies Journal, 35:4 the policies that do not vary can be held constant. From a methodological point of view this is very beneficial. Third, Swedish ALMPs can be seen as a “critical case,” meaning that here, if anywhere, we ought to expect that cooperation will improve implementation on a broad range of activities. O’Toole (1983) claims that there are very high information and expertise requirements in ALMPs. Interorganizational cooperation is therefore likely to be a good strategy for gathering the necessary resources. Furthermore, there are obvious connections between the responsibilities of the PES offices and the municipalities. This means that we have good reasons for believing that cooperation is a good strategy when it comes to both policies. Method and Measures Whereas quantitative approaches have become increasingly common and more sophisticated in the implementation literature from the United States (e.g., Brehm & Gates, 1997; Daley & Layton, 2004; Meier & McFarlane, 1996), studies based on large-N data in the European context are rare. This study is based on European quantitative data and is therefore a significant contribution.4 The PES offices are the unit of analysis and cross-section data covering 2003 is used. Questionnaires were distributed to the chief managers of all PES offices in February 2004: 268 managers answered the questionnaire, which implies a response rate of 75 percent. An analysis of the nonresponses showed no noticeable difference between respondents and nonrespondents on background characteristics. Register data from the National Labor Market Administration supplement the survey. Lastly, official statistics, in … The Policy Studies Journal, Vol. 33, No. 4, 2005 0162-895X © 2005 The Policy Studies Journal Published by Blackwell Publishing. Inc., 350 Main Street, Malden, MA 02148, USA, and 9600 Garsington Road, Oxford, OX4 2DQ. 559 Facts and Myths about Research on Public Policy Implementation: Out-of-Fashion, Allegedly Dead, But Still Very Much Alive and Relevant Harald Saetren Despite several decades of research on public policy implementation we know surprisingly little, not only about cumulative research results, but also about several other key aspects of this research field. This article tries to amend these deficiencies by presenting the results of a comprehensive literature survey. Its main purpose is to challenge, revise, and supplement some conventional wisdom about implementation research. A second motivation is to lay the foundation for and initiate a much needed synthesis of empirical research results. The main results are: The overall volume of publications on policy implementation has not stagnated or declined dramatically since the mid 1980s as is commonly asserted. On the contrary, it has continued to grow exponentially through the 1990s and into the twenty-first century. Even more surprising is that a large number of publications are located outside the core fields. Hence, the literature is substantially larger and more multidisciplinary than most com- mentators realize. Doctoral dissertations are the most ignored, but probably the richest, largest, and best source of empirical research results. Tracing the origin as well as the location of the disciplinary and geographical cradle of implementation studies must also be readjusted significantly. The ethno- centric bias of this research field toward the Western hemisphere has been, and still is, strong and some policy sectors are given much more attention than others. Although positive in many ways, the pre- dominant multidisciplinary character of implementation research still poses some serious problems with respect to theory development. Thus, I discuss whether a resurgence of interest in policy imple- mentation among policy scholars may already be occurring. Finally, I suggest that the time is long overdue for efforts to synthesize research results in a more rigorous scientific manner than has hith- erto been done. KEY WORDS: public policy implementation, bibliometric survey, origin, size, development, discipli- nary structure, relevance, research agenda Introduction Science must begin with myths and with the criticism of myths. —Karl Popper As a new field of investigation continues to grow and expand over several decades, it becomes important to take stock of what has happened and what has been achieved. If not, the knowledge and insights gained will become increasingly 560 Policy Studies Journal, 33:4 fragmented and inaccessible (Cooper & Hedges, 1994; Hunt, 1997). Quite a few state-of-the-art reviews on public policy implementation research have been pub- lished over the years (Alexander, 1985; Barrett, 2004; Goggin et al., 1990; Hill, 1997; Hill & Hupe, 2002; Ingram, 1990; deLeon, 1999a; deLeon & deLeon, 2002; Lester & Goggin, 1998; Lester et al., 1987; Linder & Peters, 1987; McLaughlin, 1987; Matland, 1995; May, 2003; O’Toole, 1986; 2000; 2004; Palumbo & Calista, 1990; Ryan, 1996; Sabatier, 1986; Schofield, 2001, Schofield & Sausman, 2004; Sinclair, 2001; Winter, 1990; Winter, 2003; Yin, 1982). They have all, in their time, pointed to important contributions and shortcomings in implementation research and offered valuable advice with respect to improvements. Nevertheless, surprisingly little is known about several key features of this type of research which will be explicated further below. Another related problem is that most reviewers fail to explain how they arrive at their many factual statements and interpretations. This is not uncommon among review articles in social sciences according to Jackson (1980, 444), but it precludes judgments regarding validity of their assertions and conclusions. For the few exceptions, see O’Toole (1986), Hill and Hupe (2002), and Sinclair (2001). As a result, a certain story about implementation research—its origin, discipli- nary foundation and development pattern, and so forth—has been retold so often that it has now become institutionalized in the form of conventional wisdom. The story has acquired this stature because it has never been challenged. Some key ele- ments of this conventional wisdom were created by Pressman and Wildavsky (1973), the presumed founding fathers of this type of research, at a time when it was much more difficult to keep track of research publications and their contents. These found- ing fathers made their claims in good faith. However, given the superior research tools and data resources at hand today, there is now little excuse when one uncrit- ically perpetuates that dubious account. It is time to set the record straight! What parts of the institutionalized story about implementation research are correct and which are not? The story is also incomplete in certain respects. That is, there are some important aspects of this research to which nobody has claimed knowledge. The primary aim of this article is to provide a more complete, factual account about implementation research at the outset of the new millennium. This entails recounting its origin, evolution, recent development, size, disciplinary foundation, and internal structure. The next two sections further elaborate the research questions and related research methodologies. Research Questions The most frequently debated issue of the 1990s has been the question of the state of implementation research. Is this a field of investigation where the volume of research has declined to the extent that it needs revitalization as many contemporary commentators have suggested (Barrett, 2004; deLeon, 1999, 2002; Lester & Goggin, 1998; Schofield, 2001; Schofield & Sausman, 2004; Winter, 1999, 2003)? Perhaps the opposite is the case as others had predicted at the outset of the previous decade (Goggin et al., 1990)? Saetren: Facts and Myths about Implementation Research 561 There is no doubt that the volume of research on public policy implementation has grown substantially since the mid-1970s, but exactly how large is it at the begin- ning of the new millennium? Can this question be answered at all, or even approx- imately? If so, what is the answer? Political science scholars and their colleagues in the subfields of public admin- istration and public policy share the sacred notion that policy implementation research is primarily their domain. Does this conviction withstand closer examina- tion? If not, what are the other academic disciplines that make important contribu- tions to implementation research? The degree of concentration or fragmentation regarding the structure of scholarly communication is essential to efforts at accumulation of knowledge in any field of investigation (Borgman, 1990). The greater the number of articles published in a few core journals the more favorable the condition should be for synthesizing that research literature and vice versa. How does implementation research fare in this vital respect? It is customarily asserted that virtually no research under the explicit label of implementation (or implementing) was carried out before the publication of Press- man and Wildavsky’s seminal book titled Implementation in 1973. Is this really true? In the same vein it might be reasonably assumed that implementation research orig- inated on the West Coast in the San Francisco Bay Area at the University of California Berkeley campus, where Wildavsky was an influential teacher and researcher during the 1960s and 1970s. Do the facts corroborate this? What policy issues do implementation scholars study and how do they match the supposedly salient ones in our societies? This question also has a geographical dimension. There is no doubt that implementation research from the beginning was predominantly a North American enterprise. The question here is to what extent the focus has shifted to other regions of the world and their policy challenges as the research field matured? Another key issue is the relevance of implementation research, and why it became unfashionable. Finally, one wonders if some policy scholars’ recent calls for the revival of implementation research are having any effect. If so, what direction should reenergized research efforts take? These, then, are the main questions this article purports to answer. Data and Methodology As most reviewers of implementation research have had little to say about their data sources and methodologies, I find it necessary to rectify this neglect. This study has relied on many data sources and research methods. Nonetheless, digitalized, scientific literature databases presently available at most universities as well as a related search instrument called bibliometrics,1 proved to be the most important by far. Three databases in particular, the Expanded Social Science Citation Index, World Catalogue, and the Digital Dissertations (Dissertation Abstracts), were utilized because they are interdisciplinary and together they cover all major types of publications. They contain vast amounts of information, but of variable quality.2 Still, there are 562 Policy Studies Journal, 33:4 some pieces of relevant information concerning publications listed in the databases that are not consistently registered. This applied especially to: (i) abstracts; (ii) authors’ background (professional and national), whether their publications are empirically oriented or not; (iii) research methodology employed; (iv) policy sector studied; and (v) regional focus. These deficiencies constituted the main rationale for creating our own electronically indexed and retrievable literature data file based on the Cardbox software program. Conceptual and practical matters related to searches in the three literature data- bases are discussed in the section below that deals with the size estimate issue where they are of particular relevance. Research Results The Alleged Demise of Implementation Research In the early 1990s scholars like Goggin et al. (1990, 9) were quite upbeat about the future of implementation research, proclaiming interest “is likely to grow during the 1990s and continue well into the twenty-first century. In fact, the field of public policy in the next decade will very likely be defined by the focus on implementa- tion. The nineties are likely to be the implementation era”(italics in original). This optimistic view, however, lost ground during the latter part of the 1990s as it dawned upon an increasing number of scholars that it might be misplaced. Saetren (1996) presented some preliminary data to the effect that a pronounced and steady decline in research publications on policy implementation from 1984 up to 1995 had occurred. At approximately the same time Hill (1997) and Lester and Goggin (1998) arrived at the same conclusion. Other commentators in recent years have also var- iously subscribed to the thesis of decline (Barrett, 2004; Hill & Hupe, 2002; O’Toole, 2000, 2004; Schofield, 2001; Schofield & Sausman, 2004; Winter, 2003). The facts of the matter would then seem to have been settled fairly and squarely in favor of the decline thesis. Not quite! It turns out that we can all be simultaneously right and wrong, depending on how we qualify our statements. If the total picture is examined, that is, all publications jointly (Figure 1), the data overwhelmingly refute the decline proposition. Although publications peaked and declined somewhat during the mid-1980s, their numbers had bounced back well before the end of that decade. Moreover, the trend continued unabated, even growing beyond previous levels through the 1990s and into the twenty-first century. This exponential growth amounted to the near doubling of publications during the period after 1984 compared with the previous period (Table 1). Although the output of some publications increased more rapidly than others, over time none of these trends lend support to a demise or decline thesis. This constitutes a paradox. How can it be that so many scholars us came to believe that the interest in implementation research had declined to an alarmingly low level? How did the exponential growth of research literature reported here escape our attention? The next section, which explores the disciplinary structure of implementation research, provides some clues to an answer. Saetren: Facts and Myths about Implementation Research 563 Exploring the Disciplinary Foundation of Implementation Research Many authors have long believed that studies of policy implementation are mostly carried out by those who have an interest either in political science in general, or in public administration and public policy in particular. This notion may seem to follow logically from the fact that a more fundamental understanding of implementation as an integral part of the public policy process must draw upon and bridge insights from these fields of knowledge (Winter, 2003). This is also our justi- fication for defining journals in these three subfields as the core. The heretical ques- tion raised here is: Does such a core really exist? The implication of this credo is that we should expect a clear majority of arti- cles on policy implementation to be published in core journals. Table 2 clearly shows that such is not the case at all. Articles published in noncore journals far outnum- ber those appearing in core journals. An alternative and more modest expectation can be derived from one of the so-called laws of bibliometrics, namely Bradford’s law. It states that journals in any field of investigation can be divided into three main categories, each containing roughly the same number of articles: (i) a nucleus of rel- atively few journals on the subject publishing approximately one-third of all arti- cles, defined as the core; (ii) another group consisting of many more journals that produces approximately the same number of articles, referred to as near-core; and (iii) an even more distant category of a still larger number of journals—the truly Table 1. Publications by Type and Two Time Periods Type of Publication 1933–84 1985–2003 Total Articles 1,094 2,429 3,523 Books and chapters 323 682 1,005 PhD dissertations 1,091 1,682 2,773 Total 2,498 4,803 7,301 Note: Absolute values are used for the data presented in this table. 0 50 100 150 200 25 N u m b er Year 0 300 350 400 1 9 3 3 1 9 3 8 1 9 4 3 1 9 4 8 1 9 5 3 1 9 5 8 1 9 6 3 1 9 6 8 1 9 7 3 1 9 7 8 1 9 8 3 1 9 8 8 1 9 9 3 1 9 9 8 2 0 0 3 Article B o o k s Chapters PhD dissertations Total Figure 1. Publications on Policy Implementation by Type and Year. 564 Policy Studies Journal, 33:4 noncore—which accounts for the last third of articles (Potter, 1988). To test this thesis I divided all journals registered in a database into these same categories.3 The data (Table 2) even repudiate this moderated expectation of Bradford’s law. Only in 1980 (Table 3) did I come close to attaining the law’s prediction, as 30 percent of all articles published that year were actually published in core journals. It is, however, clearly an exception to the general rule. Only 13 percent of all implemen- tation articles between 1942 and 2003 were published in core journals. Another 14 percent appeared in the near-core category. Doctoral dissertations reveal a similar pattern (not shown in Table 2) as approximately 15 percent of all are classified as belonging to the core fields of political science or public administration. This means that about three-fourths or more of these two types of publication were located outside the core. What academic disciplines, then, constitute the noncore group? It turns out that education and health journals each account for 15 percent of all implementation arti- cles (Table 2). Journals devoted to law, economics, and the environment are also impor- tant as they jointly account for another 14 percent. The education profession figures much more prominently among doctoral dissertations, as close to two-thirds have this affiliation. These data attest to the pronounced multidisciplinary character of research on public policy implementation and a surprisingly modest contribution of scholars from the core fields. They also help explain why so many of us have been oblivious to the continued rapid growth of implementation research during the 1990s: it happened disproportionately in other academic fields and in a bewil- dering array of different and largely peripheral journals (see Table 4) to which we could not pay much attention, even if so desired. There are, in other words, good excuses for our previous ignorance. Even looking at the core journals, the decline notion cannot be sustained. The volume of articles in these journals did peak during the mid-1980s and subsequently stabilized at a somewhat lower level (Figure 2). It would, however, be rather far- fetched to claim, as some do, that the whole research field has virtually disappeared. In fact the number of articles in core journals almost doubled during the period from 1985 compared with the previous period. Why then is the notion of the demise of implementation research so prevalent in spite of facts that clearly suggest the contrary? Table 2. Articles by type of journal and time periods Type of journal 1948–1984 1985–2003 Total Core 15% (159) 12% (295) 13% (454) Near core 9% (97) 16% (379) 14% (476) Non-core 76% (838) 72% (1,755) 73% (2,593) Total 100% (1,094) 100% (2,429) 100% (3,523) Non-core specified: Health 12% (126) 16% (384) 15% (510) Education 17% (178) 14% (330) 15% (508) Law 12% (131) 5% (131) 7% (262) Environment 4% (41) 6% (144) 5% (185) Economics 3% (31) 3% (74) 3% (105) Other non-core 29% (331) 28% (692) 29% (1,023) Note: Absolute values are in parentheses. Saetren: Facts and Myths about Implementation Research 565 Table 3. Several Measures of Fragmentation of Scholarly Communication in the Journal Literature on Public Policy Implementation for Selected Years 1970 1975 1980 1985 1990 1995 2000 Number of articles in core journals 1 6 30 20 12 17 16 Number of different core journals in 1 5 14 10 10 13 14 which they were published Journals to articles ratio in core 1 : 1 83% 47% 50% 83% 77% 88% category Number of articles in noncore 22 31 70 67 86 117 137 journals Number of different noncore 22 27 64 60 78 110 120 journals in which they were published Journals to articles ratio in noncore 1 : 1 87% 91% 90% 91% 94% 88% category Percentage of all articles in core 4% 16% 30% 23% 12% 13% 11% journals 0 50 100 150 N u m b er Year 200 250 300 1 9 3 3 1 9 3 9 1 9 4 4 1 9 4 9 1 9 5 4 1 9 5 9 1 9 6 4 1 9 6 9 1 9 7 4 1 9 7 9 1 9 8 4 1 9 8 9 1 9 9 4 1 9 9 9 Core Noncore Figure 2. Articles on Policy Implementation Published in Core and Noncore Journals by Year of Publication. Table 4. Articles on Public Policy Implementation in Core Journals by Type of Journal and Time Period 1953–79 1980–84 1985–89 1990–94 1995–99 2000–03 Total Political science 22% (12) 17% (18) 17% (18) 20% (12) 21% (16) 23% (13) 20% (89) Public policy 46% (25) 65% (68) 48% (50) 44% (26) 38% (29) 32% (18) 47% (216) Public administration 32% (17) 18% (19) 35% (37) 36% (21) 41% (31) 45% (25) 33% (150) (N) = 100% (54) (105) (105) (59) (76) (56) (454) Note: Absolute values are in parentheses. 566 Policy Studies Journal, 33:4 The paradox must also be explained by the surprising fact that implementation research, despite its resilience in quantitative terms, went out of fashion among policy scholars after a relative short peak period that lasted only through the first half of the 1980s. It is symptomatic that the last symposium on policy implementation research was published in Policy Studies Review in the late 1980s.4 Even stronger and more convincing support for this interpretation is provided by data on the number of book reviews on policy implementation in journals included in the Social Science Citation Index. Fifty-five percent of all book reviews (Table 5) appeared during the relative short period from 1980 to 1985 and only 16 percent date after 1990. We have already seen how the declining interest in implementation studies from around the mid-1980s coincided with a temporary stagnation and drop in publication output. More interestingly, the fact that publication volume returned to previous levels toward the end of the 1980s and has continued unabatedly ever since at even a higher level did not mean that implementation research had once again become fashionable among policy scholars. Thus, Goggin et al. (1990) were simultaneously both right and wrong. Imple- mentation research continued to grow as they predicted during the 1990s and into the twenty-first century. The field of public policy during the same period however, has not been defined by its focus on implementation. On the contrary, even leading policy scholars came to regard implementation focus as an intellectual “dead end” and has turned their attention toward other topics such as policy change, policy net- works, or governance. Indeed, implementation studies became so unfashionable that even those who still considered them important mistakenly thought they had essen- tially disappeared. This erroneous notion was exacerbated by a myopic disciplinary conception of the research field. Fortunately, as I have pointed out, the interest in policy implementation studies has been profound and widespread enough to defy narrow disciplinary boundaries. Thus, its fate has been insulated from the impact of whimsical fashion trends among scholars in any particular academic field, such as political science. The Structure of Scholarly Communication The previous analysis suggests that the structure of scholarly communication in implementation research is quite fragmented. To corroborate this we need to take a closer look at the ratio of articles to the number of different journals in which they were published (fragmentation ratio). Our findings (Table 3) lend strong support to the fragmentation hypothesis. At the alleged infancy of implementation studies in 1970, the 23 articles were published in an equally different number of journals (i.e., Table 5. Reviews of Books with Policy Implementation as the Key Word in Social Science Citation Index Journals by Time Period Time Period 1972–79 1980–85 1986–90 1991–95 1996–99 2000–03 Total Book reviews 7% (6) 55% (48) 23% (20) 8% (7) 2% (2) 6% (5) 101% (88) Note: Absolute values are in parentheses. Saetren: Facts and Myths about Implementation Research 567 a perfectly fragmented publication pattern). Only one article was published in a core journal. The closest we came to the opposite pattern, and that implied by Bradford’s law, was in 1980 when one (core) journal (Policy Studies Journal) accounted for 18 percent of all articles published that year. Nevertheless, even in that year, the one hundred articles published appeared in 78 different journals; 14 of which were of the core type. This fragmented publication pattern obviously poses great challenges for any efforts at systematic knowledge accumulation. A Closer Look at the Core The core consists of close to 60 different journals that collectively published 454 implementation articles over a 60-year time period (see Appendix 1). Approximately half of these articles were published in public policy journals, one-third in public administration journals, and one-fifth in political science journals (Table 4). It is note- worthy that the position of public administration journals relative to policy journals has been reversed during the 1990s compared to earlier. This pattern seems to reflect a growing interest in implementation research among editors and contributors to public administration journals during the 1990s. The emergence of the Journal of Public Administration Research and Theory in the late 1980s is but one of several indi- cators of this trend. The relatively modest but stable fraction of articles published in political science journals is also worth noticing. Two policy journals—Policy Studies Journal and Policy Studies Review—published the largest share of core articles followed by the Public Administration Review. These three journals alone published 30 percent of the total. At the other end of the spec- trum we have the prestigious American Political Science Review with only three arti- cles, none of which are cited frequently. The documented explosive growth of implementation research, even in recent years, invites the intriguing question of what its present size might be. The next section attempts to provide an approximate answer. What Is the Size of the Implementation Literature? Few, if any, serious efforts have ever been made to estimate the total volume of research on public policy implementation, and for good reason: it is a very demand- ing and difficult task. Some have even concluded that it is a futile endeavor (Hill & Hupe, 2002). To this I would counter that even having only an approximate esti- mate is better than having no clue at all. Furthermore, as others have emphasized, knowing something about the scale of the total universe of publications in any research area is a prerequisite as well as the starting point for any serious synthe- sizing efforts (Cooper & Hedges, 1994). There appears to be only three earlier attempts to assess the volume of research on public policy implementation (Hill & Hupe, 2002; O’Toole, 1986; Sinclair, 2001). They were, however, as far as we can ascertain, limited to a particular type of publication and time period. I know of no systematic attempt to document the volume of several types of publication over an extended time period. 568 Policy Studies Journal, 33:4 The universe of any research literature is located in a three-dimensional space. First, what concepts are most appropriate to identify the relevant literature? Second, what types of publications are to be searched with chosen key words? Third, what is the time frame for such searches? In this study, implementation as title-word was the preferred choice since Pressman and Wildavsky were so instrumental in populariz- ing the use of this term with regard to studies of policy execution.5 Although this choice can be challenged,6 it is justified as follows. First, the term implementation helps to establish a well-defined body of literature that can be retrieved by others and subjected to replication and critical examination. Second, I think it is reason- able to assume, as others have done before me (Sinclair, 2001, 81), that those who want to make a contribution toward the understanding of policy implementation are more prone to using either implementation or implementing as title words than those who do not share this goal. The latter group is obviously more peripheral to our mapping enterprise than the former. Publications vary from journal articles, books, book chapters, doctoral dissertations, reports, to research papers to name the most common ones in the reference literature. We chose to focus primarily on the first four mentioned above. The time frame was extended as far as the databases allowed. I found approximately 7,300 English pub- lications using implementation or implementing as a title word, after eliminating the most obviously irrelevant ones for our analysis.7 Table 1 shows their distribu- tion by types of publication: 548 books, 432 book chapters, 3,523 journal articles, and 2,773 doctoral dissertations.8 The most surprising in this respect was the large number of doctoral dissertations, both in absolute terms and relative to the two other types of publications. They are obviously an unexplored and untapped, rich source for future efforts to synthesize research results. To conclude, it is impossible to pinpoint with much precision the true size of research publications on policy implementation. Any size estimate critically depends on a number of steps and related judgment calls taken by the researcher during the search procedures.9 In that sense it remains an elusive utopian ideal. Nonetheless, I think our database is sufficiently large to ensure a reasonably approx- imate representation of the policy implementation literature as a whole. Now it is time to turn to the question of the origin of this elusive universe. When Did Implementation Research Start? The truism that Pressman and Wildavsky (1973) pioneered the more explicit use of implementation as a key term was in no small part created by the authors them- selves in a postscript chapter, Appendix 2, titled “Use of ‘Implementation’ in Social Science Literature”: There is (or there must be) a large literature about implementation in the social sciences—or so we have been told by numerous people. None of them can come up with specific citations to this literature but they are certain it must exist”. . . . “It must be there, it should be there, but in fact it is not” . . . “How shall we persuade others that it is fruitless to look for a literature that does not exist? . . . (p. 166) Saetren: Facts and Myths about Implementation Research 569 It has since been pointed out that many earlier scholars had studied the imple- mentation of policies using other labels. Few, however, have … The Implementation of Public Policy: Still the Missing Link Robbie Waters Robichau and Laurence E. Lynn Jr. Although theories of public policy and theories of governance both seek to establish relationships between policymaking and its consequences, they do not complement each other very well. Public policy models tend to de-emphasize that which governance theories tend to emphasize: the influence on government performance of implementation, broadly described as the actions taken by those engaged in administration (including managers at all levels, those engaged in service delivery, and third-party agents) after a policy has been lawfully promulgated by elected officials and interpreted by the courts. A comparison of a recently developed theory of public sector performance with several prominent theories of policymaking suggests that multilevel governance theories can supply what continues to be the missing link in public policy theories. At the same time, governance theories might be enriched by the process modeling of public policy theories. Introduction The increasing use of “governance” as a conceptual frame for research on the determinants of government performance has produced valuable insights into causal relationships among public choice processes, public management, service delivery, and citizen and stakeholder assessments and reactions. Paralleling these efforts, public policy theorists have developed a variety of models to depict relationships between policymaking processes and their outputs and outcomes. Although both types of research seek to relate policymaking to its consequences, they do not complement each other very well. Public policy models tend to de-emphasize that which governance theories tend to emphasize: the influence on government perfor- mance of implementation through administrative systems, broadly described as the actions taken by public managers at all levels, those engaged in service delivery, and third-party agents after a policy has been promulgated by elected officials and interpreted by the courts. This article offers a preliminary consideration of how theories of governance and of public policy might better complement each other. We juxtapose a theory of Prepared for presentation at the Next Generation Policy Workshop in Norman, Oklahoma, February 27–29, 2008. The Policy Studies Journal, Vol. 37, No. 1, 2009 21 0190-292X © 2009 Policy Studies Organization Published by Wiley Periodicals, Inc., 350 Main Street, Malden, MA 02148, USA, and 9600 Garsington Road, Oxford, OX4 2DQ. government performance that features the influence of administrative systems on outputs and outcomes (Forbes, Lynn, & Robichau, 2008) with some of the major public policy theories found in Sabatier’s edited volume, Theories of the Policy Process (Sabatier, 2007). We will argue that, because they do not conceptualize the distinction between policy outputs and policy outcomes, public policy theories tend to slight the administrative processes that constitute implementation. Erwin Hargrove argued in 1975 that implementation was the missing link in the study of public policy (Hargrove, 1975). As we see it, that link is still missing. We will show how even a parsimonious theory of public sector performance framed by a logic of governance (LOG) can provide the missing link. The discussion will proceed as follows. The next section explains why and how an “LOG” framework was used to develop a theory of public sector performance. The description of this theory will be used to provide insights into how and why government produces its outputs and outcomes. The article will then analyze several public policy theories in relation to what they are missing and neglecting to study, that is, the failure to distinguish between outputs and outcomes and the disregard of administrators’ and administrative systems’ impacts on policy outcomes. We will conclude with a discussion of how using governance theory can advance public policy theorizing and suggest that governance theories might be similarly enriched by public policy theories. A Theory of Public Sector Performance Public governance is defined in this article, following Lynn, Heinrich, and Hill (2001, p. 7), as “regimes of laws, rules, judicial decisions, and administrative prac- tices that constrain, prescribe, and enable the provision of publicly supported goods and services” through formal and informal relationships with third parties in the public and private sectors. Using concepts from positive political economy, Lynn, Heinrich, and Hill develop a multilevel “LOG” that postulates that politics, public policymaking, public management, and service delivery are hierarchically linked with one another in the determination of public policy outputs and outcomes. A large body of research uses hierarchical logic when designing empirical studies of governance and public management (Boyne, 2003; Boyne, Meier, O’Toole, & Walker, 2006; Forbes, Hill, & Lynn, 2006, 2007; Forbes & Lynn, 2005; Forbes et al., 2008; Heinrich, 2003; Hill & Lynn, 2005; Lynn, Heinrich, & Hill, 2001). Being suffi- ciently confirmed in theory and in the empirical research literature, the multilevel LOG was used by Forbes, Lynn, and Robichau as the point of departure for a theory of public sector performance which focuses on the operations of the administrative system in the determination of government performance (Forbes et al., 2008). A brief explanation of how the theory was developed will set the stage for showing how public policy theories can benefit from the governance literature and, in particular, the implementation process. Development of the multilevel theory of public sector performance used in this article drew on findings from an analysis of nearly thirteen hundred published empirical studies, each of which incorporated hierarchical and causal relationships 22 Policy Studies Journal, 37:1 among at least two levels of governance. The original LOG analyses categorized each study’s dependent and independent variables at each level under investigation using the scheme in Table 1 (e.g., in Hill & Lynn, 2005; Forbes et al., 2006, 2007). Policy studies are more likely to inform levels of citizen interests and preferences, public choice and policy design, outputs/outcomes and, to a lesser extent, stakeholder assessments than variables found at the management and service delivery level, which ultimately lie at the heart of policy output production. In subsequent analyses by Forbes et al. (2008), Scott’s conceptualization of orga- nizational effectiveness indicators were used to reclassify the independent and dependent variables in each study (Scott, 2003; Scott & Davis, 2007). Scott says that “proponents of rational, natural, and open system models privilege differing indi- cators of effectiveness” that can nonetheless be grouped into three general indicator Table 1. Logic of Governance Levels and Variables Citizen preferences and interests expressed politically 1. Expressions of citizen preferences and interests 2. Activities of private firms and organizations 3. Activities of interest groups Public choice and policy designs 1. Socioeconomic context 2. Level/type of government (e.g., state, council/manager) 3. Political atmosphere 4. Fiscal situation/budget constraints 5. Type of ownership (public, nonprofit, proprietary) 6. Mandates by internal government entities (e.g., Office of Management and Budget) 7. Mandates by elected executives 8. Mandates by legislatures 9. Court decisions 10. Other Public management 1. Initiation of administrative structures 2. Usage of management tools (policy planning, total quality management) 3. Management values and strategies Service delivery 1. Initiation of program design features 2. Fieldworker/office discretionary behavior 3. Fieldworker/office beliefs and values 4. Initiation of administrative processes and policies 5. Work/treatment/intervention 6. Client influence, behavior, and/or preference (coproduction) 7. Use of resources Outputs/outcomes Outputs; means to an end 1. By government/public sector 2. By markets/firms/private sector 3. By individuals/society Outcomes; ends 1. By government/public sector 2. By markets/firms/private sector 3. By individuals/society Stakeholder assessments of policy, agency, or program performance Robichau/Lynn: The Missing Link 23 types that “point to important distinctions regarding what is being assessed” (Scott & Davis, 2007, p. 327). Scott’s typology of effectiveness indicators has three categories: structures, pro- cesses, and outcomes. Structural indicators reflect the production function, that is, the way the organization’s work is organized. Process indicators measure the quan- tity or quality of the organization’s work, that is, effort or output. Outcome indicators purport to identify changes in an individual or organization that have been the object of some kind of intervention, service, or regulation. Thus, the effectiveness of each level of governance can be measured in terms of some combination of structures, processes, and outcomes. Analyzing the multilevel governance literature using these indicators led to the following findings. First, the presumption is warranted that implementation is gen- erally hierarchical; influences flow downward through a chain of delegation to the retail level of service delivery. Second, in the great majority of studies, policymaking took the form of structures; that is, the primary function of policymaking was to organize administrative systems to accomplish the purposes of public policies. Third, within administrative systems, management, that is, the authorized and nec- essary exercise of managerial discretion, took the form of a relatively balanced combination of structures and processes. Finally, measures of service delivery effec- tiveness were, in most cases, process indicators. The cumulative products of policy- making, management, and service delivery consisted of outputs, which comprised both process and outcome indicators of effectiveness, and the “final” outcomes of policymaking and its implementation.1 These findings are incorporated by Forbes, Lynn, and Robichau into their theory of public sector performance. Some of the findings were not so readily accepted, however. For example, most of the investigations were oriented toward outputs rather than outcomes. But few empirical models recognized or incorpo- rated an outputs-cause-outcomes logic; most used either outputs or outcomes without considering how outputs influence the ultimate outcomes of policies and their implementation. In Bureaucracy, James Q. Wilson (1989) helps helps us understand how outputs and outcomes are related. Similar to Scott’s explanation, Wilson defines outputs as “the work the agency does” and outcomes as “how, if at all, the world changes because of the outputs” or “results” (p. 158). Grouping outputs as the “work” and outcomes as the “results” enables scholars to think about the logical flow of cause and effect relationships in governance and policy studies. Based on the empirical findings concerning the political science and public administration literature, we found that distinguishing between outcomes and outputs was both possible and instrumental in advancing our theory of public sector performance. It is possible, however, that, in special cases, outputs of an agency may be unobservable and unknowable (Wilson, 1989). One consequence might be that some agencies will be able to see only outcomes without knowing how outputs influenced them. If outcomes are undesirable, then an agency faces the challenge of trying to decide how to fix the problem without knowing whether it is a question of structure or process. For example, a program like Head Start provides various services, such as 24 Policy Studies Journal, 37:1 education, health care, parental education and involvement, and nutritious meals to underprivileged children. They claim that children in their programs show improved cognitive and language abilities as well as higher health status ratings when compared to their peer groups (National Head Start Association, 2008); yet, it is hard to say which services produce which results. Outcomes could be produced from any one of these provisions or a combination of services, thereby making the determination of outputs unknowable and unobservable. Wilson (1989) proceeds to create a typology of four kinds of government agen- cies based on the extent to which both outcomes and outputs are observable and measurable. He categorizes these agencies as: production (where outcomes and outputs are recognizable); procedural (where outputs not outcomes are observable); craft (where outcomes not outputs are distinguishable); and finally, coping organi- zations (where neither outcomes nor outputs can be observed) (pp. 158–71). This useful heuristic has a twofold consequence for thinking about both gover- nance and policy theories. First, how administrators manage their agencies will be dependent on the type of agency in which they work and whether they focus on processes or outcomes. And second, either outcomes and/or outputs are frequently observable; therefore, they can often be measured or at least considered in agency performance. Wilson (1989) states that “people matter, but organization matters also, and tasks matter most of all” (p. 173). If we think of tasks as the work agencies and their agents produce, then depicting outputs as necessary to achieving outcomes seems intuitive. Further, a significant number of studies “skipped” levels in the LOG, excluding, for example, the mediating effects of management or service delivery in transform- ing policy structures into outputs and outcomes (see Figure 1). The percentages in Figure 1 are proportions of the total studies in the database that employ this particu- lar causal logic. In the absence of convincing substantive reasons for excluding these Management Structures Service Delivery Processes Enacted Policy Structures Outputs/ Outcomes 9% (Skip levels) 2.7% (No skip) 6.8% (Skip levels) 3.5% (No skip) 7.3% 3.5% 6.6% Management Processes 13.1% 8.0% Figure 1. Empirically Modeled Governance Relationships. Robichau/Lynn: The Missing Link 25 mediating effects from explanatory models, there is a high likelihood that the find- ings of such studies are incomplete or biased. In their analysis of studies concerned with health policy implementation, Forbe et al. (2007) concluded that “in general, the choices of organizational arrangements, administrative strategies, treatment quality, and other aspects of health care services by policymakers, public managers, physi- cians, and service workers, together with their values and attitudes toward their work, have significant effects on how health-care policies are transformed into service-delivery outputs and outcomes” (p. 453). The basic theory, then, incorporates both outputs-cause-outcomes logic and all mediating levels (unless it can be plausibly argued that policies have been designed so as to be self-executing, thus requiring little managerial intervention). The theory of public sector performance is depicted in Figure 2. It consists of hierarchically related public policymaking structures,2 management structures and processes, service delivery processes, outputs, and outcomes, with an acknowledgement of self-executing policies. Public Policymaking Structures Management Structures & Management Processes Service Delivery Processes Outputs as Processes Outcomes Environment/Policy Context Figure 2. Theory of Public Sector Performance. *The dotted line represents potential modeling patterns that skip the management level and represent public policies that are self-executing. 26 Policy Studies Journal, 37:1 It is important to note that these relationships all occur within a social, economic, and political context. Whereas these contextual considerations may influence gover- nance at any level in complex ways, the theory assumes that such contextual influ- ences do not nullify the fundamental hierarchical logic that links the multiple levels of governance, a logic that is the consequence of America’s constitutional scheme of governance, which defines the American version of the rule of law. The implications of this theory have the potential to contribute to public policy theorizing. First, given the hierarchical nature of governance, it is difficult, if not impossible, for policy outputs and outcomes to be produced without administrative systems. Further, it is difficult to imagine that public policy outcomes can occur without administrative system outputs. In part for this reason (and in part because outputs are easier to measure), administrative systems tend to be output-, not outcome-oriented. Analyzing Policy Theories With the above discussion as a point of departure, we next examined a selection of the public policy theories included in Sabatier (2007). This examination reveals two problematic aspects of these theories: the failure to distinguish between outputs and outcomes and the imprecise treatment of the role of administrative systems in mediating the relationship of policymaking to its ultimate consequences. Outputs and Outcomes Sabatier (2007) notes that Institutional Rational Choice (IRC) theory is “clearly the most developed of all the frameworks in this volume” (p. 9). In her chapter, on IRC theory, Ostrom discusses an action arena where actors and action situations occur that can lead to predicting outcomes inside this arena. Then, her framework moves from this arena to patterns of interactions, followed by outcomes, in which both are moderated by evaluated criteria. Ostrom (2007, p. 33) states that “evaluative criteria are applied to both the outcomes and the processes of achieving outcomes.” But these “processes” are not clearly defined, partly, it would appear, because outputs and outcomes have not been differentiated. Other policy theories exhibit similar ambiguity concerning outputs and out- comes. For instance, the Multiple Streams (MS) Framework is sensitive to how information affects choice and how inputs gets transformed to outputs; yet, Zaha- riadis (2007) does not clearly define what he means by “policy outputs” in this chapter. It might be the case that outputs are simply decisions (good or bad) that have been made, the actual policy that is produced from decisions made, or perhaps, both of them combined. Another policy approach that can be seen as providing vague descriptions about outputs and outcomes is the Network Approach. Adam and Kriesi (2007) state that “the impact of policy networks clearly shows that network structures are not only connected to specific policy outcomes (‘what’) but also to the type of change (‘how’) that creates these outcomes. A systematic analysis of the impact of policy networks Robichau/Lynn: The Missing Link 27 requires that the types of networks be linked to the potential for and types of change creating different outcomes” (p. 145). If outputs are regarded as the “work” of an agency or in this case, the “work” of a network, then the reference to “outcomes” seems inappropriate. Finally, the Social Construction and Policy Design framework (Ingram, Schneider, & deLeon, 2007) has an inherent output focus, which becomes particu- larly apparent when examining their propositions and diagram.3 The fundamental idea of this framework is that future policy design stems from past and current policy designs. Moreover, these policy designs are mediated by institutions, culture, and target populations, followed by society and policymaking dynamics (p. 96). But outputs, as distinguished from outcomes, are not clearly defined. One policy theory acknowledges a distinction between outputs and outcomes. The Advocacy Coalition Framework (ACF) depicts a relationship where policy outputs precede policy impacts (Sabatier & Weible, 2007, p. 202). But, this part of the policy subsystem, in our reading, is not defined or discussed in Sabatier and Weible’s chapter or in the previous models of ACF (Sabatier & Jenkins-Smith, 1999). From a governance perspective, lack of clarity concerning the distinction between outputs and outcomes is problematic. As discussed earlier, outcomes and outputs have distinguishable characteristics (Donabedian, 1966; Hall, 1999; Scott, 1977, 2003; Scott & Davis, 2007; Suchman, 1967). It is hard to tell whether policy theorists assume that those working in the field know the difference or, alternatively, do not think that the difference is important. Our argument is that drawing clear distinctions between outputs and outcomes is essential for understanding how administrative systems are influenced by and, in turn, influence the consequences of policymaking. Administrators and Administration Terry Moe (1990) crafts a theory of the public bureaucracy that takes into account politics and political organization in a two-tiered hierarchy in which “one tier is the internal hierarchy of the agency, and the other is the political control structure linking it to politicians and groups” (p. 122). Designed around the politics of bureau- cratic structures, Moe emphasizes how political uncertainty and the need for com- promise leads to rational bargaining among political actors that, in the end, produces technically irrational agencies. In other words, administrative systems are designed more to protect political bargains struck in order to guard various stakeholder interests than as to facilitate the achievement of outputs and outcomes. Our examination of the public policy theories discussed in the previous section suggests that the performance of administrative systems is not generally held to be of particular significance to public policy achievement. It seems intuitively clear, however, that managers are in a position to use their discretion to shape the rela- tionship between enacted policy structures and administrative system outputs. Moreover, the analysis of empirical health policy studies cited earlier (Forbes et al., 2006) supports this view. Wilson (1989) notes that public managers have preferences, and in a similar vein, Sabatier and Weible (2007, pp. 194–96) contend that they are 28 Policy Studies Journal, 37:1 often influenced by “policy core beliefs.” Lindblom and Woodhouse (1993) claim that “administrators indeed are policymakers” (p. 62). A convincing literature has established the influential role in policy implementation played by “street-level bureaucrats.” From a governance perspective, then, the relative neglect of adminis- trative systems in public policy theories seems unwarranted. We found that administrative systems are either referred to loosely through a discussion of managerial influences on the policymaking process or they are ignored altogether. The Institutional Analysis Framework, for example, is vague on whether the “action arena” and/or the “patterns of interactions” could be seen as places where bureaucracy and its’ administrators enable outputs to be achieved. Perhaps, the framework is set up so that administrative influences and implementation occur in both arenas, but without specific discussion of these matters, we are left wonder- ing why such a vital part of the policy process was not discussed explicitly. Similarly, the MS Framework considers bureaucrats as part of the policy com- munity that is involved in the policy stream, but no logic linking administrators as policy entrepreneurs to administrative system outputs is indicated. Authors of the Network Approach examine how coalitions or networks are formed. They claim that these coalitions are composed of either “state actors” or “system of interest inter- mediation” (Adam & Kriesi, 2007, p. 134). Adam and Kriesi specifically and clearly define those who belong to the “system of interest intermediation” as “political parties, interest groups, and nongovernmental organizations/social movement orga- nizations” (p. 134). These authors’ explanation of “state actors,” however, is ambigu- ous; “state actors constitute a special type, because they ‘have access to a very particular resource: their decisions are considered binding in society and are backed by the possibility of legitimate use of force’ ” (p. 134). In this context, state actors could be viewed as legislators, judges, political appointees, or administrators. For governance scholars, greater clarity and specificity of when, or even if, administra- tors and their agencies are involved in the policy process is essential. Administrative Systems Of even greater concern than the vague references to administrators is that administrative systems as part of the policy process are left out entirely. Policy process theories tend to analyze the progression of policy development through design and negotiation, and then assume that policy outcomes are a result of par- ticular policies. Yet, external policymaking is only the first stage in the logic of how outputs and outcomes are produced. To see the issue more clearly, several theories of the policy process have been modified to demonstrate visually the point that policy process theories have missing links that are logically and empirically essential to the determination of government performance. Graphically lacing the theory of the public sector performance to some of the major policy process theories enables us to see how they can complement one another in providing further explanatory specificity to current policy and gover- nance theories. By rotating the figures of policy process theories to a vertical orien- tation, a new and clearer picture emerges about what these theories are emphasizing Robichau/Lynn: The Missing Link 29 from a governance perspective. Policy theories appear to be focusing primarily on the public choice and policy level of governance and at the output/outcome level, thus, in effect skipping the mediating levels of administration. The model most closely approximating the LOG is the IRC framework (see Figure 3). Ostrom does consider levels of analysis (i.e., operational, collective choice, constitutional, and metaconstitutional) intermediate to outcomes in her IRC model. However, her main focus is on the effects of rules (which are governance structures) at each level. Her framework might be expanded to include more discussion of the relationships of administrative systems to structures and processes beyond rules. In Figure 3, though, it is difficult to tell whether policymaking structures, management structures and processes, service delivery processes, and outputs as processes all occur in the action arena, in the patterns of interactions, or various links occur before the action arena or even after patterns of interactions. In examining the MS Framework, it must first be noted that Zahariadis (2007, p. 65) regards this framework as one that “could conceivably be extended to cover the entire process of policymaking at various levels of government, it is examined here only in its capacity to explain policy formation (agenda setting and decision making).” Despite this caveat, locating where “policy outputs” occur in the poli- cymaking environment is challenging but nonetheless necessary in developing better theories of the policy process. Perhaps “policy outputs” could follow the policymaking structures, or they could be at the lower level of outputs as pro- cesses in comparison to the theory of public sector performance (see Figure 4). If Attributes of Community Physical/Material Conditions Actors Action Arena Action Situations Rules-In-Use Patterns of Interactions Outcomes Evaluative Criteria Environment/Policy Context Public Policymaking Structures Management Structures & Management Processes Service Delivery Processes Outputs as Processes Outcomes Figure 3. A Framework for Institutional Analysis Modified Hierarchically from Sabatier (2007, p. 27) Juxtaposed with the Theory of Public Sector Performance. 30 Policy Studies Journal, 37:1 the MS framework is to be expanded to account for the complete policy process or used by governance scholars to obtain greater insight into the policy process, then one step in doing so would be to clarify the role of administrative systems in output production. Zahariadis contends that some critics of this framework ques- tion whether the three steams are independent of one another and that “stream independence is a conceptual device” (p. 81). In the comparison to the theory of public sector performance, all three streams would fall into a larger category of public policymaking. The Network Approach is juxtaposed with the theory of public sector perfor- mance in Figure 5. Although Adam and Kriesi (2007) note that depicting network structures as being vertically organized violates network premises, doing so does produce an interesting way of thinking about the structures and processes of net- works and how they interact with other levels of governance. The “structure of policy networks” …