Evaluation in the Practice of Development

Standard methods of impact evaluation often leave significantgaps between what we know about development effectiveness andwhat we want to know—gaps that stem from distortions inthe market for knowledge. The author discusses how evaluationsmight better address these knowledge gaps and so be more relevantto the needs of practitioners. It is argued that more attentionneeds to be given to identifying policy-relevant questions (includingthe case for intervention), that a broader approach should betaken to the problems of internal validity (including heterogeneityand spillover effects), and that the problems of external validity(including scaling up) merit more attention by researchers. JEL codes: H43, O22     

Timing and Duration of Exposure in Evaluations of Social Programs

Impact evaluations aim to measure the outcomes that can be attributedto a specific policy or intervention. While there have beenexcellent reviews of the different methods for estimating impact,insufficient attention has been paid to questions related totiming: How long after a program has begun should it be evaluated?For how long should treatment groups be exposed to a programbefore they benefit from it? Are there time patterns in a program'simpact? This paper examines the evaluation issues related totiming, and discusses the sources of variation in the durationof exposure within programs and their implications for impactestimates. It reviews the evidence from careful evaluationsof programs (with a focus on developing countries) on the waysthat duration affects impacts.      

Using Household Surveys to Build Analytic Capacity

This article reviews the results of efforts in five countriesto build a national capacity to analyze social policy; theseefforts were undertaken in oconjunction with Living StandardsMeasurement Survey (LSMS) projects. Case studies for Bolivia,Jamaica, the Kyrgyz Republic, South Africa, and Vietnam showthat building such capacity requires explicit planning, significanttime and money, open access to data, and strong support frompolicymakers. Lessons are drawn about four aspects of buildingcapacity—training, technical assistance, research, andrecruitment. The lessons learned from these LSMS projects applyin a more general sense to other kinds of capacity-buildingprojects.      

IMF Conditionality and Country Ownership of Adjustment Programs

This article uses finance and agency theory to establish twokey propositions about International Monetary Fund (IMF) conditionalityand country ownership of IMF-supported adjustment programs.First, the authors propose that the conditionality attachedto these programs is justified. Second, the article hypothesizesthat country ownership of these programs is crucial for theirsuccess. Because IMF conditionality and country ownership areboth necessary, the challenge is designing conditionality thatmaximizes ownership while providing adequate safeguards forIMF lending. The article analyzes several recent proposals aimedat enhancing country ownership of policies contained in IMF-supportedprograms. These proposals include encouraging countries to designtheir own adjustment and reform programs, streamlining structuralconditionality, introducing flexibility in the timing of structuralpolicy measures (floating tranche conditionality), and applyingconditionality to outcomes rather than policies (outcomes-basedconditionality).      

How to deal with dissent among experts. Risk evaluation of EMF in a scientific dialogue

This article describes the results of a scientific discourse which aimed at exploring the reasons for differences in expert health risk assessment of radio frequency electromagnetic fields of mobile telephony.

It starts with describing the structure of the discourse. Then, the reasons for the conflicting risk assessments are discussed. Differences are due to the selection and evaluation of relevant scientific studies by applying different scientific quality standards, to the methods used for generating a research synthesis and an overall risk evaluation. Consensus could be achieved regarding the selection of and the quality requirements for the scientific studies used for risk assessment as well as their significance for risk evaluation. However, dissent remained about the synthesis of scientific evidence into an overall risk evaluation and about the relevance of the precautionary principle for risk evaluation and its implications for the risk assessment framework.

Based on the analysis of these problems, a transparent, consistent and rational procedure for risk assessment is suggested to facilitate a risk characterization which better meets the demands of policy making and the public for an appropriate risk evaluation.     

OPTIONS FOR DISMANTLING TRADE RESTRICTIONS IN DEVELOPING COUNTRIES

Although many countries have adopted freer trade regimes orare in the process of reforming quantitative restrictions, suchreforms are not without shortterm costs. Sudden removal of quantitativerestrictions can release a surge of demand for imports thatcan deplete foreign exchange reserves and force rapid adjustmentby domestic industries. Five methods to more gradually liberalizequantitative restrictions are discussed: raising quota ceilingsuntil they are non-binding; eliminating quotas product by product;converting quotas to equivalent tariffs and then reducing them;auctioning quota licenses; and converting quotas to tariff quotas,and then liberalizing these. The various instruments are evaluatedaccording to their implications for the adjustment period, forgovernment revenue, for the balance of payments, and for theextent to which they are likely to produce political disruptions.      

Inside Decentralization: How Three Central American School-based Management Reforms Affect Student Learning Through Teacher Incentives

Despite decentralization reforms of education systems worldwide,there is little empirical evidence about the processes throughwhich decentralization can improve student learning. Proponentstheorize that devolving decisionmaking authority to the locallevel can improve communication, transparency, and accountability,making teachers and school principals more responsible for betterperformance and more capable of bringing it about. Yet someresearch has shown that decentralization can increase inequalityand reduce learning for disadvantaged students. This articlereports on retrospective evaluations of three Central Americanschool-based management reforms. Using matching techniques,these evaluations investigate whether the reforms enhanced studentlearning and how they affected management processes and teachercharacteristics and behaviors. The evidence indicates that allthree reforms resulted in substantive changes in managementand teacher characteristics and behavior and that these changesexplain significant portions of resultant changes in studentlearning. This article contributes to the understanding of howdecentralization reforms can improve learning and shows howeducation reforms, even when not conceptualized as affectingteacher incentives, can generate important changes for teachersthat, in turn, affect student learning. JEL codes: I21, I28, H52, H75     

Appraising Workfare

Workfare programs aim to reduce poverty by providing low-wagework to those who need it. They are often turned to in a crisis.This article offers some simple analytical tools that can beused to rapidly appraise the cost-effectiveness of an existingworkfare operation as a basis for deciding whether the programshould be expanded. For pedagogic purposes, two stylized versionsof a range of programs found in practice are analyzed: one fora middle-income country, the other for a low-income country.The cost of a given gain to the poor is about the same for bothprograms, although the components of that cost are very different,with implications for the timing of benefits. The author pointsto program design changes that could enhance the impact on poverty.      

ECONOMIC ANALYSIS FOR HEALTH PROJECTS

This paper applies to the health sector a method of projectanalysis advocated recently by Devarajan, Squire, and Suthiwart-Narueput.A health project evaluation should establish a firm justificationfor public involvement; establish the counterfactual—whatwould happen with and without the project; and determine thefiscal effect of the project and the appropriate levels of feesin conjunction with project evaluation. The evaluation shouldalso acknowledge the fungibility of project resources and examinethe incentives both for high-level public servants to shiftgovernment resources away from project-funded activities tothose that have not been evaluated and for lower-level contractorsand civil servants to provide good or bad service. Market failuresin health services and insurance markets should serve as a startingpoint for economic analysis, not as a reason to ignore economicsin health projects. Project outputs should be predicted aftertaking into account the reaction of consumers and providersin the private sector as well as market structures of supply,demand, and equilibrium for health services.      

VALUING AND ACCOUNTING FOR LOAN GUARANTEES

To achieve certain policy objectives, governments frequentlyprovide private borrowers with loan guarantees that cover someor all of the risk that the borrower will be unable to repaythe loan. Such guarantees are extremely valuable, and theirvalue increases with the riskiness of the underlying asset orcredit, the size of the investment, and the duration of theloan. The flip side of a guarantee's value to a lender is itscost to the government. Such a cost is not explicit but is realnevertheless. When providing guarantees, governments thereforemust establish accounting, valuation, and risk-sharing mechanisms.This article describes methods of valuing guarantees; reportsestimates of the value of guarantees in different settings;and summarizes new methods of accounting designed to anticipatelosses, create reserves, and channel funds through transparentaccounts to ensure that the costs of guarantees are evidentto government decisionmakers.      

THE THREE FACES OF THE INTERNATIONAL COMPARISON PROJECT

Purchasing power parities (PPPs), this article confirms, arethe correct converters for translating GDP and its componentsfrom own-currencies to dollars (the usual numeraire); the alternativemeasure, exchange rates, obscures the relationship between thequantity aggregates of different countries. Drawing on the reportsof the United Nations International Comparison Project (ICP),the article contends that exchange rates systematically understatethe purchasing power of the currencies of low-income countriesand thus exaggerate the dispersion of national per capita incomes.Where full-scale (benchmark) PPP estimates are not available,estimates based on shortcut methods better approximate whatthe benchmark estimates would be than do the exchange rate conversions.The ICP results also illuminate price and exchange rate relationshipsamong countries by providing a measure of the difference inthe levels of prices in different countries. ICP price comparisonsfor components of GDP make possible the analysis of comparativeprice and quantity structures of different countries and providethe raw materials for many types of analytical studies.      

INTERPRETING RECENT RESEARCH ON SCHOOLING IN DEVELOPING COUNTRIES

Policymakers in developing countries have long been troubledby the undesirable, but apparently unavoidable, choice betweenproviding broad access to education and developing high-qualityschools. Recent evidence, however, suggests that this is a badway to think about human capital development. Grade repetitionand high dropout rates lead to a significant waste of resourcesin many school systems. Students in quality schools, however,respond in ways that reduce such inefficiencies, perhaps evensufficiently to recoup immediately investments in quality. Promoting high-quality schools, however, is more difficult thanmany have thought, in part because research demonstrates thatthe traditional approach to providing quality—simply providingmore inputs—is frequently ineffective. Existing inefficienciesare likely to be alleviated only by the introduction of substantiallystronger performance incentives in schools and by more extensiveexperimentation and evaluation of educational programs and schoolorganizations. Incentives, decentralized decisionmaking, andevaluation are alien terms to education, in both industrialand developing countries, but they hold the key to improvementthat has eluded policymakers pursuing traditional practices.      

ACHIEVING SOCIAL OBJECTIVES THROUGH PRIVATE TRANSFERS: A Review

Private interhousehold cash transfers are an important sourceof income in many developing countries. Among the countrieswhose experience is reviewed in the article, the proportionof all households receiving private transfers ranges from afifth to a half. The amounts received are large, particularlywhen compared with the incomes of the poorest households. Understanding more about these transfers is important for designingpolicy because, among other things, these remittances providesocial and economic benefits similar to those of public programs,such as unemployment insurance, pension support, educationalcredit, and health assistance. As such, private transfers maysupplement or overlap with public transfers, and, if privatedonors give less as public transfers increase, the effect ofpublic programs on beneficiaries would be less than originallyintended. Or the transfers may alter the distributional effectsof public programs: for again, if private donors give less aspublic transfers increase, they share in some of the benefitsof public programs.      

THE IMPACT OF HEALTH AND NUTRITION ON EDUCATION

Child health and nutrition are strongly associated with educationalachievement. But associations do not necessarily indicate causality;estimates generally are likely to be biased in one directionor the other. As a result analysts and policymakers should havemuch less confidence in findings about the effect of healthon schooling success than has been claimed in previous surveys.The evidence is more nuanced and qualified than is often recognizedbut may still support the conclusion that health may have considerableeffects on postschooling productivity. Policy implications pointtoward providing public subsidies for the collection, analysis,and dissemination of information about the links between healthand education; and toward providing services to improve thehealth of poor children.      

Managing Government Exposure to Private Infrastructure Risks

The privatization of infrastructure should lead to the developmentof new infrastructure, improvements in the operation of existinginfrastructure, and a reduction in budgetary subsidies. Whethercountries reap the full benefits of privatization, however,depends on how risks are allocated. If, as is often the casein developing countries, governments assume risks that shouldbe borne by investors, they may reduce incentives for efficiencyand incur significant liabilities. To solve these problems,governments need to improve their policies and restrict theirrisk bearing to certain political and regulatory risks overwhich they have direct control. When a government provides guarantees,it should attempt to measure their cost and improve the waythey are handled in the accounts and budgets. Measurement andbudgeting are critical to improving decisions about the provisionof guarantees, to improving project selection and contract design,and to protecting governments from unknowingly entering intocommitments that might jeopardize future budgets.      

Targeting Revisited

Public spending programs aimed at alleviating poverty can eitherbe broadly targeted at categories ofspending or narrowly targetedat types of people. Each approach has benefits and costs tothe poor. It is often claimed that narrow targeting of the poorwill allow governments to reduce pQverty more effectively andat lower cost. But narrow targeting often has hidden costs,and once these costs are considered, the most finely targetedpolicy may not have any more effect on poverty than a broadlytargeted one. Both approaches also have hidden benefits, althoughless is known about their impact. Targeting can help, but itis not a cure-all. Reducing poverty calls for broadly targetedsocial sector spending combined with narrower targeting of cashand in-kind transfers to spec groups. It is also important forgovernments to experiment with schemes that offer better incentives,to carefully monitor the costs and outcomes, and to be flexibleand pragmatic in their policy responses.      

WHEN DO HETERODOX STABILIZATION PROGRAMS WORK?: Lessons from Experience

What advantages and disadvantages does the heterodox strategyoffer to stabilization programs in countries with chronic highinflation? Heterodox stabilization programs, in our definition,are those that support orthodox policies— that is, tightfiscal policy and a fixed exchange rate—with the initial,temporary use of incomes policies— that is, price andwage controls. This evaluation, based on several heterodox programs,successful and unsuccessful, from the 1960s and 1980s in LatinAmerican countries and Israel, affords four principal lessons: * The rapid reduction in inflation at the beginning of heterodoxprograms (which usually comes about at small cost) is the easypart; the problem is to maintain price stability over time. * Incomes policies in heterodox stabilization programs are justifiedonly in countries with high chronic inflation, where persistentinflation is more pervasive and problematic. * There is a case for a bigger fiscal adjustment in heterodoxthan in orthodox programs because of the risk that a programwith price controls may be misperceived as a populist devicefor achieving price stability without adjusting. * The failure of a heterodox program is more likely to destabilizeinflation than is the failure of an orthodox program.      

USING AUCTIONS TO ALLOCATE AND PRICE LONG-TERM CREDIT

Most long-term credit in developing countries is allocated throughnegotiated agreements between government institutions and financialintermediaries or final borrowers, and often at administeredrates. Yet many developing countries have no long-term creditmarket whose interest rates can be used as benchmarks for theseloans. If credit is priced improperly, it will be allocatedinefficiently and the development of capital markets may bestunted. In light of the generally disappointing experiencewith conventional methods of allocating development credit,some countries have introduced credit auctions as an alternative.Among the advantages are greater transparency and fairness,lower transaction costs, and increased competition and efficiency.Among the disadvantages are a greater vulnerability to collusion,which can lead to lower interest rates and revenue, and a tendencyto attract the least desirable participants (adverse selection)and to lend for riskier projects (moral hazard), which can leadto lower repayment rates and a higher probability of default.All these factors can lead to inefficiency in the allocationof funds. This article suggests ways to lessen these negativeeffects and presents various elements of auction design thataffect the efficiency of credit auctions and their suitabilityto specific circumstances. When properly designed, auctionscan be used in a variety of environments to allocate developmentcredit more efficiently than current methods do.      

CASH FLOW OR INCOME?: The Choice of Base for Company Taxation

Cash flow and equity income (or income) are two alternativebases advocated for taxes on businesses throughout the world.Although in practice most tax systems are hybrids with elementsof both types of taxes, recent literature has stressed the meritsof the cash flow tax because it is simple in concept and itdoes not distort decisions about capital expenditures and financing.But international issues and administrative complexities—particularlytax evasion—present problems that must be sorted out beforea cash flow tax can be implemented.      

CHARGING FOR ROADS

The article discusses two theoretical methods of measuring roaduse costs and designing a system of road user charges. The firststates that road damage externalities are zero and road damagecosts are equal to the traffic-related fraction of maintenanceexpenditure. The second states that, with constant returns andoptimal road capacity, congestion charges should recover theremaining total overhead costs. Vehicles should be charged thesecosts, and additional pure taxes on passenger vehicles shouldbe guided by principles of indirect taxation. Although roaduser charges alone may fail to cover the total highway budget,the additional pure taxation is likely to more than cover theshortfall. The article argues that an appropriate system oftaxes and charges can be devised to meet these requirementswithout adversely effecting the rest of the economy.