What Are We Really Achieving? Building an Evaluation Culture in Migration and Development
Policymakers are increasingly interested in understanding how migration impacts development and how to harness its benefits and reduce its costs.
This is not surprising, given that migrant remittances are now more important than official development aid, and that there is a mounting interest in the contributions of migrants' "social remittances" (skills, know-how, and networks) to development. As a result, a growing body of research has focused on how to measure the broader social and economic effects of migration on development.
Much less attention, however, has been given to assessing the impact of the growing number of migration programs and projects, which either directly or indirectly affect the ways in which migration and development interact. This is an important oversight, given the recent calculation by the United Nations Department of Economic and Social Affairs that $250 million is being spent on programs specifically designed to shape migration and development outcomes.
This lack of understanding about the ways in which migration programs affect development reflects a wider lack of knowledge about the effects — and effectiveness — of migration policymaking and programming in general.
While most policymakers responsible for implementing migration programs would no doubt agree that it is important to know what works and what doesn't, and that such knowledge is key to good management of migration, in reality little has been done to explore what impacts migration programs have.
This article examines why there is a lack of an "evaluation culture" in the migration and development policy arena, and uses lessons learned from the development community — which has invested considerably in impact evaluation in recent years — to make a case for impact evaluation studies that can contribute to more evidence-based migration programming.
In particular, this article refers to the growing number of migration projects and programs that are specifically designed to have an impact on development, and seeks to illustrate how evaluation research can enhance migration's development outcomes.
|
Terminology
|
||
|
The Lack of an Evaluation Culture
A recent study of evaluation practices in several Organization for Economic Cooperation and Development (OECD) countries found that national and international public agencies operating in the field of migration undertake relatively few impact evaluation activities.
Similarly, in a survey conducted on behalf of the Global Forum for Migration and Development (GFMD) in 2010, there was little evidence to suggest that migration and development programs and policies were being assessed in a rigorous manner to ascertain their impacts. In fact, just nine GFMD participant countries responded that they had carried out any assessment work at all.
In discussing this topic, it is important to examine the culture of evaluation, rather than simply the number of evaluations conducted, because the obstacles to conducting evaluations often seem to be more political than technical. Some are specific to migration, while others are linked to a more general reluctance to support evidence-based policymaking, and impact evaluation-based policymaking in particular.
So why do governments seem to be reluctant to invest in impact evaluations? Why is there a lack of an evaluation culture in the migration policy world?
There is sometimes what might almost be called a "fear factor" associated with evaluating the impact of government programs. Decision makers and project managers often fear bad news; i.e., that a particular program or project is not having the desired outcome. Negative evaluation findings could potentially result in budget cuts or a loss of prestige for a particular program. If a program has a political profile, it can generate criticism of policymakers by the media, amongst nongovernment organizations (NGOs), and the wider public. In short, policymakers may simply fear that a rigorous evaluation will highlight more problems than solutions.
Impact evaluations may also be considered a costly investment, requiring substantial financial resources and significant commitments of time. Very rigorous impact evaluations that use experimental or quasi-experimental techniques to identify the counterfactual can cost hundreds of thousands — or even millions — of dollars.
There are also potential problems with the timing of impact evaluations. A rigorous evaluation could take three or four years to conduct, and governments may not have time to wait for the results of such assessments. Moreover, policymakers tend to be reactive, which may inhibit them from investing in impact evaluation work that may require taking a longer-term and more strategic perspective. Less expensive small-scale rapid impact evaluations are also possible, but the results may be less reliable.
An additional obstacle to sustaining an evaluation culture is that impact evaluations, especially the more rigorous ones, require a certain level of technical expertise that is not available everywhere and that relatively few governments possess in-house. This fact not only affects the ability of governments to conduct impact evaluations, but also their interest in them and sometimes even their ability to absorb the findings should an evaluation be conducted.
In addition to the generic obstacles to conducting impact evaluations described above, there are some specific challenges linked to migration and development evaluations in particular.
The usual fear factor on the part of policymakers associated with evaluation is raised several notches when the evaluation is to be applied to a migration program. Migration is a contentious and often highly politicized issue in many countries, and the public, the media, and the political opposition often place a high level of scrutiny on a government's performance in this area. Under such circumstances, using a less careful policy design process and hoping for the best may be preferable to potentially exposing problems and failures.
And while evaluation capabilities may be weak across governments, migration policy officials in particular often lack the capacities required to promote an evaluation culture. There is little training available to government officials to enhance their ability to commission and assess the results of evaluation studies. Unlike in the humanitarian and development fields, there are no dedicated training courses or training materials available on the evaluation of migration programs.
Their job is also made harder by the fact that there are particular technical challenges associated with applying evaluation techniques to migration programs. Evaluation means trying to examine what would have happened in the absence of the program. The most reliable way to do this is by measuring the outcomes of an intervention for a group of persons affected by the intervention, and comparing them with the outcomes obtained by a group not affected by it (a control group), and who, crucially, have been selected in a random manner so that there are no systematic differences between the groups except the effect of the intervention. This can be difficult in the case of migration programs, however.
Many of the effects that migration policy has are not confined to the individuals and households who directly participate in that intervention, making it hard to clearly define an intervention group and a control group.
Take health programs, which often use impact evaluations, for example: It may be relatively easy to calculate the effects of a nutritional supplement in a child's diet because the main effect is on that child's health, and it is hard to see how these effects would "spill over" and impact upon others. Similarly calculating the impact of a temporary migration scheme, however, is much harder. This is because the effects of temporary migration are not just felt by individuals and households directly involved, but can be spread much more widely throughout society. An important effect, for instance, might be on governance, because outward migration might reduce pressure upon the government for reform.
In addition, a major challenge for migration evaluation is the lack of reliable data. A 2009 report from the Commission on International Migration Data for Development Research and Policy cites the nonexistence or inaccessibility of "detailed, comparable disaggregated data on migrant stocks and flows as the greatest obstacle to the formulation of evidence-based policies to maximize the benefits of migration for economic development."
More broadly, in terms of developing an evaluation culture in the field of migration, investment in evaluation work has not kept pace with the rapid growth and diversity of migration programs over the last ten years. The International Organization for Migration (IOM) alone implements more than three times as many migration projects and programs today than it did a decade ago. These projects range from providing training to government officials to assisting victims of trafficking or persons being resettled in another country.
A second, yet related, complicating factor in the field is that this recent expansion of migration programming is related to the reframing of migration and migration policymaking towards development.
Until recently, migration has tended to be thought of as primarily affecting outcomes in receiving countries, and the debate was particularly dominated by voices from northern destination states. As a result, policies and programs have tended to be designed in relation to those parties' interests, and evaluated from the same mindset.
The creation of the GFMD in 2007 has helped to promote much more discussion about the impact of migration policies and programs on both sending and receiving countries. In 2010, for example, the GFMD Ad-hoc Working Group on Policy Coherence, Data, and Research organized a seminar on "Assessing the Impacts of Migration and Development Policies."
|
Find Out More
|
||
|
Though ideas about migration have evolved substantially over the last decade or so, however, programming has changed more slowly. An increasing number of interventions are being designed to help developing countries benefit from migration, but the bulk of migration programs and projects have not been planned with any thought to development impacts. And very few of either kind of program — those that recognize migration's effects on development and those that don't — have yet to be evaluated specifically from this perspective.
Lastly, while "communities of practice" have been established in other areas, with the results of evaluations being shared so that all can benefit from the lessons learned, the results of the relatively few existing migration and development evaluations tend to be scattered and not shared systematically between states. Some national authorities and international agencies have conducted important evaluations in recent years, but this information tends to be difficult to find; there is no database or clearing house to facilitate the sharing of the results of these evaluations.
Overcoming Obstacles: Closing the Evaluation Gap
Confronting this rather significant list of challenges may seem daunting, but there are indications that the perceived obstacles to conducting impact evaluations can be overcome. Indeed, there are some specific and relatively doable actions that the migration and development community can take to help close the "evaluation gap."
Lessons can be drawn from the experiences of the development community, where there has been a significant rise in the number of impact evaluations conducted in recent years. Over the past seven years alone, for example, the number of impact evaluations conducted by the World Bank rose substantially, from less than 50 active or completed evaluations in 2004 to approximately 300 in 2010.
Some countries have also increasingly recognized the importance of conducting impact evaluations of development projects. Mexico, for example, passed legislation in 2000 that requires impact evaluations of its social development programs, and the government created the National Council on Evaluation of Social Programs, which is intended to help ensure the quality and integrity of evaluations. Similarly, South Africa has recently established a Department of Performance Monitoring and Evaluation.
These examples can demonstrate the value of conducting impact evaluations for policymakers. Evaluations make it easier to identify what works and what doesn't, giving precise estimates of the benefits of a particular policy or program: statistics that can offer a much better defense for a policy than more general statements. For instance, the evaluation of Progresa / Oportunidades, a well-known program of conditional cash transfers in Mexico, was able to demonstrate that the policy led to a 10 percent drop in poverty levels, a 30 percent reduction in the depth of poverty, and a 9 percent increase in school enrollment.
Where success can be demonstrated, impact evaluations seem important in leveraging increased support for a policy or program, increasing confidence on the part of funders that their money is being well spent. Even when the program is not an all-around success, the evaluation can prompt additional investment for an improved program based on the lessons learned. This, for example, took place in Cambodia following the evaluation of a scholarship fund. While some key goals were not met (e.g., improved results by students), the evaluation allowed for the identification of possible improvements, and a revitalized second stage of the project was supported by a funder.
In addition to the examples available in the development world, there are a small number of evaluations that have been conducted in the migration field that demonstrate how the commonly perceived political and technical obstacles to conducting rigorous evaluations can be overcome.
New Zealand, for example, is one of only a handful of countries that has conducted a rigorous evaluation of the impact of its circular migration program. In April 2007, New Zealand's Recognized Seasonal Employer (RSE) policy was launched with 5,000 workers from Tonga, Vanuatu, Kiribati, Samoa, and Tuvalu working in the horticulture and viticulture industries for up to seven months at a time. The policy seeks to solve labor needs in selected industries while simultaneously contributing to the development of the countries of the Pacific Islands.
An Evaluation Advisory Group consisting of key stakeholders from academia and governmental and nongovernmental organizations in both the sending states and New Zealand was created to advise on research tools and review the reports prepared by the project team. The results of the evaluation of the development objectives of the RSE scheme were very positive. The incomes of RSE households, where one family member participated in the program, were raised by about 35 percent, even after subtracting worker expenses. Additionally, RSE households in Tonga were 12 to 15 percent more likely to purchase durable assets than were non-RSE households, and Ni-Vanuatu RSE households were 27 to 30 percent more likely to make such purchases.
This example also demonstrates the way to begin building a better stock of data on migration, which will enable more rigorous and informed research and policymaking, including greater evaluation possibilities. It is important to prioritize data collection, and to draw on existing data collection tools and approaches such as those implemented in the RSE evaluations and the many existing approaches recommended by the Commission on International Migration Data for Development Research and Policy. Putting these into practice, and making the data widely available, would increase policy-relevant migration knowledge exponentially.
Regarding the fiscal concerns related to conducting impact evaluations, it is important to keep in mind that investment in evaluation studies may represent only a tiny proportion of rising expenditures in migration programs. Moreover, spending on an evaluation can save policymakers money whatever the outcome. If the evaluation identifies that a project is successful and should be scaled up, it ensures money is well spent. If it highlights aspects that are working as well as those that aren't, it allows for the program to be made more effective and efficient. And if it identifies programs that aren't working and should be brought to a close, it is saving money that can be better spent elsewhere.
Nonetheless, for some countries and smaller programs, the up-front cost of conducting an impact study may be deemed too expensive. There are ways, however to reduce costs, including through cost sharing. Governments could come together to identify a few key policies and programs of mutual interest that could be the subject of a "thematic evaluation", and within a couple of years there would be a sound stock of evidence available on how key policies and programs might best be designed. This would be an invaluable resource for policymakers globally. This approach would also encourage countries of origin, transit and destination to work together to conduct joint evaluations to ensure that mobility enhances migration and development outcomes.
A recent report by the Center for Global Development makes a strong case for this approach, asserting that, because governments can't possibly conduct evaluations in all areas of policy on their own, identifying the most important policy questions and pursuing studies around those questions collectively would be an efficient and cost effective way to gain insights into common themes.
In terms of the difficulty of applying the most rigorous research techniques — such as the identification of intervention and control groups — to some aspects of migration policy and programming, this perceived obstacle is not a reason to avoid pursuing evaluation. The aim of an impact evaluation must always be to get the most accurate possible information on the effectiveness of the policy. This means being ambitious and realistic in defining what the best possible information might be, and how it might be uncovered, in each case. The issue to be assessed, the finances available, the skills that can be accessed, the timelines being worked to, and the political context in which the evaluation is happening will all shape the approach that is most suitable.
Lastly, concerning the current lack of knowledge sharing in the migration and development field, there is plenty of potential for sharing what has already been done and the skills and capacities that already exist. It would not be too onerous a task to gather together existing migration evaluations and draw out the key lessons, both for conducting future evaluations and in terms of implications for policy. Similarly, experts and governments that have some experience in this area could share their knowledge through handbooks, trainings, and other capacity building tools. There are relatively easy ways to make more of what we have.
Building an Evaluation Culture
Many of the perceived obstacles to conducting impact evaluations of migration and development projects and programs can be overcome. Simply recognizing this fact and the contribution that impact evaluations could make to migration and development policymaking is the first step to overcoming one of the most significant barriers of all: the lack of an evaluation culture. If an evaluation culture starts to develop, it should generate the momentum required to rise above the technical problems.
Deciding to pursue evaluation, and applying political will to overcome the various barriers and constraints that have prevented progress so far, would be valuable contributions to migration and development policymaking. While there are reasons — particularly, the fear of unwelcome findings — that deter governments from implementing evaluations, the corollary is limited learning about how effectively policies and programs are contributing to key outcomes such as development.
Ultimately, governments are responsible for delivering improvements in people's lives, and it is in their interest to know what effects their policies and programs are having. If impact evaluations can help to show how best to reintegrate returning migrants, or enable migrants to find work that matches their skill levels, or encourage the skilled to return more often, then governments will want to seize this potential.
Moreover, not knowing what works will eventually become a problem for governments, which are required to resolve issues of public and political concern. This is particularly the case with migration policy, because stakeholder concern about the issue is so great. Migration's highly political nature, in other words, makes it all the more important that governments identify what works in migration policy and programming.
One easy way to proceed is to start with less contentious policies and programs, where the trade-offs and associated costs are small. Measures to promote the sending of remittances, for example, or to encourage diaspora involvement in the country of origin, are likely to be less contentious, and therefore easier to evaluate, than interventions aimed at border control. In this way the migration policymaking community can start to gain familiarity with evaluation techniques, their usefulness to policymaking, and the contributions they can make to development and migration outcomes.
In other words, they can start to see a way towards building an evaluation culture.
The views expressed in this article are those of the authors and do not reflect the views of the International Organization for Migration.
Sources
Ardittis, S. and Laczko, F. (2008). Assessing the Costs and Impacts of Migration Policy: An International Comparison. IOM/Eurasylum, Geneva. Available Online.
Chappell, Laura et al. (2010). Development on the Move: Measuring and Optimising Migration's Economic and Social Impacts. Institute for Public Policy Research: London, and Global Development Network: New Delhi.
Gertler, Paul J, et al. (2011) Impact Evaluation in Practice. World Bank, Washington DC. Available Online.
Gibson, John and Mckenzie, David. (2010). The Development Impact of New Zealand's RSE Seasonal Worker Policy: New Zealand's Recognised Seasonal Employer (RSE) Scheme. Working Papers in Economics. Department of Economics, University of Waikato. Available Online.
Lucas, Robert E. B. (2005). International Migration and Economic Development: Lessons from Low-Income Countries. Edward Elgar Publishing. Northampton, Mass.
Martin, Susan. (2009). Global Forum for Migration and Development, Athens, background paper for Round Table 3, Policy and Institutional Coherence and Partnerships, Latest Data and Research Findings.
Ramasamy, Sankar, et al. (2008). The Recognised Seasonal Employer Policy: Seeking the Elusive Triple Wins for Development through International Migration. Pacific Economic Bulletin, 23 (3), 171-186.
Savedoff, William D., Ruth Levine, and Nancy Birdsall. (2006). When will we ever learn. Improving lives through Impact Evaluation. Center for Global Development, Washington, DC. Available Online.
United Nations General Assembly. (2010). International Migration and Development. Report of the Secretary-General. A 65/203. United Nations, New York. Available Online.