Measuring Regulatory Performance: A Practitioners' Guide to Perception Surveys

Now more than ever, OECD countries are investing significant resources in regulatory policies and reforms. At the same time, governments are under increasing pressure to explain such reforms and their benefits to the public. Perception surveys are an important part of this process and they are being used by OECD members to measure how citizens and businesses view regulation in their countries.

The OECD has developed a guide that: helps officials use perception surveys to evaluate and communicate the results of reform processes;clearly explains the challenges involved in the design and use of business and citizen perception surveys – and ways to overcome them and helps to get the most out of survey results, whether conducted internally or by external experts.

Perception surveys are an increasingly integral component of a business- and citizen-centred approach to regulatory reform, as a means to assist governments with better results in an open, democratic system. This guide helps officials planning perception surveys or engaging external expertise to use perception surveys for evaluating and

communicating progress in regulatory reform. It explains the challenges involved in the design and use of business and citizen perception surveys – and ways to overcome them. It will also help officials responsible for writing and evaluating tenders for surveys judge the quality of consultants’ work and get the most out of survey results. The guide is written in non-technical language for a broad audience, drawing on examples from the regulatory field.

The following ten key policy messages are presented in this guide:

1. Understanding and improving the perception of the regulatory environment matters to performance. Positive perceptions of regulations can influence investment decisions and promote respect for and compliance with regulations. (Introduction)

2. Perception surveys are increasingly used in OECD countries to evaluate the performance of regulatory reform programmes, in particular in the area of reducing administrative burdens. Perception surveys are further used to obtain information on the level of awareness and confidence in regulatory reform programmes among businesses and citizens, and as a diagnostic tool to identify areas of concern to business and citizens in order to inform future regulatory reforms. (Chapter 1)

3. If pitfalls in survey design are ignored, survey results become unusable for policy makers. There are a surprising number of pitfalls in designing surveys. For example, even the order and phrasing of questions can affect responses and the quality of survey results. (Chapter 2)

4. Using good practice methodologies will improve the quality of results considerably and help to avoid pitfalls. For example, it is advisable to run pilot surveys to identify questions that respondents have difficulty understanding and then adjust questions accordingly. (Chapter 3)

5. Perceptions and hence survey results are shaped by many factors; the actual quality of regulations is only one of them. For example, perceptions of the quality of regulations can be influenced by trust in government, the current economic situation, experience with front-line service, prior expectations and the content of government (and general media) communication. (Chapter 4)

6. It is necessary to look beneath survey results. The same survey results may be driven by very different underlying factors. In-depth questions and selected qualitative research techniques can prove very valuable in bringing to light the reasons for the results and drawing concrete policy conclusions from survey results. (Chapter 4 and 5)

7. Irritation from experiences with regulation and frontline service can account for a significant degree of business and citizens’ dissatisfaction with regulation. This experience is often more negative than might be suggested by the measurable costs of administrative burdens. (Chapter 4 and 5)

8. Perception surveys also have their limitations. Experience suggests the likelihood of a disparity between the perceived quality of regulations as reported by business and citizens and the measurable results of regulations. For example, in many countries surveys have tended to reveal negative perceptions of the quality of regulations while more fact-based measurements have shown an improvement. This appears to apply particularly to programmes targeted at reducing administrative burdens. (Chapter 5)

9. A comprehensive evaluation system should include different types of indicators, each revealing different information for policy evaluation. Discrepancies in results can highlight the need for deeper analysis to evaluate and inform policies. (Chapter 5)

10. Perception surveys are an integral part of a two-way communication strategy with stakeholders. They can serve as a means to communicate stakeholder views to the government, and discussion of the results can lead to fruitful exchanges between government and stakeholders in the case of regulatory reform. Survey results can also help to evaluate the success of the government’s communication strategy by assessing stakeholders’ level of awareness of recent initiatives. (Chapter 5)

Measuring Regulatory Performance: A Practitioners' Guide to Perception Surveys

x

Identifying and evaluating large scale policy interventions: what questions can we answer?

The current literature on empirical development economics contains a vigorous debate as regards the variety of approaches to understanding the development paradigm. Economic and regulatory policies, human and physical resources, and governance are all seen as drivers of economic development and yet there is no agreement on the exact roles each play in driving development. The fundamental puzzle retains two questions: Do the quality of one of these factors matter more for sustained economic growth and development? Is it the case that their relative importance is homogenous across countries and stages of development?

One strand of development economics contends that a combination of a stable macro-economy with market and investment climate-friendly set of economic policies is the necessary and sucient condition for economic growth, which then drives other concepts of development, such as living standards and health outcomes. This view is associated with the Washington Consensus, which was the dominant paradigm of the 1980s and 1990s. It is under this consensus that multilateral lending agencies advised successive developing country governments on the need to fix budgets, cut inflation, and watch the economy take off with the dynamism of its private businesses. The experience of various countries, especially in Latin America that followed structural reforms to heart and yet ended up with sub-par growth performance in the 1990s however was a crude blow to this approach. This prompted even the World Bank, one of the proponents of the Washington Consensus to focus beyond stable macro-economic conditions and business-friendly reforms, as it did on its report on the growth experience of the 1990s.

A rival strand, which goes back to Gerschenkron’s theories on backwardness, believes in the context-specific and more subtle nature of development. Recent proponents contend that institutions matter, and well-governed countries are likely to grow faster, be more effective in alleviating poverty, and are also likely to be more successful in creating equitable growth and redistribution. A radical strand of this approach is one taken by Rodrik, Hausmann and Velasco,

as the authors deviate significantly from a model of growth that specifies a nature of homogeneity in reforms to catalyze growth. Their growth diagnostics approach attempts to identity binding constraints, which are notions of the most important areas where reforms can bring the largest and quickest benefits. In this framework, improving trade barriers can either strongly accelerate the growth process; or have no impact on growth trends at all, depending on the need of the economy at the hour. The binding constraint of an economy hence dictates how successful particular reforms are.

Empirical work on development macroeconomics has not been able to converge on one strand or another. Initial theoretical and empirical work into growth accounting delivered results that both trends in total factor productivity and capital accumulation (physical and human) have significant roles in driving growth, while productivity trends appear to be more important. These results however move us no further in the above questions, because various economic policies, governance efficiency and institutional quality all affect trends in productivity and capital accumulation. Without a strong macro-economic environment, it is hard to create significant private investment. Rampant corruption mostly goes along with weak incentives for firms to compete

and hence to lower productivity growth. We can think of numerous other examples, but the point remains; the question of relative impact of reforms in one dimension against another is still very much left unanswered.

A large part of the problem is the inability of empirical macro-research to converge on methods and techniques that can attribute causal effects of interventions on outcomes. Traditionally, OLS regression methods have been used on country cross-section data to study the links between macro policy/governance interventions and outcome variables. These methods have been criticized for the lack of accuracy emanating from problems of omitted variable bias and endogeneity, model uncertainty and heterogeneity. Researchers have attempted to solve some of these problems by using instrumental variables approach, for example Acemoglu et al. (2001).

These studies are important steps towards estimating causal inference in cross-country studies and yet problems remain. Strong external instruments are hard to find, and even if they exist, the effect obtained from such regressions are limited to the variation that is affected by the instrument. These problems make the instrumental variables approach limited in its ability to evaluate macro interventions. This has led some researchers to use dynamic panel data methods,

which account for endogeneity issues and yet other problems remain, such as the instability of results from cross-country regressions when variables, time periods and countries in the data change.

Our goal in this paper is not to attempt to resolve the two aforementioned questions in their entirety. Such an endeavor requires a much larger and longer research agenda. We merely attempt to identify specific cases of reforms in the disparate dimensions, ranging from macroeconomic anddebt dynamics to governance reforms in the last decade, and attempt to assess their impact on development indicators such as economic growth. We do this by using recently developed methods of inference, which help us attribute with larger certainty the impact on outcome variables to the interventions themselves. By selecting case studies where changes in one dimension were not accompanied by changes in other dimensions, the paper provides a window of understanding into whether reforms in certain dimensions affected development outcomes by themselves or not.

We also perform robustness checks to see whether the results of these methods are sensitive to minor modeling changes. We believe that by building a larger catalogue of such results, we can attempt to start understanding the questions highlighted above. In order to complete this task, we need to first identify significant large scale interventions in policy and governance dimensions. Chapter II of the paper explains our methodology on how to identify and measure large recent interventions on policy and governance dimensions. Chapter III explains the methodology we use to understand how successful these interventions have been in contributing towards developm

World Bank.Author: Dhungana, Sandesh.Document Date: 2011/12/01.Document Type: Policy Research Working Paper.Report Number: WPS5918

Identifying and evaluating large scale policy interventions: what questions can we answer? x