How to Measure Institutions

1. How to Measure Institutions Stefan Voigt Philipps-University Marburg Talk prepared for World Bank Seminar on “The Empirics of Governance”, May 1 - 2, 2008

2. How to Measure Institutions I. Introduction • Statement „Institutions Matter“ has become commonplace. • In order to hold any water, two conditions need to be met: • The world does not solely exist of institutions; • Institutions can be empirically ascertained. • This talk: a number of considerations regarding measurement of institutions, 4 main messages: • Try to measure specific institutions rather than aggregate institutional systems; • Objective indicators are generally preferable over subjective ones; • Measure both de jure AND de facto; • Ability to measure does not imply capacity to change institutions. Voigt: How to Measure Institutions

3. I. Introduction (2) Structure of the Talk: • Critique: Have We Been Measuring Policies all these years? • Defining Institutions: A Proposal • Measurement Follows Theory: Assumptions and Implications • Measuring Institutions: Some Pragmatic Proposals • Conclusions and Outlook Voigt: How to Measure Institutions

4. II. Critique: Have We Been Measuring Policies all these years? Glaeser et al. (2004):Yes ! • Institutions are supposed to • Constrain behavior • Be stable over time. • Many of the frequently used measures (they cite the ICRG, the World Bank Governance Indicators and Polity IV) would neither capture policy constraints nor would they be stable. They would, rather, measure outcomes, i.e. policy choices. • The subjectivity of these measures makes them likely to be influenced by income levels (or other outcome variables).  Two serious points! And: Two pitfalls to evade! Voigt: How to Measure Institutions

5. III. Defining Institutions: A Proposal • Attempt to treat Checks and Balances Institutions Within the NIE more generally • Institutions := Commonly known rules used to structure recurrent interaction situations that are endowed with a sanctioning mechanism. • A more fine-grained delineation according to who sanctions: • Convention (self-enforcing); • Ethical rule (self-commitment of actor); • Customs (informal societal control); • Private rule (organized private enforcement); • State law (organized state enforcement). • (1) – (4): sanctioning does not rely on state  „internal institutions“ Voigt: How to Measure Institutions

6. III. Defining Institutions (2) Distinction between Political and Economic Institutions frequent • E.g. Acemoglu et al (2004): • Economic institutions „determine the incentives of and the constraints on economic actors“; • Political institutions „determine the constraints on and the incentives of the key actors, this time in the political sphere“. • Political institutions determine economic institutions. • Implementation of political institutions precarious as it is unclear who could sanction non-compliance  checks and balances as one possible solution. Voigt: How to Measure Institutions

7. IV. Measurement Follows Theory: Assumptions and Implications • Assumption 1: Effects of institutions due to their material content and their factual implementation. • Implication 1: Measure both de jure and de facto. • Assumption 2: The constraining effect of institutions is a function of their factual enforcement which depends on the behavior of those who are to sanction non-compliance. • Implication 2: Behavior of enforcers (police, prosecutors, judges, prison staff, but also the press, lobby groups, public at large) needs to be explicitly taken into account.  Messy but important! Voigt: How to Measure Institutions

8. IV. Measurement Follows Theory (2) • Assumption 3: Factual behavior is likely to be determined by a number of institutions. • Implication 3: Identify all institutions that potentially constrain behavior. • Assumption 4: The factual enforcement of political institutions is often highly precarious and a function of internal institutions. • Assumption 5: Internal institutions are largely exempt from intentional modification • Implication 4: If factual enforcement of political institutions depends on internal institutions, then political institutions should not be entirely at odds with them! Voigt: How to Measure Institutions

9. V. Measuring Institutions: Some Pragmatic Proposals • Before attempting any measurement, a clear concept of the institution is needed. • Predict behavior that would be observed were the institution truthfully implemented. • E.g. if judges are appointed for twelve years, expected average tenure should be twelve years. • „Measure“ the factually observed behavior. • Less difficult regarding political than economic than political (because number of actors is smaller); • Sometimes, appropriately chosen sample will do (e.g. only highest court judges). • Measure behavior over appropriately chosen period of time. Voigt: How to Measure Institutions

10. V. Measuring Institutions (2) • Compare (2) with (3). • Appropriate „measuring rod“ depends on the issue at stake: dummy vs. continuous variable etc.; • Observed variation might also play a role. • If at all possible, rely on objective measures.  Subjective measures often tainted by dependent variable; • Subjective measures influenced by many diverse and implicit theories; • Objective measures (how many times has X happened between 1990 and 2000?) can be verified. • Yes, objective and de facto can be mixed! Voigt: How to Measure Institutions

11. V. Measuring Institutions (3) • Make coding transparent. • Allows other researchers to use the data for their research questions (and recode, reweigh etc.) • If aggregating institutions into indicators, use standard techniques. Voigt: How to Measure Institutions

12. V. Measuring Institutions (4) Some Examples • Central Bank Independence (and accountability) has been measured drawing on objective indicators; but focus clearly on de jure de facto remains to be done; • E.g. number of changes in legal bases, qualification requirements, development of governor‘s salary … • Also true for regulatory agencies (telecom, energy, water, …); • Judicial Independence; Voigt: How to Measure Institutions

13. V. Measuring Institutions (5) Some Examples (cont.d) • Most relevant aspects of the rule of law. • Generality of rules, • their abstractness, • certainty of law, • Absence of retroactive legislation; • Expropriation without compensation; • Habeas corpus; • Protection of confidence; • Principle of proportionality … Voigt: How to Measure Institutions

14. VI. Conclusions and Outlook • Ability to measure institutions combined with econometric findings showing their significance for explaining variation in output variables is not sufficient for reforming or modernizing institutions at will. • Beware of premature policy conclusions! • Many (most?) of the currently available indicators suffer from serious flaws  major data collection exercises ahead of us! Voigt: How to Measure Institutions

15. Thank you for your attention! Voigt: How to Measure Institutions