Corrupt Indexes, Part I

Over the course of my stay so far, I've had a number of conversations with various people about the quality of governance in Nepal. Everybody I've spoken to thinks that the government here is failing to meet the needs of its citizens. Here are a just few reasons why they might feel that way...

• The roads are riddled with potholes and are generally falling apart. Even when there is no traffic, cabs rarely go faster than 30 mph as any faster and they risk blowing out a tire or wrecking their shocks. Basic traffic control mechanisms like stop lights, traffic signs, etc. either don't exist or don't function.
• Kathmandu has no water purification infrastructure that I am aware of. Everyone purifies their own water because the government can't do it for them. Similarly, the rivers that run through Kathmandu are visibly polluted.
• Only 40% of Nepalis have access to electricity. Kathmandu experiences load-shedding blackouts that go as long as 20 hours a day during the late winter months. This is particularly egregious since the country's hydroelectric infrastructure currently taps only 2.765 Billion KWh/y - roughly 1% of the country's hydroelectric potential. 

Road maintenance, water quality control, and electricity production - these all look like services that either the state or a state-regulated private entity should provide. The key question, though, is why the Nepali government has failed to do so. I've heard a wide variety of answers to that question. Poor decision making, general political strife (the parliament has failed to elect a new PM 7 times), a lack of resources. Take your pick. And then there's one explanation that I find very interesting: the government is corrupt.

The corruption argument is strange for one particular reason. Corruption is usually a secretive activity. It's unlikely that the ordinary people I'm talking to really know about corruption in their government. I mean they may know that you have to pay a bribe here or there if you want something, but this is not the sort of corruption we're talking about. The stories I have heard conjure up massive embezzlement schemes that literally leech most of the state's resources into the pockets of politicians. I don't think that this is the sort of thing ordinary people really know about. In other words, corruption is sort of the boogeyman of governance. It's hearsay.

This got me curious about what we actually know about Nepali corruption and, more generally, how corruption is measured. As some of you already know, I had some minor stomach troubles last Friday. As a result, I spent most of the day at home doing some research. I learned a little about Nepali corruption (mostly from here). But I learned a lot about corruption indexes - the tools NGOs, aid donors, and others use to quantify levels of corruption in specific countries. The rest of this post is about those indexes.

(OK before I go on, a few disclaimers. I am not a statistician. I've tried to keep all of what follows accurate, non-misleading, etc... but if there's an error that I make, let me know in the comments and I'll fix it...)

Two things surprised me. First, some of these indexes are assembled using methodologies that almost completely undermine their results. Second, despite huge methodological flaws (and essentially meaningless results) economists, aid agencies, and the media consistently draw major (and equally bogus) conclusions from these same indexes.

To illustrate this, let's examine Transparency International's Corruption Perception Index (conveniently abbreviated as CPI, which isn't at all easy to confuse with the Consumer Price Index...) The CPI "ranks countries in terms of the degree to which corruption is perceived to exist among public officials and politicians." Perceived by who? Business people and "experts." Here's a step by step break down of their methodology.

1. Transparency International gathers their raw data from a number (13 this year) different surveys that ask either business people or experts questions about corruption in a particular country. Each survey is conducted in a different set of countries and asks a different set of questions.
2. Transparency International then converts the results of these surveys to the same numerical scale and averages the results for each country.
3. The result of this aggregation are then published yearly as the CPI.

You can find much more detail about the methodology used to construct the CPI here. The media loves these sorts of indexes and especially the CPI. In some countries, the motherlands ranking in the newest CPI report is front page news.

Now, let's go through several ways in which the CPI methodology is flawed. First, the CPI averages different surveys. They claim this makes their results more reliable. That might be true if all these surveys were conducted in all the countries that the CPI ranks. But they're not.

Take a hypothetical: only 3 of the surveys that the CPI uses might be conducted in Nepal, while 8 are conducted in China. Now recall that all the surveys ask differently worded (and sometimes just plain different) questions. So let's say that the new CPI report tells us that Nepal is more corrupt than China. Does that mean that Nepal really is more corrupt than China. Well, no, primarily because the CPI measures perceptions of corruption as opposed to actual perception. But worse still we can't even say with any certainty that Nepal is perceived to be more corrupt than China. That might be what the data indicates, but alternately it could just be that the 3 surveys used in Nepal asked questions that were worded in such a way as to generate relatively more negative perceptions of corruption than the 8 surveys used in China.

Ok so that's a problem. Indexes are supposed to offer the ability to compare a number of things either between each other, or over time. That's what indexes are for. And yet here we have a Corruption Perceptoins Index that doesn't really allow for comparability of corruption between countries because of the methodology with which the index is constructed.

Oh well. At least, the CPI for a particular country offers some comparability over time, right? Nope. The CPI doesn't aggregate the same set of surveys for each country each year. In other words, the 1999 CPI might have aggregated 3 surveys for Nepal while the 2005 CPI might have aggregated 8 surveys, or 3 completely different surveys, for Nepal. And this doesn't even account for the possibility that these surveys might periodically change the questions they ask or the wording of those questions.

So take another hypothetical: say that Nepal ranks better in the 2009 CPI than in the 1999 CPI. Does that mean corruption is becoming less of a problem in Nepal? The CPI can't tell you. It could be that corruption is becoming less of an issue. But it could just as easily be that the differences between the surveys that were aggregated in 1999 and 2009 are the reason for the improved ranking. In that case, the use of different sets of surveys gives the appearance of improvement when in fact there is none.

Also, what does it mean that the different surveys that the CPI uses actually correspond to each other quite closely? Transparency International argues that this should give us more confidence in the accuracy of the data that they are aggregating. I'm not so sure. It could just be that the different surveys have an influence on each other or on their respondents. I.e. you might be more likely to perceive high levels of corruption in your country if you know that some other surveys say that people in your country perceive high levels of corruption. It could also be that all the surveys that the CPI aggregates are reproducing the same systemic error. All these surveys assume or suggest that perceptions of corruption closely track actual levels of corruption. What if that assumption turns out to be completely false? Then all the surveys the CPI uses contain the same systemic error. That these surveys provide similar results would not somehow make this less of an error.

One final complaint. Even if the CPI were methodologically more robust, it wouldn't provide much information that you could use to formulate anti-corruption policy. As policy wonks would say, the CPI is non-actionable information. It doesn't tell you what kinds of corruption are a problem, where in the government (or the private sector) corruption tends to occur, etc. This has got to be ridiculously frustrating for governments that know they have a corruption problem and actually want to do something about it.

The worst part of all this, of course, is that the media (and economists, and aid agencies, and etc) will be irresponsible enough to pick up the CPI and misuse it because they didn't bother to understand its methodology. They do it every year. Anecdotally, I have been reading papers that look for a correlation between corruption and economic growth (or the lack thereof). I have yet to see a single paper that doesn't plug the CPI into an equation and draw a conclusion based on that. It seems intuitive that corruption should hamper economic growth but I'm worried (i.e. I'm concerned but not completely certain) that because all these economists misuse the CPI, no one has empirically demonstrated that relationship.

OK. That's enough ranting for now. Look for Part II in the near future. It will be about good news - there are some corruption indexes that don't make these mistakes and they're doing really quality work. In the meantime, if you're interested, here are two  papers that go into the methodological problems with some corruption indexes and ways to avoid those problems.