Several people are comparing Chris Giles’s piece in the Financial Times, which criticizes the data Thomas Piketty used in his book Capital in the 21st Century, to the Reinhart-Rogoff (R-R) incident from last year. That was when Carmen Reinhart and Kenneth Rogoff’s paper ”Growth in a Time of Debt,” which found that growth went negative above a 90 percent debt-to-GDP threshold, was criticized by Thomas Herndon, Michael Ash, and Robert Pollin (HAP). HAP found data and methodology errors in R-R, and now Giles finds data and methodology errors in Piketty. (I wrote about Giles’s article here.)
So the critiques must be similar, right? No. They are quite different, and in fact there are at least four ways in which they are practically the opposite of each other: in their transparency; in the object of their criticism; in the severity of their critiques; and in their democratic implications.
Transparency and Accessibility of the Data
Piketty’s data is public. That is why we are debating it, because that’s how Giles went about critiquing it. R-R kept their data hidden for years as their policies shaped the international debate over austerity.
R-R based their argument on post-war debt and growth, but their site had no spreadsheet saying “here are the countries and growth rates we used for the post-war period.” Instead they offered links to various other sites for growth data, without clarifying which ones they used. If you tried to replicate the data yourself, as many did, you’d find 110 high-debt data points, but R-R only used 96. Again, it wasn’t clear which were being used.
It’s a minor point, but one worth emphasizing. I can think of at least three sets of economists who stated publicly that R-R had not released their data between 2010 and 2012 . This was before Carmen Reinhart sent their raw data to an innocuous graduate student named Thomas Herndon, which formed the basis for HAP.
Attacking the Data Versus Attacking the Argument
Giles is questioning Piketty’s underlying, original data. HAP took the data that R-R provided for granted, even though it likely would have similar questions, and instead criticized what they did with said data.
A lot of people are pointing out that creating brand new data sets, especially using data that spans countries and centuries, will necessarily involve a lot of difficult calls around merging and splicing various sources. To put that a different way, it would be odd if someone went back into the raw, underlying data and didn’t find some difficult calls that could be questioned.
Critics took R-R’s underlying data for granted in the debate. Perhaps they shouldn’t have. As Bivens and Irons of EPI pointed out in their 2010 discussion, R-R use gross debt, which seems inappropriate compared to debt held by the public if the story they’re telling is about debt and economic outcomes. Yeva Nersisyan and Randy Wray argued that R-R also did a poor job of noting whether a debt was denominated in its own currency.
Those are good points, but they’re not what HAP focused on. They looked at the methodology and construction of results and took the R-R data as given instead of nitpicking the underlying data calls — calls which are always fraught with ambiguity. Critics generally didn’t try to undermine the data R-R presented in This Time is Different; they took on a supplemental argument tacked onto that data, and the problems they found were less subjective and much more devastating.
The Actual Problems Identified Were Far Different in Scale
Giles focused his analysis on the most speculative data chapter in the book. According to Piketty, inequality in the ownership of wealth is one of the two channels that can lead to greater income inequality, but it’s the less important and far more speculative one, developed at the end of the book and added with many, many caveats by the author himself. This chapter is also at the farthest edge of the research frontier, as evidenced by the fact that new research on this topic is still breaking. Even if the whole chapter collapses, there are still very open questions about the growth of capital stock, how much of the economic pie capital will take home, the rise in labor inequality, and many other topics that comprise a much bigger part of the book.
In contrast, within 72 hours of HAP, support for the idea that there was a debt “threshold” collapsed. John Taylor said that the G20, a far cry from a group of liberal bloggers, omitted specific deficit or debt-to-GDP targets as a result of HAP’s critique. What happened?
First, the actual methodological problem was more important in R-R. It became clear that the choices made in weighting and averaging radically overstated the effects of one year from New Zealand in which R-R recorded a negative 7.6 percent change in GDP. But more generally, HAP showed that the final results were very sensitive to minor data adjustments.
This gets confused in the subsequent narrative, but R-R largely accepted the numbers of HAP. In fact, they said that the smaller numbers HAP found were in line with their new research, which found a smaller decline and correlation between debt and GDP, implicitly abandoning their 2010 paper that had become the focus of world policy. But they still argued that a negative relationship was present.
Since the data was made available by HAP, it took only 24 hours before other researchers found major problems that R-R’s response did not address. Specifically, the economist Arin Dube showed that “simple exercises suggest that the raw correlation between debt-to-GDP ratio and GDP growth probably reflects a fair amount of reverse causality. We can’t simply use correlations like those used by R-R (or ones presented here) to identify causal estimates.” In other words, low growth led to a higher debt-to-GDP ratio, not the other way around.
There was no convincing answer forthcoming from R-R about this issue. A month later, the economist Miles Kimball and Yichuan Wang found that they “could not find even a shred of evidence in the Reinhart and Rogoff data for a negative effect of government debt on growth.”
That no other researchers have used Giles’s findings to immediately disprove, or at least cast doubt on, Piketty’s central arguments is telling. This could still happen, so it’s important to be critical. But the general work in Capital, leaving aside the question of inequality of the ownership of capital in Chapter 10, has evolved over decades and has had its tires kicked many, many times. The debt threshold of R-R never passed peer review, and it is unlikely it could have given the obvious reverse causality issues.
The Difference in Democratic Accountability
It seems like everyone who brings up Capital in the 21st Century has to immediately remark about how impossible it would be to do anything about Piketty’s findings given our current reality. Did you hear that Piketty’s solutions in Capital are impractical? They’re impractical, you know. A global wealth tax? Impractical!
But some people, when they act, create their own reality. Even though it had never been replicated, R-R’s paper was immediately moved to the center of elite discussion. It was one of the most cited pieces of evidence during the Great Recession. It became a justification for austerity, and it was one of the central economic arguments for the Ryan Plan, the budget that Mitt Romney would have tried to put into place had he won the 2012 election.
Some people want to argue that the R-R Excel error was no big deal. And in an econometric sense, they might have a point. But in a political sense, it mattered. It showed that hundreds of millions of people’s lives were being guided by a piece of research with an error that literally anyone would have found if R-R had let another set of human eyeballs look at it.
This is why democratic accountability is so important with economics. It’s good to see Piketty checked here, even if the concerns are overplayed; as Piketty says, the distribution of wealth “is too important an issue to be left to economists, sociologists, historians, and philosophers. It is of interest to everyone, and that is a good thing.” Indeed it is, just as austerity and government budgets are. First, “Government Debt and Economic Growth.” Bivens and Irons of EPI, July 2010, footnote 5: “The actual data used in the [R-R] study have not been made available to the public by the authors.”
Second, “Not Following Professional Ethics Matters Also.” Dean Baker, July 2010: “Mr. Rogoff and Ms. Reinhart have declined to adhere to standard ethics within the economics profession and have refused to share the data on which they base their conclusion with other researchers.”
Third, “Is High Public Debt Always Harmful to Economic Growth?” Minea and Parent, Feburary 2012, footnote 4: “Our efforts for obtaining the database used by RR were…unfortunately unsuccessful.”