Around the horn, 20 February
Today on TAL we start a new feature: a quick round-up of a couple of newspaper editorials that pissed me off philosophically and conceptually. Not substantive criticism, but simply an ongoing catalog of places where public intellectuals and other commentators engage in analytically dubious reasoning. Not surprisingly, expect to hear a lot about methodology here…
First off comes
this editorial about social security from
The Washington Post. The substance of the article is a fairly interesting comparison of the terms and metaphors that FDR and GWB use to talk about the Social Security program, noting that where FDR characterized the program as an obligation, GWB talks about "ownership." Lots of examples ensue. So far, so good -- and then there's this bit of wisdom: "How we talk about policy says a lot about how we think about it." Which means
what, exactly? We can take Bush's public statements as indicating something about his genuinely-held personal beliefs about Social Security? The fact that there is controversy about the language deployed in the debate about Social Security indicates that we "think" different things about it now than we did in 1935? (And who's this "we" who was thinking things in 1935 and is still around to think things, collectively, in 2005?)
What we have here is the classic balk committed by liberal individualists who are trying to be social constructionists, in which public articulations are reduced to epiphenomenal indicators of the subjective beliefs which are supposed to
really drive behavior. Although the author seems quite sensitive to the nuances of the public debate and to the different kinds of programs envisioned by the FDR framing and the GWB framing, what is missing is any real sense of the dynamics of framing itself, and hence any real explanation of
why the debate now seems to be about ownership and individual retirements when once upon a time it was about the obligation of society towards its retirees. Implicitly, standing in place of any kind of defensible explanation, we have the vague notion of "thinking," suggesting that this shift is due to "us" changing "our" (collective?) mind. What about the mutations of discourse that make such a shift
possible in the first place, and the concrete deployments of cultural resources that produce
this shift and not others? In short, what happened to the
politics of the debate about Social Security?
Second,
this gem of an editorial (also from the
Post) suggesting that we would have better intelligence estimates if analysts would assign a numerical value to two aspects of their analyses: their confidence in the quality of the evidence that they've used, and their confidence in the conclusions that they've drawn from that evidence. This is basic conditional probability reasoning, the same sort of thing that fuels expected utility forecasts (in which one separates probability of occurrence from payoff amount, and then multiplies down the probability tree to give expected values, and repeats for each step of the process involved). And if this would generate reliable numbers, then yes, we could use Bayesian and other statistical techniques to determine which estimates were the best ones to use as a guide for policymaking.
One small problem: the numbers involved in this procedure are
meaningless. The author says that his modest proposal is based on the success of "data-driven analyses" like those used in sabremetric baseball management and on Wall Street. But a key difference is that in those relatively closed social environments, the numbers in question are generated
by the process under observation, like the movement of a stock or a player's success at getting on base. What the author proposes is instead like asking scouts to estimate how confident they are in their projections of a player's future performance, which is precisely what traditional scouts have been doing for years -- albeit without trying to assign precise numbers to their level of confidence (which strikes me as
more honest than pretending that one can assign a precise number to a subjective level of certainty). The author's other misleading parallel concerns the prominence of Bayesian techniques in modern spam filtering software; Bayesian techniques rely on a massive amount of data (including a massive number of judgments by individual end users about what constitutes "spam"; filter components are weighted based on the relative frequency of some user's designation of particular kinds of e-mail as "spam," and do
not rely on the user's determination that a particular piece of mail is 40% likely to be junk…which would be a meaningless determination in any event) so that updates of prior probability estimates are
meaningful, as opposed to simply representing the needless quantification of a hunch. Once again, this works in relatively closed social systems characterized by repeated acts, and is not likely to be revealing in other settings.
All four of the regular readers of this blog will know that when it comes to baseball, as well as other relatively closed social systems (or social systems that can be safely presumed to be approximately closed for analytical purposes, of which I'd argue that there are considerably fewer in political and social life than we'd like to think), I'm a big partisan of quantitative/statistical/comparative techniques for knowledge-construction. I'd much rather rely on numerical data when managing a baseball team (even a fantasy baseball team) than on "hunches" and "feelings" about particular players (with the exception of Mike Mussina, whom I
always rank higher up in the pre-draft order than his numbers merit; I've seen him do enough amazing things in his career that I generally think he's worth the risk. Let's not forget that he threw six
perfect innings in the ALCS last year against the eventual champion Boston Red Sox, and has twice carried perfect games even deeper, once coming within one out…). Why do I do this? Because
the numbers are meaningful in baseball, just as they are meaningful if one is trying to ease traffic congestion at a busy intersection. But numbers like the ones called for by the op-ed author are
meaningless. The fact that the Holy Writ Of Neopositivist Social Science -- a.k.a. King, Keohane, and Verba's
Designing Social Inquiry -- also calls for such likelihood estimates on the part of researchers doesn't change the basic issue: such numbers can tell one a lot about the
analyst, but I would be seriously skeptical that they can tell us diddly-squat about the
analysis. And I am 1.0 confident in that conclusion.
So much silliness, so little time -- since I have papers to grade and papers to write. So these two will have to suffice for today.
[Posted with ecto]
