Causal Analysis in Theory and Practice

Judea Pearl writes:

A new book on causality came out last month, Hunting Causes and Using Them by Nancy Cartwright (Cambridge University Press, 2007.) Cartwright is a renown philosopher of science who has given much thought to the methodology of econometrics, and I was keenly curious to read her take on the current state of causality in economics.

Cartwright summarizes what economists such as Heckman, Hoover, Leroy and Hendry said and wrote about causal analysis in economics, she occasionally criticizes their ideas, and further discusses related works by philosophers such as Hausman and Woodward, but what I found surprising is that she rarely tells us how WE OUGHT to think about causes and effects in economic models. Given that economists admit to the chaotic state of affairs in their court, the role of philosophy should be, in my opinion, to instill clarity and provide coherent unification of the field. This I could not find in the book.

Additionally, and this naturally is my main concern, Cartwright rejects the surgery method as the basis of counterfactual and causal analysis and, in so doing, unveils and reinforces some of the most serious misconceptions that have hindered causal analysis in the past half century (see my earlier posting on Heckman's articles.)

I will focus on the latter point, for this will illuminate others.

Cartwright description of surgery goes as follows (quoting from p. 246):

"Pearl gives a precise and detailed semantics for counterfactuals. But what is the semantics a semantics of? What kinds of counterfactuals will it treat, used in what kinds of contexts? Since Pearl introduces them without comment we might think that he had in mind natural language counterfactuals. But he presents only a single semantics with no context dependence, which does not fit with natural language usage.

Worse, the particular semantics Pearl develops is unsuited to a host of natural language uses of counterfactuals, especially those for planning and evaluation of the kind I have been discussing. That is because of the special way in which he imagines that the counterfactual antecedent will be brought about: by a precision incision that changes exactly the counterfactual antecedent and nothing else (except what follows causally from just that difference). But when we consider implementing a policy, this is not at all the question we need to ask. For policy and evaluation we generally want to know what would happen were the policy really set in place. And whatever we know about how it might be put in place, the one thing we can usually be sure of is that it will not be by a precise incision of the kind Pearl assumes.

Consider for example Pearl's axiom of composition, which he proves to hold in all causal models – given his characterization of a causal model and his semantics for counterfactuals. This axiom states that `if we force a variable (W) to a value w that it would have had without our intervation, then the intervention will have no effect on other variables in the system' (Pearl 2000, p. 229). This axiom is reasonable if we envisage interventions that bring about the antecedent of the counterfactual in as minimal a way as possible. But it is clearly violated in a great many realistic cases. Often we have no idea whether the antecedent will in fact obtain or not, and this is true even if we allow that the governing prinicples are deterministic. We implement a policy to ensure that it will obtain – and the policy may affect a host of changes in other variables in the system, some envisaged and some not." (Cartwright 2007, pp. 246-7)

To summarize, Cartwright claims are:

1. Surgery requires that equations be modular
2. Normally, economic equations are not modular
3. Surgery does not give answers to practical questions about the efficacy of real life interventions, because those are accompanied with unintended, implementation-dependent side effects.

Succinctly, my answers to Cartwright are:

1. Surgery does not require modularity.
2. Economic equations, by and large, ARE modular
3. Surgery, when taken seriously, does give the answers to questions about real life interventions.

I will now elaborate on each of these three points:

1. Surgery does not require modularity.
   Surgery, and the whole semantics and calculus built around it, does not assume that in the physical world we always have the technology to incisively modify the mechanism behind each structural equation while leaving all others unaltered. Symbolic modularity does not assume physical modularity. Surgery is a symbolic operation and makes no claims about the physical means available to the experimenter, nor about possible connections that might exist between the mechanisms involved.

   Symbolically, one can surely change one equation without altering another and proceed to define quantities that rest on such "atomic" changes. Whether the quantities defined in this manner have anything to do with changes that can be physically realized is a totally different question that can only be addressed once we have a formal description of the interventions available to us.

   An example will help.

   Smoking cannot be stopped by any legal or educational means available to us today, but cigarette advertising can. This appears to be a violation of modularity — one cannot change the mechanism behind smoking without changing the mechanism behind advertisement. Does that forbid scientists from speaking about "the effect of smoking on cancer"? Should that inhibit mathematicians from defining formally what is meant by "the effect of smoking on cancer" just because the symbolic surgery invoked in the definition of this concept is not directly implementable? Quite the opposite. By defining and analyzing such ideal quantities we often find that these could be inferred indirectly, from experiments that ARE physically implementable (see Causality, pages 88-89). Such opportunities would be missed were scientists to abide by Cartwright's doctrine and refrain from defining, formalizing and analyzing those "ideal" interventions, or "impostor counterfactuals" as she calls them.

   Thus, despite the fact that the system is non-modular from implementation viewpoint, the mathematics of symbolic surgery permits us to estimate the desired, non-implementable quantities from surrogate implementable experiments and, not least important, prove that the result is valid.

2. Economic equations, by and large, ARE modular.
   In the example above, an orthodox opponent of surgery may argue that although smoking ban is not enforcible, smoking nevertheless is conceptually stoppable without affecting cigarretes advertising, hence the example does not properly represent an INHERENT type of non-modularity, a type that governs most economic systems.

   Let us examine this kind of non-modularity as described on page 15 of Cartwright's book.

   "When Pearl talked about this recently at LSE he illustrated this requirement with a Boolean input-output diagram for a circuit. In it, not only could the entire input for each variable be changed independently of that for each other, so too could each Boolean component of that input. But most arrangements we study are not like that. They are rather like a to
   aster or a carburettor."

   At this point, Cartwright provides a 4-equation model of a car carburettor and concludes:

   "Look at equation (1). The gas in the chamber is the result of the pumped gas and the gas exiting the emulsion tube. How much each contributes is fixed by other factors: for the pumped gas both the amount of airflow and a parameter a, which is partly determined by the geometry of the chamber; and for the gas exiting the emulsion tube, by a paramenter a', which also depends on the geometry of the chamber. The point is this. In Pearl's circuit-board, there is one distinct physical mechanism to underwrite each distinct causal connection. But that is incredibly wasteful of space and materials, which matters for the carburettor. One of the central tricks for an engineer in designing a carburettor is to ensure that one and the same physical design – for example, the design of the chamber – can underwrite or ensure a number of different causal connections that we need all at once.

   Just look back at my diagrammatic equations, where we can see a large number of laws all of which depend on the same physical features – the geometry of the carburettor. So no one of these laws can be changed on its own. To change any one requires a redesign of the carburettor, which will change the others in train. By design the different causal laws are harnessed together and cannot be changed singly. So modularity fails." (Cartwright 2007, pp. 15-16)

   Thus, for Cartwright, a set of equations that share parameters is inherently non-modular; changing one equation means modifying at least one of its parameters and, if this parameter appears in some other equation, it must change as well, in violation of modularity.

   Heckman, readers should recall, makes similar claims: "Putting a constraint on one equation places a restriction on the entire set of internal variables." "Shutting down one equation might also affect the parameters of the other equations in the system and violate the requirements of parameter stability" (Heckman, Sociological Methodology, page 44)

   These fears and warnings are illusionary. Shutting down an equation does not necessarily mean meddling with its parameters, it means overruling that equation, namely, leaving the equation in tact but lifting the outcome variable from its influence.

   Let's take a simple example to illustrate this point.

   Assume we have two objects under free fall condition. The respective accelerations, a₁ and a₂ of the two objects are given by the equations:

   a₁ = g     (1)
   a₂ = g     (2)

   where g is the earth gravitational pull. The two equations share a parameter, g, and appear to be non-modular in Cartwright's sense; there is indeed no way to modify the gravitational parameter in one equation without a corresponding change in the other. However, this does not mean that we cannot intervene on object 1 without touching object 2. Assume we grab object 1 and bring it to a stop. Mathematically, the intervention amounts to replacing Eq. (1) by

   a₁ = 0     (1')

   while leaving Eq. (2) in tact:

   a₂ = g     (2)

   Setting g to zero in Eq. (1) is a symbolic surgery that does not alter g in the physical world but, rather, sets a₁ to 0 by bringing object 1 under the influence of a new force, f, emanating from our grabbing hand. Thus, Eq. (1') is a result of two forces:

   a₁ = g + f/m₁     (1''')

   where f = – gm₁, which is identical to (1).

   This operation of adding a term to the rhs of an equation to ensure constancy of the lhs is precisely how Haavelmo (1943) envisioned surgery in economic settings. Why his wisdom disappeared from the teachings of his disciples in 2007 is one of the great mysteries of economics (see Hoover (2004), "Where Have All the Causes Gone?").

   This same operation can be applied to Cartwright carburettor, for example, the gas outflow can be fixed without changing the chamber geometry by installing a flow regulator at the emulsion tube. It definitely applies to economic systems, where human agents are behind most of the equations; the lhs of the equations can be fixed by exposing agents to different information, rather than changing parameters in the physical world. A typical example emerges in job discrimination cases. To test the "effect of gender on hiring" one need not physically change applicant's gender; it is enough to change employers awareness of the applicant's gender.

   I am yet to see an example of an economic system which is not modular in the sense described here.

   I now return to the third topic, concerning practical interventions, and elaborate on my claim that:

3. Surgery, if taken seriously, does give the answers to questions about real life interventions.
   My point is that a calculus based on surgery is precisely the mathematical tool we need for answering questions about the efficacy of real life interventions, like the one described by Cartwright, which modify several mechanisms at once, and are loaded with unintended side effects.

   Let us return to the example of cigarette advertisement. Suppose a policy maker is interested in the impact of cigarette advertising on lung cancer. Can we truly say that the effect of smoking on cancer, a quantity defined by non-realizable surgery, is irrelevant to the policy maker, just because political considerations forbid direct intervention on smoking? Quite the contrary. We often find that the target quantity can be decomposed into atomic quantities and the decomposition allows us both to define, analyze and decide whether the target quantity is identifiable in a given experimental context (see Causality, pages 81-82). Chapter 3 and 4 of Causality demonstrate through ample examples how, counter to intuition or conventional wisdom, the impact of complex interventions can be predicted from non experimental data using the calculus of atomic interventions.

   Ironically, by shunning mathematics based on "impostor counterfactuals" (i.e., surgery) Cartwright, like Heckman, Leroy, Dawid and others, condemns scientists to ineptness in handling "genuine counterfactuals."

   Science and mathematics are full of auxiliary abstract quantities that are not directly measured or tested, but serve to analyze those that are. Pure chemical elements do not exist in nature, yet they are useful in understanding the behavior of alloys and compounds. Negative numbers do not exist in isolation, yet they are essential in understanding and manipulating positive numbers.

I personally have found that, invariably, questions about interventions and experimentation, ideal as well as non-ideal, practical as well as epistemological, can be formulated precisely and managed systematically using the atomic intervention as a primitive notion. I am confident that Cartwright's ideas, too, would benefit from the clarity offered by this mathematical language.