Causal Analysis in Theory and Practice

In his 2005 article "The Scientific Model of Causality" (Sociological Methodology, vol. 35 (1) page 40,) Jim Heckman reviews the historical development of causal notions in econometrics, and paints an extremely complimentary picture of the current state of this development.

As an illustration of econometric methods and concepts, Heckman discusses the classical problem of estimating the causal effect of Y₂ on Y₁ in the following systems of equations

Y₁ = a₁ + c₁₂Y₂ + b₁₁X₁ + b₁₂ X₂ + U₁     (16a)
Y₂ = a₂ + c₂₁Y₁ + b₂₁X₁ + b₂₂ X₂ + U₂     (16b)

where Y₁ and Y₂ represent, respectively, the consumption levels of two interacting agents, and X₁, X₂, the levels of their income.

Unexpectedly, on page 44, Heckman makes a couple of remarks that almost threw me off my chair; here they are:

"Controlled variation in external (forcing) variables is the key to defining causal effects in nonrecursive models. It is of some interest to readers of Pearl (2000) to compare my use of the standard simultaneous equations model of econometrics in defining causal parameters to his. In the context of equations (16a) and (16b), Pearl defines a causal effect by "shutting one equation down" or performing "surgery" in his colorful language."

"He implicitly assumes that "surgery," or shutting down an equation in a system of simultaneous equations, uniquely fixes one outcome or internal variable (the consumption of the other person in my example). In general, it does not. Putting a constraint on one equation places a restriction on the entire set of internal variables. In general, no single equation in a system of simultaneous equation uniquely determines any single outcome variable. Shutting down one equation might also affect the parameters of the other equations in the system and violate the requirements of parameter stability."

I wish to bring up for blog discussion the following four questions:

1. Is Heckman right in stating that in nonrecursive systems one should not define causal effect by surgery?
2. What is the causal effect of Y₂ on Y₁ in the model of Eqs. (16a -16b) ??
3. What does Heckman mean when he objects to surgery as the basis for defining causal parameters?
4. What did he have in mind when he offered "… the standard simultaneous equations model of econometrics" as an alternative to surgery "in defining causal parameters"?

The following are the best answers I could give to these questions, but I would truly welcome insights from other participants, especially economists and social scientists (including Jim Heckman, of course).

1. My answer to Question 1 is that there is no difference between recursive and nonrecursive systems in defining causal effects, in both cases causal effects are well defined by the surgery procedure. Thus, I disagree with Heckman's first sentence above, and for two reasons. First, it is not "Controlled variation in external variables" that is the key to defining causal effects" but rather, hypothetical variation in internal (endogenous) variables. Second, this is true for both recursive and nonrecursive systems. In fact, Haavelmo (1941), to whom Heckman attributes the idea of surgery (see his page 1 and, again, page 40) introduced his argument using a nonrecursive system of equations.
2. My answer to question 2 is simply: c₁₂. This is what the surgery procedure dictates and, since the surgery emulates the desired hypothetical variations, this is what we must accept. To witness, we shut off Eq. (16b), replace it with Y₂ = y₂, and we find that, in the resulting mutilated system Y₁ changes by an among c₁₂ with a unit change in y₂. Cut and dry; no ifs, no buts. This is the beauty of mathematical definitions; all the ifs and buts are pushed to discussions about whether the model is valid. But, once this is established, clarity prevails and argumentation gives way to computation.

   Heckman himself admits so on page 40 of his article, stating: "Applying Haavelmo's argument to (16a) and (16b), the causal effect of Y₂ on Y₁ is c₁₂. This is the effect on Y₁ of fixing Y₂ at different values, holding constant the other variables in the equation." And, again, on page 41, he states: "Without any further information on the variables of (U₁, U₂) and their relationship to the causal parameters, we cannot isolate the causal effects c₁₂ and c₂₁ from the reduced form regression coefficients." Thus, Heckman twice bestows the title "causal effect" onto the term c₁₂, so, even those who trust Heckman more than they trust the logic of surgery must surrender to the latter and state: "The causal effect of Y₂ on Y₁ is: c₁₂".

3. My answer to question 3 is: I am at a loss. Heckman does not explicate his objections to the surgery definition and, until such time that he does, we will not be able to tell whether he is wrong about surgeries alone or about other aspects of causal modelling as well.

   Let us examine Heckman's second paragraph, for it gives a clue on whether he understands the surgery procedure the way we do, and the way it was understood by its original conceptualizers (e.g., Haavelmo, Marschak, Strotz, and Wold). Re-quoting, Heckman says: He [Pearl] implicitly assumes that "surgery," or shutting down an equation in a system of simultaneous equations, uniquely fixes one outcome or internal variable (the consumption of the other person in my example)." This is not the way surgery is defined in the literature (see Causality, section 7.2.1). The surgery does not merely "shut down" an equation but also replaces it with the constant equation, e.g., Y₂ = y₂. The uniqueness of Y₂ under this condition is guaranteed.

   I wish to believe that this oversight is the only source of the misunderstanding but, unfortunately, Heckman's next sentence reveals a second source. It reads: "In general, it does not. Shutting down one equation might also affect the parameters of the other equations in the system." Heckman must be thinking about some other operation, not surgery. When we shut down equation Eq. (16b), the other equation (16a) stands before us in full day light, and proudly boasts how all its parameters remain unaltered. Recall, the surgery procedure is a man made symbolic operation, defined precisely to embody the requirement of invariance, or as Heckman puts it: "the requirements of parameter stability."

   Judging by his assertion that surgery may violate parameter stability, it seems that Heckman conflates the DEFINITION of causal effect with the practical means available for TESTING it. Whereas the latter can be crude or inadequate, the former is always pure and incisive.

   This conflation is reflected again on page 43, where Heckman states: "If for a given model, the parameters of (16a) or (16b) shift when external variables are manipulated, or if external variab
   les cannot be independently manipulated, causal effects of one internal variable on another cannot be defined within that model (italic in the original)." The first part of the sentence speaks about physical manipulation, while the second speaks about a mathematical definition. The sentence as a whole, implies that the causal effect of Y₂ on Y₁ depends on the technology available to the experimenter — hardly a dependency we would expect from a "definition".

   On page 44, Heckman argues again that the definition of causal effects must depend on the physical manipulations available to the experimenter. He describes a situation in which the behavior of agent 2 changes, depending on whether U1 is randomized or chosen naturally . He then says: "At issue is whether such a randomization would recover c₁₂. It [randomizing U₁] might fundamentally alter agent 1's response to Y₂ if that person is randomly assigned as opposed to being selected by the agent. Judging the suitability of an invariance assumption entails a thought experiment — a purely mental act." Again, Heckman confuses the causal effect c₁₂ as encoded in (16), with some perturbed causal effect c'₁₂ that prevails when certain interventions are implemented to measure c₁₂. Obviously, if an experimental intervention modifies c₁₂ or has other side-effects, the modification must be encoded in the model and assessed whether the original c₁₂ can be recovered from data obtained under such imperfect manipulation. But this does not change the fact that the original c₁₂ (prior to intervention) is the parameter of interest and, more importantly, that c12 can be defined within the original unperturbed model Eqs (16a) and (16b) by a mental act of shutting down eq. (16b), fixing every variable on the rhs to a constant, Y₁=y₁, X₁=x₁, X₂=x₂, and computing mathematically the partial derivative (with respect to y₁) of the expected value of Y under this mathematical surgery.

   The quantity computed this way, d E[Y|do(y₁,x₁,x₂)]/ d y₂ would be none other but Heckman's target quantity, c₁₂ — the causal effect of Y₂ on Y₁ under normal behavior of the two agents, as described by the model. It does not matter if we do or do not have practical means to shut down equation (16b) without affecting eq. (16a), and it does not matter if every practical manipulation available to us affects other parameters in the system, including c₁₂.

   In summary, I submit that, contrary to Heckman's assertion, the causal effect of Y₂ on Y₁ is well defined within the model of Eqs. (16) (through the surgery procedure) even "when the parameters of (16a) or (16b) shift when external variables are manipulated," and even "if external variables cannot be independently manipulated." Technical feasibility ought not to alter definitions, and the surgery semantics ensures that it does not.

4. This brings us to Question 4: Whether "… the standard simultaneous equations model of econometrics" define causal parameters differently than the surgery does.

   I find this question intriguing, because I was always under the impression that the surgery procedure encodes in precise and unambiguous terms how causal parameters are defined in "… the standard simultaneous equations model of econometrics," and that this standard has been restored by Heckman and his co-workers after decades of confusion and neglect (see Causality, pp. 136-138). Was I wrong? Is economics on the wrong path again?

Instead of speculating on the status of simultaneous equations in economics, I will share with readers a couple of email messages that I sent to professor Heckman in April 27 1995, after giving a talk to his group at the University of Chicago.

Dear Professor Heckman,

One topic which we did not have a chance to discuss, is the current status of structural equations in econometrics, as manifested, for example, in textbooks and research publications and. You said that the textbooks I selected (Maddala, Goldberger, Intriligator) as not typical. Can you refer me to a textbook which, in your opinion, does a good job of introducing structural equations and demonstrating their role in policy analysis. Of special interest to me would be any book, article, or report which deals squarely with the operation of exogenizing endogenous variables, an operation which I have not seen discussed anywhere, with the exception of Strotz and Wold. Any reference to such operation would be appreciated because there is no sense for me going around accusing econometricians for not practicing what they in fact do practice. I would much rather justify the "surgery" semantics by citing prevailing practices.

I wish to emphasize though that my perception of how economists have distorted the teachings of the founding fathers (Wright and Haavelmo) is based not only on textbooks and confused publications, but on face to face conversations with colleagues in your profession. I am sure it will come as a shock to you, but more than 50% of the economists I surveyed are in the opinion that: "Structural equations are just a parsimonious summary of joint distributions". And 50% of the rest do not know the operational difference between structural and non-structural equations (aside from the cliche that the former is more "meaningful" of more "law-like")

Finally, 50% of the rest do not know how to solve the little puzzle in my handout (Ed Leamer was the only one who solved problem 1-2, and he admits that there must be something wrong in econometric education if students cannot solve it blind folded (not to mention esteemed scholars)). It is quite possible that your students are exceptional, but I would still be interested to know if they can solve it correctly; even given a more vivid presentation than the one I cooked up in the seminar. I am anxious to know the results.

Heckman replied that the best book, out of print, is by Carl Christ (1966).

My message of April 28, 1995 reads:

Thanks for the reference — our library has the Carl Christ book and I will get it soon.

I hope you agree, though, that if we need to go to 1966 for a good book on structural equations, then something must have gone sour in the past 30 years.

We are now in year 2007, and I am eager to learn whether clarity shines on the econometric front. Heckman's 2005 paper indicates that we might not be there yet.

A Challenge to Students of Economics

Readers might find it audacious of me, a student of computer science, to correct Jim Heckman on matters concerning economic modelling. My daring emanates, I confess, not from expertise in economics, I have none, but from familiarity with some basic concepts in logic and computer science — two sciences that specialize in modelling human knowledge by symbols. One of the first thing computer science students learn is that a model is an "oracle," namely, a mathematical object from which even a stupid robot should be able to compute answers to every query of possible future interest to the model builder. In our example, this means that, given a causal model like Eq. (16a-b), every causa
l parameter should be computed from the model itself, without sending the modeller back to the drawing board for additional information, say, if external variables can or cannot be independently manipulated. Thus, parameters such as causal effects, direct effects, indirect effects, counterfactual predictions, probabilities of counterfactuals, independence of counterfactuals and more and more, all should be computed automatically from the equations alone. They cannot depend on external circumstances such as whether a certain physical manipulations are technologically feasible, or whether a given physical manipulation has undesirable side effects.

More specifically, if the causal effect of treatment X on outcome Y is defined by the counterfactual expression E(Y_x' – Y_x), where x and x' are two treatment levels, then we need to know how a robot should calculate E(Y_x) from a well specified economic model given by a general set of equations such as (16). Economics will become a mature science when an economist of the stature of Jim Heckman gives his readers a general procedure of calculating E(Y_x) from any set of equations, parametric as well as non-parametric, recursive as well as nonrecursive. Needed is a half-page procedure that even a robot can follow, free of references to previous literature, which takes a model as input and computes E(Y_x) as output.

Is economics ready for the challenge?

========Judea Pearl