{"id":40,"date":"2007-05-17T01:00:42","date_gmt":"2007-05-17T08:00:42","guid":{"rendered":"http:\/\/www.mii.ucla.edu\/causality\/?p=49"},"modified":"2007-05-17T01:00:42","modified_gmt":"2007-05-17T08:00:42","slug":"more-on-where-economic-modeling-is-heading","status":"publish","type":"post","link":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/2007\/05\/17\/more-on-where-economic-modeling-is-heading\/","title":{"rendered":"More on Where Economic Modeling is Heading"},"content":{"rendered":"

Judea Pearl writes:<\/strong><\/p>\n

My previous posting in this forum raised questions regarding Jim Heckman's analysis of causal effects, as described in his article, "The Scientific Model of Causality" (Sociological Methodology<\/em>, Vol. 35 (1) page 40.)<\/p>\n

To help answer these questions, Professor Heckman was kind enough to send me a more recent paper entitled: "Econometric Evaluation of Social Programs," by Heckman and Vytlacil (Draft of Dec. 12, 2006. Prepared for The Handbook of Econometrics<\/em>, Vol. VI, ed by J. Heckman and E. Leamer, North Holland, 2006.)<\/p>\n

This paper indeed clarifies some of my questions, yet raises others. I will share with readers my current thoughts on Heckman's approach to causality and on where causality is heading in econometrics.<\/p>\n

(Post edited 5\/4: revisions in red, thanks to feedback from David Pattison<\/font><\/font>)<\/strong><\/em>
(Post edited 5\/17: correction and new comments by LeRoy and Pearl)<\/em><\/strong> <\/p>\n

<\/p>\n

A New Definition of Causal Effects<\/strong>
In their Handbook paper, Heckman and Vytlacil (HV) provide a new definition of causal effects, based on "external-variations," instead of shutting down equations (i.e., "surgery"). The definition is described semi-formally on page 77 and footnote 81 of their paper. The following is my extrapolation of their method as it applies to multi-equations and nonlinear systems.<\/p>\n

Given a system of equations:<\/p>\n

                                              Yi<\/sub> = fi<\/sub><\/em>(Y, X, U<\/em>)      i = 1, 2,…, n<\/em><\/p>\n

where X<\/em> and U<\/em> is sets of observed and unobserved external variables, respectively, the causal effect of Yj<\/sub><\/em> on Yk<\/sub><\/em> is computed in four steps:<\/p>\n

    \n
  1. Choose any member Xt<\/sub><\/em> of X<\/em> that appears in fj<\/sub><\/em>.<\/li>\n
  2. If Xt<\/sub><\/em> appears in any other equation as well, exclude it from that equation (e.g., set its coefficient to zero if the equation is linear or replace Xt<\/sub><\/em> by a constant).<\/li>\n
  3. Solve for the reduced form
                                           Yi<\/sub><\/em> = gi<\/sub><\/em>(X, U<\/em>)      i = 1, 2, …., n<\/em>
    of the resulting system of equations.<\/li>\n
  4. Compute the partial derivative\n

                                        dYk<\/sub><\/em>\/dYj<\/sub><\/em> using the ratio dgk<\/sub><\/em>\/dXt<\/sub><\/em> : dgj<\/sub><\/em>\/dXt<\/sub><\/em><\/li>\n<\/ol>\n

    The resulting ratio gives "the causal effect of Yj<\/sub><\/em> on Yk<\/sub><\/em>."<\/p>\n

    (Note: In the two-equations model discussed in my previous posting<\/p>\n

                      Y1<\/sub> = a1<\/sub> + c12<\/sub>Y2<\/sub> + b11<\/sub>X1<\/sub> + b12<\/sub> X2<\/sub> + U1<\/sub><\/em>     (4.8a)
                      Y2<\/sub> = a2<\/sub> + c21<\/sub>Y1<\/sub> + b21<\/sub>X1<\/sub> + b22<\/sub> X2<\/sub> + U2<\/sub><\/em>     (4.8b)<\/p>\n

    the external-variation definition yields c12<\/sub><\/em> as the causal effect of Y2<\/sub><\/em> on Y1<\/sub><\/em>, which is identical to the result obtained by the surgery definition. In general, the two results may differ, as demonstrated below<\/strike><\/font>.)<\/p>\n

    External Variation vs. Surgery<\/strong>
    In comparing their definition to the one provided by the surgery procedure, HV write (page 79): "Shutting down an equation or fiddling with the parameters … is not required to define<\/em> causality in an interdependent, nonrecursive system or to identify causal parameters. The more basic idea is exclusion<\/em> of different external variables from different equations which, when manipulated, allow the analyst to construct the desired causal quantities."<\/p>\n

    The following are my thoughts on this idea of HV.<\/p>\n

      \n
    1. In general, "exclusion" involves the removal of a variable from an equation and amounts to "fiddling with the parameters." It is, therefore, a form of "surgery" – a modification of the original system of equations — and would be subject to the same criticism one may raise against "surgery." I will refute such criticism in items 3 and 4 below, noting that if it ever has a grain of validity, the criticism would apply equally to both methods. I will then argue that "surgery" is a more basic idea than "exclusion", more solidly motivated and more appropriate for policy evaluation tasks.<\/li>\n
    2. The idea of relying exclusively on external variables to reveal internal cause-effect relationships has its roots in the literature on IDENTIFICATION (e.g., as in the studies of "instrumental variables") because such variables transmit the only manipulations present in observational studies. This restriction however, is unjustified in the context of DEFINING causal effect, since "causal effects" are meant to quantify effects produced by NEW external manipulations, not necessarily those shown explicitly in the model. Moreover, every causal structural equation model, by its very nature, provides an implicit mechanism, for emulating such external manipulations via surgery.\n

      Indeed, most policy evaluation tasks are concerned with NEW external manipulations which exercise direct control over endogenous variables, namely, surgeries. Take for example a manufacturer deciding whether to double the current price of a given product after years of letting the price track the cost, i.e., price = f<\/em>(cost<\/em>). Such decision amounts to removing the equation price = f<\/em>(cost<\/em>) in the model at hand, (i.e., the one responsible for the available data), and replacing it with a constant equal to the new price. This removal is necessary for evaluating the decision at hand, and no external variables can help us avoid it.<\/p>\n

      Or take the example of evaluating the impact of terminating an educational program for which students are admitted based on a set of qualifications . The equation admission = f<\/em>(qualifications<\/em>) will no longer hold under program termination, and no external variable can simulate the new condition (i.e., admission = 0<\/em>) save for one that actually neutralizes (or "ignores", or "shuts down") the equation admission = f<\/em>(qualifications<\/em>).<\/p>\n

      (NOTE: the method used in Haavelmo (1943) to define causal effects is mathematically equivalent to surgery, not to external variation. Instead of replacing the equation Yj<\/sub> = fj<\/sub><\/em>(Y, X, U<\/em>) with<\/p>\n

      Yj<\/sub> = yj<\/sub><\/em><\/p>\n

      as would be required by surgery, Haavelmo writes Yj<\/sub> = fj<\/sub><\/em>(Y, X, U<\/em>) + xj<\/sub><\/em> where xj<\/sub><\/em> is chosen so as to
      \nmake Yj<\/sub><\/em> constant Yj<\/sub> = yj<\/sub><\/em>. Thus, since xj<\/sub><\/em> liberates Yj<\/sub><\/em> from any residual influence of fj<\/sub><\/em>(Y, X, U<\/em>), Haavelmo's method is equivalent to that of surgery. Heckman's method of external variation leaves Yj<\/sub><\/em> under the influence fj<\/sub><\/em>.)<\/li>\n

    3. Definitions based on external variation have the obvious flaw that the target equation may not contain any observable external variable. In fact, in many cases the set of observed external variables in the system is empty. Additionally, a definition based on a ratio of two partial-derivatives does not generalize easily to non-linear systems with discrete variables. Thus, those who accept Heckman's restrictions would be deprived of the many identification techniques now available for instrument-less models (see Causality,<\/em> chapter 3 and 4) and, more embarrassingly yet, they would be unable to even ask whether causal effects are identified in any such model — identification questions are meaningless for undefined quantities.\n

      Fortunately, liberated by the understanding that definitions can be based on purely symbolic manipulations, we can modify Heckman's proposal and ADD fictitious external variables to any equation we desire. The added variables can then serve to define causal effects in a manner similar to the four steps in (2) (assuming continuous variables). This brings us closer to surgery, with one basic difference of leaving Yj<\/sub><\/em> under the influence of fj<\/sub><\/em>(Y,X,U<\/em>).<\/p>\n<\/li>\n

    4. Having argued that definitions based on "external variation" are conceptually ill-motivated, we now explore whether these definitions yield correct causal effects, identical to those defined by the surgery logic.\n

      Consider a system of 3 equations:
      Y1<\/sub> = aY2<\/sub> + cY3<\/sub> + U1<\/sub>
      <\/em> Y2<\/sub> = bY1<\/sub> + X + U2<\/sub><\/em>
      Y3<\/sub> = dY1<\/sub> + U3<\/sub><\/em>
      Needed: the causal effect of Y2<\/sub><\/em> on Y1<\/sub><\/em>.<\/p>\n

      The system has one external variable, X<\/em>, which appears in the second equation alone, hence no exclusion is necessary. Applying the "external variation" procedure (following the 4-steps above), the reduced form yields: <\/p>\n

                                              dY1<\/sub>\/dX = a\/(1-ba-cd)      dY2<\/sub>\/dX= (1-cd)\/ (1-ab-cd)<\/font><\/em><\/p>\n

      and the causal effect of Y1<\/sub><\/em> on Y2<\/sub><\/em> calculates to:<\/p>\n

                                                                     dY1<\/sub>\/dY2<\/sub> = a\/(1-cd)<\/em><\/span><\/font> <\/p>\n

      In comparison, the surgery procedure yields the following modified system of equations:
      Y1<\/sub> = aY2<\/sub> + cY3<\/sub> + U1<\/sub><\/em>
      Y2<\/sub> = y2<\/sub><\/em>
      Y3<\/sub> = dY1<\/sub> + U3<\/sub><\/em><\/p>\n

      from which we obtain for the causal effect of Y2<\/sub><\/em> on Y1<\/sub><\/em>;<\/p>\n

                                                                       dY1<\/sub>\/dy2<\/sub> = a\/(1-cd)<\/em><\/font><\/p>\n

      an identical expression to that obtained from the "external variation" procedure.<\/p>\n

      It is highly probable that the results of the two procedures always coincide, though I cant see an easy proof. Perhaps readers can provide the answer.<\/span><\/font><\/li>\n<\/ol>\n

      Criticism 1:<\/strong> Parameter Stability<\/strong><\/p>\n

      "Shutting down one equation might also affect the parameters of the other equations in the system and violate the requirement of parameter stability" (HV page 79).<\/p>\n

      In the physical world, creating the conditions dictated by a "surgery" may sometimes affect parameters in other equations. The same applies to exclusion, which is a form of surgery (see item 2 above). For example, some parameters may depend on the excluded variable or on the coefficient of the excluded variable. However, we are dealing here with symbolic, not physical manipulations. Our task is to craft a meaningful mathematical definition of "the causal effect of one variable over another" from a symbolic system called a "model." This permits us to manipulate symbols at will, while ignoring the physical consequences of these manipulation. Physical considerations need not enter the discussion of DEFINITION.<\/p>\n

      Criticism 2:<\/strong> Equation ambiguity in non-causal systems<\/strong><\/p>\n

      "In general, no single equation in a system of simultaneous equations uniquely determine any single outcome variable" (HV page 79). Heckman and Vytlacil refer here to systems containing non-directional equations, namely, equations in which the equality sign does not stand for the non-symmetrical relation "is determined by" or "is caused by" but for symmetrical algebraic equality. In econometrics, such non-causal equations usually convey equilibrium or resource constraints; they impose equality between the two sides of the equation but do not endow the variable on the left hand side with a special status of an "outcome" variable.<\/p>\n

      The presence of non-directional equations creates ambiguity in the surgical definition of the counterfactual Yx<\/sub><\/em>, which calls for replacing the equation determining X<\/em> with the constant equation X=x<\/em>. If X<\/em> appears in several equations, and if the position of X<\/em> in the equation is arbitrary, then each one of those equations would be equally qualified for replacement by X=x<\/em>, and the value of Yx<\/sub><\/em> (i.e., the solution for Y<\/em> after replacement) would be ambiguous.<\/p>\n

      (Note that this problem does not occur in directional nonrecursive systems (i.e., systems with feedbacks) since in such systems each variable is an "outcome" of precisely one equation.)<\/p>\n

      HV paper creates the impression that equation ambiguity is a flaw of the surgery definition and does not plague the exclusion-based definition. However, this is not the case. In a system of non-directional equations, we have no way of knowing which external variable to exclude from which equation to get the right causal effect.<\/p>\n

      For example: Consider Eqs. (4.8a)-(4.8b) in HV page 75.<\/p>\n

      Y1<\/sub> = a1<\/sub> + c12<\/sub>Y2<\/sub> + b11<\/sub>X1<\/sub> + b12<\/sub> X2<\/sub> + U1<\/sub><\/em>     (4.8a)
                                           Y2<\/sub> = a2<\/sub> + c21<\/sub>Y1<\/sub> + b21<\/sub>X1<\/sub> + b22<\/sub> X2<\/sub> + U2<\/sub><\/em>   &n
      \nbsp; (4.8b)<\/p>\n

      Suppose we move Y1<\/sub><\/em> to the lhs of (4.8b) and get:<\/p>\n

                                        Y1<\/sub> = <\/em>[a2<\/sub> – Y2<\/sub> + b21<\/sub>X1<\/sub> + b22<\/sub> X2<\/sub> + U2<\/sub><\/em>]\/c21<\/sub><\/em>     (4.8b')<\/p>\n

      To define the causal effect of Y2<\/sub><\/em> on Y1<\/sub><\/em>, we now have a choice of excluding X2<\/sub><\/em> from (4.8a) or (4.8b'). The former yields c12<\/sub><\/em>, while the latter yields 1<\/em>\/c21<\/sub><\/em>. We see that the ambiguity we have in choosing an equation for surgery now translates into ambiguity in choosing an equation and an external variable for manipulation.<\/p>\n

      Remark: Methods of breaking this ambiguity were proposed by Simon (1953) and are discussed in some detail in (Pearl 2000, Causality<\/em>, page 226-228).<\/p>\n

      Summary<\/strong>
      To summarize, the idea of constructing causal quantities by exclusion<\/em> and manipulation of external variables, while soundly motivated in the context of identification problems, has no logical basis when it comes to model-based definitions. It may yield erroneous results in nonrecursive systems, and suffers from problems of ambiguity in non-directional systems.<\/span><\/strike><\/font> Definitions based on surgery, on the other hand, enjoy generality, semantic clarity and immunity from "parameter instability" concerns.<\/p>\n

      So, where does this leave econometric modeling? Is the failure of the "external variable" approach central or tangential to economic analysis and policy evaluation?<\/p>\n

      In almost every one of his recent articles Jim Heckman stresses the importance of counterfactuals as a necessary component of economic analysis and the hallmark of econometric achievement in the past century. For example, the first paragraph of HV article reads: "they [policy comparisons] require that the economist construct counterfactuals. Counterfactuals are required to forecast the effects of policies that have been tried in one environment but are proposed to be applied in new environments and to forecast the effects of new policies." Likewise, in his Sociological Methodology<\/em> article (2005), Heckman states: "Economists since the time of Haavelmo (1943, 1944) have recognized the need for precise models to construct counterfactuals … The econometric framework is explicit about how counterfactuals are generated and how interventions are assigned…"<\/p>\n

      I totally agree with Heckman on the centrality of counterfactuals in economic analysis. However, I am not aware of even one econometric article or textbook in the past 40 years in which counterfactuals or causal effects are properly defined. Economists working within the potential-outcome framework of the Neyman-Rubin model take counterfactuals as undefined primitives, totally detached from the knowledge encoded in structural equations models. Economists working within the structural equations framework are busy estimating parameters while treating counterfactuals as metaphysical ghosts that should not concern ordinary mortals. They trust leaders such as Heckman to define precisely what the policy implications are of the structural parameters they labor to estimate, and to relate them to what their colleagues in the potential-outcome camp are doing.<\/p>\n

      Fortunately, a simple and precise unification of the two approaches can be achieved using the mathematical properties of the surgery operation (see Causality<\/em>, page 98-102). Economists will do well resurrecting the basic surgery ideas of Haavelmo (1943) Marschak (1950) and Strotz and Wold (1960) and re-invigorating them with the logic of Graphs and counterfactuals developed in the past two decades.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Judea Pearl writes: My previous posting in this forum raised questions regarding Jim Heckman's analysis of causal effects, as described in his article, "The Scientific Model of Causality" (Sociological Methodology, Vol. 35 (1) page 40.) To help answer these questions, Professor Heckman was kind enough to send me a more recent paper entitled: "Econometric Evaluation […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,13],"tags":[],"class_list":["post-40","post","type-post","status-publish","format-standard","hentry","category-discussion","category-economics"],"_links":{"self":[{"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/40","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=40"}],"version-history":[{"count":0,"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/40\/revisions"}],"wp:attachment":[{"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=40"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=40"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/causality.cs.ucla.edu\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=40"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}