Are economists smarter than epidemiologists? (Comments on Imbens’s recent paper)
In a recent survey on Instrumental Variables (link), Guido Imbens fleshes out the reasons why some economists “have not felt that graphical models have much to offer them.”
His main point is: “In observational studies in social science, both these assumptions [exogeneity and exclusion] tend to be controversial. In this relatively simple setting [3-variable IV setting] I do not see the causal graphs as adding much to either the understanding of the problem, or to the analyses.” [page 377]
What Imbens leaves unclear is whether graph-avoiding economists limit themselves to “relatively simple settings” because, lacking graphs, they cannot handle more than 3 variables, or do they refrain from using graphs to prevent those “controversial assumptions” from becoming transparent, hence amenable to scientific discussion and resolution.
When students and readers ask me how I respond to people of Imbens’s persuasion who see no use in tools they vow to avoid, I direct them to the post “The deconstruction of paradoxes in epidemiology”, in which Miquel Porta describes the “revolution” that causal graphs have spawned in epidemiology. Porta observes: “I think the “revolution — or should we just call it a renewal”? — is deeply changing how epidemiological and clinical research is conceived, how causal inferences are made, and how we assess the validity and relevance of epidemiological findings.”
So, what is it about epidemiologists that drives them to seek the light of new tools, while economists (at least those in Imbens’s camp) seek comfort in partial blindness, while missing out on the causal revolution? Can economists do in their heads what epidemiologists observe in their graphs? Can they, for instance, identify the testable implications of their own assumptions? Can they decide whether the IV assumptions (i.e., exogeneity and exclusion) are satisfied in their own models of reality? Of course the can’t; such decisions are intractable to the graph-less mind. (I have challenged them repeatedly to these tasks, to the sound of a pin-drop silence)
Or, are problems in economics different from those in epidemiology? I have examined the structure of typical problems in the two fields, the number of variables involved, the types of data available, and the nature of the research questions. The problems are strikingly similar.
I have only one explanation for the difference: Culture.
The arrow-phobic culture started twenty years ago, when Imbens and Rubin (1995) decided that graphs “can easily lull the researcher into a false sense of confidence in the resulting causal conclusions,” and Paul Rosenbaum (1995) echoed with “No basis is given for believing” […] “that a certain mathematical operation, namely this wiping out of equations and fixing of variables, predicts a certain physical reality” [ See discussions here. ]
Lingering symptoms of this phobia are still stifling research in the 2nd decade of our century, yet are tolerated as scientific options. As Andrew Gelman put it last month: “I do think it is possible for a forward-looking statistician to do causal inference in the 21st century without understanding graphical models.” (link)
I believe the most insightful diagnosis of the phenomenon is given by Larry Wasserman:
“It is my impression that the “graph people” have studied the Rubin approach carefully while the reverse is not true.” (link)
[…] [link] for attempted […]
Pingback by Causal Analysis in Theory and Practice » Fall Greetings from UCLA Causality Blog — October 29, 2014 @ 6:10 am
My impression is that many economists’ aversion to DAGs (or any other diagrammatic depiction of a problem) is due, in part, to their fear that causal diagrams will be substituted for theory, e.g., one simply can draw diagrams and arrows instead of working through a model like utility maximization and the comparative statics to derive predictions. They fail to see that DAGs are not a substitute for theory, but a means of clarifying theory, and particularly, a means of identifying estimation problems that one is likely to encounter when one moves from theoretical constructs to observed data. I’ve had some success on the latter point with the following example. Econometric texts devote a lot of time to omitted variable bias, usually characterized as a correlation of the error term in a regression equation with one or more explanatory variables. But the troublesome correlation could be due either to omitted variable bias (spurious correlation, unobserved confounder) or to the omitted variable being a mediator of the relationship between X and Y. Those two cases have very different implications for causal inference, and in the case of an unobserved mediator, you may not have wanted to include it in the model even if you could observe it, because your interest is in the full effect of X on Y, not the partial effect controlling for the (unobserved) mediator.
Comment by Bryan Dowd — October 29, 2014 @ 12:32 pm
Bryan,
I agree with your observation that graph-averse economists “fail to see that DAGs are not a substitute for theory, but a means of clarifying theory”. But this could not be the whole story, because it does not explain why the aversion is inflicted so prominently and vocally among “potential outcome” adherents.
I believe this blog would benefit from learning more about your experience with economists’s #1 confusion: Regression vs. structural equations. Bryant Chen and I wrote a piece about it: “Regression and Causation: A Critical Examination of Six Econometrics Textbooks” http://ftp.cs.ucla.edu/pub/stat_ser/r395.pdf. But we would welcome additional evidence that econometric textbooks are criminally responsible for perpetuating the confusion.
Comment by judea pearl — October 29, 2014 @ 7:49 pm
Hi Judea,
I would offer a simpler (and less derisive to economists!) explanation: econometricians in particular tend to be more comfortable with equations with diagrams. Am I wrong in thinking that anything that can be expressed with a DAG can be expressed with equations?
Consider the examples above: Bryan’s omitted mediator is commonly referred to as a “bad control” in econometrics, and the issue, in contrast to his implication, well-known among applied econometricians. There is currently interest in sophisticated approaches to this problem, e.g., Heckman and Pinto (2013):
https://ideas.repec.org/p/hka/wpaper/2013-006.html
I also don’t agree with your claim that econometricians are hopelessly confused over the interpretation of our models (although I do agree many popular econometrics textbooks are not so good on this point), and as you know I wrote a semi-rejoinder to your piece with Chen:
http://chrisauld.com/2013/10/08/remarks-on-chen-and-pearl-on-causality-in-econometrics-textbooks/
Best, Chris.
Comment by Chris Auld — October 30, 2014 @ 9:55 am
My concern with any types of graphical presentation that suggests a logical progression toward a causal end, is that such approaches often hide the fact that the underlying data may be unreliable. Epidemiologists tend to forget that we are dealing with observational data rather than controlled experiments. Bias is always incompletely controlled (sometimes to a great extent) because we do not fully understand the important underlying risk factors (or disease pathophysiology for that matter). Neither can we adequately measure important risk factors either for reasons of cost or because our tools (eg, records and questionnaire responses) inadequately represent these data. Therefore, while I do not necessarily disagree with graphical presentations (ie, they can clarify ones view), I do suggest that these tools be taken with a grain of skepticism, especially when turned into dogmatic theory (as is sometimes done).
Comment by John — October 31, 2014 @ 12:49 pm
Dear John,
Can you educate us as to where we can find evidence that epidemiologists tend to forget that we are dealing with observational data. I found them extremely careful about this distinction. Also, I have not seen graphical representations “turned into dogmatic theory”, have you?
Researchers using graphical representations invariably use SEM and counterfactual representations as well, so it is unfair to label them “dogmatic”. If you are referring to my critics of Imbens , please note that I was very specific in identifying HOW economists can benefit from graphical representations. I said: “Can they, for instance, identify the testable implications of their own assumptions? Can they decide whether the IV assumptions (i.e., exogeneity and exclusion) are satisfied in their own models of reality?”
If you think they can do these tasks, please enlighten us.
This is not an opinion, John, certain things are tractable and certain things are not. The tasks I am talking about are intractable to a graph-free mind. I therefore do not see dogmatism here but factual observations of how certain cultures are missing out on the causal revolution.
Comment by judea pearl — November 1, 2014 @ 7:54 am
I am glad that in his discussion of my paper Judea Pearl provides a link to the paper. I suggest the reader look at the paper and decide for him or herself whether it reflects the attitudes Judea ascribes to me. I find it difficult to have a scholarly discussion with Judea when he feels the need to constantly use ad hominems to attack, and to question the motives of, those who disagree with him. Judea and others using graphical models have developed a very interesting set of tools that researchers in many areas have found useful for their research. Other researchers, including myself, have found the potential outcome framework for causality associated with the work by Rubin (see my forthcoming book with Rubin, that will be available in January 2015 for a general discussion of that framework) more useful for their work. In my view that difference of opinion does not reflect “economists being scared of graphs”, or “educational deficiencies” as Judea claims, merely legitimate heterogeneity in views arising from differences in preferences and problems. The “educational deficiencies” claim, and similarly the comment about my “vow” to avoid causal graphs is particularly ironic given that in the past Judea has presented, at my request, his work on causal graphs to participants in a graduate seminar I taught at Harvard University.
In the discussion on page 377 I explore the reasons why economists have not adopted the graphical methods. As reflected in Judea’s quote from my paper, I write that in the three variable instrumental variables case I do not see much gain in using a graphical model. Nothing in Judea’s comment answers that question. Instead Judea asks whether I refrain from using graphical models “to prevent those `controversial assumptions’ from becoming transparent, hence amenable to scientific discussion and resolution.” It is disappointing that simply because of a disagreement on a substantive issue, Judea feels the need to question other researchers’ integrity.
It may clarify my views to give a longer quote from the paper:
“Now consider a more complicated setting such as the ‘hypothetical longitudinal study represented by the causal graph shown in Figure 2,’ in the comment by Shpitser, or Figure 1 in Pearl (1995). Here, identification questions are substantially more complex, and there is a strong case that the graph-based analyses have more to contribute. However, I am concerned about the relevance of such examples in social science settings. I would like to see more substantive, rather than hypothetical, applications where a graph such as that in Figure 2 could be argued to capture the causal structure. There are a large number of assumptions coded into such graphs, and given the difficulty in practice to argue for the absences of one or two arrows in instrumental-variables or no-unobserved-confounders applications in social sciences, I worry that in practice it is difficult to convince readers that such a causal graph fully captures all important dependencies. In other words, in social sciences applications a graph with many excluded links may not be an attractive way of modeling dependence structures.”
What I am concerned with is not so much the question whether economists can “decide whether the IV assumptions (i.e., exogeneity and exclusion) are satisfied in their own models of reality” but whether causal graphs or potential outcomes may it easier to assess whether the assumption are accurate approximations to what is going on in the real world. Personally I have found the potential outcomes, and, for example, the concepts of nevertakers/alwaystakers/compliers to be useful in assessing the plausibility of the exclusion restrictions. See the discussion on page 344 of my Statistical Science paper. Judea may not find that argument persuasive, and that is fine. However, I strongly disagree with the comment that: “Certain things are tractable and certain things are not. The tasks I am talking about are intractable to a graph-free mind.” I do not know how Judea could possibly be so sure of that!
My guess is that what will convince economists and other social scientists of the value of the graphical approach as Judea sees it is not attacks on their integrity, but persuasive real-world examples.
Comment by guido imbens — November 1, 2014 @ 9:49 pm
Dear Guido,
I apologize if my comments on your survey paper were taken as a personal attack; this was not my intention. My comments meant to express a critical view of a subculture which your survey seems to endorse (i.e., the “potential outcome” framework (PO)), and the scientific harm caused by some practices within this subculture. I have posted a summary of my views on PO here, which I urge you and readers to read and discuss; it ends with the following paragraph:
In summary, the PO framework offers a useful analytical tool (i.e., an algebra of counterfactuals) when used in the context of a symbiotic SCM analysis. It may be harmful however when used as an exclusive and restrictive subculture that discourages the use of process-based tools and insights.
But I will now leave cultural differences aside and focus on substantive issues because, despite my persistent requests and challenges in the past, I have not been able to engage in such discussions with any researcher who expresses doubts concerning the usefulness of graphs. I am doing so here primarily as a service to the hundreds of researchers (including social scientists) who are using graphs as a second language, and who were surprised, if not bewildered by the doubts expressed in your survey which, perhaps unintentionally, also cast doubts on the soundness and effectiveness of their methods and tools.
I will start at your backyard of a 3-variable IV setting, Here, the main issues are the choice and justification of the instrument which, according to your survey, is a complex mental process involving lots of “controversial assumptions.” Your post (above) adds to it another issue: “whether causal graphs or potential outcomes make it easier to assess whether the assumptions are accurate approximations to what is going on in the real world.”
Let us start with the latter. We have a bunch of assumptions, and we are wondering whether they are good approximation of reality. Obviously, we cannot examine those assumption in the real world. We must examine them in some mental REPRESENTATION of the real world. We call such a representation “a model”. Further, once we represent our assumptions in some symbolic way, what is the criterion against which we should judge their plausibility? I believe you would agree with me that the criterion should be the scientific knowledge that we possess about the domain. Once we agree on these two premises, we can go to the next step and address your question “whether causal graphs or potential outcomes make it easier to assess…” which is now translated into the question: whether causal graphs or potential outcomes are more faithful representations of the way people store scientific knowledge.
This may appear to be a psychological question, which can be dismissed with: “I am not a psychologist”. But since we are aiming at understanding, not winning an argument, we can submit the question to a simple test. We can think of a problem or a story, for which the scientific assumptions are obvious and indisputable, you express the assumptions in PO, I express them in a graph, and we ask a researcher whether he/she can recognize the problem from our descriptions. I actually performed a version of this experiment in the seminar I gave to your group at Stanford ( and in dozens other lectures I gave) and you can guess the results. No one can tell from the PO representation whether it faithfully represents the story, whether one ignorability condition is redundant (follows from the others) or whether some ignorability conditions are missing.
We can try it in this forum, if you wish, you choose the story. The story I chose is on page 232-233 of Causality, with the two representations standing side by side. Please judge for yourself.
The reasons for this perception of representativeness is also obvious. People store scientific knowledge in the form of set of qualitative causal influences, namely, in graphs. The closer we make our assumptions fit the format in which scientific knowledge is stored, the more reliable would our judgment be about the plausibility of the assumptions.
Let us now address the question of assessing the plausibility of the exclusion (and exogeneity) restrictions. Here I stated “This is not an opinion. Certain things are tractable and certain things are not. The tasks I am talking about are intractable to a graph-free mind.” You expressed strong disagreement with this comment and added “I do not know how Judea could possibly be sure of that!”
Let me explain how. There was a time when people settled differences by saying: Oh, it is just a different approach to the same problem, each person may choose the argument he/she finds most persuasive, let a thousand flowers bloom, there are dozen roads to Rome, etc. That era has ended with the development of objective criteria of tractability and feasibility.
We are comparing here two representations of the same problem, one in the form of a graph, the other in the form of a set S of ignorability and exclusion statements which, logically speaking, are equivalent to the graph. As usually happens in complexity analysis, the initial representation can make the whole difference. Geiger (1990) has proven that the task of checking whether a given conditional independence statement follows from a set of N such statements is exponentially hard. Verma and Geiger proved (1990) that the complexity of testing whether a conditional independence statement follows from a given graph is polynomial. The reason for this complexity difference is that the graph structure embodies all the implications of the graphoid axioms (See http://en.wikipedia.org/wiki/Graphoids). In other words, the graph acts as a logical engine that makes certain answers explicit, while other representations leave them implicit.
You can see a demonstration of this tractability issue in the paper by Heckman and Pinto http://ftp.cs.ucla.edu/pub/stat_ser/r420.pdf who labored hard to circumvent d-separation and to derive conditional independencies directly from the graphoid axioms. The result is pages after pages of derivations (on a 4-variable problem) that would take two lines using graphs.
The same applies to the other task I mentioned: “Can they [economists] identify the testable implications of their own assumptions?” Deciding if a set S of ignorability and exclusion statements has testable implications is hard (intractable), while the graphs gives you the answer almost explicitly (in polynomial time.) I discuss this issue in a recent paper “Graphoids over Counterfactuals” http://ftp.cs.ucla.edu/pub/stat_ser/r396.pdf
One can argue, or course, that in a 3-variable IV setting we do not need this tractability. True, but we need it in the process of reducing a complex web of “controversial assumptions” to a simple 3-variable problem.
This is more or less the source of my confidence in claiming that, by disposing of graphs one disposes of an indispensable tool of analysis, and, consequently, one will be missing out on the causal revolution.
I hope these considerations help explain what I mean by “indispensable” and why it is that those who use graphs once never question their usefulness. Try it.
Judea
Comment by judea pearl — November 2, 2014 @ 6:54 am
Dear Judea,
Thanks for your comments.
First, I appreciate your apology. Beyond the personal attacks, however, I resent the misleading title of your original post: “are economists smarter than epidemiologists?’’ I suggested nothing of the sort in my paper, and your the suggestion that I or other economists do not respect the intellect or work of epidemiologists casts dispersions that are not appropriate. I would appreciate a rectification. Similarly, I find offensive your comment that you welcome additional evidence that “econometric textbooks are criminally responsible for perpetuating the confusion.’’ I have a great deal of admiration for the authors of the six textbooks you discussed in your paper, even if I may not always agree with their points of view. It is comments like these that certainly have made me, and perhaps others, reluctant to heed your “persistent requests and challenges … to engage in such discussions with any researcher who expresses doubts concerning the usefulness of graphs.’’
Second, your comment about speaking on behalf of the “hundreds of researchers (including social scientists) who are using graphs as a second language, and who were surprised, if not bewildered by the doubts expressed in your [imbens’] survey,’’ made me laugh out loud. I think in this discussion we ought to speak for ourselves, and not hide behind so many others. You wish to promote the work on causal graphs you have been heavily involved in. That is your prerogative, and such is the nature of academics. My views on the causal inference are different, as exposited in the work I have done over the years and in the forthcoming book with Rubin. As economists might say, these different views compete in the market place of ideas. Incidentically, I find your comment that “those who use graphs once never question their usefulness,’’ very revealing and at the same time very odd: I find it healthy to constantly question the usefulness of what I do myself, as well as what others do, that is an important part of my research philosophy. Obviously if I write something that leaves many researchers bewildered, I have failed and those researchers should ignore my writings. (I am surprised though, and somewhat flattered if anything I wrote left hundreds of researchers bewildered, and am even more surprised that so many researchers would reach out to you to serve as an intermediary rather than simply send an email to me.) If this were a common experience, I would rethink my work and attempt to figure out why it was perceived as so confusing. Similarly, you may wish to question more self-critically why despite your repeated efforts the methods you wish to promote fail to gain a foothold in some disciplines. The absolutist view that it can only be the fault of the researchers in those fields, rather than anything to do with merits of those methods, or your exposition, is unlikely to be productive.
Third, let me now turn to a substantive example of why I find the potential outcomes framework to be more helpful than causal graphs for the problems I study. Consider one of the canonical examples in the economics literature, the draft lottery example (Joshua Angrist, American Economic Review, 1989). Angrist is interested in the causal effect of veteran status on earnings, using the draft lottery number as an instrument for veteran status. In a simplified version of the problem the lottery number is a binary indicator, high/low, with those who have low lottery number having priority for the draft. Suppose I want to assess whether the draft lottery number is a valid instrument for veteran status. The draft lottery number was randomly assigned, so I am sure there are no unmeasured variables that affect both the lottery number and earnings. I am also sure the draft lottery number affects veteran status. What I am not sure about is whether there is a direct effect of the draft lottery number on earnings (an arrow from the draft lottery number to earnings). It is the absence or presence of such a direct affect that is key to the applicability of the instrumental variables methods. If there is no direct effect, I know what to do, and do not need causal graphs: earlier work (e.g., Imbens and Angrist, 1994, Angrist, Imbens and Rubin 1996, coincidentally both using the potential outcomes approach) established that we can identify the local average treatment effect (the average effect for compliers). To assess the presence or absence of a direct effect of the draft lottery number on earnings I find it helpful to consider the question separately for always-takers (men who would serve irrespective of their draft lottery number) and never-takers (men who would not serve, irrespective of their draft lottery number. In this application I find the exclusion restriction (the assumption of no direct effect of the instrument on the outcome) plausible for always-takers. It is more complicated for never-takers. If an individual is a never-taker because of a medical condition, the exclusion restriction again is plausible. However, if there is a substantial number of never-takers who would have to take active steps to stay out of the military had they been assigned a low draft lottery number (e.g., stay in formal education longer, or go to Canada), I would be more worried about the exclusion restriction because such behavior could directly affect subsequent earnings. Such considerations may lead me to search for information on the prevalence of such behavior. Personally I find the potential outcome set up, and its attendant characterization of the compliance types as never-takers, always-takers, compliers, and possibly defiers, to be helpful in elucidating possible violations of the exclusion restriction. Neither the traditional econometrics set up with linear models and constant coefficients, nor the causal graphs, are in my view as helpful in discovering possible violations. Moroever, I see nothing in your work that suggests that my views here are untenable as a matter of fact. You may disagree with these views, but I do not see how mathematical arguments can shed light on them.
Respectfully,
Guido Imbens
Comment by guido imbens — November 4, 2014 @ 12:57 am
Dear Guido,
Please permit me to skip the personal insults and continue with our substantive discussion because, for me, this is such a rare opportunity, I cant let it pass.
So, in response to my assertion that the usefulness of graphs is not a matter of personal opinion or convenience but an objective, demonstrable fact, you presented the story of Angrist’s draft lottery where you find it convenient to check the exclusion restriction by thinking separately about the four response types: “always takers”, “never takers””compliers” and “defiers”.
You also explain why it is more convenient. In your words:
“If an individual is a never-taker because of a medical condition, the exclusion restriction again is plausible. However, if there is a substantial number of never-takers who would have to take active steps to stay out of the military had they been assigned a low draft lottery number (e.g., stay in formal education longer, or go to Canada), I would be more worried about the exclusion restriction because such behavior could directly affect subsequent earnings.”
I could not have chosen a more vivid demonstration of graph-guided reasoning. Notice that after mentioning the response type you seek an explanation and you say the word “because”, and, then, it is the causal explanation (e.g., medical condition, education, escape to Canada) that leads you to judge whether it has an effect on the outcome, not the response type. This is precisely what I meant in claiming that judgment of plausibility rests on our stored scientific knowledge and that we store that knowledge in the form of qualitative cause effect relationships, read: a graph.
Let us continue this example a bit further. What if you judge that there is a direct effect from the instrument, Z, to the outcome? Are we done? No!. You and I know that such an effect is made up of several processes which might be intercepted by a bunch of intermediate variables and if we stratify on those variables we might turn Z back into a good instrument. But can we choose those intermediate variables correctly? In my paper on Haavelmo (R391), I give a toy example where one choice works and the other does not. Is the human mind capable of deciding between the two? Again, this is not a psychological question. It is a mathematical question whose answer is: Yes, with the help of graphs, No without graphs.
Without this thrilling experience of seeing an intractable problem turn into a toy it is really hard, Guido, to appreciate the power of graphs. I am sure those who never held a flashlight do not appreciate its powers. I do not blame you, therefore, for accusing me of self promotion, if not narrow mindedness. But I would rather take these accusations than give up my belief that some flashlights do advance science, I can tell by the rapid movement of their users.
Oh, what happened with the guessing game I proposed yesterday? Care to play? You choose a story.
Best
Judea
Comment by judea pearl — November 4, 2014 @ 7:14 am
Dear Guido,
This is an addendum to my previous comment.
10.1. Readers asked me to clarify what I meant by a “flashlight”, in the last paragraph. In describing the graph-averse culture, I usually resort to three metaphors: One is a flashlight, where I imagine the famous guy who was searching for his lost wallet under the lamppost, instead of grabbing a flashlight and search where he lost it. The second metaphor is the “telescope”, which some 17th century astronomers labeled “a work of the devil” (this is a historical fact). The third is the “multiplication table”; it takes a little effort to learn by heart, and it is not very clear to a 7 year old child why he needs to invest this effort but, if he doesn’t, he would find himself constantly apologizing to people around him: “I have not found it to be very useful in my work”. And when people would show him toy problems to demonstrate how useful multiplication can be, he would dismiss them with: “I don’t care about toy problems, I deal with important problems.” The last metaphor is the most representative of the computational advantages we get from graphs and of arguments cultivated in certain circles to justify the exclusion of graphs from their tool sets.
10.2. It might be useful to add a word of explanation as to why I claimed that the mental steps you described in examining the exclusion assumptions are steps in a causal graph. When you argued, for example, that some “never takers” might leave the country (to Canada) which will affect their income, you were tracing a path in a graph, going from the lottery outcome, to objection to serve, to leaving the US, to staying 2 years in Canada, to missing on education, to difficulties in getting a job, to low salary. This causal path resides in our mind and each link along this path is traced when we ask: “Can we think of a reason why a low lottery number may, by itself, affect a person salary?” The response type, in our case “never taker,” is a derivable property of the functional relationships in the model, it is not an object that resides explicitly in the model.
Comment by judea pearl — November 5, 2014 @ 2:13 am
Dear Judea,
Thanks for the two responses, which I will reference below as Pearl 1 and Pearl 2.
First of all, I am mystified by your metaphors. When I write things along the lines of “I have not found [causal graphs] to be very useful in my work,” it is not at all intended as an apology. It is intended as an acknowledgement that these are my views, that there may be legitimate differences of opinion on these issues, and most importantly that I am willing to entertain alternative views. Why you link that to a seven-year old refusing to learn multiplication tables, I have no idea! It is probably the same world view that makes you compare those who hold opinions different from yours to 17th century astronomers decrying the invention of the telescope, or denounce textbook writers who do not discuss your work in sufficient detail as “criminally responsible.’’ As much as I admire the work you have done in the area of causality, I think this is an unproductive, and, to be honest, disrespectful, attitude that prevents constructive dialogue between researchers with different views.
Regarding the instrumental variables example: in the paper that motivated this exchange I wrote that I found it helpful to think in terms of potential outcomes to assess whether the absence of a direct effect of the draft lottery number on earnings was a reasonable assumption. I elaborated on that by discussing in the context of that specific empirical example how thinking in terms of compliance types (never-takers, always-takers, compliers, and defiers, defined in terms of potential outcomes) facilitated my evaluation of that assumption, and concluded that, at least for me, causal graphs did not bring the same clarity to the problem as the potential outcomes approach did. The response in Pearl 1 did not address that point at all, instead pivoting to a discussion of a very different question, one I did not discuss in my paper, namely what to do if one concluded that there was in fact a direct effect of the lottery number on earnings and the instrument was not valid. I was tempted to conclude that you conceded the point and agreed with me on the value of the potential outcome representation in the assessment of the critical assumptions in the instrumental variables setting. However, Pearl 2, after a day’s work, took a different tack. Now the claim is that what I did was in fact a causal graph, I just did not realize that! Apparently I was not the only one, it appears that Pearl 1 had not realized that either. Historically, however, the claim by Pearl 2 is not correct. The notions of never-takers, always-takers, compliers and defiers, and the insights that they led to were developed explicitly in a potential outcome framework. I know that because I was there.
Our conversation started with my challenge to your claim that “ Can they [economists] decide whether the IV assumptions (i.e., exogeneity and exclusion) are satisfied in their own models of reality? Of course they can’t; such decisions are intractable to the graph-less mind.’’ I explained that, in my view (risking here another comparison to seven-year olds), the potential outcome framework was valuable in that it aided the assessment of the IV assumptions. I was told I was wrong on that and was promised “objective criteria of tractability and feasibility.’’ All I got was mystical and increasingly derogatory metaphors. That leaves me with little appetite for continuing this discussion.
I leave it to you to write another (or multiple) responses why you are in fact correct and I am mistaken, but I will focus my attention on discussions where there are more prospects for progress. The readers of your blog can decide whether you made a convincing case for your position.
Respectfully,
Guido Imbens
Comment by guido imbens — November 5, 2014 @ 4:14 pm
Dear Guido,
I am going to change gears. I will take your advice and will communicate with readers of this blog without involving you personally. It seems that whenever we communicate directly, there is a tendency to become defensive, take offense, hit back. misinterpret metaphors, and so on, with the end result that substantive issues get neglected. So, In the next few days I will be posting here a few notes on the main issues before us:
1. Can a graph-free mind identify the testable implications of its own assumptions?
2. Can a graph-free mind decide whether the IV assumptions (i.e., exogeneity and exclusion) are satisfied in its own models of reality?
Before we part, however, I would like to explain where I see the main source of our differences. You portray me as a rude grumpy old man who cannot tolerate anyone who “does not discuss my work in sufficient detail”, or who has “different opinion” or who holds “different views”. My position is that we are not dealing here with “opinions” nor with “views” but with a simple and useful tool, not much different from the flashlight, telescope or the multiplication table, that has objective measures of merits, and which has been treated as an “opinion” (and worse) by anything but scientific reasoning.
For the sake of this conversation, just imagine that you are convinced of the merits of multiplication, and that you discover that 30% of high-school teachers do not teach multiplication, saying “I haven’t found it useful in my work.” Imagine further that this mantra is being repeated for 20 years, and that 30% of graduating high-school students are doing arithmetic with addition only (it is doable) and, worse yet, those students begin to repeat their teachers’ mantra, and refrain from testing their math proficiency on any problem with a known solution, saying: “I only work on hard problems” (It is a quote from one of your colleagues, not my invention).
So, assuming that you view things that way, how would you go about convincing the 30% math teachers that they are doing harm to their students? Market of ideas? No way; they can’t read the multiplication sign. Objective performance test? No way; their students only take tests in addition.
Don’t you think that a little unconventional jolt is appropriate here?
Well, we are in the midst of an unconventional jolt, and I am glad I have the opportunity to give your students an objective test of graph-less inference. I hope you encourage them to take it.
It will be posted here shortly.
I will post it even if no one takes it. Why? Because I get a kick thinking how a future historian, way into the future, would be mused to discover that, as late as 2014, some math teachers were still arguing about the merits of multiplication.
Comment by judea pearl — November 5, 2014 @ 11:34 pm
Dear Judea,
If I understood Guido correctly, what he’s asking is for you to solve his IV problem (not other IV examples) more efficiently than he did (I guess in a similar fashion you did for Heckman’s paper).
-Conrad.
Comment by Conrad — November 6, 2014 @ 8:43 am
Dear Conrad,
Once an problem is reduced to a 3-variable IV problem, there is not much one can do for efficiency. However, a causal inference task does not start as a 3-variable IV problem. It starts, as Imbens says, with a whole bunch of controvertial assumptions about the existence or non-existence of influences among variables, and about dependencies among omitted factors, etc. The question before us is how to do this reduction from a web of possible influences to a 3-variable IV setup. My preference is to represent the web as a web and reason on whether the reduction is justified. The graph-free approach is to reason about the web in your head and write down the 3-variable setup if head says OK. Here, graphs play a dual role. First as a transparent representation of the mental web. Second, as a computational engine checking whether the reduction is legitimate. The last step is not trivial, even with a web of 5 variables. Most mortals fail on this task.
judea
Comment by judea pearl — November 6, 2014 @ 9:58 pm
Reply to Chris Auld,
I wish your explanation would be true, ie., that econometricians in particular tend to be more comfortable with equations with diagrams. If this were the case it would be easy to show them the virtues of both.
Diagrams are just qualitative abstractions of the equations, which should allow every red bloooed economist to draw one in his sleep, and answer difficult problems about exogeneity and indentification.
That fact that they are not used gives me the suspicion that the hurdle lies elsewhere.
Regarding mediation, I was glad to see Heckman and Pinto getting into this business, which became very lively about 10 years ago in Health sciences. I hope they learn graphs because, otherwise, they will not be able to
compete with the epidemiologists.
Thank you for alerting me to your semi-rejoinder, it is really a valuable piece of research. I recommend that all our readers read it. http://chrisauld.com/2013/10/08/remarks-on-chen-and-pearl-on-causality-in-econometrics-textbooks/
I was looking forward to examine the book by Triveli etal, which you designated as the most forward-thinking. But they have only one section on causality.
Can you name one economist who knows the meaning of beta? I am not kidding. I know that they all “dig it” and some also “understand it” , but do you know anyone who defines it mathematically?
Even Sir David Hendry says: “the meaning of beta depends on the statistics of the error..”
Judea
Comment by judea pearl — November 7, 2014 @ 6:40 am
Dear Conrad,
Following your exchange with Judea, we would like to present concrete examples of how graphical tools can help determine whether a variable qualifies as an instrument. We use the example of job training program which Imbens used in his paper on instrumental variables.
In this example, the goal is to estimate the effect of a training program (X) on earnings (Y). Imbens suggested proximity (Z) as a possible instrument to assess the effect of X on Y. He then mentioned that the assumption that Z is independent of the potential outcomes {Yx} is a strong one, noting that this can be made more plausible by conditioning on covariates.
To illustrate how graphical models can be used in determining the plausibility of the exclusion restriction, conditional on different covariates, let us consider the following scenarios.
Scenario 1. Suppose that the training program is located in the workplace. In this case, proximity (Z) may affect the numbers of hours employees spend at the office (W) since they spend less time commuting, and this, in turn, may affect their earnings (Y).
Scenario 2. Suppose further that the efficiency of the workers (unmeasured) affects both the number of hours (W) and their salary (Y). (This is represented in the graph through the inclusion of a bidirected arrow between W and Y.)
Scenario 3. Suppose even further that this is a high-tech industry and workers can easily work from home. In this case, the number of hours spent at the office (W) has no effect on earnings (Y). (This is represented in the graph through the removal of the directed arrow from W to Y.)
Scenario 4. Finally, suppose that worker efficiency also affects whether they attend the program because less efficient workers are more likely to benefit from training. (This is represented in the graph through the inclusion of a bidirected arrow from W to X.)
The following figures correspond to the scenarios discussed above.
The reasons we like to work with graphs on such problems is, first, we can represent these scenarios clearly and unambiguously and, second, we can derive the answer in each of these scenarios by inspection of the causal graphs. Here are our answers: (We assume a linear model. For nonparametric, use LATE.)
Scenario 1.
Is the effect of X on Y identifiable? Yes
How? Using Z as an instrument conditioning on W and the effect is equal to r_{zy.w} / r_{zx.w}.
Testable implications? (W independent X given Z)
Scenario 2.
Is the effect of X on Y identifiable? No
How? n/a.
Testable implications? (W independent X given Z)
Scenario 3.
Is the effect of X on Y identifiable? Yes
How? Using Z as an instrument and the effect is equal to r_{zy} / r_{zx}.
Remark. Conditioning on W disqualifies Z as an instrument.
Testable implications? (W independent X given Z)
Scenario 4.
Is the effect of X on Y identifiable? Yes
How? Using Z as an instrument and the effect is equal to r_{zy} / r_{zx}.
Conditioning on W disqualifies Z as an instrument.
Testable implications?
In summary, the examples demonstrate Imben’s point that judging whether a variable (Z) qualifies as an instrument hinges on substantive assumptions underlying the problem being studied. Naturally, these assumptions follow from the causal story about the phenomenon under study. We believe graphs can be an attractive language to solve this type of problem for two reasons. First, it is a transparent representation in which researchers can express the causal story and discuss its plausibility. Second, as a formal representation of those assumptions, it allows us to apply mechanical procedures to evaluate the queries of interest. For example, whether a specific set Z qualifies as an instrument; whether there exists a set Z that qualifies as instrument; what are the testable implications of the causal story.
We hope the examples illustrate these points.
Bryant and Elias
Comment by eb — November 7, 2014 @ 3:50 pm
Follow-up reply to Chris Auld,
Dear Chris,
Enticed by your positive review of Cameron, A. Colin; Trivedi, Pravin K. (2005) book “Microeconometrics”,I bought a Kindle copy and examined the way they treat causation. I was disappointed. Although the word “causal” appear more than 100 times, and although they try hard to explain the difference between “causal interpretation” and other interpretations, the book suffers from the same confusion that plagues other econometric texts: notational ambiguity between regression and structural parameters.
This ambiguity leads to paragraphs such as this one:
“The regression (4.5) is then a regression of one endogenous variable, y, on another, s, and so does not measure the causal impact of an exogenous change in s. The conditional mean function here is not causally meaningful because one is conditioning on a factor, schooling, that is endogenous. Indeed, unless we can argue that s is itself a function of variables at least one of which can vary independently of u, it is unclear just what it is unclear just what it means to regard “alpha” as a causal parameter.”
Here they tie the interpretation of parameters to endogeneity conditions. However, the interpretation of structural parameters remain causal regardless of whether variables are exogenous or endogenous. It is identifiability that depends on such considerations, not interpretation. The criterion cited: “s is itself a function of variables at least one of which can vary independently of u” is not valid even as an identification criterion. An outcome variables (s) can be a function of endogeneous variables, none of which varies independently of u and, still, all parameters have identified causal-effect interepretations.
But following your pointer to this book was worth the effort; I discovered the source of this confusion. It all comes from Sargan’s definition of what an economic model is. Cameron and Trivedi site it: “For example, Sargan (1988, p. 27) states: A model is the specification of the probability distribution for a set of observations. A structure is the specification of the parameters of that distribution.”
As I have argued again and again, there is nothing more misleading than this definition. (see here, http://ftp.cs.ucla.edu/pub/stat_ser/r391.pdf ) But I cannot afford to make more enemies among econometric textbook authors, so I will let you and your colleagues continue this campaign. Strange, we are now 70 years after Haavelmo, and economists still don’t get it.
Judea
Comment by judea pearl — November 8, 2014 @ 1:44 am
I am glad to see Bryant and Elias join the conversation. Let me make one comment, one comment/question, and raise one question.
First, the comment. I think the examples they introduce are interesting ones, and these examples clearly show the tremendous power of causal graphs in answering questions about the identifiability of complex systems given the specification of the graph. I agree with their claim that “We believe graphs can be an attractive language to solve this type of problem.” In fact, this is the type of situation I had in mind when I wrote “ Here, identification questions are substantially more complex, and there is a strong case that the graph-based analyses have more to contribute” in the rejoinder to my Statistical Science paper that started the discussion.
Second, the first comment/question. My original comment was about the instrumental variables setting, as captured in a graph like Figure (1) without the additional variable W. Thus, the graph is simply Z => X => Y with an additional unobserved variable affecting both X and Y but not Z (an arc between X and Y). In that case I raised the issue of assessing the validity of the instrument, and whether a causal graph or a potential outcome set up would make it easier to do so. Proposing meaningful generalizations of the instrumental variables set up is a creative process: there are many ways of doing so. Bryant and Elias suggest a particular set of generalizations starting with the causal graph, involving an additional observed variable W with various scenarios concerning the dependence between W and the original variables in the model. I have no argument with that. If they feel that the causal graph is a natural way for coming up with those generalizations, they should do so. My claim was simply that in the Angrist draft lottery example I found it useful to think of generalizations that involved direct effects of Z on Y that were present for some subpopulations (e.g., never-takers) but not for others (e.g., always-takers). These generalizations are different from the ones Bryant and Elias suggest. I found those generalizations important in practice and easier to conceptualize in a potential outcome framework than in a graph, and, in fact, I still have not seen a natural way of conceptualizing those in a causal graph.
Third, the question. In passing Bryant and Elias mention that they “assume a linear model. For nonparametric, use LATE.” How should we think about identification of the causal effects of X on Y in, say in a simplified version of Figure (1), without W and the arrows from Z to W and from W to Y? It is well known that outside of linear models the average effect of X on Y is not identified, although it can be bounded. How do we read from the graph that the average effect of X on Y for compliers (the local average treatment effect or LATE) is identified? Such a calculation is straightforward in the potential outcomes approach where the LATE concept originates. I looked in Judea’s book, and realized that despite the multiple discussions of instrumental variables, there is no discussion of LATE, or the fact that instrumental variables methods can identify the average effect for a subpopulation, namely that of compliers. Judea may not find LATE important, and he is fully entitled to that view, but were one interested in LATE, how would one consider identification questions such as those in a causal graph?
Respectfully,
Guido Imbens
Comment by guido imbens — November 8, 2014 @ 5:05 pm
[…] a recent posting on this blog, Elias and Bryant described how graphical methods can help decide if a […]
Pingback by Causal Analysis in Theory and Practice » Causal inference without graphs — November 9, 2014 @ 3:47 am
Dear Guido and other bloggers,
In my earlier response to Guido (number 13 above) I promised him to post an “objective test of graph-less inference” so that his students, as well as other curious researchers could examine seriously and concretely if such a thing as a “graph-less causal inference” could exist. I have just posted the test under a new title: “Causal Inference without Graphs”. If you know anyone who truly believes that such a thing exists, please encourage him/her to take the test. But, as I stated before, even if no one takes it, I am glad it is posted in the public sphere. Why? (repeating myself) Because I get a kick thinking how a future historian, way into the future, would be mused to discover that, as late as 2014, some math teachers were still arguing about the merits of multiplication.
Judea
Comment by judea pearl — November 9, 2014 @ 5:10 am
Earlier, Judea said:
10.2. It might be useful to add a word of explanation as to why I claimed that the mental steps you described in examining the exclusion assumptions are steps in a causal graph. When you argued, for example, that some “never takers” might leave the country (to Canada) which will affect their income, you were tracing a path in a graph, going from the lottery outcome, to objection to serve, to leaving the US, to staying 2 years in Canada, to missing on education, to difficulties in getting a job, to low salary. This causal path resides in our mind and each link along this path is traced when we ask: “Can we think of a reason why a low lottery number may, by itself, affect a person salary?” The response type, in our case “never taker,” is a derivable property of the functional relationships in the model, it is not an object that resides explicitly in the model.
The thing that this leaves me wondering about is how you reconcile differences in parsimony across a model with relatively few parameters being estimated (e.g., the IV approach used in econometrics) vs a model with many more parameters (e.g., the graphical approach mentioned by Judea)? I don’t think anyone would argue that what Judea illustrated is a very likely path from the onset of the assignment to the outcome, but given the issues implied by a full specification of these potential paths the computational and data demands necessary to test this type of a model seems to be stifling for research. The concern that I would have is that it would partition the data space so heavily that the impact on statistical power would negate the usefulness of illustrating and fitting a model that more closely resembles likely realities rather than being willing to make assumptions to advance a more parsimonious model that could be tested further in the future (e.g., you get an estimate of the effect of veteran status and then investigate that sample in greater detail to develop a graph based model of the process that led to the observed outcome).
Personally, I don’t view Judea and Guido’s approaches as irreconcilable but rather see them as serving different purposes to arrive at the same goal. In the IV case, it is looking at the most parsimonious model to explain differences in salary that result from military service. In graphical case, it is looking at the processes – and sub-processes – that occur between military service and salary. You could arrive at the same answer as the IV approach using the graphical approach, but in a sense they appear to be modeling/answering fundamentally different questions.
Comment by Billy Buchanan — November 9, 2014 @ 10:12 am
Dear Guido,
We would first like to thank you for taking the time to read and respond to our post. Like you, we are concerned with your model, figure (1) without the W. What we are saying is that this model is a product of a mental simplification process from a complex mental representation of reality, in which many variables like W are present, any of which may violate the exclusion property of Z. We hope you agree with us that every Z has consequences, call them W_1, W_2,…, W_n (you mentioned a couple: going to Canada, being late in getting to the job market, etc.). We are modeling the process through which a researcher takes a messy web with W_1,…W_n and filters it down to a model without any W, or to a model like figure (3), where W resides in the model but is declared harmless.
Additionally, in your paper on IVs, you write that the “strong” assumption that the instrument be independent of the potential outcomes can be made more plausible by conditioning on covariates. We believe that we have demonstrated how graphs are essential for considering such additional variables.
As to your third question, on LATE and never takers, our strategy is to first check if identification is feasible non-parametrically and, if not, to resort to stronger assumptions, such as monotonicity, linearity, no-interactions, and special response-types (e.g., compliers). Graphical methods are necessary and sufficient for the first phase (non-parametric analysis). When stronger assumptions are invited, graphical methods work symbiotically with algebraic methods to fully exploit the powers of those assumptions. Chapters 7-9 of Causality are devoted to this symbiosis. The purpose of our posting was to demonstrate that, even if one insists on pure LATE analysis in a 3-variable IV setting, one must resort to graphical methods in the process of filtering possible threats to the IV conditions, and justifying the 3-variable IV model.
Appreciating your input,
Bryant and Elias
Comment by eb — November 9, 2014 @ 8:44 pm
Dear Bryant and Elias,
Thanks for your response. I think we are actually quite close, and as Billy Buchanan writes, that our differences are not impossible to reconcile. When you write “Graphical methods are necessary and sufficient for the first phase (non-parametric analysis). When stronger assumptions [such as monotonicity, linearity, no-interactions, and special response-types (e.g., compliers)] are invited, graphical methods work symbiotically with algebraic methods to fully exploit the powers of those assumptions,” I read that as acknowledging that causal graphs are not sufficient to study concepts like the local average treatment effect, and that in fact potential outcomes may be very helpful (dare I say essential?) there. That was in the end all I claimed in the paper: for some causal problems (and these include problems I have focused on in my work) the potential outcomes approach is very useful! And, going back to the Koch and West review, that is why I completely agree with the Koch and West conclusion that “ there is much to be gained by also considering the other [other than the approach in Judea’s book “Causality”] major approaches to causal inference,” a claim that Judea disputed.
After that there are just minor quibles left. I do not think you fully substantiated your claim that “we have demonstrated how graphs are essential for considering such additional variables.” What you showed is that for your way of thinking about possible violations of the instrumental variables set up you find the causal graphs useful. You cannot easily put my thought process that sometimes depends on the distinction between nevertakers and alwaystakers into a causal graph, so I don’t really buy the “essential” part of your claim. To see that it is not essential, also note that the findings concerning non-identifiability of the average treatment effect in the instrumental variables setting were well established in the econonometric literature before causal graph methods were used in that setting. As I have written repeatedly in this thread, I think in general it makes little sense to claim that one’s own approach is “necessary and sufficient.” Even if you were to believe that this is true (which I do not think really makes a lot of sense), it might make sense to leave open the (remote) possibility that you are wrong. There is room for different views here!
Finally, the claim that “Chapters 7-9 of Causality are devoted to this symbiosis [of causal graphs and algebraic methods]” is a bit odd: these chapters never actually mentions the local average treatment effect result whose identification we are discussing here.
Respectfully,
Guido Imbens
Comment by guido imbens — November 9, 2014 @ 11:54 pm
Dear Guido,
I am heading to Japan soon and will have limited ability to reply to your comments in the next few days, I would like however to make three brief points.
First, we are not against potential outcomes in any way. Formally speaking, a graphical model is nothing more than a parsimonious encoding of a collection of counterfactual statements (i.e., potential outcomes). The only discussion left is about the language used to reason with these collection of potential outcome statements, namely, which language offers a more cognitively meaningful basis for researchers to judge the plausibility of their assumptions, a computationally more efficient baseline to allow procedures to efficiently compute the queries, and so on. Again, no one is against potential outcomes, the question is just how to dress them, and this choice has strong consequences. I understand that the intent of Judea’s latest post was to discuss precisely these issues (link here), I hope he made them clear.
Second, it is not accurate to say that “You cannot easily put my thought process that sometimes depends on the distinction between nevertakers and alwaystakers into a causal graph”. Please, open pages 262-266 in Causality and you will see exactly the ‘always takers’, ‘never takers’, and the other response types there. These types follow naturally from the functional relationships (structural equation model) and are encoded through the exogenous (unobserved) variable U. These types are derivative of a deeper interpretation following a process-based type of reasoning, but they are there. So, the reasoning that you proposed has a mirror in a graph. Note that a type like “never takers”, for example, is represented through one possible instantiation of the variable U.
Third, your statement “What you showed is that for your way of thinking about possible violations of the instrumental variables set up you find the causal graphs useful” indicates that you consider our way to be somewhat uncommon. I would be interested to know what would be your way of guarding against possible violation of the instrumental variables set up. Can you explicitly write down the process through which the types (‘never takers’, ‘always taker’, etc) will appear in your reasoning? I think you would agree that explicating one’s thought process is an important component in communication and inference. Graphs happened to facilitate this explication but perhaps there are other representations that are as effective. I am also curious to see how this thought process can be expanded to handle the examples with 4 variables given in our first post.
Appreciating your input,
Elias
Comment by eb — November 10, 2014 @ 4:01 am
Dear Elias,
Have a good trip to Japan!
I am glad to read that your comment “we are not against potential outcomes in any way.’’ If you agree that it is about “which language offers a more cognitively meaningful basis for researchers to judge the plausibility of their assumptions’’ it is difficult to see what the basis is for your earlier absolutist claims that “graphs are essential,’’ that “Graphical methods are necessary and sufficient,’’ “one must resort to graphical methods.’’ As I wrote earlier, I think such blanket statements are misleading and incorrect.
Pages 262-266 in Judea’ book say nothing about local average treatment effects. The discussion suggests we all agree that LATE is easier to discuss in a potential outcome framework, again showing that both parts of the claim that graphical methods are necessary and sufficient are incorrect. Your own comments taken at face value imply that for some questions/settings causal graphs are superior, and for some questions/settings potential outcomes are superior. I completely agree with that.
Respectfully,
Guido Imbens
Comment by guido imbens — November 10, 2014 @ 10:41 am
Dear Guido,
Judea has given his views on LATE/principal stratification in this paper
http://ftp.cs.ucla.edu/pub/stat_ser/r382.pdf
I just thought you would be interested to take a look.
-Conrad.
Comment by Conrad — November 10, 2014 @ 5:32 pm
Dear Conrad,
I gave my view on LATE/principal stratification in a paper on the same Journal IJB in response to Judea’s questions about LATE and PS: Mattei A., Mealli F., A Refreshing Account of Principal Stratification, International Journal of Biostatistics, 1, 1-37.
I think the paper reinforces the claim that, while graphs can be a powerful tool discovering identification strategies in some settings, they cannot easily handle identifiability/estimation of causal effects defined for latent but meaningful subpopulations, a setting where potential outcomes have shown their usefulness.
Fabrizia
Comment by Fabrizia Mealli — November 10, 2014 @ 6:13 pm
Dear Fabrizia,
Thank you for sharing the article. I have views similar to yours when it comes to interaction or latent subpopulations. In graphs interaction is assumed to exist implicitly. Beyond that I find graphs to be quite useful.
-Conrad.
Comment by Conrad — November 10, 2014 @ 9:03 pm
Dear Guido,
I dont understand your logic.
The basis for the claim that “graphs are essential” is that they match our picture of reality, while the alternatives do NOT offer “a cognitively meaningful language to judge the plausibility of assumptions”. If you know of an alternative that does, please unveil it (without graphs) in the context of the two examples I discuss in “Causal Reasoning w/o Graphs”, Nov. 9, 2014, where two languages are presented side by side, from the same starting point.
You keep on bringing the 3-variable IV setting as a proof that some settings exist in which potential outcome (PO) can operate without graphs. But you ignore the fact that PO can operate in this setting only because the setting was already trimmed and groomed from a larger setting, and that PO cannot handle the trimming and grooming on its own. Elias challenged you with several requests to show how this trimming can be accomplished, even in a simple 4-variable model, and you keep bringing up, again, the 3-variable setting as a proof of your claim.
A fisherman enters a restaurant, orders a fried fish and tells his friend: You see, some fish need no catching.
Every causal inference problem requires judgmental knowledge, hence, a process-based language for representing what we know, in the format in which our experience is stored. Graphs provide such a language and, unless someone convinces us that human experience is stored in the form of sentences such as
X _||_ {Y(0),(Y(1)} | W1,W2,W3..
we have a fairly solid basis for claiming that graphs are essential, even for 3-variable IV’s or other groomed exercises and, perhaps all fish need catching.
Comment by judea pearl — November 11, 2014 @ 2:19 am
Judea and his collaborators wrote repeatedly that causal graphs were essential and indispensable for analyzing causal problems. Some of the comments included “Graphical methods are necessary and sufficient,” “one must resort to graphical methods,” “the assumptions needed are … comprehensible only in the language of graphs.”
I disagreed with that claim and asserted that for at least for some problems potential outcomes were a more attractive approach.
Judea took a hard line on this, responding: “There was a time when people settled differences by saying: Oh, it is just a different approach to the same problem, each person may choose the argument he/she finds most persuasive, let a thousand flowers bloom, there are dozen roads to Rome, etc. That era has ended with the development of objective criteria of tractability and feasibility.”
The counter example I then introduced concerned identification of the local average treatment effect in an instrumental variables setting. (Because Judea’s claims are about all causal settings / problems, to counter his claims logic dictated it would suffice to give a single counter example.) This more focused discussion led Judea to make the very revealing admission in his latest post: “no one should expect the graph alone to tell us if LATE is identified.” Indeed! I could not agree more. So, potential outcomes are useful for some problems, and causal graphs are useful for some problems. Let a thousand flowers bloom, and let’s dispense with the absolutist statements!
Respectfully,
Guido Imbens
Ps it was nice to have your collaborators Elias and Bryant join the discussion, and especially refreshing because they did not feel the need to resort to offensive metaphors. In that regard your earlier apology seems to be short-lived.
Comment by guido imbens — November 11, 2014 @ 1:23 pm
Dear Guido,
In a later posting I will justify formally each and every one of my “absolutist” claims, from “Graphical methods are necessary and sufficient” to “the assumptions needed are comprehensible only in the language of graphs”. In this short comment, I would like only to compare the social and educational impacts of my “absolutist” claims against your seemingly more tolerant and pluralistic approach, as summarized in mild assertions such as: “at least for some problems potential outcomes were a more attractive approach” or “[economists] have not felt that graphical models have much to offer them.”
My “absolutist” approach has resulted in a generation of researchers who are equally versed in potential outcomes and graphical models; they can prove the LATE estimand with the same ease that they can spot conditional ignorability in a 10-variable model. At the same time, your “thousand flowers” banner is beeing used as an excuse to exclude one of the two languages from the vocabulary of thousands of students and readers. It has spawned a generation of willfully blinded researchers who are totally ignorant of graphs, who cannot tell ignorability even in their own models, and who are absolutely convinced that “It is possible for a forward-looking statistician to do causal inference in the 21st century without understanding of graphical models” (quoted from Andrew Gelman).
One does not need to go very far to see how it works. In your upcoming book with Don Rubin you say: “In our own work, perhaps influenced by the type of examples arising in social and medical sciences, we have not found this [graphical] approach to facilitate the drawing of causal inferences, and we do not discuss it in detail in this text. [page 25] (In fact you do not discuss it AT ALL, in this nor in any other text). The pattern is familiar; start by contextualizing the exclusion to “our own work”, give a mild generalization to the entirety of the “social and medical sciences,” and thousands of readers will then take it as a license (if not a decree) for a career-long handicap. (And this is a 2015 book.)
So let us not wave the “thousand flowers” banner too high. Let us start by counting the number of flowers in our own garden, pay some attention to the educational impact of banner waving, and consider seriously whether a more “absolutist” approach is not urgently needed, to get “thousand flowers” to bloom.
——–
REMARK: This post is dedicated to next generation researchers and to Lady History who, way into the future, would be mused to discover that, as late as 2014, some math teachers were still questioning the essential role of multiplication in arithmetics.
Comment by judea pearl — November 11, 2014 @ 9:17 pm
[…] has been going on for the past two decades and, judging by the recent exchange with Guido Imbens (link), we are not closer to an understanding than we were in 1995. Even an explicit demonstration of how […]
Pingback by Causal Analysis in Theory and Practice » On the First Law of Causal Inference — November 29, 2014 @ 3:53 am
whoah this weblog is wonderful i like reading your posts.
Keep up the great work! You recognize, many persons are looking around
for this information, you could aid them greatly.
Comment by Jimmy — December 18, 2014 @ 2:54 am
This is a very short and modest contribution from epidemiology. Truly modest because of my wide ignorance on the complex issues that you guys are tackling in fascinating and truly exciting ways. Nevertheless, I think this comment may provide an interesting glimpse into what is going on in epidemiology. At least I find very relevant the comparisons and analogies that Judea is drawing among different disciplines, including economics and epidemiology.
The current deconstruction of paradoxes that is going in epidemiology, public health, and medicine (or, if you wish, in some of the health, life, and social sciences) is one among several signs that a profound methodological renewal is taking place.
A profound renewal of methods for clinical and epidemiological research is taking place. Perhaps for some basic sciences as well. The new methodological approaches have already explained −and sometimes, deconstructed− long puzzling paradoxes, including the benefits of obesity in diabetics, and of smoking in low birth weight. Theoretical and practical achievements of the new methods also comprise the elucidation of the causal structure of long-disputed questions as Berkson’s bias and Simpson’s paradox, or the controversies on estrogens and endometrial cancer, and on some adverse effects of hormone replacement therapy. These are all signs that the new methods can go deeper and beyond the methods in current use. The current theoretical and methodological renewal –or perhaps “revolution”– may be changing deeply how clinical and epidemiological research is conceived and performed, how we assess the validity and relevance of findings, and how causal inferences are made. Clinical researchers better keep an eye on DAGs and related concepts.
As many others, I believe we can have deep respect and appreciation for “purely” theoretical discoveries and, at the same time, for discoveries with “simple” practical implications (e.g., discoveries that improve medical care, discoveries that help develop better policies for the primary prevention of diabetes or obesity, discoveries that widen our knowledge on the causes of human diseases). Accordingly, I’d like to suggest that the debate that is going on in this blog could devote some additional efforts to assess the practical contributions that the methodological revolution is making. Yes, just the old and always relevant test against some aspect of reality. In medicine, in economics, in education… Because, if a method is working in the real world, how likely is it that the method is flawed?
Finally: I think it will not be long before the ongoing methodological innovations become mainstream in clinical research, as they are becoming in epidemiologic research. And why should the innovations not affect some areas of basic research as well? For instance, how many biochemical and genetic relationships are currently considered causal −not even perceived as paradoxical− and yet due to problems as conditioning on colliders?
Comment by Miquel Porta — January 4, 2015 @ 6:15 am
[…] discussion with Guido Imbens on why some economists avoid graphs at all cost (link) has moved on to another question: “Why some economists refuse to benefit from the First […]
Pingback by Causal Analysis in Theory and Practice » Winter Greeting from the UCLA Causality Blog — January 27, 2015 @ 7:53 am
[…] with Guido Imbens regarding computational barriers to graph-free causal inference, click here). Graph avoiders, should reckon with this […]
Pingback by Causal Analysis in Theory and Practice » Flowers of the First Law of Causal Inference (3) — April 24, 2015 @ 10:54 pm
lustra na wymiar piotrków
See this lustra na wymiar for yourself.Glass furniture,building and much more in Piotrków Trybunalski
Trackback by kabiny prysznicowe piotrków — May 19, 2015 @ 12:33 pm