Causal Analysis in Theory and Practice

Judea Pearl writes:

Antonio Forcina writes:

I have tried to read Judea and Ilya's paper on effect of treatment on the treated which sound much more general than anything else I have read on the subject before. Unfortunately I was unable to follow their proof and could not find an instance where the ETT effect is identifiable. The only instance where I new that ETT was identifiable is with an instrumental variable under certain restrictions, instead I imagine that identifiability here means without restrictions other than those encoded in the DAG.The most clear treatment of the subject I new until now is in a paper by Hernan and Robins in Epidemiology 2006; and I do not understand why the discussion on Forcina's paper by Robins, Vander Weele and RIchardson is so popular.

Judea Pearl writes:

Recently, there have been several articles and many blog entries concerning the question of what measurements should be incorporated in various methods of causal analysis.The statement below is offered by way of a resolution that (1) summarizes the discussion thus far, (2) settles differences of opinion and  (3) remains faithful to logic and facts as we know them today.

The resolution is reached by separating the discussion into three parts:  1.  Propensity score matching  2. Bayes analysis 3. Other techniques

1. Propensity score matching. Everyone is in the opinion that one should screen variables before including them as predictors in the propensity-score function.We know that, theoretically, some variables are capable of increasing bias (over and above what it would be without their inclusion,) and some are even guaranteed to increase such bias.

1.1 The identity of those bias-raising variables is hard to ascertain in practice. However, their
general features can be described in either graphical terms or in terms of the "assignment mechanism", P(W|X, Y0,Y1),if such is assumed.

1.2 In light of 1.1, it is recommend that the practice of adjusting for as many measurements as possible should be approached with great caution. While most available measurements are bias-reducing, some are bias-increasing.The criterion of producing "balanced population" for
matching, should not be the only one in deciding whether a measurement should enter the propensity score function.

2. Bayes analysis. If the science behind the problem, is properly formulated as constraints over the prior distribution of the "assignment mechanism" P(W|X, Y, Y0,Y1), then one need not exclude any measurement in advance; sequential updating will properly narrow the posteriors to reflect both the science and the available data.

2.1 If one can deduce from the "science" that certain covariates are "irrelevant" to the problem at hand,there is no harm in excluding them from the Bayesian analysis. Such deductions can be derived either analytically, from the algebraic description of the constraints, or graphically, from the diagramatical description of those constraints.

2.2 The inclusion of irrelevant variables in the Bayesian analysis may be advantageous from certain perspectives (e.g., provide evidence for missing data) and dis-advantageous from others (e.g, slow convergence, increase in problem dimensionality, sensitivity to misspecification).

2.3 The status of intermediate variables (and M-Bias) fall under these considerations. For example, if the chain Smoking ->Tar-> Cancer represents the correct specification of the problem, there are advantages (e.g., reduced variance (Cox, 1960?)) to including Tar in the analysis even though the causal effect (of smoking on cancer) is identifiable without measuring Tar, if Smoking is randomized. However, misspecification of the role of Tar, may lead to bias.

3. Other methods. Instrumental variables, intermediate variables and confounders can be identified, and harnessed to facilitate effective causal inference using other methods, not involving propensity score matching or Bayes analysis. For example, the measurement of Tar in the example above, can facilitate a consistent estimate of the causal effect (of Smoking on Cancer) even in the presence of unmeasured confounding factors, affecting both smoking and cancer. Such analysis can be done by either graphical methods (Causality, page 81-88) or counterfactual algebra (Causality, page 231-234).

Thus far, I have not heard any objection to any of these conclusions, so I consider it a resolution of what seemed to be a major disagreement among experts. And this supports what Aristotle said (or should have said): Causality is simple.

Judea

Judea Pearl's exchange with Andrew Gelman about Donald Rubin's approach to propensity scores.

Dear Andrew,

In your last posting you have resolved the clash between Rubin and Pearl — many thanks.

You have concluded: "Conditioning depends on how the model is set up,” which is exactly what I have been arguing in the last five postings.

But I am not asking for credit. I would like only to repeat it to all the dozens of propensity-score practitioners who are under the impression that Rubin, Rosenbaum and other leaders are strongly in favor of including as many variables as possible in the propensity score function, especially if they are good predictors of the treatment assignment.

Let me quote you again, lest it did not reach some of those practitioners:

"They (Rubin, Angrist, Imbens) don't suggest including an intermediate variable as a regression predictor or as a predictor in a propensity score matching routine, and they don't suggest including an instrument as a predictor in a propensity score model." (Gelman posting 2009)

Our conclusion is illuminating and compelling:

When Rosenbaum wrote: "there is little or no reason to avoid adjustment for a true covariate, a variable describing subjects before treatment" (Rosenbaum, 2002, p. 76], he really meant to exclude instrumental variables, colliders and perhaps other nasty variables from his statement.

And when Rubin wrote (2009):

"to avoid conditioning on some observed covariates,… is nonscientific ad hockery."

he really did not mean it in the context of propensity-score matching (which was the topic of his article.)"

And when Gelman wrote (in your first posting of this discussion):

(which, if we look carefully at the reason for this exclusion, really means to exclude almost ALL variables that affect treatment assignment). And when Rubin changed the definition of "ignorability" (2009) "to be defined conditional on all observed covariates" he really meant to exclude colliders, instrumental variables and other trouble makers, he simply did not bother to tell us that (1) some variables are trouble makers, and (2) how to spot them.

If you and I accept these qualifications, and if you help me get the word out to those poor practitioners, I don’t mind it if you tell them that all these exceptions and qualifications are well known in the potential-outcome subculture and that these prove that Pearl's approach was wrong all along. But, please get the word out to those poor propensity- score practitioners, because they are conditioning on everything they can get their hand on.

I have spoken to many of them, and they are not even aware of the problem.

They follow Rubin's advice, and they are scared to be called "unprincipled" — I am not.

Andrew Gelman replies:

I agree with you that the term "unprincipled" is unfortunate, and I hope that all of us will try to use less negative terms when describing inferential approaches other than ours.

Regarding your main point above, it does seem that we are close to agreement. Graphical modeling is a way to understand the posited relations between variables in a statistical model. The graphical modeling framework does not magically create the model (and I'm not claiming that you ever said it did), but it can be a way for the user to understand his or her model and to more easily to communicate it to others.

I think you're still confused on one point, though. It is not true that Rubin and Rosenbaum "really meant to exclude colliders, instrumental variables and other trouble makers." Rubin specifically wants to include instrumental variables and other "trouble makers" in his models–see his 1996 paper with Angrist and Imbens. He includes them, just not as regression predictors.

I agree with you that Rubin would not include an instrument or an intermediate outcome in the propensity score function, and it is unfortunate if people are doing this. But he definitely recommends including instruments and intermediate outcomes in the model in an appropriate way (where "appropriate" is defined based on the model itself, whether set up graphically (as you prefer) or algebraically (as Rubin prefers).

Judea Pearl replies:

We are indeed close to a resolution.

Let us agree to separate the resolution into three parts:

1. Propensity score matching
2. Bayes analysis
3. Other techniques

1. Propensity score matching. Here we agree (practitioners, please listen) that one should screen variables before including them in the propensity-score function. Because some of them can be trouble-makers, name, capable of increasing bias over and above what it would be without their inclusion, and some are guaranteed to increase that bias.

1.1 Who are those trouble makers, and how to spot them, is a separate question that is a matter of taste. Pearl prefers to identify them from the graph and Rubin prefers to identify them from the probability distribution P(W|X, Y₀,Y₁) which he calls "the science",

1.2 We agree that once those trouble makers are identified, they should be excluded (repeat: excluded) from entering the propensity score function, regardless of how people interpreted previous statements by Rubin (2007, 2009), Rosenbaum (2002) or other analysts.

2. Bayes analysis. We agree that, if one manages to formulate the "science" behind the problem in the form of constraints over the distribution P(W|X, Y, Y₀,Y₁) and load it with appropriate priors, then one need not exclude trouble makers in advance; sequential updating will properly narrow the posteriors to reflect both the science and the data. One such exercise is demonstrated in section 8.5 of Pearl's book Causality, which purposefully include an instrumental variable to deal with Bayes estimates of causal effects in clinical trials with non-compliance.(Mentioned here to allay any fears that Pearl is "confused" about this point, or is unaware of what can be done with Bayesiam methods)

2.1 Still, if the "science" proclaims certain covariates to be "irrelevant", there is no harm in excluding them EVEN FROM a BAYESIAM ANALYSIS, and this is true whether the "science" is expressed as distribution over counterfactuals (as in the case of Rubin) or as a graph, based directly on the subjective judgments that are encoded in the "science". There might actually be benefits to excluding them, even when measurement cost is zero.

2.2 Such irrelevant variables are, for example,
colliders, and certain variables    affected by the treatment, e.g., Cost<—Treatment —-> Outcome.

2.3 The status of intermediate variables (and M-Bias) is still in the open. We are waiting for detailed analysis of examples such as Smoking —>Tar—>Cancer with and without the Tar.  There might be some computational advantages    to including Tar in the analysis, although the target causal effect (of smoking on cancer) is insensitive to Tar if Smoking is randomized.

3. Other methods. Instrumental variables, intermediate variables and confounders can be identified, and harnessed to facilitate effective causal inference using other methods, not involving propensity score matching or Bayes analysis. The measurement of Tar, for example (see example above)  can be shown to enable a consistent estimate of the causal effect (of Smoking on Cancer) even in the presence of confounding factors affecting both smoking and cancer (page 81-84 of Causality).

Shall we both sign on this resolution? 

Judea Pearl writes to Andrew Gelman about differences between Donald Rubin's and Pearl's approaches.

Dear Andrew,

I think our discussion could benefit from the distinction between "theories" and “approaches." A theory T is a set of mathematical constraints on what can and cannot be deduced from a set of premises. An approach is what you do with those constraints, how you apply them, at what sequence, and in what language.
 
In the context of this distinction I say that Rubin’s theory T is equivalent to Pearl’s. While the approach is different, equivalence of theories means that there cannot be a clash of claims, and this is a proven fact. In other words if there is ever a clash about a given problem, it means one of two things, either the theory was not applied properly or additional information about the problem was assumed by one investigator that was not assumed by the other.
 
Now to the "approach". Below is my analysis of the two approaches, please check if it coincide with your understanding of Rubin's approach.
 
Pearl says, let us start with the science behind each problem, e.g., coins, bells, seat-belts, smoking etc.. Our theory tells us that no causal claim can ever be issued if we know nothing about the science, even if we take infinite samples. Therefore, let us articulate what we do know about the science, however meager, and see what we can get out of the theory.  This calls for encoding the relationships among the relevant entities, coins, bells and seat-belts, in some language, call it L, thus creating a "problem description" L(P). L(P) contains variables, observed and unobserved factors, equations, graphs, physical constraints, processes, influences, lack of influences, dependencies, etc, whatever is needed to encode our understanding of the science behind the problem P.
 
Now we are ready to solve the problem. We take L(P) and appeal to our theory T:Theory, theory on the wall, how should we solve L(P)? The theory says: Sorry, I don’t speak L, I speak T.
 
What do we do? Pearl's approach says: take the constraints from T, and translate them into new constraints, formulated in language L, thus creating a set of constraints L(T) that echo T and tell us what can and what cannot be deduced from certain premises encoded in L(P).Next, we deduced a claim C in L(P) (if possible)or we proclaim C to be "non-deducible". Done.
 
Rubin's approach is a bit different. We again look at a problem P but, instead of encoding it in L, we skip that part and translate P directly into a language that the theory can recognize; call it T(P). (It looks like P(W|X, Y₁, Y₂) according to Rubin's SIM article (2007)) Now we ask: Theory, theory on the wall, how should we solve T(P)? The theory answers: Easy, man! I speak T. So, the theory produces a claim C in T, and everyone is happy.
 
To summarize, Pearl brings the theory to the problem, Rubin takes the problem to the theory.
 
To an observer from the outside the two approaches would look identical, because the claims produced are identical and the estimation procedures they dictate are identical. So, one should naturally ask, how can there ever be a clash in claims like the one concerning covariate selection?
 
Differences will show up when researchers begin to deviate from the philosophies that govern either one of the two approaches. For example, researchers might find it too hard to go from P to T(P). So hard in fact that they give up on thinking about P, and appeal directly to the theory: Theory, theory on the wall, we don’t know anything about the problem, actually, we do know, but we don’t feel like thinking about it. Can you deduce claim C for us?
 
If asked, the theory would answer: "No, sorry, nothing can be deduced without some problem description. "But some researchers may not wish to talk directly to the theory, it is too taxing to write a story and coins and bells in language of P(W|X, Y₁, Y₂)..So what do they do? They fall into a lazy mode, like: "Use whatever routines worked for you in the past. If propensity scores worked for you, use it, take all available measurements as predictors. the more the better." Lazy thinking forms subcultures, and subcultures tend to isolate themselves from the rest of the scientific community because nothing could be more enticing than methods and habits, especially when they reinforced by respected leaders, And especially when habits are supported by convincing metaphors. For example, how can you go wrong by "balancing" treated and untreated units on more and more covariates. Balancing, we all know, is a good thing to have; is even present in randomized trials. So, how can we go wrong? An open-minded student of such subculture should ask: "The more the better? Really? How come? Pearl says some covariates might increase bias? And there should be no clash in claims between the two approaches. "An open minded student would also be so bold as to take a pencil and paper and consult the theory T directly, asking: Do I have to worry about increased bias in my specific problem?" And the theory would answer: You might have to worry, yes, but I can only tell you where the threats are if you tell me something about the problem, which you refuse to do.
 
Or the theory might answer: If you feel so shy about describing your problem, why don’t you use the Bayesian method; this way, even if you end up with unidentified situation, the method would not punish you for not thinking about the problem, it would just produce a very wide posterior, The more you think, the narrower the posterior. Isn't this a fair play?
 
To summarize:
 
One theory has spawned two approaches, The two approaches have spawned two subcultures.Culture-1 solves problems in L(P) by the theoretical rules of L(T) that were translated from T into L. Culture-2 avoids describing P, or thinking about P, and relies primarily on metaphors, convenience of methods and guru's advise.
 
Once in a while, when problems are simple enough, (like the binary Instrumental Variable problem), someone from culture 2 would formulate a problem in T and derive useful results. But, normally, problem-description avoidance is the rule of the day. So much so, that even 2-coins-one-bell problems are not analyzed mathematically by rank and file researches; they are sent to the gurus for opinion.
 
I admit that I was not aware of the capability of Bayesian methods to combine two subpopulations in which a quantity is unidentified and extract a point estimate of the average, when such average is identified. I am still waiting for the bell-coins example worked out by this method — it would enrich by arsenal of techniques. But this would still not alter my approach, namely, to formulate problems in a language close to their source: human experience.
 
In other words, even if the Bayesian method will be shown capable of untangling the two subpopulations, thus giving researchers the assurance that they have not ignored any data, I would still prefer to encode a problem in L(P), then ask L(T): Theory, theory on the wall, look at my problem and tell me if perhaps there are measurements that are redundant. If the answer is Yes, I would save the effort of measuring them, and the increased dimensionality of regressing on them, and just get the answer that I need from the essential measurements. Recall that, even if one insists on going the Bayesian route, the task of translating a problem into T remains the same. All we gain is the luxury of not thinking in advance about which measurements can be avoided, we let the theory do the filtering
automatically.   I am now eager to see how this is done; two-cons and  one bell. Everyone knows the answer: coin-1 has no causal effect on coin-2 no matter if we listen to the bell or not. Lets see Rev. Bayes advise us correctly: ignore the bell.

Andrew Gelman wrote a follow up to his original post:

To follow up on yesterday's discussion, I wanted to go through a bunch of different issues involving graphical modeling and causal inference.

Contents:
– A practical issue: poststratification
– 3 kinds of graphs
– Minimal Pearl and Minimal Rubin
– Getting the most out of Minimal Pearl and Minimal Rubin
– Conceptual differences between Pearl's and Rubin's models
– Controlling for intermediate outcomes
– Statistical models are based on assumptions
– In defense of taste
– Argument from authority?
– How could these issues be resolved?
– Holes everywhere
– What I can contribute

Andrew Gelman (Columbia) recently wrote a blog post motivated by Judea Pearl's paper, "Myth, Confusion, and Science in Causal Analysis. " In response, Pearl writes:

Dear Andrew,

Thank you for your blog post dated July 5. I appreciate your genuine and respectful quest to explore the differences between the approaches that I and Don Rubin are taking to causal inference.

In general, I would be the first to rally behind your call for theoretical pluralism (e.g., "It make sense that other theoretical perspectives such as Pearl's could be useful too.") We know that one can prove a theorem in geometry by either geometrical or algebraic methods, depending on the problem and the perspective one prefers to take–only the very dogmatic would label one of the methods "unprincipled".

My article, "Myth, confusion and Science in Causal Analysis", is written with this dual perspective in mind, fully accommodating the graphical and potential-outcome conceptualizations as interchangeable, "A theorem in one approach is a theorem in another," I wrote.

However, when adherents of the one-perspective approach make claims that mathematically contradict those derived from the dual-perspective approach, one begins to wonder whether there is something more fundamental at play here.

(more…)

A letter from Judea Pearl to the Editor of Statistics in Medicine:

Dear Editor,

I read with great interest Donald Rubin’s paper "The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials" (2007, 26(1):20-36) [1], as well as the Letter To The Editor by Ian Shrier (2008, 27(14):2740-2741) [2], and Author Reply by Don Rubin (2008, 27(14):2741-2742) [3].

Shrier’s Letter posed an important question which remains unanswered in Rubin’s reply. I here venture to answer this question and to clarify related issues concerning the interpretation of propensity scores (PS) and their role in causal inference.

Shrier’s question was whether, asymptotically, the use of PS methods as described by Rubin may actually increase, not decrease, bias over and above a crude, unadjusted comparison between treated and untreated subjects. The answer is: Yes, and the M-graph cited by Shrier (see also [4, 5]) provides an extreme such example; the crude estimate is bias-free, while PS methods introduce new bias.

This occurs when treatment is strongly ignorable to begin with and becomes non-ignorable at some levels of e. In other words, although treated and untreated units are balanced in each stratum of e, the balance only holds relative to the covariates measured; unobserved confounders may be highly unbalanced in each stratum of e, capable of producing significant bias. Moreover, such imbalance may be dormant in the crude estimate and awakened throughthe use of PS methods.

There are other features of PS methods that are worth emphasizing.

First, the propensity score e is a probabilistic, not a causal concept. Therefore, in the limit of very large sample, PS methods are bound to produce the same bias as straight stratification on the same set of measured covariates. They merely offer an effective way of approaching the asymptotic estimate which, due to the high dimensionality of X, is practically unattainable with straight stratification. Still, the asymptotic estimate is the same in both cases, and may or may not be biased, depending on the set of covariates chosen.

Second, the task of choosing a sufficient (i.e., bias-eliminating) set of covariates for PS analysis requires qualitative knowledge of the causal relationships among both observed and unobserved covariates. Given such knowledge, finding a sufficient set of covariates or deciding whether a sufficient set exists are two problems that can readily be solved by graphical methods [6, 7, 4].

Finally, experimental assessments of the bias-reducing potential of PS methods (such as those described in Rubin, 2007 [1]) can only be generalized to cases where the causal relationships among covariates, treatment, outcome and unobserved confounders are the same as in the experimental study. Thus, a study that proves bias reduction through the use of covariate set X does not justify the use of X in problems where the influence of unobserved confounders may be different.

In summary, the effectiveness of PS methods rests critically on the choice of covariates, X, and that choice cannot be left to guesswork; it requires that we understand, at least figuratively, what relationships may exist between observed and unobserved covariates and how the choice of the former can bring about strong ignorability or a reasonable approximation thereof.

Judea Pearl
UCLA

REFERENCES

1. Rubin D. The design versus the analysis of observational studies for causal effects: Parallels with the designof randomized trials. Statistics in Medicine 2007; 26:20–36.
2. Shrier I. Letter to the editor. Statistics in Medicine 2008; 27:2740–2741.
3. Rubin D. Author’s reply (to Ian Shrier’s Letter to the Editor). Statistics in Medicine 2008; 27:2741–2742.
4. Greenland S, Pearl J, Robins J. Causal diagrams for epidemiologic research. Epidemiology 1999; 10(1):37–48.
5. Greenland S. Quantifying biases in causal models: Classical confounding vs. collider-stratification bias.pidemiology 2003; 14:300–306.
6. Pearl J. Comment: Graphical models, causality, and intervention. Statistical Science 1993; 8(3):266–269.
7. Pearl J. Causality: Models, Reasoning, and Inference. Cambridge University Press: New York, 2000.

In response to an email discussion between Sander Greenland and Tyler VanderWeele concerning the semantics of causal dags, I have offered the following comment:

The phrase "what is causal about … " can be answered at three distinct levels of discussion: 1. Interpretation, 2. Construction and 3. Validation.

1. Interpretation

Causal dag is a model, and, like any other model, it is a symbolic object that acts as an oracle, that is, it produces answers to a set of queries of interest. In our case, the queries of interest are those concerning interventions and counterfactuals.

A distinction should be made here between "Causal Bayesian Networks" which are oracles of interventional queries only (see Causality Def. 1.3.1 pp. 23,) and functional or counterfactual dags, which are smarter oracles (given all the functions), capable of answering interventional as well as counterfactual queries, via Eq. (3.51), pp 98. For example, Fig. 1.6 (a) is a Causal Bayesian Network but not a functional dag of the process described.

2. Construction

What questions should investigators ask themselves during the construction of a causal dag, so as to minimize the likelihood of misspecification?

Given the judgmental nature of the assumptions embedded in the dag, this is a cognitive question, touching on the way scientists encode experience.

Three observations:

2.1 Scientists encode experience in the form of scientific laws, that is, counterfactual value-assignment processes. To determine the value x that variable X takes on, Nature is envisioned as examining the values s of some other variables, say S, and deciding according to some function x=f(s) what value X would be assigned. This is a more fundamental conception of causation than intervention, and it applies to non-manipulable variables as well, hence my favorite counter-slogan: "Of course causation without manipulation," or "Causation precedes manipulation."

2.2 To match the nature of scientific thought, the construction of causal dags is best done in counterfactual vocabulary. People judgment about statistical dependencies emanate from their judgment about cause-effect relationships, and people's judgments about causation emanate from counterfactual thinking; (otherwise, why would David Hume and David Lewis be tempted to define the former in term of the latter and not the other way around? see pp. 238-9). Accordingly, questions such as "What other factors determine X beside S?" or "Are the omitted factors determining X correlated with those determining Y?" can be quite meaningful to investigators and, hence, can be answered reliably, which explains why students find it easier to think in terms of "the existence of a hidden common parent of two or more nodes."

2.3 There is, I admit, some finger crossing in such judgment, as there is in any judgment, but the amount of guesswork is much much less than in the highly respected "Let us assume strong ignorability" which no mortal understands, except through translation into "no correlated hidden factors."

3. Validation

Given a causal dag, are its predictions compatible with the set of observations and experiments that one can perform?. The three conditions of Def. 1.3.1 pp. 23, are sufficient for guaranteeing that ALL observational and interventional queries be answered correctly. What does it mean? It means that, once conditions i-iii are satisfied, we can predict the effects of any policy, atomic as well as compound, deterministic as well as stochastic, that can be specified in terms of direct changes onto a given set of variables in the study. (By this we exclude unanticipated side-effects).In reality, students who construct dags do not think interventions, nor is it reasonable to assume that all interventional distributions would be available to us (as is assumed in Def 1.3.1). Still, it is healthy to have a sufficient set of empirical validation criteria such as Def 1.3.1.