Causal Analysis in Theory and Practice

The European Journal of Personality just published an article by James Lee, titled
“Correlation and Causation in the Study of Personality”
European Journal of Personality, Eur.J.Pers. 26: 372-390 (2012) DOI:10.1002/per.1863.
Link: http://onlinelibrary.wiley.com/doi/10.1002/per.1863/pdf,
or here.

Lee’s article is followed by Open Peer Commentaries
http://onlinelibrary.wiley.com/doi/10.1002/per.1865/full,
or here.

(Strikingly, the commentary by Rolf Steyer declares the do-operator to be self-contradictory. I trust readers of this blog to spot Steyer’s error right away. If not, I will post.)

Another recent paper on causation in psychological research is the one by Shadish and Sullivan,
“Theories of Causation in Psychological Science”
In Harris Cooper (Ed-in-Chief), APA Handbook of Research Methods in Psychology, Volume 1, pp. 23-52, 2012.
http://www.cs.ucla.edu/~kaoru/shadish-sullivan12.pdf

While these papers indicate a healthy awakening of psychological researchers to recent advances in causal inference, the field is still dominated by authors who have not heard about model-based covariate selection, testable implications, nonparametric identification, bias amplification, mediation formulas and more.

Much to do, much to discuss,
Judea

Andrew Gelman and his blog readers followed-up with the previous discussion (link here) on his methods to address issues about causal inference and transportability of causal effects based on his “hierarchical modeling” framework, and I just posted my answer.

This is the general link for the discussion:
http://andrewgelman.com/2012/07/examples-of-the-use-of-hierarchical-modeling-to-generalize-to-new-settings/

Here is my answer:
http://andrewgelman.com/2012/07/examples-of-the-use-of-hierarchical-modeling-to-generalize-to-new-settings/#comment-92499

Cheers,
Bareinboim

In the past week, I have been engaged in a discussion with Andrew Gelman and his blog readers regarding causal inference, selection bias, confounding, and generalizability. I was trying to understand how his method which he calls “hierarchical modelling” would handle these issues and what guarantees it provides. Unfortunately, I could not reach an understanding of Gelman’s method (probably because no examples were provided).

Still, I think that this discussion having touched core issues of scientific methodology would be of interest to readers of this blog, the link follows:
http://andrewgelman.com/2012/07/long-discussion-about-causal-inference-and-the-use-of-hierarchical-models-to-bridge-between-different-inferential-settings/

Previous discussions took place regarding Rubin and Pearl’s dispute, here are some interesting links:
http://andrewgelman.com/2009/07/disputes_about/
http://andrewgelman.com/2009/07/more_on_pearlru/
http://andrewgelman.com/2009/07/pearls_and_gelm/
http://andrewgelman.com/2012/01/judea-pearl-on-why-he-is-only-a-half-bayesian/

If anyone understands how “hierarchical modeling” can solve a simple toy problem (e.g., M-bias, control of confounding, mediation, generalizability), please share with us.

Cheers,
Bareinboim

Our attention was called to a new attack on graphical models and structural equation models (SEM), this time in the name of “correctness”..

The article in question is: Cloak and DAG: A response to the comments on our comment by Martin A. Lindquist and Michael E. Sobel (L&S) Forthcoming , NeuroImage http://www.sciencedirect.com/science/article/pii/S1053811911013085

The advice that L&S give to NeuroImaging researchers reads as follows:

“For if fMRI researchers continue to use their “familiar approach”, drawing diagrams and fitting SEMs without realizing the assumptions they are making, many of the causal inferences thereby generated will be incorrect, and the development and use of alternative ways of studying effective connectivity will be stifled.”

L&S’s warning of the importance of scrutinizing assumptions is admirable. Yet readers of NeuroImage will have difficulty understanding why they are judged incapable of scrutinizing causal assumptions in the one language that makes these assumptions transparent, i.e., diagrams or SEM, and why they are threatened with “incorrect inferences” for not rushing to translate meaningful assumptions into a language where they can no longer be recognized, let alone justified.

For a simple example, …

Click here for the full post.

The following paradox was brought to our attention by Pablo Lardelli from Granada (Spain).

Pablo writes:

1. Imagine that you design a cohort study to assess the causal effect of X on Y, E[Y|do(X=x)]. Prior knowledge informs you that variable M is a possible confounder of the process X—>Y, which leads you to assume X<---M--->Y.

To adjust for the effect of this confounder, you decide to design a matched cohort study, matching on M non exposed to exposed. You know that matching breaks down the association between X and M in the sample.
[……]
The problem arises when you draw the DAG […] and realize that S is a collider on the path X—>S<---M and, since we are conditioning on S (because the study sample is restricted to S=1) we are in fact opening a non causal path between X and Y (through M) in the sample. But this stands in contradiction to everything we are told by our textbooks. Click here for full discussion of matching in DAGs, persistent-unfiathfulness and unit-to-unit interactions.

Stan Mulaik called my attention to a recent article by Grice, Shlimgen and Barrett (GSB) (linked here http://psychology.okstate.edu/faculty/jgrice/personalitylab/OOMMedForm_2011A.pdf ) which is highly critical of structural equation modeling (SEM) in general, and of the philosophy and tools that I presented in “The Causal Foundation of SEM” (Pearl 2011) ( http://ftp.cs.ucla.edu/pub/stat_ser/r370.pdf.)  In particular, GSB disagree with the conclusions of the Mediation Formula — a tool for assessing what portion of a given effect is mediated through a specific pathway.

I responded with a detailed account of the disagreements between us (copied below), which can be summarized as follows:

Summary

1. The “OOM” analysis used by GSB is based strictly on frequency tables (or “multi-grams”) and, as such, cannot assess cause-effect relations without committing to some causal assumptions. Those assumptions are missing from GSB account, possibly due to their rejection of SEM.

2. I define precisely what is meant by “the extent to which the effect of X on Y is mediated by a third variable, say Z,” and demonstrate both, why such questions are important in decision making and model building and why they cannot be captured by observation-oriented methods such as OOM.

3. Using the same data and a slightly different design, I challenge GSB to answer a simple cause-effect question with their method (OOM), or with any method that dismisses SEM or causal algebra as unnecessary.

4. I further challenge GSB to present us with ONE RESEARCH QUESTION that they can answer and that is not answered swiftly, formally and transparently by the SEM methodology presented in Pearl (2011). (starting of course with the same assumptions and same data.)

5. I explain what gives me the assurance that no such research question will ever be found, and why even the late David Friedman, whom GSB lionize for his staunch critics of SEM, has converted to SEM thinking at the end of his life.

6. I alert GSB to two systematic omissions from their writings and posted arguments, without which no comparison can be made to other methodologies:
(a) A clear statement of the research question that the investigator attempts to answer, and
(b) A clear statement of the assumptions that the investigator is willing to make about reality.

Click here for the full response.

=======Judea

Dear Nancy,

This letter concerns the issue of “causal plurality” which came up in my review of your book “Hunting Causes and Using Them” (Cambridge 2007) and in your recent reply to my review, both in recent issue of Economics and Philosophy (26:69-77, 2010).

My review:
http://journals.cambridge.org/action/displayFulltext?type=1&fid=7402268&jid=&volumeId=&issueId=&aid=7402260

Cartwright Reply:
http://journals.cambridge.org/action/displayFulltext?type=1&fid=7402292&jid=&volumeId=&issueId=&aid=7402284

I have difficulties understanding causal pluralism because I am a devout mono-theist by nature, especially when it comes to causation and, although I recognize that causes come in various shades, including total, direct, and indirect causes, necessary and sufficient causes, actual and generic causes, I have seen them all defined, analyzed and understood within a single formal framework of Structural Causal Models (SCM) as described in Causality (Chapter 7).

So, here I am, a mono-theist claiming that every query related to cause-effect relations can be formulated and answered in the SCM framework, and here you are, a pluralist, claiming exactly the opposite. Quoting:

“There are a variety of different kinds of causal systems; methods for discovering causes differ across different kinds of systems as do the inferences that can be made from causal knowledge once discovered. As to causal models, these must have different forms depending on what they are to be used for and on what kinds of systems are under study.

If causal pluralism is right, Pearl’s demand to tell economists how they ought to think about causation is misplaced; and his own are not the methods to use. They work for special kinds of problems and for special kinds of systems – those whose causal laws can be represented as Pearl represents them. HC&UT argues these are not the only kinds there are, nor uncontroversially the most typical.“

I am very interested in finding out if, by committing to SCM I have not overlooked important problem areas that are not captured in SCM. But for this we need an example; i.e., an example of ONE problem that cannot be formulated and answered in SCM.

The trouble I have with the examples sited in your reply is that they are based on other examples and concepts that are scattered on many pages in your book and, thus, makes it hard to follow. Can we perhaps see one such example, hopefully with no more than 10 variables, described in the following format:

Example: An agent is facing a decision or a question.

Given: The agent assumes the following about the world: 1. 2. 3. ….
The agent has data about …., taken under the following conditions.
Needed: The agent wishes to find out whether…..

Why use this dry format, you may ask, when your book is full with dozens of imaginative examples, from physics to econometrics? Because if you succeed in showing ONE example in this concise format you will convert one heathen to pluralism, and this heathen will be grateful to you for the rest of his spiritual life.

And if he is converted, he will try and help you convert others (I promise) and, then, who knows? life on this God given earth would become so much more enlightened.

And, as Aristotle used to say (or should have) May clarity shine on causality land.

Sincerely,

Judea Pearl

Opening remarks

A few days ago, Dan Sharfstein posed a question regarding the “well-defineness” of “direct effects” in situations where the mediating variables cannot be manipulated. Dan’s question triggered a private email discussion that has culminated in a posting by Thomas Richardson and Jamie Robins (below) followed by Judea Pearl’s reply.

We urge more people to join this important discussion.

Thomas Richardson and James Robins’ discussion:

Hello,

There has recently been some discussion of mediation and direct effects.

There are at least two issues here:

(1) Which counterfactuals are well defined.

(2) Even when counterfactuals are well defined, should we include assumptions that identify effects (ie the natural direct effect) that could never be confirmed even in principle by a Randomized Controlled Trial (RCT).

As to (1) it is clear to most that all counterfactuals are vague to a certain extent and can be made more precise by carefully describing the (quite possibly only hypothetical) intervention you want the counterfactual to represent. For this reason,  whether you take manipulation or causality as ontologically primary, we need to relate causation to manipulation to clarify and make more precise which counterfactual world we are considering.

On (2) we have just finished a long paper on the issue, fleshing out considerably an argument I (Jamie) made at the American Statistical Association (in 2005) discussing a talk by Judea on natural (pure and total) direct effects.

“Alternative Graphical Causal Models and the Identification of Direct Effects”

It is available at
http://www.csss.washington.edu/Papers/wp100.pdf.

Here is a brief summary:

Click here for the full post.

Best wishes,

Jamie Robins  and Thomas Richardson

Judea Pearl’s reply:

1.
As to the which counterfactuals are “well defined”, my position is that counterfactuals attain their “definition” from the laws of physics and, therefore, they are “well defined” before one even contemplates any specific intervention. Newton concluded that tides are DUE to lunar attraction without thinking about manipulating the moon’s position; he merely envisioned how water would react to gravitaional force in general.

In fact, counterfactuals (e.g., f=ma) earn their usefulness precisely because they are not tide to specific manipulation, but can serve a vast variety of future inteventions, whose details we do not know in advance; it is the duty of the intervenor to make precise how each anticipated manipulation fits into our store of counterfactual knowledge, also known as “scientific theories”.

2.
Regarding identifiability of mediation, I have two comments to make; ‘ one related to your Minimal Causal Models (MCM) and one related to the role of structural equations models (SEM) as the logical basis of counterfactual analysis.al basis of counterfactual analysis.

Click here for Judea’s reply.

Sander Greenland from UCLA writes:

The machinery in your book addresses only issues of identification and unbiasedness. Of equal concern for practice is variance, which comes to the fore when (as usual) one has a lot of estimators with similar bias to choose from, for within that set of estimators the variance becomes the key driver of expected loss (usually taken as MSE (mean-squared-error = variance+bias^2). Thus for example you may identify a lot of (almost-) sufficient subsets in a graph; but the minimum MSE attainable with each may span an order of magnitude. On top of that, the financial costs of obtaining each subset may span orders of magnitudes. So your identification results, while important and useful, are just a start on working out which variables to spend the money to measure and adjust for. The math of the subsequent MSE and cost considerations is harder, but no less important.

Judea Pearl replies:

You are absolutely right, it is just a start, as is stated in Causality page 95. The reason I did not  emphasize the analysis of variance in this book was my assumption that, after a century of extremely fruitful statistical research, one would have little to add to this area.

My hypothesis was:

Once we identify a causal parameter, and produce an estimand of that parameter in closed mathematical form, a century of statistical research can be harnessed to the problem, and render theestimation task a routine exercise in data analysis. Why spend energy on areas well researched when so much needs to be done in areas of neglect?

However, the specific problem you raised, that of choosing among competing sufficient sets, happens to be one that Tian, Paz and Pearl (1998) did tackle and solved. See Causality page 80, reading: “The criterion also enable the analyst to search for an optimal set of covariates — a set Z that minimizes measurement cost or sampling variability (Tian et al, 1998).” [Available at http://ftp.cs.ucla.edu/pub/stat_ser/r254.pdf] By “solution”, I mean of course, an analytical solution, assuming that cost is additive and well defined for each covariate. The paper provides a polynomial time algorithm that identifies the minimal (or minimum cost) sets of nodes that d-separates two nodes in a graph. When applied to a graph purged of outgoing arrows from the treatment node, the algorithm will enumerate all minimal sufficient sets, i.e., sets of measurements that de-confound the causal relation between treatment and outcome.

Readers who deem such an algorithm useful, should have no difficulty implementing it from the description given in the paper; the introduction of variance considerations though would require some domain-specific expertise.

Michael Foster from University of North Carolina writes:

I’m an economist here in the UNC school of public health and trying to work on the intuition of MSM for my non-methodologists collaborators. My bios and epi colleagues can give me mechanical answers but are short on intuition at times. Here are two questions:

1. Consider a regressor that is a confounding variable but that is also a victim of unobserved confounding itself. Why does weighting with this troublesome covariate not cause bias that regression causes (collider bias)? In this case, I’m principally thinking about past exposures and how to handle them in an analysis of dynamic treatment. Marginal structural models (MSM) including them in calculating the weights; Robins suggests that including them as covariates in the outcome equation produces the “null paradox”.

Here’s my answer. A confounding variable has two characteristics–it is related to the exposure and to the outcome. When we weight with that variable, we break the link between the exposure and that variable. However, other than the portion due to the exposure, we do not eliminate the relationship between the covariate and the outcome. In that way (by not breaking both links), we avoid the bias created by the collider issue.

1. How do I know what variables to include in the numerator of the MSM weight?

Here’s my answer: I would include in the weights those variables that will be included in the analysis of the outcome. Their presence in the denominator of the weight is essentially duplicative–we’re accounting for them there and in the outcome model.

Judea Pearl replies (updated 11/19/2009):

Your question deals with the intuition behind “Inverse Probability Weighting” (IPW), an estimation technique used in several frameworks, among them Marginal Structural Models (MSM). However, the division by the propensity score P(X=1| Z=z) or the probability of treatment X = 1 given observed covariates Z = z, is more than a step taken by one estimation technique; it is dictated by the very definition of “causal effect,” and appears therefore, in various guises, in every method of effect estimation — it is a property of Nature, not of our efforts to unveil the secrets of Nature.

Let us first see how this probability ends up in the denominator of the effect estimand, and then deal with the specifics of your question, dynamic treatment and unobserved confounders.

Click here for the full post.

As always, we welcome your views on this topic. To continue the discussion, please use the comment link below to add your thoughts. You can also suggest a new topic of discussion using our submission form by clicking here.