Causal Analysis in Theory and Practice

June 11, 2018

A Statistician’s Reaction to The Book of Why

Filed under: Book (J Pearl) — Judea Pearl @ 12:37 am

Carlos Cinelli brough to my attention a review of The Book of Why, written by Kevin Gray, who disagrees with my claim that statistics has been  delinquent in neglecting causality. See https://www.kdnuggets.com/2018/06/gray-pearl-book-of-why.html I have received similar reactions from statisticians in the past, and I expect more in the future. These reactions reflect a linguistic dissonance which The Book of Why describes thus: “Many scientists have been quite traumatized to learn that none of the methods they learned in statistics is sufficient even to articulate, let alone answer, a simple question like ‘What happens if we double the price?'” p 31.

I have asked Carlos to post the following response on Kevin’s blog:

————————————————
Kevin’s prediction that many statisticians may find  my views “odd or exaggerated” is accurate. This is exactly what I have found in numerous conversations I have had with statisticians in the past 30 years. However, if you examine my views closely,  you will find that they are not as thoughtless or exaggerated as they may appear at first sight.

Of course many statisticians will scratch their heads and ask: “Isn’t this what we have been doing for years, though perhaps under a different name or not name at all?” And here lies the essence of my views. Doing it informally, under various names, while refraining from doing it mathematically under uniform notation has had a devastating effect on progress in causal inference, both in statistics and in the many disciplines that look to statistics for guidance. The best evidence for this lack of progress is the fact that, even today, only a small percentage of practicing statisticians can solve any of the causal toy problems presented in the Book of Why.

Take for example:

1. Selecting a sufficient set of covariates to control for confounding
2. Articulating assumptions that would enable consistent estimates of causal effects
3. Finding if those assumptions are testable
4. Estimating causes of effect (as opposed to effects of cause)
5. More and more.

Every chapter of The Book of Why brings with it a set of problems that statisticians were deeply concerned about, and have been struggling with for years,  albeit under the wrong name (eg. ANOVA or MANOVA) “or not name at all.” The results are many deep concerns but no solution.

A valid question to be asked at this point is what gives humble me the audacity to state so sweepingly that no statistician (in fact no scientist) was able to properly solve those toy problems prior to the 1980’s. How can one be so sure that some bright statistician or philosopher did not come up with the correct resolution of the Simpson’s paradox or a correct way to distinguish direct from indirect effects? The answer is simple: we can see it in the syntax of the equations that scientists used in the 20th century.  To properly define causal problems, let alone solve them, requires a vocabulary that resides outside the language of probability theory. This means that all the smart and brilliant statisticians who used joint density functions, correlation analysis, contingency tables, ANOVA, Entropy, Risk Ratios, etc., etc., and did not enrich them with either diagrams or counterfactual symbols have been laboring in vain — orthogonally to the question — you can’t answer a question if you have no words to ask it. (Book of Why, page 10)

It is this notational litmus test that gives me the confidence to stand behind each one of statements that you were kind enough to cite from the Book of Why. Moreover, if you look closely at this litmus test, you will find that it not just notational but conceptual and practical as well. For example, Fisher’s blunder of using ANOVA to estimate direct effects is still haunting the practices of present day mediation analysts. Numerous other examples are described in the Book of Why and I hope you weigh seriously the lesson that each of them conveys.

Yes, many of your friends and colleagues will be scratching their head saying: “Hmmm… Isn’t this what we have been doing for years, though perhaps under a different name or not name at all?” What I hope you will be able to do after reading “The Book of Why” is to catch some of the head-scratchers and tell them: “Hey, before you scratch further, can you solve any of the toy problems in the Book of Why?” You will be surprised by the results — I was!
————————————————

To me, solving problems is the test of understanding, not head scratching. That is why I wrote this Book.

Judea

1. Kevin Gray review of “Book of Why” was posted on KDnuggets, and we also posted Judea’s reply on KDnuggets to continue the discussion https://www.kdnuggets.com/2018/06/gray-pearl-book-of-why.html

Comment by Gregory Piatetsky — June 11, 2018 @ 12:03 pm

2. Judea

You are indeed exaggerating. There are statisticians who are unfamiliar with causal inference.
But as you know, there are plenty of statisticians who have worked on causal inference for many, many years.
There is an entire subfield of statistics devoted to causal inference. They even have conferences.
It is misleading to lump all statisticians together.

Larry Wasserman

Comment by Larry Wasserman — June 14, 2018 @ 10:08 am

3. Dear Larry,
I am really glad you have joined this discussion,
partly because you really have a global perspective
of the field, including its causal subfield, and partly
because you were the first author ever to include
a causal treatment of Simpson’s paradox in a statistic textbook. Kudos.

But why am I exaggerating?
Isn’t it true that (quoting) “even today, only a small percentage of practicing
statisticians can solve any of the
causal toy problems presented in the Book of Why?”
How many statistics textbooks do you know that have
“causal” in their index?
What percentage of graduating PhD’s in statistics
can use potential outcome notation or graphical
models to solve the Simpson paradox correctly,
as you did in your textbook. Isn’t it “a small
percentage” as I state above?

The “entire subfield of statistics devoted to causal
inference” and the conferences that you mention
are a very recent phenomenon. It is part of the
“Causal Revolution” that I introduce in the Book
of Why, whose modern awakening I attribute to Rubin,
Robins, Greenland and other “statisticians who have
worked on causal inference for many, many years.”

I do not think however that there were “plenty” of such statisticians, but the few that have contributed to the Causal Revolution are notably mentioned in the Book. Please alert me if I have missed any.

I also do not think this amounts to “lumping all statisticians together”. On the contrary, I think it amounts to making a clear distinction between statisticians who have contributed to the Causal Revolution and the great majority of mainstream statisticians who refrained from using causal notation.

I hope I have made it even clearer in the reply that I just sentto Kevin Gray, on kdnuggets.com.

Judea

Comment by Judea Pearl — June 14, 2018 @ 1:15 pm

4. Well that is an interesting empirical question: how many statisticians can answer your four questions.
I wish it were 100 percent but I agree that it isn’t.
But I wish there were a way to find out

Comment by Larry Wasserman — June 14, 2018 @ 3:36 pm

5. Dr. Pearl consider joining twitter to engage in these discussions of causal inference.

Comment by Boback — June 20, 2018 @ 5:09 pm