A Statistician’s Re-Reaction to The Book of Why
Responding to my June 11 comment, Kevin Gray posted a reply on kdnuggets.com in which he doubted the possibility that the Causal Revolution has solved problems that generations of statisticians and philosophers have labored over and could not solve. Below is my reply to Kevin’s Re-Reaction, which I have also submitted to kdhuggets.com:
Dear Kevin,
I am not suggesting that you are only superficially acquainted with my works. You actually show much greater acquaintance than most statisticians in my department, and I am extremely appreciative that you are taking the time to comment on The Book of Why. You are showing me what other readers with your perspective would think about the Book, and what they would find unsubstantiated or difficult to swallow. So let us go straight to these two points (i.e., unsubstantiated and difficult to swallow) and give them an in-depth examination.
You say that I have provided no evidence for my claim: “Even today, only a small percentage of practicing statisticians can solve any of the causal toy problems presented in the Book of Why.” I believe that I did provide such evidence, in each of the Book’s chapters, and that the claim is valid once we agree on what is meant by “solve.”
Let us take the first example that you bring, Simpson’s paradox, which is treated in Chapter 6 of the Book, and which is familiar to every red-blooded statistician. I characterized the paradox in these words: “It has been bothering statisticians for more than sixty years – and it remains vexing to this very day” (p. 201). This was, as you rightly noticed, a polite way of saying: “Even today, the vast majority of statisticians cannot solve Simpson’s paradox,” a fact which I strongly believe to be true.
You find this statement hard to swallow, because: “generations of researchers and statisticians have been trained to look out for it [Simpson’s Paradox]” an observation that seems to contradict my claim. But I beg you to note that “trained to look out for it” does not make the researchers capable of “solving it,” namely capable of deciding what to do when the paradox shows up in the data.
This distinction appears vividly in the debate that took place in 2014 on the pages of The American Statistician, which you and I cite. However, whereas you see the disagreements in that debate as evidence that statisticians have several ways of resolving Simpson’s paradox, I see it as evidence that they did not even come close. In other words, none of the other participants presented a method for deciding whether the aggregated data or the segregated data give the correct answer to the question: “Is the treatment helpful or harmful?”
Please pay special attention to the article by Keli Liu and Xiao-Li Meng, both are from Harvard’s department of statistics (Xiao-Li is a senior professor and a Dean), so they cannot be accused of misrepresenting the state of statistical knowledge in 2014. Please read their paper carefully and judge for yourself whether it would help you decide whether treatment is helpful or not, in any of the examples presented in the debate.
It would not!! And how do I know? I am listening to their conclusions:
- They disavow any connection to causality (p.18), and
- They end up with the wrong conclusion. Quoting: “less conditioning is most likely to lead to serious bias when Simpson’s Paradox appears.” (p.17) Simpson himself brings an example where conditioning leads to more bias, not less.
I dont blame Liu and Meng for erring on this point, it is not entirely their fault (Rosenbaum and Rubin made the same error). The correct solution to Simpson’s dilemma rests on the back-door criterion, which is almost impossible to articulate without the aid of DAGs. And DAGs, as you are probably aware, are forbidden from entering a 5 mile no-fly zone around Harvard [North side, where the statistics department is located].
So, here we are. Most statisticians believe that everyone knows how to “watch for” Simpson’s paradox, and those who seek an answer to: “Should we treat or not?” realize that “watching” is far from “solving.” Moreover, the also realize that there is no solution without stepping outside the comfort zone of statistical analysis and entering the forbidden city of causation and graphical models.
One thing I do agree with you — your warning about the implausibility of the Causal Revolution. Quoting: “to this day, philosophers disagree about what causation is, thus to suggest he has found the answer to it is not plausible”. It is truly not plausible that someone, especially a semi-outsider, has found a Silver Bullet. It is hard to swallow. That is why I am so excited about the Causal Revolution and that is why I wrote the book. The Book does not offer a Silver Bullet to every causal problem in existence, but it offers a solution to a class of problems that centuries of statisticians and Philosophers tried and could not crack. It is implausible, I agree, but it happened. It happened not because I am smarter but because I took Sewall Wright’s idea seriously and milked it to its logical conclusions as much as I could.
It took quite a risk on my part to sound pretentious and call this development a Causal Revolution. I thought it was necessary. Now I am asking you to take a few minutes and judge for yourself whether the evidence does not justify such a risky characterization.
It would be nice if we could alert practicing statisticians, deeply invested in the language of statistics to the possibility that paradigm shifts can occur even in the 21st century, and that centuries of unproductive debates do not make such shifts impossible.
You were right to express doubt and disbelief in the need for a paradigm shift, as would any responsible scientist in your place. The next step is to let the community explore:
- How many statisticians can actually answer Simpson’s question, and
- How to make that number reach 90%.
I believe The Book of Why has already doubled that number, which is some progress. It is in fact something that I was not able to do in the past thirty years through laborious discussions with the leading statisticians of our time.
It is some progress, let’s continue,
Judea
Anyone in doubt of whether there is a Causal Revolution should search Amazon.com’s Book Department using the word ‘causal’. Further, one should note the number of major programs now establishing research groups and graduate programs dedicated to causal analysis and inference. Some may choose for their own reasons to ignore this field, but they should not doubt its importance for Scientists.
Comment by Jim Grace — June 16, 2018 @ 4:47 pm
Jim,
The numerical evidence you bring up is assuring, but I am more gratified by the educational awakening that I see through readers who write: “Gee, how come no one told me about causal graphs in grad school?” or, “I finally understand what my professor tried to say.”
The current level of causal misunderstanding among scientists is astounding. You can easily meet, for example, senior level PhD’s who think that causal inference is circular, because we start with causal assumptions. Others are surprised to hear that there is a barrier between the interventional and counterfactual levels of the Ladder.
If the Editors of Science Magazine only knew what their readers do not know they would cover the Book of Why in five consecutive issues. But science moves slow and, by the time editors wake up, their readers are already way ahead.
Judea
Comment by Judea Pearl — June 20, 2018 @ 5:21 am
Professor Pearl,
Could one of the issues be that civilization looks at the function of time narratively, as in the point of the present/reference moving past to future, whereas the physical explanation is change turning future to past? Potential to actual to residual. Tomorrow becomes yesterday, because the earth turns.
Then reality is not so much the evolution of information, but the conserved and thus always and only present energy dynamically driving this process? One could, simplistically, describe reality as this dichotomy of energy and form. Energy manifesting form, as form defines energy. Consider as evidence that over a few billion years of evolution, animate organisms developed a central nervous system to process information, as well as the digestive, respiratory and circulatory systems to process the energy driving it. Consequently the academic focus on information tends to ignore the medium of energy driving it.
As energy is dynamic, it is constantly changing form. Thus energy goes from prior to succeeding forms, past to future, while forms, coalescing and dissolving, go future to past. Consider a factory; The product goes start to finish, future to past. While the production line goes the other way, consuming material and expelling product, prior to succeeding. Life is similar, in that individuals go birth to death, while the species goes the other direction, shedding old generations and growing new. Which also goes to the knowledge centric and problem centric views of computer processing, as one focuses on the nodes of knowledge, while the other on the processes creating and dissolving them.
Physics likes to reduce time to measures of duration, in order to create information, but duration is the present, as forms come and go. There is no underlaying dimension. Different clocks run at different rates because they are separate actions. A faster clock simply uses energy quicker. Like metabolism. Time is asymmetric because action is inertial. The earth turns one direction, not both.
As a measure and effect of action, frequency and change, time has more in common with temperature, than space. Our minds function sequentially, as we are mobile organisms and need to navigate, from which narration emerges. Our bodies and emotions are more temperature and feedback oriented.
There is a feedback loop between energy pushing out, as form coalesces in. Consider that galaxies are radiation expanding out, as mass coalesces in. As such we live in a thermodynamic reality and understanding the forces at work in any particular situation can be understood in terms of where in the cycle and structures it is located.
Consider politics, for example; Every society is going to have liberal and conservative instincts, as one is the organic and emotional energies pushing out, like youth, while the civil and cultural forms coalesce in, as with age and structure, creating that tension animating society. We are driven by our emotions and steered by our rationality. Or rationalizations.
Cheers,
JBM
Comment by John Merryman — September 9, 2018 @ 11:17 pm
I am both impressed and grateful for the gracious and analytical discussion; we could use a lot more of it.
Comment by Ron Burke — October 3, 2018 @ 11:57 am