In my last posting on Lord’s paradox, I have argued that statisticians have given up prematurely on resolving Lord’s Paradox, and some have even proclaimed it to be beyond the province of statistical analysis.

Here are two classical examples: The first is taken from Lord’s himself (1967), the second from Holland and Rubin (1983) who considered their analysis to be a “resolution.”

Lord states:

“What is the `explanation’ of the paradox? There are as many different explanations as there are explainers.”

In the writer’s opinion, the explanation is that with the data usually available for such studies, there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups. The researcher wants to know how the groups would have compared if there had been no preexisting uncontrolled differences. The usual research study of this type is attempting to answer a question that simply cannot be answered in any rigorous way on the basis of available data.”

Comment: Unlike many of his contemporaries who rushed to proclaim the two statisticians equally “right”, each trying to estimate another quantity, Lord is very clear as to what that quantity should be: “how the groups would have compared if there had been no preexisting uncontrolled differences.” He notes correctly that this counterfactual question simply cannot be answered in any rigorous way on the basis of available data.

Had he lived today, he would have said “on the basis of any data” and he would have continued to ask:

“What extra-data information would allow us to answer this important question, and how we should answer it once the information is provided.”

Holland and Rubin (1983) “resolved” the paradox by evading it. They noted correctly that Lord’s question is causal, hence requiring causal assumptions. However, since such assumptions are not testable in the available data everything goes, that is, each of the two statisticians can be right, depending on what assumptions they entertain.

In their words: “Since [the two statisticians’ assumptions] cannot be tested with the available data, acceptance or criticism of [them] must be based on intuition and/or subject-matter experience” (p. 11).

For statisticians in 1984, “subject matter” information meant the end of the problem, rather than opening it up to interesting analysis in which the “subject matter” information is expressed formally, combined with data, and answers Lord’s question: “how the groups would have compared if there had been no preexisting uncontrolled differences.”

To the best of my knowledge, Lord’s question has yet to receive a general answer in the statistical literature.

References

Lord, F.M., “A paradox in the interpretation of group comparisons,” Psychological Bulletin, (68): 304-305, 1967.

Holland, P. and Rubin, D., “On Lord’s paradox,” in H. Wainer and S. Messick (Eds.), Principals of Modern Psychological Measurement, Hillsdale, NJ: Lawrence Earlbaum, 3-25, 1983.

]]>