Complements, Substitutes and What's New in AI
Back in 2009, Charles P. Friedman published a paper, the title of which was "A "Fundamental Theorem" of Biomedical Informatics. Here's the abstract of that paper:
This paper proposes, in words and pictures, a “fundamental theorem” to help clarify what informatics is and what it is not. In words, the theorem stipulates that a person working in partnership with an information resource is “better” than that same person unassisted. The theorem is applicable to health care, research, education, and administrative activities. Three corollaries to the theorem illustrate that informatics is more about people than technology; that in order for the theorem to hold, resources must be informative in addition to being correct; and that the theorem can fail to hold for reasons explained by understanding the interaction between the person and the resource
This is not, by any stretch of the imagination, a brand new idea. A high school student can tell you that this is just complements and substitutes by another name. Steve Jobs comes to mind, because what is this, if not the "a bicycle for the mind" quote in disguise?
Sure, with a narrow application, granted. But surely the same idea. Well, not quite, and to be perfectly clear, I am not trying to diss either the author or the paper here, from from it.
But yes, the core idea at the heart of the whole thing is very much complements and substitutes, absolutely.
Except for the fact that we live in a very rapidly changing world:
In a clinical vignette-based study, the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources, although it may improve components of clinical reasoning such as efficiency. GPT-4 alone demonstrated higher performance than both physician groups, suggesting opportunities for further improvement in physician-AI collaboration in clinical practice
In the study whose conclusion I have excerpted above, fifty doctors (resident and attending, all with training in family medicine, internal medicine or emergency medicine) were assigned patients over remote video conferencing facilities. Some participants were given access to GPT-4 (the treatment group), while others were not. All participants had access to conventional diagnostic references.
The idea was to "assess diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps".
And just in case it wasn't clear yet, the study found that there was next to no difference in the diagnostic reasoning score for the two groups. That is, doctors were at about the same level, regardless of whether or not they had access to GPT-4.
But there was a clear winner: GPT4, all by itself, without any help form the doctors, scored 15.5 percentage points higher than the conventional resources group. For those who are wondering: 95% CI 1.5 to 29, p=0.03.
Once again, for your reading pleasure, the fundamental theorem of biomedical informatics says that a person working in partnership with an information resource is "better" than that same person unassisted.
At least in one specific, limited case, it would seem that we now have evidence that the bicycle sans the mind is, well, "better".
What, I wonder, would Paul Samuelson have said today?