Artificial intelligence (AI) models are already in widespread use by physicians, patients, and other parties in the healthcare system. Could these models be quietly steering us towards expensive or unnecessary treatments? Might they be favoring some patients over others when used in triage? These questions may sound farfetched, even conspiratorial, but we need only consider our own interactions with the healthcare system to understand how AI may be misaligned with our values.
One such interaction took place for one of us (I.S.K.) when attending a cocktail party in Cambridge, Massachusetts, while in clinical training. A dentist was explaining to his friend that he was considering improvements to his boat before the summer. He joked that it would depend on whether he could arrange enough dental implants during the spring. By chance, he was the dentist of I.S.K., who could not help wondering what led the dentist to recommend a procedure the following month. That memory remains vivid because it crystallized a truth: no matter how competent one’s clinician, if their values in diagnosing and treating their patient are not aligned with the patient’s preferences, then trust and efficacy are subverted.
