Introduction

The influx of artificial intelligence (AI) systems into certain areas of healthcare delivery operations and administration has prompted substantial debate regarding i) whether such algorithms can be useful in clinical decision-making as well, and if so, ii) how these algorithms should interface with physicians. While the window of discourse has firmly centered on the idea that algorithms should augment physicians’ clinical decision-making, the primary assumption that physicians will interact directly with these AI systems remains unchallenged. However, as the number of complex models that exist at a physician’s disposal for any clinical scenario grows - some that perform one specific clinical prediction task, others that are more flexible like emerging generative AI tools—we believe it is impractical for all physicians to be expected to reliably select, use, interpret, and clinically translate the output of these highly specialized tools. The field of radiology evolved to help clinically translate the output of complex imaging machines. The role of a clinical pharmacist in overseeing safe and effective drug use is critical for every hospital. Likewise, having a specialist guide the physician on appropriate use and clinical translation of AI systems will be critical in realizing their full potential in clinical decision-making and patient care.

Direct physician-AI interaction yields underwhelming results

Ample evidence suggests that directly giving physicians access to algorithmic tools and AI decision support systems does not reliably improve their decision-making. Brennan et al.1 describe a study in which an algorithm consistently outperforms surgeons in predicting patients’ post-surgical outcomes. Yet when surgeons were given the algorithm’s output and asked to update their prediction, their predictive performance did not improve. This phenomenon holds true for generative AI tools as well; a recent study by Chen and Rodman et al.2 found that, while the independent diagnostic performance of a large language model (LLM) far outperformed human physician diagnostic capabilities, allowing the human to use the LLM as an aid did not significantly improve their performance.

The fundamental problem that arises with deployment of direct physician-facing AI tools is that it forces individuals with almost no training or familiarity with these tools to somehow incorporate them into their decision-making with little guidance. Imagine if general practitioners were expected to fully understand and clinically translate the unprocessed output of MRI machines without the aid of radiologists? While most physicians see great potential in the use of AI to improve care, they do not feel adequately informed or equipped to start using it directly3.

Current efforts are necessary but insufficient

To prepare future physicians for integration of algorithms in clinical care, some have proposed the addition of probabilistic reasoning to medical school education4,5. Given the growing complexity of AI systems, this is necessary but not sufficient to ensure algorithms are used safely and effectively. Algorithmic tools are just as complex as those of a radiologist or surgeon, and can similarly be misused or misinterpreted6—e.g., applied to the wrong patient population, to draw the wrong clinical conclusion, or to justify the wrong intervention. It is implausible for every physician to understand the indications and limitations of each algorithm. As an example, infectious disease specialists are needed to provide guidance on antibiotic selection and use, despite medical school education that includes teaching on antibiotics. Similarly, the decision of how to interpret and integrate imaging results into a patient’s care is facilitated by a radiologist, not by the patient’s physician alone.

Model explainability efforts (such as “Model Facts” labels7 or heat maps overlaid on imaging studies) have also been proposed as a way to help physicians use algorithms directly in clinical practice by sharing the tool’s indications, applicable patient population, performance metrics, and justification for its prediction, thereby ostensibly giving the physician enough information to decide if the model is appropriate or not and how to interpret its output. While this is an intuitively appealing concept, current explainability resources do not consistently appear to improve decision-making. Jabbour et al.8 found that providing heatmaps on chest x-rays that help to explain how the algorithm arrived at its prediction did not mitigate the negative effects of incorrect algorithmic predictions on the radiologists reading those chest x-rays.

Possible explanations for the limited efficacy of explainability efforts are numerous. First, physicians are not trained to fully understand, be convinced by, or know how to act on these efforts. These efforts can often be interpreted in many different ways instead of providing clarity9. They also place an unrealistic burden on the developer creating these explainability resources: it is impossible to anticipate every scenario where a physician might consult a model. Finally, these resources are static and therefore cannot be adapted to the nuances of the clinical scenario. A specialist with both algorithmic fluency and clinical understanding is necessary to fill this gap.

Algorithmic consultants as stewards of safe and effective use of AI in clinical practice

While these current efforts are important to facilitate physician-AI collaboration, their limited efficacy leaves large gaps for improvement. Furthermore, precedents from other fields like radiology and clinical pharmacy show us that specialists are required to unlock the clinical value of new scientific advances and technologies and ensure they do not cause harm.

We envision an algorithmic consultant as someone who plays a role similar to clinical pharmacists in hospitals today. A pharmacist’s clinical responsibilities in a hospital can be categorized into two buckets: providing guidance to providers at the point-of-care with individual drug-related decisions, and governing a hospital’s ecosystem of drugs. Similarly, an algorithmic consultant’s two roles would be traditional point-of-care consultations with physicians seeking input from an AI system on a specific clinical scenario, and in overseeing and managing the hospital’s ecosystem of algorithms.

Point-of-care guidance

At the point-of-care, clinical pharmacists provide guidance on appropriate medication selection and drug dosing. They combine their unique knowledge of pharmacokinetics and pharmacodynamics, comprehension of relevant literature, and understanding of the clinical scenario to answer these questions. Likewise, at the point of care, an algorithmic consultant would provide guidance on model selection and model output interpretation, and would combine their knowledge of data science with an understanding of the clinical scenario to provide guidance. Importantly, this specialist would not need to have some special understanding of the numerous weights and parameters of inscrutably large models in order to do their job effectively. Instead, they would focus on elements such as a model’s training cohort and how well it aligns with the patient, the strengths and weaknesses of the model reported in technical articles and uncovered during local validation, and the types of clinical questions the model was designed to shed light on. Figure 1A offers an example of their role at the point of care.

Fig. 1: The dual roles of algorithmic consultants in a provider-facing role and a system-level role, and how it mirrors the dual roles of a clinical pharmacist today.
Fig. 1: The dual roles of algorithmic consultants in a provider-facing role and a system-level role, and how it mirrors the dual roles of a clinical pharmacist today.The alternative text for this image may have been generated using AI.
Full size image

Figure 1A: The point-of-care workflow of an algorithmic consultant, which is modeled after that of an inpatient clinical pharmacist. Figure 1B: A clinical pharmacist’s organizational governance responsibilities (e.g., managing an institution’s formulary), and the parallel role of an algorithmic consultant in governing an institution’s AI models through their lifecycle. (Author’s own work.).

As mentioned earlier, direct physician-AI interaction sometimes does not improve decision-making. Point-of-care guidance by an algorithmic consultant addresses many of the known contributors to this problem by selecting the best tool, helping physicians accurately update their clinical reasoning5,10, adjusting for the model’s limitations and biases11, and possibly even recognizing the physician’s cognitive biases12.

Institutional algorithmic governance

Algorithmic consultants would also govern the models available at an institution. A hospital’s ecosystem of models will likely soon be as complex and numerous as its ecosystem of drugs. Similar to how pharmacists oversee and maintain a hospital’s formulary, algorithmic consultants would maintain the organization’s ecosystem of models. This includes roles in model deployment and governance, such as selection and vetting of third-party models, implementing guardrails around the types of patients, scenarios, and physicians who can use new models, ensuring algorithmic fairness, and re-training or phasing out old models. At an organizational level, they would offer insight into which models from academia or industry would be most useful for consideration, validate the performance of candidate models, and fine-tune models to a hospital’s local patient population. Figure 1B offers an example of their role in governance. Many institutions have proposed AI governance frameworks in recent years13,14,15,16; algorithmic consultants would be knowledgeable about the strengths and limitations of each, and it would be up to individual institutions and their informatics leadership to decide on specific processes. As a secondary priority, they would also enable greater information exchange between physicians and the bioinformatics community, identifying clinical gaps where tools are needed and communicating this to developers.

As an example, recently, an AI speech-to-text transcription tool used at many institutions for clinical documentation came under great scrutiny for the degree of hallucinations it produced in its clinical notes17. The existence of an algorithmic consulting service governing body at the hospital level could prevent such events by tightly regulating the deployment of appropriate tools ready for clinical use only. In the case of this third-party tool, an algorithmic consultant’s role would be to evaluate the tool’s performance on local data, identify its weaknesses, and deem it not ready for clinical use.

This proposed role would be a specialization within clinical informatics focused exclusively on the unique challenges of clinical AI (Table 1). In addition to their unique point of care responsibilities, their other duties—such as managing the AI lifecycle and governing the AI ecosystem—diverge from the tasks of clinical informaticians who tend to focus on electronic health records and other digital health tools. An algorithmic consultant’s job would be to operationalize the strategy and vision of a chief health AI officer (CHAIO)18 or the AI-specific priorities of a chief medical informatics officer (CMIO).

Table 1 The five core topics of a clinical informatics training program, as outlined by the American Board of Preventive Medicine21 (column 1), and the current content focus of each topic (column 2)
Full size table

Challenges and considerations

As with the introduction of an AI tool, the introduction of an algorithmic consultation service would entail cost. Some might argue that the use of an algorithmic consultant—compared to direct physician use—defeats the purported efficiencies of using AI in clinical practice. However, we hypothesize that an algorithmic consultation service would substantially improve the efficacy of the AI systems they manage. The act of facilitating and interpreting an AI system’s output is a genuine addition of value, and is currently a task that physicians are unfairly asked to take on, without corresponding training or compensation. While only a small fraction of the activities a clinical pharmacist engages in have direct financial implications (e.g., selecting more cost-effective drugs for a hospital’s formulary), much of their value arises from their proven abilities to reduce adverse events, morbidity, and mortality thereby not only incurring a net cost saving but also fundamentally enabling the daily safe and effective clinical operations of a hospital19. The return on investment of an algorithmic consultation service follows the same logic: some of their activities will have direct financial implications (such as promoting the use of tools that make clinical processes more efficient), but most of their benefits would arise from their crucial role in enabling safe use of new technology for patient care.

Another consideration is liability. While liability for algorithmic output is a complex issue for which no universally accepted solution yet exists, we think this area is a unique strength of an algorithmic consultation service. New AI systems are entering clinical practice rapidly and will continue to do so regardless of when solutions to the issue of liability emerge; in the interim, it is important to introduce measures that minimize the risk of using these tools in everyday practice. While the presence of an algorithmic consult service does not solve the issue of liability, it would certainly lower overall risk for physicians and health systems using AI: physicians would no longer feel like they are taking on unnecessary risk by using complex tools since they are being guided by an expert, and health systems would de-risk their AI offerings by having specialists that carefully vet and curate an ecosystem of safe models using the governance principles we described earlier.

The other limitation surrounds the adoption of models and utilization of consultants in the first place. Given the limited bandwidth of practitioners to pick up and internalize new models, would they know to consult an algorithmic specialist when confronted with an uncertain clinical scenario or a complex model? Following the successful model of the clinical pharmacist, we propose that algorithmic specialists engage in background surveillance as AI systems are used by providers and intervene when necessary. When specific provider activities are detected in the electronic health record, a pharmacist is notified to review the case and given the option for involvement. For example, at some institutions, if certain medications are ordered at doses that do not make sense for a specific patient, this activity is flagged, and a pharmacist may contact the provider to offer advice. Similarly, if a provider independently begins using a disease-specific algorithm to help manage an inpatient who does not have that condition, an algorithmic consultant might be automatically alerted to consider stepping in and offer alternatives if appropriate.

Finally, our argument is based on the assumption that direct physician-AI collaboration will be an ongoing challenge. Indeed, fundamental incompatibilities between human intuition-based decision-making and statistical inference by algorithms are unlikely to change soon. However, there is a possibility in the future that the field of clinical AI will advance to a point where complex algorithmic output is designed in some way to integrate more easily into human judgment without the aid of specialists.

Future directions

As we begin to enter an era of algorithmic medicine, it will be important for all physicians to have some basic AI literacy4; an algorithmic consultation service does not obviate this need. Medical school curricula should incorporate basic principles of probabilistic reasoning and data science so future physicians are prepared to interact with these tools and may collaborate productively with algorithmic consultants.

The identity of an algorithmic consultant is one who can enable bidirectional communication between AI and physicians and therefore has both clinical and technical training. While a large supply of people with this skill set is not currently available, specialty training in clinical informatics is becoming gradually more popular. Clinical informatics training programs recruit and produce individuals with the closest skill set and interests to those that an algorithmic consultant would need. Therefore, the most suitable avenue for training algorithmic consultants would be within existing clinical informatics fellowship programs. In particular, we recommend that a specialized training pathway be offered for algorithmic consultants within clinical informatics fellowships for those who are interested. The current clinical informatics fellowship curriculum20 would be adapted to equip algorithmic consultant trainees with skills to provide point-of-care guidance on AI use, institution-level AI governance, and more. This proposed specialized curriculum is outlined in Table 1.

Conclusion

The potential of AI systems is greatly diminished if they are unable to convince physicians to alter their behaviors; evidence shows these tools are not capable of accomplishing this unaided. To successfully affect provider behaviors, an active interpretation gap must be filled by someone who knows the nuances of the technology and also nuances of the clinical scenario. We propose that an algorithmic consultation service fills this gap. While it would be a big undertaking, the act of consultation could have substantial impact on (i) the ability of an algorithm to influence physician behavior, (ii) physician satisfaction with AI systems overall, and (iii) downstream patient outcomes. The broader promise of artificial intelligence in improving clinical care has yet to be realized - algorithmic consultants may be instrumental in bridging the gap between model and practitioner.