Table 2 Learning elements selected for inclusion, organized by core theme and round in which consensus was achieved

E1

Identify key regulatory issues surrounding data sharing between healthcare institutions, academic institutions, and private organizations.

1

10

Leader, Health Advocate

E2

Analyze the implications of these regulatory issues on data-sharing practices in healthcare.

1

10

Leader, Health Advocate

E3

Apply appropriate response strategies to comply with regulatory requirements related to data sharing between healthcare institutions.

1

5, 10

Leader, Health Advocate, Professional

E4

Explain the importance of data privacy in the context of using artificial intelligence (AI) with healthcare data.

1

10

Communicator, Health Advocate, Scholar

E5

Define equitable AI and explain its importance in promoting fairness and avoiding bias in AI applications.

1

9, 10

Health Advocate

E6

Define and differentiate between the different types of biases that can appear in AI, including algorithmic, data, and user biases.

1

10

Scholar, Professional

E7

Identify real-world examples of each type of bias and their impact on the effectiveness of AI applications.

1

10

Scholar, Health Advocate

E8

Develop strategies to mitigate and prevent the occurrence of biases in AI applications.

1

5, 10

Scholar, Leader

E9

Apply strategies to promote the use of equitable AI and advocate for its implementation.

1

10

Health Advocate, Leader

E10

Define patient rights and the ethical considerations related to using AI in healthcare.

1

9, 10

Health Advocate, Professional

E11

Explain the importance of respecting patient rights when using AI and describe the potential benefits of doing so.

1

9, 10

Health Advocate, Professional

L1

Define data governance and explain its importance when working with AI.

1

10

Leader, Scholar

L2

Explain the importance of confidentiality in healthcare data when using AI.

1

9, 10

Professional, Health Advocate

L3

Identify potential risks to data privacy and best practices when using AI, including relevant legal and regulatory requirements.

1

10

Professional, Scholar

L4

Apply appropriate confidentiality measures to ensure the privacy and security of healthcare data when using AI.

1

4, 5, 10

Professional, Scholar

L5

List and explain the various concerns surrounding liability when using AI in healthcare.

1

9, 10

Professional, Health Advocate

L6

Apply strategies to mitigate liability risks associated with the use of AI in healthcare.

1

9, 10

Professional, Leader

L7

Explain the importance of shared decision-making with AI and the physician’s role in shared decision-making with AI.

1

5, 9, 10

Communicator, Collaborator

L8

Understand the legal implications of shared decision-making with AI.

1

10

Leader, Professional

L9

List the key issues surrounding the copyright of AI.

2

10

Leader, Scholar

L10

Identify the key components of a data governance framework and how they relate to AI.

2

10

Leader, Scholar

L11

Apply appropriate data governance measures when working with AI.

2

5, 10

Leader, Scholar, Professional

T1

Define and differentiate between statistical concepts of accuracy, F1 score, sensitivity, specificity, positive predictive value, negative predictive value, odds ratio, relative risk, positive and negative likelihood ratios.

1

3, 12

Scholar

T2

Interpret and apply these statistical concepts to real-world healthcare scenarios.

1

3, 4, 12

Scholar, Medical Expert

T3

Understand, interpret, and explain the different types of statistics (descriptive vs inferential).

1

3, 4

Scholar

T4

Understand, interpret, and explain the different types of data (numerical vs categorical).

1

3, 4

Scholar

T5

Understand, interpret, use, and explain common terminology used in AI.

1

3, 4, 10

Scholar

T6

Identify the different domains of healthcare where AI has been successfully applied.

1

3, 4, 10

Scholar, Medical Expert

T7

Evaluate the strengths and benefits of using AI in each domain, including improved accuracy, efficiency, and cost-effectiveness.

1

4, 10

Scholar, Health Advocate

T8

Explain how AI has impacted the quality of patient care and the healthcare industry as a whole.

1

10

Scholar, Health Advocate

T9

Identify the limitations and challenges of using AI in different domains of healthcare.

1

3, 10

Scholar, Health Advocate

T10

Predict and anticipate how the workflow of physicians may change with the implementation of AI.

1

4, 10

Scholar, Leader

T11

Identify techniques that will better facilitate the implementation of AI.

1

10

Scholar, Leader

T12

Understand the basic concepts and principles of machine learning.

1

4

Scholar

T13

Identify and differentiate between different types of machine learning, including supervised, unsupervised, and reinforcement learning.

1

4

Scholar

T14

Evaluate the strengths and limitations of each type of machine learning and their applications in healthcare.

1

4, 10

Scholar

T15

Identify and differentiate between different types of regression analyses, including linear, logistic, and Poisson regression.

1

4

Scholar

T16

Understand the concept of model selection in machine learning.

1

4

Scholar

T17

Understand the basic concepts and principles of deep learning.

1

4

Scholar

T18

Understand the different applications of deep learning in healthcare, including image analysis, natural language processing, and time series analysis.

1

4

Scholar, Medical Expert

T19

Understand the basic concepts and principles of natural language processing (NLP).

1

4

Scholar

T20

Identify and differentiate between different applications of NLP in healthcare, including clinical documentation, patient communication, and disease surveillance.

1

4, 6, 7, 10, 12

Scholar, Medical Expert

T21

Evaluate the impact of NLP on the quality and efficiency of healthcare processes.

1

4, 6, 7, 10

Scholar, Medical Expert

T22

Evaluate the strengths and limitations of each type of deep learning and their applications in healthcare.

2

4, 10

Scholar

T23

Identify and differentiate between different types of models, including decision trees, random forests, and support vector machines.

2

4, 10

Scholar

T24

Evaluate the strengths and limitations of each type of model and their applications in healthcare.

2

4, 10

Scholar

T25

Develop skills in data preprocessing, feature engineering, model selection, and evaluation.

2

10

Scholar

T26

Apply these skills to solve real-world problems in healthcare using AI tools.

2

4, 10

Scholar, Medical Expert

T27

Evaluate the economic impact of AI adoption in healthcare, including the costs associated with implementation and maintenance.

3

10, 12

Scholar, Health Advocate

T28

Analyze the potential cost savings and revenue generation opportunities associated with using AI in healthcare.

3

10, 12

Scholar, Health Advocate

T29

Define and differentiate between big data and traditional data sets.

3

–

Scholar

A1

Analyze and interpret data, including AI model input and output, to inform decision-making.

1

4, 5, 8

Medical Expert, Scholar

A2

Integrate evidence from AI models into clinical decision-making practices in healthcare.

1

3, 4, 5

Medical Expert, Scholar

A3

Critically evaluate the integrity, reliability, and applicability of research on AI applications in healthcare.

1

10

Scholar, Professional

A4

Create research questions that are well-designed and specific to AI research.

2

10

Scholar

A5

Collect and manage data effectively for AI research.

2

10

Scholar

A6

Apply principles of data stewardship to ensure the quality and security of AI data.

2

10

Scholar, Professional

A7

Validate AI models using appropriate statistical methods to ensure their accuracy and reliability for research purposes.

2

10

Scholar, Professional

A8

Use different functions and tools to visualize data in order to gain insights from it.

2

10

Scholar

A9

Preprocess data appropriately for AI research by cleaning, transforming, and selecting relevant features.

3

10

Scholar

A10

Evaluate and select appropriate algorithms for specific AI problems, based on their strengths and limitations.

3

3, 4, 5, 8, 10

Scholar, Medical Expert

A11

Execute and interpret error analysis in machine learning and deep learning models.

3

3, 4, 5, 8, 10

Scholar, Medical Expert

C1

Develop strategies for establishing and maintaining positive relationships with colleagues involved in the AI side of healthcare, such as data scientists.

1

7, 9

Collaborator, Leader

C2

Distinguish between the roles of a physician, other healthcare providers, and data scientists to promote clear communication.

1

10

Collaborator, Communicator

C3

Engage in shared decision-making with colleagues focused on the AI aspect of healthcare to promote patient-centered care.

1

7, 8, 9

Collaborator, Communicator

C4

Reflect on one’s own roles and limitations in the context of AI in healthcare, including ethical considerations and potential biases, to promote responsible use of AI tools.

1

10

Professional, Collaborator

C5

Identify opportunities for learning and self-improvement with respect to one’s AI abilities, including training programs and online resources, to ensure that one’s skills and knowledge remain up-to-date.

1

10

Scholar, Professional

C6

Identify, select, and navigate credible sources to learn about AI in healthcare, including peer-reviewed publications, expert opinion, and government reports, to ensure that one is using accurate and reliable information.

1

10

Scholar, Professional

C7

Explain the importance of patient inclusion when designing AI for healthcare to ensure that AI tools are designed and implemented in a way that reflects the needs and values of the patient population.

1

10

Health Advocate, Communicator

Cm1

Predict and anticipate how patient interactions may change with the implementation of AI.

1

7

Communicator, Health Advocate

Cm2

Develop effective communication strategies to disseminate AI-related knowledge and research to colleagues in the healthcare industry.

1

7

Communicator, Scholar

Cm3

Develop patient-friendly materials to disseminate AI-related knowledge and research to patients.

1

5, 10, 12

Communicator, Health Advocate

Cm4

Demonstrate empathetic communication skills when discussing the use of AI in patient care, including patient-centered approaches that encourage patient trust and autonomy.

1

4, 5, 6, 7, 9, 10, 12

Communicator, Health Advocate

Cm5

Manage disagreements and emotionally charged conversations related to AI effectively, including techniques for de-escalation and conflict resolution.

1

7

Communicator, Professional

Cm6

Collect and synthesize relevant information from patients and other sources for use in AI analysis.

1

7

Communicator, Scholar

Cm7

Appropriately interpret and document results from AI analyses for use in patient care and other healthcare decision-making processes.

1

6, 7, 10

Communicator, Medical Expert

Q1

Evaluate patient feedback to identify areas of improvement for AI in healthcare.

1

10

Health Advocate, Scholar

Q2

Propose solutions to improve the capability of AI in healthcare based on patient feedback and experience.

1

10

Health Advocate, Leader

Q3

Analyze current applications of AI in healthcare to identify areas for improvement.

1

10

Health Advocate, Professional

Q4

Evaluate community health needs and propose solutions using AI to address these needs.

1

10, 12

Health Advocate, Leader

Q5

Integrate patient feedback into the development and implementation of AI in healthcare.

1

10, 12

Health Advocate, Communicator

Q6

Apply principles of user-centered design to improve the user experience of AI in healthcare.

1

10, 12

Health Advocate, Scholar