Table 3 In-depth review questions
Question | No | Yes | Uncertain | |
|---|---|---|---|---|
1b | 1.1 Software as a medical device (SaMD) Has the FDA cleared or approved the AI as SaMDa? • The business owner is responsible for producing the FDA letter. | Continue to 1.2 | Continue to 1.3 | n/a |
1.2 SaMD Could the software meet the FDA definition of software as a device? a. The AI acquires, processes, or analyzes a medical image or signal related to a patient’s health. If this statement is TRUE, answer “YES”. b. The AI displays medical information about a patient, study or guideline. If this statement is TRUE, answer “NO”. c. The AI provides recommendations to a health care professional about prevention, diagnosis, or treatment of a disease AND provides the basis for recommendations, so the health care professional is not relying solely on the AI output for decision making. If this statement is TRUE, answer “NO”. | Moderate risk Continue to 2 | High risk Continue to 2 | High risk Continue to 2 | |
1.3 SaMD Is the FDA approval of the AI as SaMD for the intended use within the healthcare organization? • The vendor and/or business owner are responsible for providing the FDA confirmation letter and all supporting documentation or data to allow for this determination. | High risk | Moderate risk | n/a | |
2 | Potential for significant adverse effects Potential adverse effects are notable and could have a significant negative impact on patients, teammates, individuals, or the enterprise? • There should be adequate evidence of implementation in other similar settings to support a ‘NO’ answer. • The business owner is responsible for identifying supporting documentation | Moderate risk | High risk | High risk |
3 | Adverse effects and workflows Potential adverse effects are not minor but are adequately addressed by workflows to mitigate/control the risk? | High risk | Moderate risk | High risk |
4 | Net benefit There is substantial evidence that supports the benefits outweigh the risks that are expected from AI implementation? • Evidence should include implementation in other similar settings to support a ‘YES’ answer. • The business owner is responsible for identifying supporting documentation. | High risk | Moderate risk | High risk |
5 | AI features, equity in depth If the AI uses features that include characteristicsc historically used to discriminate, then adequate evidence is provided for how they influence the output in the context of the intended use? | High risk | Moderate risk | High risk |
6 | AI output, equity in depth Adequate evidence is provided that the AI solution performs well in all key subgroups? • E.g., a model appropriately ranks patients according to risk and does not systematically underestimate or overestimate risk. | High risk | Moderate risk | High risk |
7 | Access, equity Is the AI system equally accessible to those who may benefit? | High risk | Moderate risk | High risk |
8 | Medical billing, coding, human resource a. Is an output of the AI that is related to medical billing or medical coding made part of a patient’s permanent record or released to a third party without the intervention of a human? b. Does the AI rank or categorize applicants or teammates for an intended use that is HR related? | Moderate risk | High risk | n/a |
9 | Privacy/transparency a. Does the AI solution record an individual without their knowledge? b. Is the organization ethically obligated to provide an explicit explanation that AI is being used or need to consent that AI is being used, but that is not part of solution or workflow?d (e.g., based on potential risk(s) or if no human is in the loop) c. Does the AI solution analyze personal data that may lead to profiling or categorizing of individuals (excluding risk scoring for clinical diagnosis or clinical workflow prioritization)? | Moderate risk | High risk | n/a |
10 | Development and validity Transparent reporting of development and validation steps is available AND no concerns are identified when evaluated against contemporary published AI reporting standards. If this statement is TRUE, answer “YES”. • Answer ‘NO’, if supporting evidence is insufficient. • Answer ‘NO’, if concerns are present regarding the general validity of the model. • Answer ‘NO’ if the AI solution’s methods or outputs are “blackbox”e and the AI implementation creates the potential for anything more than minor adverse effects on patients, employees, or individuals. | High risk | Moderate risk | n/a |
11 | External performance a. Substantive evidence of external performance exists to the level that a local validation is not required? OR b. The development and validation data/environment are expected to be so similar to the local data/environment that local confirmation is NOT necessary (e.g., radiology imaging)? | High risk | Moderate risk | n/a |
12 | Human oversight Is the AI solution directly or indirectly tied to workflow(s) that automate an action, documentation, or patient communication without human review, which may adversely affect patient health outcomes? | Moderate risk | High risk | n/a |
Carry forward low-risk screening questions that are high risk | ||||
13 | Sensitive data Does the AI interface with data that may require special consideration? a. Recording individuals b. Facial recognition c. Fingerprints d. Genetic data e. Claims/payor data f. Other sensitive data | n/a | High risk | High risk |
14 | Trust Is there reasonable potential that the AI may negatively impact trust between provider (or health system) and patient(s)? | n/a | High risk | High risk |
15 | Vulnerability considerationsf Does the AI implementation intersect with any of the following healthcare settings/functions/populations: • Beginning of life (pre, peri, neo-natal) • End of life (hospice, DNR/code status, palliative care, advance directives) • Consent for treatment/research • Capacity for decision making | n/a | High risk | High risk |
16 | Other concern(s) Does the reviewer have any other significant concerns about the AI not captured within the low-risk screen? | n/a | High risk | High risk |
