Diagnosing rare diseases, such as choroid neoplasias, remains a critical challenge. Here, the authors develop a multimodal concept-based interpretable model (MMCBM) to distinguish uveal melanoma from hemangioma and metastatic carcinoma, obtaining performance comparable to senior ophthalmologists in a large cohort of Asian patients with choroid neoplasms.