Table 9 Results of the summarization models on the objective_results division, test set 1.

Model	Evaluation score on the objective_results division
Model	ROUGE-1	ROUGE-2	ROUGE-L	BERTScore	BLEURT	medcon	Average
Retrieval-based
train_UMLS	30.26	14.89	29.87	66.24	37.25	8.91	34.35
train_sent	40.52	18.21	38.87	73.33	45.79	12.45	41.03
BART-based
BART	0.00	0.00	0.00	0.00	5.45	0.00	1.36
BART (Division)	30.48	19.16	27.80	66.64	43.07	21.56	39.27
BART + FT_SAMSum	20.79	0.46	20.67	54.54	28.32	0.77	24.40
BART + FT_SAMSum (Division)	29.45	18.01	26.63	66.43	40.75	20.17	38.01
BioBART	17.50	0.00	17.50	52.44	25.33	0.00	22.36
BioBART (Division)	35.38	14.33	32.79	68.40	47.63	15.69	39.81
LED-based
LED	0.00	0.00	0.00	0.00	5.45	0.00	1.36
LED (Division)	14.04	4.97	11.08	48.86	9.61	7.86	19.09
LED + FT_pubMed	0.00	0.00	0.00	0.00	5.45	0.00	1.36
LED + FT_pubMed (Division)	10.48	3.64	8.32	42.43	7.13	8.86	16.48
OpenAI (wo FT)
Text-Davinci-002	41.48	20.12	39.95	70.61	50.79	24.42	44.92
Text-Davinci-003	44.92	25.21	43.84	72.35	55.87	29.37	48.90
ChatGPT	34.50	17.75	30.84	66.68	48.51	22.28	41.29
GPT-4	37.65	19.94	35.73	68.33	48.50	26.73	43.67

Similar to objective_exam, BART and LED full note generation models suffered a significant drop at the objective_results division. This may be attributable to the higher sparsity of this division, low amounts of content (sometimes only 2-3 sentences), and the appearance of text later in the sequence. The OpenAI were in general better performant with BART division-based models as next best.

Quick links

Search