Table 1 Scientific tests for bio-macromolecular modeling, continuously running on our testing server framework.

From: Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Test suite	Tests	Refs.	Test author	Quality measures	Targets	nstruct	Runtime in CPUh
Antibodies	antibody_grafting	⁵⁵	Jeliazko Jeliazkov	Fraction residues within rmsd to native	48	1	3
	antibody_h3_modeling	⁵⁶		Score vs. rmsd	6	500	3000
	antibody_snugdock	⁵⁷		I_sc vs. I_rmsd	6	500	3000
Carbohydrates	glycan_dock, (dock_glycans)^*	^58,59	Jason Labonte, Morgan Nance	I_sc vs. L_rmsd	6	1000	1100
Carbohydrates	glycan_structure_prediction	⁶⁰	Jared Adolf-Bryfogle	Score vs. rmsd	4	500	950
Comparative modeling	RosettaCM	⁶¹	Jason Fell	GDT-MM	16	200	1800
Design	ddg_alanine_scan	⁶²	Ajasja Ljubetič	R, MAE, fraction correctly classified	19: 381	1	3
Design	SEWING	⁶³	Frank Teets	MotifScorer, InterModelMotifScorer	1	100	75
Design	enzyme_design	⁶⁴	Rocco Moretti	Various sequence recoveries	50	1	50
Design	design_fast	⁶⁵	Jack Maguire, Chris Bahl	Score vs. seqrec	48	100	2600
Design, interfaces	cofactor_binding_sites	⁶⁶	Amanda Loshbaugh	rank top, position profile similarity	7	200	170
design, immune system	mhc_epitope_energy	⁶⁷	Brahm Yachnin	Degree of de-immunization, among others	50	100	2000
docking	protein_protein_docking	⁶⁸	Shourya SR Burman	I_sc vs. I_rmsd	10	5000	833
docking	ensemble docking	⁶⁹	Ameya Hamalkar	I_sc vs I_rmsd	3	5000	3000
FlexPepDock	FlexPepDock	⁷⁰	Ziv Ben-Aharon	reweighted I_sc vs backbone I_rmsd	2	200	70
fragments	fragment_picking	⁷¹	Justyna Krys, Dominik Gront	rmsd	10	400	2000
fragments	make fragments pipeline	⁷¹	Daniel Farrell	Coverage, precision	65	1	3000
ligand docking	ligand_docking	⁵⁰	Shannon Smith	Delta_Isc vs. ligand_rmsd	50	200	2000
ligand docking	ligand_scoring_ranking	⁵⁰	Shannon Smith	Spearman and Pearson correlation coefficient	57: 285	1	2
loop modeling	loop_modeling_CCD	⁷²	Phuong Tran, Shane Ó Conchúir	Score vs. loop_rmsd	7	500	500
	loop_modeling_KIC	⁷³		Score vs. loop_rmsd	7	500	620
	loop_modeling_KIC_fragments	⁷⁴		Score vs. loop_rmsd	7	500	760
	loop_modeling_NGK	⁷⁵		Score vs. loop_rmsd	7	500	570
membrane protein-energy function	mp_f19_energy_landscape^#	³⁷	Rituparna Samanta, Rebecca Alford	ddG, depth and title angle	4	1	10
	mp_f19_decoy_discrimination	³⁷		Score vs. rmsd, Wrms	4×100	1	2000
	mp_f19_sequence_recovery	³⁷		sequence recovery, Kullback-Leibler divergence	130	1	500
	mp_f19_ddG_of_mutation	⁷⁶		Pearson correlation coefficient	3	1	1
membrane proteins	mp_dock	⁷⁷	Julia Koehler Leman, Rebecca Alford	I_sc vs. I_rmsd	10	1000	200
	mp_domain_assembly	⁷⁸		Score vs. rmsd	5	5000	700
	mp_lipid_acc	⁷⁹		Accuracy	223	1	2
	mp_relax	⁷⁷		Score vs. rmsd	4	100	40
	mp_symdock	⁷⁷		I_sc vs. rmsd	5	1000	140
PDB diagnostic	PDB_diagnostic	NA	Steven Lewis, William Hansen, Sergey Lyskov	Read-in error type	entire PDB	1	1000
peptide structure prediction	simple_cycpep_predict	⁴⁸	Vikram K. Mulligan	Score vs. rmsd, PNear	1	~800,000	320
peptide structure prediction	peptide_pnear_vs_ic50	⁵¹	Vikram K. Mulligan	IC50 vs. folding energy	7	80,000	400
refinement	relax_cartesian	³²	Julia Koehler Leman	Score vs. rmsd	12	100	120
	relax_fast	⁸⁰		Score vs. rmsd	12	100	120
	relax_fast_5iter	⁸⁰		Score vs. rmsd	12	100	120
RNA	rna_denovo_favorites	⁸¹	Andy Watkins	Score vs. rmsd	12	200	120
RNA	stepwise_rna_favorites	⁸²	Andy Watkins	Score vs. rmsd	12	200	240
RosettaNMR	abinitio_RosettaNMR_rdc	⁸³	Georg Kuenze, Julia Koehler Leman	Score vs. rmsd	3	2000	170
RosettaNMR	abinitio_RosettaNMR_pcs	⁸³	Georg Kuenze, Julia Koehler Leman	Score vs. rmsd	3	2000	1400

The number of tests is constantly being expanded. The test suite is the overall application, the test is the specific test, implemented by the test author(s). The quality measures are evaluated to choose a pass/fail criterion. The targets are the number of different proteins (or biomolecules) tested on, nstruct is the number of models built for each target, and the runtime in CPU hours is the total runtime over all targets.
^*The dock_glycans test has been superceded by glycan_dock.
^#The mp_f19_energy_landscape test has been renamed to mp_f19_tilt_angle.

Back to article page

Table 1 Scientific tests for bio-macromolecular modeling, continuously running on our testing server framework.

Search

Quick links