Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

iCOBRA: open, reproducible, standardized and live method benchmarking

Nature Methods volume 13, page 283 (2016)Cite this article

Subjects

Access through your institution
Buy or subscribe

To the Editor:

Modern life science research tasks often involve ranking or classifying items. For example, in studies of differential expression, genes can be ranked by the estimated P value or, using a cutoff on the P value, classified as either 'significantly different' or 'not significantly different' between conditions of interest. A wide range of computational methods dedicated to these tasks exist1,2,3, many of which rely on accurate quantification of underlying entities such as abundance levels. As methods are developed and refined, static benchmarking studies quickly become outdated. Moreover, a standard way to present results from method comparisons is lacking, and raw reference data are not always made available. This often makes it difficult for method researchers to reproduce published evaluations or explore them from different angles. Here we present iCOBRA (for “interactive comparative evaluation of binary classification and ranking methods”), a benchmarking platform for both users and developers of methods that promotes open, standardized and reproducible evaluations. iCOBRA consists of an R package and a flexible, interactive web application that can rapidly evaluate methods for binary classification, ranking and continuous target estimation against a ground truth. In addition, we have collected a set of benchmarking data sets in standard formats (a link is provided at https://github.com/markrobinsonuzh/iCOBRA) to lower barriers for new method developers as well as to facilitate standardized method evaluations in the future. We envision that this resource will be extended over time, and we encourage the community to contribute data (for example, simulations) and method assessments. In Supplementary Note 1, we show how iCOBRA can be used to exactly reproduce and visualize results from recent benchmarking studies.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Screenshot of the iCOBRA interactive application interface.

References

  1. Robinson, M.D., McCarthy, D.J. & Smyth, G.K. Bioinformatics 26, 139–140 (2010).

    Article  CAS  Google Scholar 

  2. Paulson, J.N., Stine, O.C., Bravo, H.C. & Pop, M. Nat. Methods 10, 1200–1202 (2013).

    Article  CAS  Google Scholar 

  3. Anders, S., Reyes, A. & Huber, W. Genome Res. 22, 2008–2017 (2012).

    Article  CAS  Google Scholar 

  4. Sing, T., Sander, O., Beerenwinkel, N. & Lengauer, T. Bioinformatics 21, 3940–3941 (2005).

    Article  CAS  Google Scholar 

  5. Soneson, C. Bioinformatics 30, 2517–2518 (2014).

    Article  CAS  Google Scholar 

  6. Fijorek, K., Fijorek, D., Wisniowska, B. & Polak, S. Bioinformatics 27, 3439–3440 (2011).

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Molecular Life Sciences, University of Zurich, Zurich, Switzerland

    Charlotte Soneson & Mark D Robinson

  2. Swiss Institute of Bioinformatics (SIB), Zurich, Switzerland

    Charlotte Soneson & Mark D Robinson

Authors
  1. Charlotte Soneson
    View author publications

    Search author on:PubMed Google Scholar

  2. Mark D Robinson
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Mark D Robinson.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Information

Supplementary Notes 1 and 2 (PDF 592 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Soneson, C., Robinson, M. iCOBRA: open, reproducible, standardized and live method benchmarking. Nat Methods 13, 283 (2016). https://doi.org/10.1038/nmeth.3805

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/nmeth.3805

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics