![]() We assessed the performance of metagenome assembly, binning and taxonomic profiling programs when encountering major challenges commonly observed in metagenomics. The CAMI portal is also open to submissions, and the benchmarks generated here can be used to assess and develop future work. Although benchmarking has been performed before 2, 3, this is the first community-driven effort that we know of. To generate a comprehensive overview, we organized a benchmarking challenge on data sets of unprecedented complexity and degree of realism. CAMI aims to evaluate methods for metagenome analysis comprehensively and objectively by establishing standards through community involvement in the design of benchmark data sets, evaluation procedures, choice of performance metrics and questions to focus on. We tackle these challenges with a community-driven initiative for the Critical Assessment of Metagenome Interpretation (CAMI). Furthermore, the state of the art in this active field is a moving target, and the assessment of new algorithms by individual researchers consumes substantial time and computational resources and may introduce unintended biases. These results are extremely difficult to compare owing to varying evaluation strategies, benchmark data sets and performance criteria. The evaluation of computational methods has been limited largely to publications presenting novel or improved tools. Tremendous progress has been achieved 1, but there is still much room for improvement. ![]() The biological interpretation of metagenomes relies on sophisticated computational analyses such as read assembly, binning and taxonomic profiling. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. ![]() Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. ![]() Nature Methods volume 14, pages 1063–1071 ( 2017) Cite this article Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |