The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions: (i) Can deep CNN be trained with data collected from crowdsourcing?, (ii) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)?, (iii) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.     «

The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy anno...     »

biological organs; cancer; data aggregation; image classification; image denoising; learning (artificial intelligence); medical image processing; biomedical image database; breast cancer histology imaging; conventional machine-learning methods; convolutional neural network; crowd annotation datasets; crowd flower API; crowd sourcing layer; data aggregation; deep CNN learning; ground-truth data; image annotation tasks; learning annotation models; learning process; mitosis detection; noisy annotations; self-implemented web-platform; Biomedical imaging; Computational modeling; Crowdsourcing; Data models; Machine learning; Noise measurement; Robustness; Aggregation; crowdsourcing; deep learning; gamification; online learning     «

biological organs; cancer; data aggregation; image classification; image denoising; learning (artificial intelligence); medical image processing; biomedical image database; breast cancer histology imaging; conventional machine-learning methods; convolutional neural network; crowd annotation datasets; crowd flower API; crowd sourcing layer; data aggregation; deep CNN learning; ground-truth data; image annotation tasks; learning annotation models; learning process; mitosis detection; noisy annotat...     »