Large-scale medical image annotation with crowd-powered algorithms

Citations Over TimeTop 10% of 2018 papers

Abstract

Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.

Related Papers

• → Multi-label Crowdsourcing Learning with Incomplete Annotations(2018)12 cited
• TourExplain : A Crowdsourcing Pipeline for Generating Explanations for Groups of Tourists(2018)
• → Selective Verification Strategy for Learning From Crowds(2018)2 cited
• → Object Detection, Tracking, and Motion Segmentation for Object-level Video Segmentation(2016)30 cited
• → Does History Help? An Experiment on How Context Affects Crowdsourcing Dialogue Annotation(2013)