CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

Citations Over TimeTop 1% of 2023 papers

Abstract

An increasing number of public datasets have shown a marked impact on automated organ segmentation and tumor detection. However, due to the small size and partially labeled problem of each dataset, as well as a limited investigation of diverse types of tumors, the resulting models are often limited to segmenting specific organs/tumors and ignore the semantics of anatomical structures, nor can they be extended to novel domains. To address these issues, we propose the CLIP-Driven Universal Model, which incorporates text embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models. This CLIPbased label encoding captures anatomical relationships, enabling the model to learn a structured feature embedding and segment 25 organs and 6 types of tumors. The proposed model is developed from an assembly of 14 datasets, using a total of 3,410 CT scans for training and then evaluated on 6,162 external CT scans from 3 additional datasets. We rank first on the Medical Segmentation Decathlon (MSD) public leaderboard and achieve state-of-the-art results on Beyond The Cranial Vault (BTCV). Additionally, the Universal Model is computationally more efficient (6× faster) compared with dataset-specific models, generalized better to CT scansfrom varying sites, and shows stronger transfer learning performance on novel tasks.

Related Papers

• → Embedding as a modeling problem(1998)167 cited
• → PixelSNE: Pixel‐Aligned Stochastic Neighbor Embedding for Efficient 2D Visualization with Screen‐Resolution Precision(2018)2 cited
• → A solution to the reversible embedding problem for finite Markov chains(2016)2 cited
• → A reversible and multiple-layer embedding strategy with high capacity(2010)