AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts

Citations Over TimeTop 10% of 2023 papers

Abstract

Sparsely activated Mixture-of-Experts (MoE) is becoming a promising paradigm for multi-task learning (MTL). Instead of compressing multiple tasks’ knowledge into a single model, MoE separates the parameter space and only utilizes the relevant model pieces given task type and its input, which provides stabilized MTL training and ultra-efficient inference. However, current MoE approaches adopt a fixed network capacity (e.g., two experts in usual) for all tasks. It potentially results in the over-fitting of simple tasks or the under-fitting of challenging scenarios, especially when tasks are significantly distinctive in their complexity. In this paper, we propose an adaptive MoE framework for multi-task vision recognition, dubbed AdaMV-MoE. Based on the training dynamics, it automatically determines the number of activated experts for each task, avoiding the laborious manual tuning of optimal model size. To validate our proposal, we benchmark it on ImageNet classification and COCO object detection & instance segmentation which are notoriously difficult to learn in concert, due to their discrepancy. Extensive experiments across a variety of vision transformers demonstrate a superior performance of AdaMV-MoE, compared to MTL with a shared backbone and the recent state-of-the-art (SoTA) MTL MoE approach. Codes are available online: https://github.com/google-research/google-research/tree/master/moe_mtl.

Related Papers

• → An Object Detection and Pose Estimation Approach for Position Based Visual Servoing(2017)5 cited
• → Self-monitoring to improve robustness of 3D object tracking for robotics(2011)4 cited
• → Tracking in 3D: Image Variability Decomposition for Recovering Object Pose and Illumination(1999)15 cited
• → Foreground object segmentation from binocular stereo video(2005)2 cited
• → 6-DOF object localization by combining monocular vision and robot arm kinematics(2017)1 cited