Deployment of inference as a service at the US CMS Tier-2 data centers

Abstract

Coprocessors, especially GPUs, will be a vital ingredient of data production workflows at the HL-LHC. At CMS, the GPU-as-a-service approach for production workflows is implemented by the SONIC project (Services for Optimized Network Inference on Coprocessors). SONIC provides a mechanism for outsourcing computationally demanding algorithms, such as neural network inference, to remote servers, where requests from multiple clients are intelligently distributed across multiple GPUs by a load-balancing service. This talk highlights the recent progress in deploying SONIC at selected U.S. CMS Tier-2 data centers. Using realistic CMS Run3 data processing workflows, such as those containing transformer-based algorithms, we demonstrate how SONIC is integrated into the production-like environment to enable accelerated inference offloading. We will present developments from both the client and server sides, including production job and data center configurations for NVIDIA and AMD GPUs. We will also present performance scaling benchmarks and discuss the challenges of operating SONIC in CMS production, such as server discovery, GPU saturation, fallback server logic, etc.

Related Papers

• → MEASUREMENTS OF WAVE IMPACTS AT FULL SCALE: RESULTS OF FIELDWORK ON CONCRETE ARMOUR UNITS(1996)9 cited
• → Serving Away From Home: How Deployments Influence Reenlistment(2002)19 cited
• → Defining and Checking Deployment Contracts for Software Components(2006)15 cited
• → Understanding deployment from the perspective of those who have served(2016)5 cited
• → Large-Scale Deployment of Tablet Computers in Brazilian Public Schools: Decisive Factors and an Implementation Model(2017)1 cited