ORNIC: A High-Performance RDMA NIC with Out-of-Order Packet Direct Write Method for Multipath Transmission

Citations Over TimeTop 16% of 2024 papers

Abstract

Remote Direct Memory Access (RDMA) technology provides a low-latency, high-bandwidth, and CPU-bypassed method for data transmission between servers. Recent works have proved that multipath transmission, especially packet spraying, can avoid network congestion, achieve load balancing, and improve overall performance in data center networks (DCNs). Multipath transmission can result in out-of-order (OOO) packet delivery. However, existing RDMA transport protocols, such as RDMA over Converged Ethernet version 2 (RoCEv2), are designed for handling sequential packets, limiting their ability to support multipath transmission. To address this issue, in this study, we propose ORNIC, a high-performance RDMA Network Interface Card (NIC) with out-of-order packet direct write method for multipath transmission. ORNIC supports both in-order and out-of-order packet reception. The payload of OOO packets is written directly to user memory without reordering. The write address is embedded in the packets only when necessary. A bitmap is used to check data integrity and detect packet loss. We redesign the bitmap structure into an array of bitmap blocks that support dynamic allocation. Once a bitmap block is full, it is marked and can be freed in advance. We implement ORNIC on a Xilinx U200 FPGA (Field-Programmable Gate Array), which consumes less than 15% of hardware resources. ORNIC can achieve 95 Gbps RDMA throughput, which is nearly 2.5 times that of MP-RDMA. When handling OOO packets, ORNIC’s performance is virtually unaffected, while the performance of Xilinx ERNIC and Mellanox CX-5 drops below 1 Gbps. Moreover, compared with MELO and LEFT, our bitmap has higher performance and lower bitmap block usage.

Related Papers

• Design Guidelines for High Performance RDMA Systems.(2016)
• → RDMA Communciation Patterns(2020)14 cited
• → Performance profiling and design choices of an RDMA implementation using FPGA devices(2024)1 cited
• → Towards RDMA-Based High Performance Network Technologies(2017)