Scalable Data Management on Next-Generation Networks beyond RDMA

• Applicant: Professor Dr. Carsten Binnig
• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: since 2017
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 361478211

The efficient use of networks plays a substantial role for the scalability of distributed in-memory DBMSs. While in the past, communication networks have been slow and passive, modern networks have significantly been changing. Modern networks not only enable high bandwidth and low latency communication via Remote-Direct-Memory-Access (RDMA) but more recently also provide ample opportunities to offload computation to network devices by in-network-processing (INP). However, simply replacing the traditional network without making use of these capabilities does not lead to immediate performance benefits. The overarching goal of this proposal is thus to investigate the opportunities of redesigning distributed DBMSs for the capabilities of modern (i.e., high-speed and programmable) networks.In Phase I of the priority program, we have focused on the question how distributed in-memory DBMSs should be redesigned to make best use of RDMA. The results of Phase I can be summarized as follows: (1) We have shown that using RDMA is an important prerequisite to leverage the high-bandwidth and low-latency of high-speed networks. (2) We have proposed a new distributed DBMS architecture called the Network-Attached-Memory (NAM) architecture which makes efficient use of RDMA and provides many advantages w.r.t. scalability (e.g., better load balancing) over classical distributed database architectures. (3) Based on the NAM-architecture, we have shown how core building blocks ranging from the storage layer over query processing have to be redesigned for RDMA to best support scalability for different workloads (OLTP and OLAP).While RDMA provides significant performance benefits for scalable in-memory DBMSs, it also has many inherent limitations. A main reason is that the limited set of RDMA operations often leads to complex DBMS protocols. In Phase II, we will thus use the programmability of the next generation of high-speed network cards and switches to address these inherent limitations. The main directions of the proposal are: (1) First, we want to leverage the programmabilty of high-speed networks to develop a set of smart remote memory operations that extend the RDMA protocol. (2) Second, based on these smart remote memory operations, we will revisit the design of core components of distributed DBMS for OLAP and OLTP. (3) Finally, we will evaluate the end-to-end efficiency of our new techniques by running benchmarks such as TPC-C or TPC-H.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2037: Scalable Data Management on Future Hardware