Proactive Network, I/O and Storage Steering for Multiple DAWs on Shared Infrastructures (B04)

• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems ; Data Management, Data-Intensive Systems, Computer Science Methods in Business Informatics
• Term: since 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 414984028

During DAW execution, energy-intensive components such as CPUs, GPUs and main memory often idle in the time of distributed data access or network data transfers. Thus, optimizing data access and network usage is crucial for efficiency in a given infrastructure. In FONDA I, we de-vised a framework to model inter-task data dependencies to derive information about bottle-necks and to monitor task and DAW execution to steer resource use favorably. In FONDA II we generalize and extend these findings to multi-DAW and multi-center aspects, considering net-work, I/O and storage usage across distributed environments and shared infrastructures. Suitable approaches must account for unexpected changes in resource availability and demand, re-quiring solutions differing substantially from the Phase I single-DAW, single-site setting.

• DFG Programme: Collaborative Research Centres
• Subproject of: SFB 1404: FONDA - Foundations of Workflows for Large-Scale Scientific Data Analysis
• Applicant Institution: Humboldt-Universität zu Berlin
• Project Heads: Professor Dr. Alexander Reinefeld, until 6/2024; Professor Dr. Björn Scheuermann; Dr. Florian Schintke