ML-Augmented Database Systems over Multi-Modal Data using Natural Language Queries

• Applicant: Professor Dr. Carsten Binnig
• Subject Area: Security and Dependability, Operating-, Communication- and Distributed Systems
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 545611510

In this proposal, we present our vision of ML-augmented databases. In stark contrast to the existing line of work on ML for databases which solely focus on enhancing the performance of databases using ML, our approach seeks to leverage ML to extend databases with new functionalities to tackle the limitations of existing systems. In particular, we envision a new class of ML-augmented databases to enable natural language querying over multi-modal data without the need to first transform the data into tabular format. Recently, large language models (LLMs) such as GPT-4 already demonstrate capabilities to support natural language question answering on multi-modal data sources. We call this an LLM-first approach since the LLM is used itself for question answering. While such an LLM-first approach might seem to solve the problem of multi-modal question answering as outlined above, we argue that the LLM-first approach has significant downsides. For example, LLMs have inherent hard-to-address limitations, such as hallucinations, which are a consequence of their generative nature. Moreover, query answering with LLMs is a black-box since users cannot trace how the answer came to be. Even more severe are the high overheads of LLMs, which prevent them from efficiently provide question answering on large datasets. In this proposal, with ML-augmented databases we suggest a new approach for questions answering on multi-modal data. In contrast to the LLM-first approach, ML-augmented databases represent a database-first solution, where the core idea revolves around leveraging first principles from databases, such as query plans and query optimization, to enable efficient and robust query answering. These principles are further extended with the proficiency of LLMs in comprehending natural language questions and multi-modal data.

• DFG Programme: Research Grants
• International Connection: France
• Partner Organisation: Agence Nationale de la Recherche / The French National Research Agency
• Cooperation Partner: Professor Paolo Papotti