Automated Translation of Functional Big Data Queries to SQL

Big data analytics frameworks like Apache Spark and Flink enable users to implement queries over large, distributed databases using functional APIs. In recent years, these APIs have grown in popularity because their interfaces abstract away much of the minutiae programming required by traditional query languages SQL. However, convenience comes at a cost are often less efficient than SQL counterparts. Motivated this observation, we present new technique for automatically transpiling While our approach is based on standard paradigm counterexample-guided inductive synthesis, it uses novel column-wise decomposition split synthesis task into smaller subquery problems. We implemented as tool called RDD2SQL translating RDD empirically evaluate effectiveness set real-world queries. Our results show that (1) most can be translated SQL, (2) very effective automating translation, (3) performing translation offers significant performance benefits.