Apache Spark: Build monitoring UI and related infrastructure for Spark SQL and structured streaming
- Mentors
- yhuai
- Organization
- Apache Software Foundation
As more efforts have been made to Spark SQL, there are more demands on providing more information and metrics through Spark UI in an effective way. Specifically, users and developers want to (1) browse information for every individual Spark SQL sessions, (2) get continuous updates from jobs created by structured streaming, and (3) have an easy and intuitive way to debug runtime issues. In this work, we aim to make efforts to response the above three demands by adding features, improvements, and related infrastructures to Spark UI for Spark SQL and structured streaming.