Elaborating the new stream processing system utilizes serverless framework with fast scaling policy
- Mentors
- wonook, Jeongyoon Eo
- Organization
- Apache Software Foundation
- Technologies
- java, ApacheNemo
- Topics
- scalability, streaming system, low-latency
In this project, I will focus on building a concrete baseline for stream processing system scaling on a serverless framework. First, choose an adequate system for scaling based on the purpose of a scaling, dealing with events with high input rate. Second, examine the dataflow reshaping mechanism on stream query DAG and evaluate it though some experiment, since it is often helpful for efficient scaling on serverless framework. Finally, elaborate scaling policy to reasonably determine when to scale up on a serverless framework. This process mainly focuses on input rate on the stream that can cause high latency spike toward the system.