Implementing Work Stealing on Nemo
- Mentors
- Taegeon Um, Won Wook SONG
- Organization
- The Apache Software Foundation
This project aims to optimize the performance of Nemo by dynamically splitting skewed tasks of a job based on the information collected from runtime metrics, executing them in parallel, and merging them afterwards. The goal is achieved through the following four steps: First, gather relevant metrics during execution. Second, using the gathered statistics, determine which task is skew(i.e. needs a longer time to finish) periodically. Third, Split the skewed tasks in half and allocate the latter splitted half to the new task. And lastly, execute the new task and check its performance.