Contributor
Gábor Horváth

[FLINK-3599] Code Generation in Serializers and Comparators of Apache Flink


Mentors
mbalassi
Organization
Apache Software Foundation

The current implementation of the serializers can be a performance bottleneck in some scenarios. These performance problems were also reported on the mailing list recently. I plan to implement code generation into the serializers to improve the performance.

I did some preliminary benchmarks on the Nine Men’s Morris)application being developed by Gábor Gévay and Márton Balassi using a profiler. The runtime of the application was 47 seconds with the default serializers and 35 seconds with the handwritten ones. After switching to handwritten serializers, the number of samples in java.lang.reflect, org.apache.flink.api.java.typeutils.runtime, and sun.reflect was reduced to 36% from 54%. The majority of the rest of the calls to java.lang.reflect are from the comparators. The number of garbage collections was also significantly reduced. Similarly, by adding handwritten serializer to the POJO word count example could reduce the runtime from 9.5s to 8s on a 100mb input file. Note that, these benchmarks did not measure the potential speedups from the rework of the tuple serializer and the comparators.