Performance difference in Spark write stage when running on different kernels

Question

When running a simple Spark application (GroupByTest) on two different kernel versions (3.10 and 4.18)), I noticed that write performance is 13%-15% better on the newer kernel. I tested this using the exact same two machines, switching only the system image between runs. I'm trying to determine the cause of this performance difference. I generated flame graphs for both runs and observed a different call stack for the write operation, which likely indicates an implementation change in the newer kernel. However, I’m still unsure if this is the primary reason for the performance improvement

New kernel

Old kernel