Thanks for the response. This means that the build not deterministic and leads to cache misses.
I've noticed that the KSP task adds the outputs of the KAPT task. In the first execution these will be empty, in the second execution these will be populated meaning that the KSP task must re-execute because the inputs have changed.