why does allTest not fail if it failed previously? I worked around (I still let it fail if it failed without the need to execute the tests again) but I realised that gradle's test task does not have the same behaviour. And what surprised me even more is that :jvmTest also executes the test again if it failed previously.

Try running

cleanAllTests

first

I am more wondering about the reason for the different behaviours. Personally I don't mind/prefer if no tests are executed again whereas cleanAllTests would execute all tests again (also from platforms which might have been run with success)