Anybody else face issues with #jacoco branch coverage when upgrading kotlin? Blocked from upgrading from 1.3.72 to 1.4+ because a lot of new 'ghost' branches pop up as uncovered. Some might be addressed in the next jacoco release based on the release notes, but it's been one of the few impactful pain points I've had with kotlin. Unfortunately been blocked from upgrading for months on this, curious if there's a plan to address this better going fwd?

Are you saying you're prevented from using the latest kotlin because of the code coverage report from jacoco?

@Byron Katz Because of build failures from test coverage reqs suddenly not being met across the codebase, to be specific

Then I'd suggest to temporarily relax the gates to let that change through and have them back up again after

Tried but unfortunately the compromise would be quite drastic, e.g. from 85% to 50% in many cases, and across lots of packages, for an unknown amount of time. Branch coverage has been very helpful in ensuring diff code paths get covered that developers forget or don't notice. This is kind of an ongoing battle between jacoco and kotlin bytecode, where jacoco is constantly needing to catch up with kotlin compilation changes

Hmm. For what it's worth I use jacoco a lot in my project looking for places I missed and while it's a bit annoyingly pedantic about not having checked every branch in some functional mishmash, I generally am able to work through that and derive good value.

Yeah Jacoco v0.8.6 works quite well with kotlin 1.3.x. They resolved lots of issues we faced when first adopting kotlin in 2019, and it's been pretty smooth since that release (easy to handle any potential discrepancies). And valuable indeed as you said, well worth it. However upgrading kotlin has proved challenging for us. Hoping the next Jacoco release is just as helpful in this regard, and that it comes soon :) But discrepancies are more so in the hands of the kotlin team with compilation modifications, hence asked here. Would be good to have the jacoco lib more in sync with kotlin bytecode changes

Or maybe it's time for Kococo 😉

probably not what you want to hear but regular code coverage is meaningless 😬

http://pitest.org works pretty well with kotin, and gives you mutation coverage which is what really matters.

exactly @christophsturm i’m a big fan of PIT 🙂

I like PIT, it's good. But Jacoco is also very good for certain purposes. I can certainly understand why people would think plain code coverage isn't useful - the way it's instituted most of the time, the fact that developers don't typically apply good testing practices. It's all endemic.

good news is jacoco 0.8.7 just came out, and the build now passes for the kotlin upgrade to 1.4.32 🙂

for pitest to work well you need a pretty well designed test suite, the tests should be fast and be able to run in parallel. But i guess thats how a test suite should be anyway 🙂

But i guess thats how a test suite should be anyway

But i guess thats how a unit test suite should be anyway 😉

I've generally found PIT craps out if I throw it at a ton of tests - have you figured out a configuration where that doesn't happen?

But i guess thats how a unit test suite should be anyway

right. and you should only use the unit test suite with pit, because integration test coverage is cheating 🙂

actually the run in parallel part is true for any test suite 🙂

“it depends” if i want to test a full-fledged scenario like from browsing the catalog to final checkout i’ll probably have small tests that depend on each other

hmm if it needs an order i treat it as one test. and i can not run the later parts separate anyway. but i can still run these long running tests in parall with multiple browsers, web servers etc.

the problem when you treat is as one test is that if it breaks, you’ve no clue where it did that’s one of the issues of integration tests if you define steps and chain them, it becomes much clearer

well i know what assertion failed

but the function split doesn’t tell you at first glance where it failed

what test runner would you use for that? I have not found one that i liked yet.

testng junit 4 frowned upon test ordering and i don’t have experience enough in junit 5 to pinpoint you to a possible existing feature