Anybody else face issues with #jacoco branch cover...

# server

[Husam]

Anybody else face issues with #jacoco branch coverage when upgrading kotlin? Blocked from upgrading from 1.3.72 to 1.4+ because a lot of new 'ghost' branches pop up as uncovered. Some might be addressed in the next jacoco release based on the release notes, but it's been one of the few impactful pain points I've had with kotlin. Unfortunately been blocked from upgrading for months on this, curious if there's a plan to address this better going fwd?

[Husam]

Jacoco changelog for reference: https://www.jacoco.org/jacoco/trunk/doc/changes.html

[Byron Katz]

Are you saying you're prevented from using the latest kotlin because of the code coverage report from jacoco?

[Husam]

@Byron Katz Because of build failures from test coverage reqs suddenly not being met across the codebase, to be specific

[Byron Katz]

Then I'd suggest to temporarily relax the gates to let that change through and have them back up again after

[Husam]

Tried but unfortunately the compromise would be quite drastic, e.g. from 85% to 50% in many cases, and across lots of packages, for an unknown amount of time. Branch coverage has been very helpful in ensuring diff code paths get covered that developers forget or don't notice.
This is kind of an ongoing battle between jacoco and kotlin bytecode, where jacoco is constantly needing to catch up with kotlin compilation changes

[Byron Katz]

Hmm. For what it's worth I use jacoco a lot in my project looking for places I missed and while it's a bit annoyingly pedantic about not having checked every branch in some functional mishmash, I generally am able to work through that and derive good value.

[Husam]

Yeah Jacoco v0.8.6 works quite well with kotlin 1.3.x. They resolved lots of issues we faced when first adopting kotlin in 2019, and it's been pretty smooth since that release (easy to handle any potential discrepancies). And valuable indeed as you said, well worth it. However upgrading kotlin has proved challenging for us.
Hoping the next Jacoco release is just as helpful in this regard, and that it comes soon :) But discrepancies are more so in the hands of the kotlin team with compilation modifications, hence asked here. Would be good to have the jacoco lib more in sync with kotlin bytecode changes

[Byron Katz]

Or maybe it's time for Kococo 😉

[nfrankel]

probably not what you want to hear
but regular code coverage is meaningless
😬

[christophsturm]

http://pitest.org works pretty well with kotin, and gives you mutation coverage which is what really matters.

[nfrankel]

exactly @christophsturm
i’m a big fan of PIT 🙂

[Byron Katz]

I like PIT, it's good. But Jacoco is also very good for certain purposes. I can certainly understand why people would think plain code coverage isn't useful - the way it's instituted most of the time, the fact that developers don't typically apply good testing practices. It's all endemic.

[Husam]

good news is jacoco 0.8.7 just came out, and the build now passes for the kotlin upgrade to 1.4.32 🙂

[Husam]

Better late than never!

[Husam]

@nfrankel I wouldn't say useless, but they have to be written well (perhaps that's what you were alluding to by 'regular'?). This line from pitest is totally true tho, in terms of what code coverage "guarantees":

"It is therefore only able to identify code that is definitely not tested."

And that's also useful to know - how much code is def not tested; at total risk

[Husam]

Good to know ppl have had good experiences with pitest and kotlin, will def look into setting it up! Looking forward to it

[christophsturm]

for pitest to work well you need a pretty well designed test suite, the tests should be fast and be able to run in parallel. But i guess thats how a test suite should be anyway 🙂

[nfrankel]

But i guess thats how a test suite should be anyway

But i guess thats how a unit test suite should be anyway
😉

[Byron Katz]

I've generally found PIT craps out if I throw it at a ton of tests - have you figured out a configuration where that doesn't happen?

[christophsturm]

But i guess thats how a unit test suite should be anyway

right. and you should only use the unit test suite with pit, because integration test coverage is cheating 🙂

[christophsturm]

actually the run in parallel part is true for any test suite 🙂

[christophsturm]

@Byron Katz for me the number of tests did not really matter as long as they are well written, and don’t depend on global state

[nfrankel]

actually the run in parallel part is true for any test suite 🙂

“it depends”
if i want to test a full-fledged scenario like from browsing the catalog to final checkout
i’ll probably have small tests that depend on each other

[christophsturm]

hmm if it needs an order i treat it as one test. and i can not run the later parts separate anyway. but i can still run these long running tests in parall with multiple browsers, web servers etc.

[nfrankel]

the problem when you treat is as one test is that if it breaks, you’ve no clue where it did
that’s one of the issues of integration tests
if you define steps and chain them, it becomes much clearer

[christophsturm]

well i know what assertion failed

[christophsturm]

you can still define steps by seperating the test into functions. but from the test runners perspective i think it should be just one test

[nfrankel]

but the function split doesn’t tell you at first glance where it failed

[christophsturm]

what test runner would you use for that? I have not found one that i liked yet.

[nfrankel]

testng
junit 4 frowned upon test ordering
and i don’t have experience enough in junit 5 to pinpoint you to a possible existing feature