Starting a thread about memory usage since the previous one is long enough 🙂

Probably lots of optimization could go into that stuff. The

Description

class is getting deprecated anyway in 4.7

i wonder if https://github.com/Kotlin/kotlinx.collections.immutable could be helpful for cases where you want to have immutability without copying around

We don’t have property tests, but I have a hunch that there are some optimisations around

forall

that can be made as well

are you using data tests ?

tough question 😄 I think we do, but not many. We have more of the older style

forall(row(…

tests, and not the new API that allows data classes

or the new new api that I haven't written docs on yet 😂

i have had problems with duration too. but i think with 1.5 its no longer experimental

no it's still experimental

ok but inline classes are no longer experimental

right

Yay 😄 Yeah I’m not really up to date with those, I know some people preferred the old approach so they kept using it + we have more older tests that we didn’t migrate. Also there were some bugs initially that bit us. Either way we use both, but definitely more of the older API. I know one particular test has ~4 nested levels of the old forall and those tests take super long, even though it’s just 2/3 rows for each forall (so under 100 test cases, not that much). But that’s probably not related to memory

so duration will probably not break with every update 🙂

duration has broken with every single update so far

yea duration even hit us as a non-library — we can’t update kotlin without updating coroutines because of the duration library, and we’ll have to migrate super convenient

Int.seconds

etc. apis to whatever they did instead 😞 not fun

Yes Int.seconds was awesome, and now it's Duration.seconds(int), so ugly

why was that removed? i really like that in ruby on rails.

@wasyl are you on 4.5 ?

i guess it could still be a separate library

Yes, 4.5.0

@christophsturm so when are you starting to join the kotest team 😂 😛

i guess it could still be a separate library

someone already did that 😄 https://github.com/eygraber/kotlin-duration-extensions

i could contribute to the idea plugin and make it run failfast tests too :))

You could just make kotest support what you're missing 🙂

the reason why i did not do this earlier and instead started my own test runner was that it was just easier for me to start from scratch than to figure out how kotest works. but you are right, its definately a good option for the future. I’m just a bit afraid that kotest is harder to refactor because there are so many features that have to be taken into account.

That's what happens when people start using your framework. They have requests and the feature set grows. Kotest tries to be general purpose, but I don't think that precludes being fast. I would say that outside of the test runner classes, it's very straightforward. And most of the complexity comes from trying to be junit friendly. I would like to ignore junit platform as it's not well suited to a modern test framework in my opinion, but of course you can't do that unless you are going to provide a gradle plugin (which we now have) and an intellij plugin (which we've had for ages) but there are still certain things that are junit first (eg the way intellij hooks into gradle for example). Over the past year or so I've tried to go in a direction of being less a junit platform engine and more a completely standalone framework.

right, but i think its also important to say no to features. but you are right there is no reason why kotest should be slower than failfast. definately something to look at, maybe i should try to implement the failfast dsl as kotest dsl and see if i hit any roadblocks.

We do say no to features a lot of the time 🙂 I'm not even sure failfast is noticeable faster, it's just that you're comparing failfast's own test suite with kotest's own test suite. The latter is like 10,000 tests. Certainly I don't think the DSL matters - it's just nested test lambdas at the end of the day, and that's the same for kotest, spek, failfast, whatever.

yes its definately possible that kotest is faster.

Imagine how good Kotest would be with more people contributing. We're still a small team of 4 regular contributors and a handful of semi regular contributors. I suggested to the current maintainer of Spek that we fold together as he was looking for someone to take over that project (seems like development has slowed but it still has a lot of users) but I don't think he was keen.

in failfast i have 2 test executors, one for the initial execution, and then one that uses that map to just execute single tests. it may seem redundant and stupid but if you really want high core utilization from the start i see no other way

Pretty much yeah

and i have a `dependency { }`method that creates or waits for a test dependency, and it does that in a separate coroutine dispatcher to free the cpu core for another test.

take a look at the concurrency stuff in kotest, specifically, start with SpecLauncher and follow it through. Allows the users to customize how concurrency works.

ok i will

@sam how do I know snapshot release version for https://github.com/kotest/kotest/actions/runs/842320540?

what about speed ?

Ah, I was looking for buildnumber — it’s there in action title

master #230

. About speed: I don’t see any significant improvement , both versions finish on average within 1 second of each other

I guess reflection is not slow then

let me look at memory profiler

How many times does it get called, it's not clear

Would that caching I did above make sense as well? You mentioned

Description

will be deprecated, will the logic be moved somewhere or be reworked completely?

I'm adding a replacement that gives you more information about the test at runtime (like the tags that were present at runtime, not just compile time). It's early days but it will replace description eventually, although description will hang about for a long while yet for compatibility. https://github.com/kotest/kotest/pull/2239

This is somewhere after ~200 or 300 tests (not specs)

I've cached various methods in description by putting them in lazy {} blocks, see if this improves things https://github.com/kotest/kotest/actions/runs/842480289

~1GB lower memory footprint, slight speed improvement (~10%, but this is not very scientific), and still kotlin.reflect APIs churning through Strings and ArrayLists in the profiler.

yeah if you can find that out that would be good

I’d guess this is majority of our assertions

actual full classes, not fields ?

Yes, we have a lot of those

wonder why testcase to string is being called

log("Executing active test $testCase with context $context")

in

TestCaseExecutor:148

yep, I'm currently replacing them with functions

Yeah with inline functions that should be practically free with logging enabled anyway I guess?

yes

There’s also

TestCaseExecutor:179

for example, I can grab some more unless you’re just replacing all logs everywhere 😄

the latter 🙂

Looks good 🙂

addressed your feedback and merged

Will do!

this should be a big improvement 🙂

seems like the build failed

Yeah I added 233

30% speed improvement 😮 And very interesting, peak memory during tests execution is down to ~1G

instead of 7gb?

Yes, but!

test discovery maybe ?

Yeah most likely

if its first couple of seconds

This is first GC at ~1.6s, memory is slighly over 4GB. After that it doesn’t go over 1.2GB

sounds like discovery

Yeah. About how many tests: this is just one module, so 180 specs, around 1k `it`s (and

InstancePerLeaf

strategy)

ok so a reasonable number of test lambdas

Sadly startup profiling is broken for me, so I can’t profile that early in the execution process 😞

ok, well not much we can do about discovery anyway

Yeah it’s ok. The 4GB memory could use an improvement if possible, but without profiling it’s difficult to say

Yes that's fair

Great improvement other than that regardless, many thanks 🙂 I’ll profile some more and let you know the results, but probably in a couple of days

alright, I'll take a look at discovery as well

Will do, what does it do?

doesn't scan inside jars on the classpath

Yeah, if the jvm knows which are included jars that could help

yep that's a good idea too

(I’m also thinking that e.g. kotlin or androidx folks won’t be able to use Kotest, right? 😄)

it would still need to un-zip the jars to look for your packages tho

Not much improvement with this property, sadly 😞 neither speed nor memory, I added

"-Dkotest.framework.discovery.jar.scan.disable=true"

property

hmmm ok