Hi folks, when you read a book on testing it often says that a test must be repeatable In the case ...

# kotest

[Eric Ampire [MOD]]

Hi folks, when you read a book on testing it often says that a test must be repeatable

In the case of property based testing, the data is generated automatically, can we say that these tests are repeatable?

[Big Chungus]

Repeatable tests basically mean that your tests do not depend on any shared mutable state nor produce side effects. So if implemented well, property based tests can also be treated as repeatable as the input is generated from within the test scope.

[wasyl]

Non-repeatable tests would usually involve things like current time, random numbers etc. With property testing I think you usually can provide a seed for generation such that the tests run with the same data every time?

[Eric Ampire [MOD]]

It's a good question

[christophsturm]

https://kotest.io/docs/proptest/property-test-config.html#seed

[mitch]

Hey @Eric Ampire [MOD] thanks for the question! Kotest’s property testing is designed to make sure those tests are repeatable. This is achieved by what @wasyl and @christophsturm rightly mentioned as the random seed.

When a property test fails on certain input combinations, kotest will report:
• the input combination which made the test fail,
• the “simplest” input combination which also made the test fail, as well as, you’ve guessed it
• the random seed which was used to generate those input
For instance:

class SomeTest : FunSpec({
  test("dummy test should be repeatable") {
    checkAll(<http://Arb.int|Arb.int>(1..10), <http://Arb.int|Arb.int>(1..10)) { first, second ->
      first shouldBeGreaterThan second
    }
  }
})

~~~ SomeTest ~~~
  + dummy test should be repeatable: TestResult=Failure(duration=211.721007ms, cause=java.lang.AssertionError: Property failed after 1 attempts

    Arg 0: 1 (shrunk from 8)
    Arg 1: 1 (shrunk from 9)

Repeat this test by using seed 8256346135502929881

Caused by: 1 should be > 1)

you can then replay this test using the seed provided by passing PropTestConfig(seed = 8256346135502929881) in the first checkAll param.

Now on deeper discourse.

can we say that these (property) tests are repeatable?

Not necessarily.

Property tests are only repeatable when the test is properly designed. Both the System Under Test (SUT) and your arbitrary input generators (Arb<A>) need to obey certain constraints. That is:
• The SUT must be pure i.e. suspend fun sut(inputs: A...): B output purely depends on the inputs it is given, (e.g. strings.joinToString()). As a counter example, if your sut mutates an internal state which changes its behaviour such that 1 + 1 is no longer 2, but instead 0, then it is not repeatable, unless there’s a way to inject that internal state into the sut prior to the experiment, such that your sut setup function becomes sut(initialState: S, inputs: A...): B . This effectively made the sut a pure function. Real world example of such test is when you upsert an arbitrary entry in the database, and then do an operation that relates to that entry and then inspect the output.
• all Arb<A> arbitraries that are used to generate the inputs are repeatable. I.e. it and all of the arbs that it depends on relies solely on random seed. e.g. <http://Arb.int|Arb.int>(1..10). As a counter example, imagine if you create an arb that uses Instant.now() , then it won’t satisfy the repeatable requirement of an arb.

[Eric Ampire [MOD]]

Thanks @mitch for your detailed answer 👌, really helpful