I’m a little confused when it comes to shrinking. Specifically if we start with a failing sample, then the input will simply be shrunk to it’s smallest possible value. Wouldn’t it be of more value to first find a succeeding sample and then shrink to try to find the “border” between success and failure?

checkAll<Int> { it shouldBeGreaterThan 1000 }

Given the above, wouldn’t it make sense that shrinking should inform me that it is succeeding for 1000, failing for 999?

to my understanding i think that’s actually what kotest’s checkAll + shrinks intended behaviour. With shrinking i’d be interested to know what is exactly the simplest input for the minimal reproducible case I can find. theres multiple sources on this but i’m quite fond of this one https://dev.to/dubzzz/your-own-property-based-testing-framework-part-3-shrinkers-5a9j

In property based testing, whenever a failure occurs the framework will try to reduce it to something smaller so that the end-user only has to cope with a very simple and small input. It is designed to help developers focusing on the real cause without having to manually investigate several potential sources of bugs.

Let’s imagine there’s a test which has some arbs as an input and it fails on some sample of

<http://Arb.int|Arb.int>

. The framework realize that, hey maybe we’ll try

int = 0

would that fail? It goes and rerun the test (i.e. the lambda inside our

checkAll { ... }

. Does that fail? if it does, maybe looking for further more complex parameter isn’t too useful because 0 already gives a simple enough input to reproduce the test failure. If we consider to look for the boundary case, we’re inverting the problem to an unbounded search. That is a very difficult task. I don’t know if there’s a guaranteed deterministic output out of it. I need to search for literatures that covers that. There are several questions on that in general.. how might that search space look like? can that be defined for any type? What would be the risk for the test framework when dealing larger search space? For instance.. shrinks has a bounded search space, and even that we still have problems about cartesian product rerunning the test lambdas millions of times.. Now for the problem at hand: wishing to harvest the values in the rtree children, you might be interested in the following code

test("arbInt srhinking") {
   val arb = <http://Arb.int|Arb.int>(0..100)
   val shrinks = arb.sample(RandomSource.seeded(1234L)).shrinks
   shrinks.value() shouldBe 54
   shrinks.children.value.map { it.value() } shouldContainExactly listOf(
      0,
      1,
      18,
      27,
      36,
      49,
      50,
      51,
      52,
      53
   )
}

I'm thinking if we have a failing input

a

, and succeeding input

b

, then we could shrink from

a

towards

b

. Or vice versa, if

a

is smaller than

b

You would need two types of shrinks - one that shrinks and one that enlarges, and you try both until you get a failing case/passing case

I agree that enlarging is an unbounded problem. So for any single failing sample

S_f

, it would be impossible to search upwards. However, if we have at least one succeeding sample

S_s

, we can use that to constrain the problem, right? If

S_s > S_f

, We can shrink the

S_s

sample using

S_f

as lower bound. If

S_f > S_s

, can shrink

S_f

with

S_s

as lower bound. In both cases, we have a finite search space that should lead us to a boundary between success and failure. I think I need more practical experience with property testing to reason about it properly.. 🙂

if @mitch is happy with it, I am

Looking good! Thanks @Emil Kantis!

So I think as one of the final things for 5.0 I'd like to look at the checkAll syntax again see what we can improve there.