I’m a little confused when it comes to shrinking. ...

# kotest-contributors

[Emil Kantis]

I’m a little confused when it comes to shrinking. Specifically if we start with a failing sample, then the input will simply be shrunk to it’s smallest possible value. Wouldn’t it be of more value to first find a succeeding sample and then shrink to try to find the “border” between success and failure?

checkAll<Int> { it shouldBeGreaterThan 1000 }

Given the above, wouldn’t it make sense that shrinking should inform me that it is succeeding for 1000, failing for 999?

[Emil Kantis]

I’m trying to validate the shrinking going on in bind , but this left me a little confused 🙂 I actually think we construct the RTrees properly, but was a bit confused by the above. Writing a test similar to my example, the input was shrunk to 0 in all components immediately then failed. Iexpected it to try all values in the RTree 🙂

[mitch]

to my understanding i think that’s actually what kotest’s checkAll + shrinks intended behaviour. With shrinking i’d be interested to know what is exactly the simplest input for the minimal reproducible case I can find. theres multiple sources on this but i’m quite fond of this one https://dev.to/dubzzz/your-own-property-based-testing-framework-part-3-shrinkers-5a9j

In property based testing, whenever a failure occurs the framework will try to reduce it to something smaller so that the end-user only has to cope with a very simple and small input. It is designed to help developers focusing on the real cause without having to manually investigate several potential sources of bugs.

Let’s imagine there’s a test which has some arbs as an input and it fails on some sample of <http://Arb.int|Arb.int>. The framework realize that, hey maybe we’ll try int = 0 would that fail? It goes and rerun the test (i.e. the lambda inside our checkAll { ... }. Does that fail? if it does, maybe looking for further more complex parameter isn’t too useful because 0 already gives a simple enough input to reproduce the test failure.

If we consider to look for the boundary case, we’re inverting the problem to an unbounded search. That is a very difficult task. I don’t know if there’s a guaranteed deterministic output out of it. I need to search for literatures that covers that.

There are several questions on that in general.. how might that search space look like? can that be defined for any type? What would be the risk for the test framework when dealing larger search space? For instance.. shrinks has a bounded search space, and even that we still have problems about cartesian product rerunning the test lambdas millions of times..

Now for the problem at hand: wishing to harvest the values in the rtree children, you might be interested in the following code

test("arbInt srhinking") {
   val arb = <http://Arb.int|Arb.int>(0..100)
   val shrinks = arb.sample(RandomSource.seeded(1234L)).shrinks
   shrinks.value() shouldBe 54
   shrinks.children.value.map { it.value() } shouldContainExactly listOf(
      0,
      1,
      18,
      27,
      36,
      49,
      50,
      51,
      52,
      53
   )
}

[Emil Kantis]

I'm thinking if we have a failing input a, and succeeding input b, then we could shrink from a towards b. Or vice versa, if a is smaller than b

[sam]

You would need two types of shrinks - one that shrinks and one that enlarges, and you try both until you get a failing case/passing case

[sam]

the problem with going up, is that its unbounded

[sam]

whereas shrinking is bounded by the nautral "emtpy case"

[sam]

I think its also about just finding inputs that are easy for humans to use to figure out what's going on. If that's 0 or 1, it's probably easier for us to work out what the bug is than 1231023124

[Emil Kantis]

I agree that enlarging is an unbounded problem. So for any single failing sample S_f, it would be impossible to search upwards. However, if we have at least one succeeding sample S_s , we can use that to constrain the problem, right? If S_s > S_f , We can shrink the S_s sample using S_f as lower bound. If S_f > S_s, can shrink S_f with S_s as lower bound. In both cases, we have a finite search space that should lead us to a boundary between success and failure.

I think I need more practical experience with property testing to reason about it properly.. 🙂

[Emil Kantis]

Anyway, I opened a PR for the original problem now. I think the testing is okay and implementation seems good. I’ll merge it in a few days unless someone complains 🙂

[sam]

if @mitch is happy with it, I am

[mitch]

Looking good! Thanks @Emil Kantis!

[sam]

So I think as one of the final things for 5.0 I'd like to look at the checkAll syntax again see what we can improve there.

[sam]

I won't be able to go back to it for several days tho