https://kotlinlang.org logo
#kotest-contributors
Title
# kotest-contributors
e

Emil Kantis

09/25/2021, 9:24 AM
I’m a little confused when it comes to shrinking. Specifically if we start with a failing sample, then the input will simply be shrunk to it’s smallest possible value. Wouldn’t it be of more value to first find a succeeding sample and then shrink to try to find the “border” between success and failure?
Copy code
checkAll<Int> { it shouldBeGreaterThan 1000 }
Given the above, wouldn’t it make sense that shrinking should inform me that it is succeeding for 1000, failing for 999?
I’m trying to validate the shrinking going on in
bind
, but this left me a little confused 🙂 I actually think we construct the RTrees properly, but was a bit confused by the above. Writing a test similar to my example, the input was shrunk to 0 in all components immediately then failed. Iexpected it to try all values in the RTree 🙂
m

mitch

09/26/2021, 1:10 PM
to my understanding i think that’s actually what kotest’s checkAll + shrinks intended behaviour. With shrinking i’d be interested to know what is exactly the simplest input for the minimal reproducible case I can find. theres multiple sources on this but i’m quite fond of this one https://dev.to/dubzzz/your-own-property-based-testing-framework-part-3-shrinkers-5a9j
In property based testing, whenever a failure occurs the framework will try to reduce it to something smaller so that the end-user only has to cope with a very simple and small input. It is designed to help developers focusing on the real cause without having to manually investigate several potential sources of bugs.
Let’s imagine there’s a test which has some arbs as an input and it fails on some sample of
<http://Arb.int|Arb.int>
. The framework realize that, hey maybe we’ll try
int = 0
would that fail? It goes and rerun the test (i.e. the lambda inside our
checkAll { ... }
. Does that fail? if it does, maybe looking for further more complex parameter isn’t too useful because 0 already gives a simple enough input to reproduce the test failure. If we consider to look for the boundary case, we’re inverting the problem to an unbounded search. That is a very difficult task. I don’t know if there’s a guaranteed deterministic output out of it. I need to search for literatures that covers that. There are several questions on that in general.. how might that search space look like? can that be defined for any type? What would be the risk for the test framework when dealing larger search space? For instance.. shrinks has a bounded search space, and even that we still have problems about cartesian product rerunning the test lambdas millions of times.. Now for the problem at hand: wishing to harvest the values in the rtree children, you might be interested in the following code
Copy code
test("arbInt srhinking") {
   val arb = <http://Arb.int|Arb.int>(0..100)
   val shrinks = arb.sample(RandomSource.seeded(1234L)).shrinks
   shrinks.value() shouldBe 54
   shrinks.children.value.map { it.value() } shouldContainExactly listOf(
      0,
      1,
      18,
      27,
      36,
      49,
      50,
      51,
      52,
      53
   )
}
e

Emil Kantis

09/26/2021, 3:00 PM
I'm thinking if we have a failing input
a
, and succeeding input
b
, then we could shrink from
a
towards
b
. Or vice versa, if
a
is smaller than
b
s

sam

09/26/2021, 3:14 PM
You would need two types of shrinks - one that shrinks and one that enlarges, and you try both until you get a failing case/passing case
the problem with going up, is that its unbounded
whereas shrinking is bounded by the nautral "emtpy case"
I think its also about just finding inputs that are easy for humans to use to figure out what's going on. If that's 0 or 1, it's probably easier for us to work out what the bug is than 1231023124
e

Emil Kantis

09/26/2021, 4:13 PM
I agree that enlarging is an unbounded problem. So for any single failing sample
S_f
, it would be impossible to search upwards. However, if we have at least one succeeding sample
S_s
, we can use that to constrain the problem, right? If
S_s > S_f
, We can shrink the
S_s
sample using
S_f
as lower bound. If
S_f > S_s
, can shrink
S_f
with
S_s
as lower bound. In both cases, we have a finite search space that should lead us to a boundary between success and failure. I think I need more practical experience with property testing to reason about it properly.. 🙂
Anyway, I opened a PR for the original problem now. I think the testing is okay and implementation seems good. I’ll merge it in a few days unless someone complains 🙂
s

sam

09/26/2021, 4:33 PM
if @mitch is happy with it, I am
👍 1
m

mitch

09/26/2021, 9:40 PM
Looking good! Thanks @Emil Kantis!
🙏 1
s

sam

09/26/2021, 9:41 PM
So I think as one of the final things for 5.0 I'd like to look at the checkAll syntax again see what we can improve there.
I won't be able to go back to it for several days tho
6 Views