Hello all, question about Arbs. I have a situatio...

# kotest

[James Eschner]

Hello all, question about Arbs.

I have a situation where my code is trying to generate an impossible-to-generate arb. To show a simple example:

data class Foo(val x: Boolean)

val fooArb: Arb<Foo> = arbitrary {
    Foo(x = false)
}

val stuck = fooArb.filter { it.x }.next()

Now, obviously this won't work. This is a simple example but in my case, the arb is much more complex (and transient) so the situation wasn't immediately obviously. What I would have hoped would happen is that the program would eventually time out or throw trying to generate the "next" arb, but what I am witnessing is the kotest test hangs seemingly indefinitely, making it very hard to debug.

What I am wonder is if there is a way to better detect this situation? Can I set a timeout? Force an error? Etc...

** Not a contribution

[Adam S]

Instead of filtering, try defining an 'assumption'

https://kotest.io/docs/proptest/property-test-assumptions.html

if there are too many values that fail the assumption, the test should fail

[James Eschner]

If I just want to pull a single value from an arb, is there a clean way to do that with these assumptions?

** Not a contribution

[Adam S]

.sample() I think?

[Adam S]

https://github.com/kotest/kotest/blob/0f07e00794ee61bc9cfa4cdb9c0ac9113cf570ca/kotest-property/src/commonMain/kotlin/io/kotest/property/Gen.kt#L105-L108

[James Eschner]

Well yes there is sample and I was also using next above. You were suggesting that I use an assumption, I was wondering how that would work to grab a single value

** Not a contribution

[James Eschner]

With filter it's pretty clear, I can use

fooArb.filter { it.x }.next() // substitute sample here if you'd like

with assumptions it doesn't look like, based on the docs, that it's a drop-in replacement for filter

[Adam S]

filter and `assume are similar, but different

filter acts before the test has started, while assume acts during an active test.

I rarely use filter, because there's a risk of filter out all elements (as you found out), but also it can be slow. For example, f 90% of the generated values are discarded, then that's a waste of time.

[Adam S]

With filter it's pretty clear, I can use

fooArb.filter { it.x }.next() // substitute sample here if you'd like

this doesn't look like standard usage... are you using Arb in a checkAll() or forAll()?

[James Eschner]

I do use checkAll and forAll , where appropriate. But if I don't want to incur the cost of the loop I will occasionally just call next so that I can generate a random arb. Additionally, using `checkAll`/`forAll` doesn't solve the problem at hand, if you pass a filtered arb into these functions it can still hang.

We could probably go back and forth for a bit on best practices for arb testing but the simple comment I have is that ideally calling .next() doesn't hang indefinitely and would either time out or throw an exception if there is no possible arb to be generated

** Not a contribution

[Adam S]

oh I wasn't asking because of best practices, of course it's fine to use Arbs outside of a checkAll(), it's just not typical so I didn't understand your use case. assume() requires checkAll() or forAll(), so that won't work for you

Basically your question isn't related to Arbs really, because you could have the same problem with a Sequence. This example will loop infinitely, because all generated values are smaller than 100.

generateSequence { Random.nextInt(1, 100) }
  .filter { it > 100 }
  .first()

probably the simplest workaround is to just limit the number of samples you take

generateSequence { Random.nextInt(1, 100) }
  .take(1000)
  .filter { it > 100 }
  .first()

which will fail, but at least it won't loop infinitely

You can use Arb.samples() to get a sequence of values, and then just make sure to limit them

https://github.com/kotest/kotest/blob/0f07e00794ee61bc9cfa4cdb9c0ac9113cf570ca/kotest-property/src/commonMain/kotlin/io/kotest/property/Gen.kt#L110-L116

[James Eschner]

Ah the take thing is an interesting bit, I will check it out thank you!

[James Eschner]

Stumbled across this situation again (trying to call next on an impossible-to-generate arb). The most painful part about it is the test hangs indefinitely...

I thought that maybe setting a timeout would help (at least the test would complete), but it turns out the timeout does not work...

I tried setting from code (ShouldSpec) and system properties...

timeout = 5000L
invocationTimeout = 5000L

Any ideas?

* Not a contribution *

[sam]

The timeouts only work if it's cooperative concurrency. If you're blocking (or at least not yielding), then you would need to set the blocked=true flag as well.