https://kotlinlang.org logo
Title
g

Grigory Derzhavets

06/18/2020, 9:51 AM
Hey I am using tags to filter out tests. I can not find a way how to solve a problem: when some tests are filtered out by tag, I need the runner not to run 
beforeSpec/beforeTest
 methods. As I see right now, the test_executor firstly runs these and then filters out tests. Does anybody know whether it is possible in KotlinTest? Please
Or perhaps I don't understand the idea of tags and am not using it correctly
s

sam

06/18/2020, 12:19 PM
I think what you're really asking is, how can I skip the entire spec if all tests are filtered out
g

Grigory Derzhavets

06/18/2020, 12:39 PM
In my understanding a spec means a class and a spec has several tests inside it. The case: I have a BaseTest class, that has a beforeSpec which sets up something for future tests. I have 2 classes (specs) that inherits from BaseTest. Class_A has tag_A, Class_B has tag_B. If a run all tests with a tags.include=tag_A, then
beforeSpec
will be ran twice - for Class_A, which is correct, and for Class_B, which seems like is incorrect, since execution of tests in Class_B won't happen due to tag filter
s

sam

06/18/2020, 12:46 PM
So the tags filter the tests, as you have seen, but the test discovery logic needs to have an instantiated class before it can find the tests. And it makes sense to run beforeSpec as soon as the class is created. That's why it works like it does.
However you can put tags on the spec itself
There's @Tag or @Tags can't remember which one. Put that on the spec and the entire spec will be skipped
g

Grigory Derzhavets

06/18/2020, 12:51 PM
You mean jupiter annotations, right? If yes, do they work with
-Dkotlintest.tags.include
?
I see both @Tag and @Tags
s

sam

06/18/2020, 12:52 PM
No there's a kotest specific one
are you on 4.x ?
io.kotest.core.annotation.Tags
g

Grigory Derzhavets

06/18/2020, 12:54 PM
ah, I see, I am on 3.1.8
I will try 4.x
thanks a lot, much appreciated!
s

sam

06/18/2020, 12:59 PM
Oh 4.0.x is a big big update - rename, package changes
You might want to just jump straight to 4.1.0.RC2
g

Grigory Derzhavets

06/18/2020, 1:02 PM
I joined the existing project and didn't realise it wasn't fresh 🙂
4.1.0.RC2 
is it stable enough?
s

sam

06/18/2020, 1:52 PM
I don't think anything else will be added before the final
its just waiting on me adding a blog post
🙂 1
g

Grigory Derzhavets

06/18/2020, 1:54 PM
ok, I just switched to 4.1.0. Will report here whether it resolved my problem
Tried the @Tags annotation. What I see is this: It works good if you explicitly exclude a tag (kotest.tags.exclude=Tag_A) in the test run. In this case the whole spec with @Tags("Tag_A") is skipped and beforeSpec is not executed. However, if you run with kotest.tags.include=Tag_B, then: • for test with no @Tags annotation, the test itself will be ignored, but the beforeSpec will be executed. • the test with @Tags("some_other_tag") will be ignored and its beforeSpec won't be executed.
s

sam

06/19/2020, 1:27 PM
Can you file an issue for the error case.
g

Grigory Derzhavets

06/19/2020, 1:50 PM
s

sam

06/19/2020, 1:50 PM
Thanks
g

Grigory Derzhavets

06/22/2020, 8:47 AM
I see you fixed it. When will 4.1.0 final be released?
s

sam

06/23/2020, 1:14 PM
4.1.0 is now released including this fix
g

Grigory Derzhavets

06/23/2020, 1:24 PM
Thanks!
Hey! Looks like the whole behaviour of
-Dkotest.tags.include=
parameter has been changed after I upgraded to 4.1.0 final. I have a base test that has a base tag for all the tests.
override fun tags(): Set<Tag> = setOf(MyTag)
Before the upgrade I could execute the tests with
-Dkotest.tags.include=MyTag
and the tests were executed. Now they are not executed and the test runner doesn't find any tests. It only finds a test if I use @Tags("MyTest") annotation. So looks like this parameter only works with annotations now and doesn't work when you create an io.kotest.core.Tag objects and override them in your test specs. Is this expected behaviour?
@sam please take a look above
s

sam

06/30/2020, 12:57 PM
This is a bug with a non trivial solution. I'll endeavour to get it fixed soon.
g

Grigory Derzhavets

06/30/2020, 1:07 PM
Ok, thanks a lot
Any estimates on this one?
s

sam

07/08/2020, 1:25 PM
This one is more urgent, I should get to this one by the weekend at the latest.
👍 1
I've started work on this and I'm kind of stumped as to how to get the required behaviour. Overriding tags in a spec was never intended to apply at the spec level. As the documentation states:
/**
 * Any tags added here will be in applied to all [TestCase]s defined
 * in this [Spec] in addition to any defined on the individual
 * tests themselves.
 */
fun tags(): Set<Tag> = emptySet()
So does the system need to inspect all the root tests, and skip the spec only if none of them are active.
g

Grigory Derzhavets

07/13/2020, 11:22 AM
Let me ask you first. Is this correct use of @Tags annotation?
s

sam

07/13/2020, 2:01 PM
Yes
g

Grigory Derzhavets

07/13/2020, 2:01 PM
But the test test_1 is not executed, as displayed on the screenshot
s

sam

07/13/2020, 2:02 PM
Yep, hmm
where are you putting the kotest.tags.include, just inside the intellij run config ?
g

Grigory Derzhavets

07/13/2020, 2:04 PM
yeah, in VM options
but gonna use it with gradle test task
s

sam

07/13/2020, 2:08 PM
I know what's happening
It's looking for the tag on the Spec and finding it
then looking for the tag on the test as well
but it's not on the test, and there's no inheritance rules
There seems to be lots of edge cases here. I need to spend a day thinking through these rules. I'll probably have to deprecate the @Tags and come up with something else.
I'll see how junit does it
g

Grigory Derzhavets

07/13/2020, 2:10 PM
Thats what I wanted to suggest 🙂
Just a reminder - my initial problem was to find a way to have a beforeSpec() thing executed after tests filtering by tags
s

sam

07/13/2020, 2:11 PM
you want to avoid beforeSpec if all tests are filtered out ?
g

Grigory Derzhavets

07/13/2020, 2:14 PM
not for all, but for some
s

sam

07/13/2020, 2:15 PM
if the rule was "only run beforeSpec if at least one root test is active", then it would apply for all specs, that seems reasonable
g

Grigory Derzhavets

07/13/2020, 2:20 PM
yes, it seems so my workaround would be to have a beforeTest() with some kind of flag like
s

sam

07/13/2020, 2:21 PM
There's also prepareSpec
Might be reasonable to always run prepareSpec
I guess if you want "all tests filtered out" to mean different things depending on the Spec class, then you would have to roll something yourself. Or perhaps it could be controlled in config.
g

Grigory Derzhavets

07/13/2020, 3:09 PM
Might be reasonable to always run prepareSpec
you mean that filter out test by tag in prepareSpec?
My application under tests has different configurations, which means that I need to run different tests depending on core application configuration. These test may (or may not) have a common setup. I wanted to use Tags to distinguish these tests run and at the same time to have some common setups
s

sam

07/13/2020, 3:17 PM
That's the aim of tags for sure. You tag some tests as linux, some as windows say, and if none of them are active in a particular spec, it should skip the before spec
g

Grigory Derzhavets

07/13/2020, 6:17 PM
I just got confused. Am I still able to do this in BaseTest class
override fun tags(): Set<Tag> = setOf(MyTag)
and then execute all the tests, inherited from BaseTest ?
s

sam

07/13/2020, 6:51 PM
so any tags you add by overriding that function are applied to each test. So it's as if you added the MyTag to every test in that class
g

Grigory Derzhavets

07/14/2020, 8:43 AM
right it used to be - before the upgrade to 4.1.0 - that I ran tests in gradle with
-Dkotest.tags.include=MyTag
and my tests derived from BaseTest (with tags overriden) were executed. Now they are not executed and I am kinda confused
Here is more detailed question. Please take a look, am I doing something wrong?
and it works with kotest plugin
My guess was that the problem was in system properties propagating to test executor, but I checked it and
kotest.tags.include -> MyTag
was in system properties
Double checked Seems like the -Dkotest.tags.include=ATag only works for @Tags(“ATag”) and doesn’t work for .config(tags = setOf(ATag)). Whereas -Dkotest.tags.exclude=ATag works fine.
s

sam

07/17/2020, 12:23 PM
Still some tests to work out, but close https://github.com/kotest/kotest/pull/1568
4.1.2 has been released which should address all your issues. Thanks for the help diagnosing this.
g

Grigory Derzhavets

07/20/2020, 6:41 AM
Awesome! I like the example you added. Now everything looks good. Thank you too