https://kotlinlang.org logo
Join Slack
Powered by
Hey I am using tags to filter out tests. I can not...
# kotest
g
Hey I am using tags to filter out tests. I can not find a way how to solve a problem: when some tests are filtered out by tag, I need the runner not to run 
beforeSpec/beforeTest
 methods. As I see right now, the test_executor firstly runs these and then filters out tests. Does anybody know whether it is possible in KotlinTest? Please
Or perhaps I don't understand the idea of tags and am not using it correctly
s
I think what you're really asking is, how can I skip the entire spec if all tests are filtered out
g
In my understanding a spec means a class and a spec has several tests inside it. The case: I have a BaseTest class, that has a beforeSpec which sets up something for future tests. I have 2 classes (specs) that inherits from BaseTest. Class_A has tag_A, Class_B has tag_B. If a run all tests with a tags.include=tag_A, then 
beforeSpec
will be ran twice - for Class_A, which is correct, and for Class_B, which seems like is incorrect, since execution of tests in Class_B won't happen due to tag filter
s
So the tags filter the tests, as you have seen, but the test discovery logic needs to have an instantiated class before it can find the tests. And it makes sense to run beforeSpec as soon as the class is created. That's why it works like it does.
However you can put tags on the spec itself
There's @Tag or @Tags can't remember which one. Put that on the spec and the entire spec will be skipped
g
You mean jupiter annotations, right? If yes, do they work with 
-Dkotlintest.tags.include
?
I see both @Tag and @Tags
s
No there's a kotest specific one
are you on 4.x ?
io.kotest.core.annotation.Tags
g
ah, I see, I am on 3.1.8
I will try 4.x
thanks a lot, much appreciated!
s
Oh 4.0.x is a big big update - rename, package changes
You might want to just jump straight to 4.1.0.RC2
g
I joined the existing project and didn't realise it wasn't fresh 🙂
4.1.0.RC2 
is it stable enough?
s
I don't think anything else will be added before the final
its just waiting on me adding a blog post
🙂 1
g
ok, I just switched to 4.1.0. Will report here whether it resolved my problem
Tried the @Tags annotation. What I see is this: It works good if you explicitly exclude a tag (kotest.tags.exclude=Tag_A) in the test run. In this case the whole spec with @Tags("Tag_A") is skipped and beforeSpec is not executed. However, if you run with kotest.tags.include=Tag_B, then: • for test with no @Tags annotation, the test itself will be ignored, but the beforeSpec will be executed. • the test with @Tags("some_other_tag") will be ignored and its beforeSpec won't be executed.
s
Can you file an issue for the error case.
g
s
Thanks
g
I see you fixed it. When will 4.1.0 final be released?
s
4.1.0 is now released including this fix
g
Thanks!
Hey! Looks like the whole behaviour of 
-Dkotest.tags.include=
parameter has been changed after I upgraded to 4.1.0 final. I have a base test that has a base tag for all the tests.
Copy code
override fun tags(): Set<Tag> = setOf(MyTag)
Before the upgrade I could execute the tests with 
-Dkotest.tags.include=MyTag
and the tests were executed. Now they are not executed and the test runner doesn't find any tests. It only finds a test if I use @Tags("MyTest") annotation. So looks like this parameter only works with annotations now and doesn't work when you create an io.kotest.core.Tag objects and override them in your test specs. Is this expected behaviour?
@sam please take a look above
s
This is a bug with a non trivial solution. I'll endeavour to get it fixed soon.
g
Ok, thanks a lot
Any estimates on this one?
s
This one is more urgent, I should get to this one by the weekend at the latest.
👍 1
I've started work on this and I'm kind of stumped as to how to get the required behaviour. Overriding tags in a spec was never intended to apply at the spec level. As the documentation states:
Copy code
/**
 * Any tags added here will be in applied to all [TestCase]s defined
 * in this [Spec] in addition to any defined on the individual
 * tests themselves.
 */
fun tags(): Set<Tag> = emptySet()
So does the system need to inspect all the root tests, and skip the spec only if none of them are active.
g
Let me ask you first. Is this correct use of @Tags annotation?
s
Yes
g
But the test test_1 is not executed, as displayed on the screenshot
s
Yep, hmm
where are you putting the kotest.tags.include, just inside the intellij run config ?
g
yeah, in VM options
but gonna use it with gradle test task
s
I know what's happening
It's looking for the tag on the Spec and finding it
then looking for the tag on the test as well
but it's not on the test, and there's no inheritance rules
There seems to be lots of edge cases here. I need to spend a day thinking through these rules. I'll probably have to deprecate the @Tags and come up with something else.
I'll see how junit does it
g
Thats what I wanted to suggest 🙂
Just a reminder - my initial problem was to find a way to have a beforeSpec() thing executed after tests filtering by tags
s
you want to avoid beforeSpec if all tests are filtered out ?
g
not for all, but for some
s
if the rule was "only run beforeSpec if at least one root test is active", then it would apply for all specs, that seems reasonable
g
yes, it seems so my workaround would be to have a beforeTest() with some kind of flag like
s
There's also prepareSpec
Might be reasonable to always run prepareSpec
I guess if you want "all tests filtered out" to mean different things depending on the Spec class, then you would have to roll something yourself. Or perhaps it could be controlled in config.
g
Might be reasonable to always run prepareSpec
you mean that filter out test by tag in prepareSpec?
My application under tests has different configurations, which means that I need to run different tests depending on core application configuration. These test may (or may not) have a common setup. I wanted to use Tags to distinguish these tests run and at the same time to have some common setups
s
That's the aim of tags for sure. You tag some tests as linux, some as windows say, and if none of them are active in a particular spec, it should skip the before spec
g
I just got confused. Am I still able to do this in BaseTest class
Copy code
override fun tags(): Set<Tag> = setOf(MyTag)
and then execute all the tests, inherited from BaseTest ?
s
so any tags you add by overriding that function are applied to each test. So it's as if you added the MyTag to every test in that class
g
right it used to be - before the upgrade to 4.1.0 - that I ran tests in gradle with 
-Dkotest.tags.include=MyTag
and my tests derived from BaseTest (with tags overriden) were executed. Now they are not executed and I am kinda confused
Here is more detailed question. Please take a look, am I doing something wrong?
and it works with kotest plugin
My guess was that the problem was in system properties propagating to test executor, but I checked it and 
kotest.tags.include -> MyTag
was in system properties
Double checked Seems like the -Dkotest.tags.include=ATag only works for @Tags(“ATag”) and doesn’t work for .config(tags = setOf(ATag)). Whereas -Dkotest.tags.exclude=ATag works fine.
s
Still some tests to work out, but close https://github.com/kotest/kotest/pull/1568
4.1.2 has been released which should address all your issues. Thanks for the help diagnosing this.
g
Awesome! I like the example you added. Now everything looks good. Thank you too
215 Views
Open in Slack
PreviousNext
Powered by