https://kotlinlang.org logo
Join Slack
Powered by
How is the standard output captured and sent to In...
# testballoon
c
How is the standard output captured and sent to IntelliJ? I'm seeing many tests with standard output partially missing.
👀 1
o
TestBalloon does not interfere with output processing. Is output really missing or does it appear in the whole task’s output?
m
How does it even work in Junit? If you have 2 test processes, each one running N tests, I'm guessing the output has to be interleaved?
(not missing though)
c
It does appear in the whole task output, but doesn't appear when clicking on a specific test.
👍 1
For JVM platforms, I get some output in the test-specific view, but not always all of it, and I'm not sure what makes things appear or not there. For non-JVM platforms, I don't get any output in the test-specific view.
o
These are buffering issues. Test output might not make it through output buffers when the end of a test is signaled. So the test frameworks cannot reliably assign output to the correct test. It becomes even more complicated with concurrent execution. This is basically a flaw of the global state processing model for standard I/O. Of course, one could synchronize output with test events. But that would mean paying a rather high price by slowing down tests. That’s why there’s buffering. Coroutines-based output could be a solution. Either by intercepting all standard output calls (if suitable APIs exist on all platforms) and collecting it in coroutine-local buffers, or by special output functions to that effect.
b
It does appear in the whole task output, but doesn't appear when clicking on a specific test.
Not TestBalloon-specific, though. I've observed that behaviour with Kotest since forever, IIRC
m
Yea, I don't think this can reasonnably work unless each function has its own test process, which is probably overkill
c
At some point it does need to work though, you can't really have a good dev UX without seeing the stdout of a test.
m
Unless you ask every test to not use 
System.out
but another logging API I don't see how that could work.
You could Kotlin compiler plugin intercept all the calls to 
System.out
(and 
open(1)
) but that sounds like a nightmare...
c
Well, all other test frameworks that I can name, no matter the language, are able to do this.
m
And tests running in parallel?
c
That I know fewer that can do, but JUnit and Pytest can
Rust test, too
m
Junit has the same problem. I tested it the other day
🤔 1
You need to give it very specific option to run test functions in parallel, let me check.
Copy code
@Execution(ExecutionMode.CONCURRENT)
By default only classes are run in parallel
It's a mess, you have: • processes • classes • function
And I always get confused which one runs in parallel to each one
But if 2 functions execute concurrently, I would expect 
System.out
to show interleaved output. This is a feature actually, this is how the system would behave in production.
For all matters, 
System.out
is like a network socket or something else, there's nothing special about it, it's just external state to your test.
c
But if 2 functions execute concurrently, I would expect 
System.out
to show interleaved output. This is a feature actually, this is how the system would behave in production.
True, but two different tests should be independent from each other, and the test report infra should be able to show only the output of a given test.
m
They are not
As soon as you use 
System.out
, you have external state
It's write-only state so it won't affect your test execution much but it does actually. If you have a lot of concurrent writers, the timing of your 
System.out.println()
will change so it will affect the execution of your test
(But the same is true about your CPU so probably not too much of an issue)
c
Sure, the test isn't actually independent, but that's the abstraction tests are supposed to provide. Many other things are happening on the system may also have slight impacts on the timing, but that shouldn't matter to a test framework.
(but it does actually matter for benchmark frameworks, I tried porting criterion-rs to Kotlin and it's basically impossible to get an accurate timing of a short-running process, Gradle, its workers, etc introduce way too much noise)
m
that's the abstraction tests are supposed to provide
I'm not sure about that. I think there is a little bit of a Stockholm syndrome here. We are used to this IJ test UI showing individual test output but it has been "training" us wrong and it has been showing wrong results probably forever.
Look at Logcat on Android, you need a specific 
Logcat
API and then there is a single output console where you can filter. This is the actual model
Your test framework "could" probably systrace/JVM agent/instrument your tests to clone stdout but I'm not sure I would want a test framework to do this for me implicitely. I'd rather have this being explicit by calling a dedicated logging API
o
I agree with the idea of having a dedicated test-local logging API.
🤝 2
3 Views
Open in Slack
Previous
Powered by