How is the standard output captured and sent to In...
# testballoon
c
How is the standard output captured and sent to IntelliJ? I'm seeing many tests with standard output partially missing.
👀 1
o
TestBalloon does not interfere with output processing. Is output really missing or does it appear in the whole task’s output?
m
How does it even work in Junit? If you have 2 test processes, each one running N tests, I'm guessing the output has to be interleaved?
(not missing though)
c
It does appear in the whole task output, but doesn't appear when clicking on a specific test.
👍 1
For JVM platforms, I get some output in the test-specific view, but not always all of it, and I'm not sure what makes things appear or not there. For non-JVM platforms, I don't get any output in the test-specific view.
o
These are buffering issues. Test output might not make it through output buffers when the end of a test is signaled. So the test frameworks cannot reliably assign output to the correct test. It becomes even more complicated with concurrent execution. This is basically a flaw of the global state processing model for standard I/O. Of course, one could synchronize output with test events. But that would mean paying a rather high price by slowing down tests. That’s why there’s buffering. Coroutines-based output could be a solution. Either by intercepting all standard output calls (if suitable APIs exist on all platforms) and collecting it in coroutine-local buffers, or by special output functions to that effect.
b
It does appear in the whole task output, but doesn't appear when clicking on a specific test.
Not TestBalloon-specific, though. I've observed that behaviour with Kotest since forever, IIRC
m
Yea, I don't think this can reasonnably work unless each function has its own test process, which is probably overkill
c
At some point it does need to work though, you can't really have a good dev UX without seeing the stdout of a test.
m
Unless you ask every test to not use
System.out
but another logging API I don't see how that could work.
You could Kotlin compiler plugin intercept all the calls to
System.out
(and
open(1)
) but that sounds like a nightmare...
c
Well, all other test frameworks that I can name, no matter the language, are able to do this.
m
And tests running in parallel?
c
That I know fewer that can do, but JUnit and Pytest can
Rust test, too
m
Junit has the same problem. I tested it the other day
🤔 1
You need to give it very specific option to run test functions in parallel, let me check.
Copy code
@Execution(ExecutionMode.CONCURRENT)
By default only classes are run in parallel
It's a mess, you have: • processes • classes • function
And I always get confused which one runs in parallel to each one
But if 2 functions execute concurrently, I would expect
System.out
to show interleaved output. This is a feature actually, this is how the system would behave in production.
For all matters,
System.out
is like a network socket or something else, there's nothing special about it, it's just external state to your test.
c
But if 2 functions execute concurrently, I would expect
System.out
to show interleaved output. This is a feature actually, this is how the system would behave in production.
True, but two different tests should be independent from each other, and the test report infra should be able to show only the output of a given test.
m
They are not
As soon as you use
System.out
, you have external state
It's write-only state so it won't affect your test execution much but it does actually. If you have a lot of concurrent writers, the timing of your
System.out.println()
will change so it will affect the execution of your test
(But the same is true about your CPU so probably not too much of an issue)
c
Sure, the test isn't actually independent, but that's the abstraction tests are supposed to provide. Many other things are happening on the system may also have slight impacts on the timing, but that shouldn't matter to a test framework.
(but it does actually matter for benchmark frameworks, I tried porting criterion-rs to Kotlin and it's basically impossible to get an accurate timing of a short-running process, Gradle, its workers, etc introduce way too much noise)
m
that's the abstraction tests are supposed to provide
I'm not sure about that. I think there is a little bit of a Stockholm syndrome here. We are used to this IJ test UI showing individual test output but it has been "training" us wrong and it has been showing wrong results probably forever.
Look at Logcat on Android, you need a specific
Logcat
API and then there is a single output console where you can filter. This is the actual model
Your test framework "could" probably systrace/JVM agent/instrument your tests to clone stdout but I'm not sure I would want a test framework to do this for me implicitely. I'd rather have this being explicit by calling a dedicated logging API
o
I agree with the idea of having a dedicated test-local logging API.
🤝 2
Picking this thing up: For completeness, JUnit has not solved the problem of test-attributable console output: github.com/junit-team/junit-framework/issues/4323
Considering a test-scoped logging API: Suppose we'd have something like this:
Copy code
test("top 1") {
    testOutput("hello from top 1 before delay")
    delay(2.seconds)
    testOutput("hello from top 1 after delay")
}
Then test-attributable output would be output that's always reported in the context of the test it belongs to. (Details in 🧵)
In contrast, global console output would be reported at the time it "appears". In concurrency scenarios, such output may show up along with reporting about an unrelated test, and may interleave in unpredictable ways. Then this is what's currently possible in the existing test infrastructure with a
testOutput
function: • Test-attributable console output: Android (S), Native (TC), Wasm/WASI (TC). • Global console output: JS/Browser, JS/Node, Wasm/JS/Browser, Wasm/JS/Node, JVM. • Test-attributable output as extra data in HTML and XML reports: JVM (but not via Kotlin/Multiplatform due to KT-89413). With the new Browser DSL for JS (using Playwright) coming to Kotlin 2.5, we could add: • Test-attributable (instead of global console) output: JS/Browser, Wasm/JS/Browser. (S) = The platform requires forced sequential execution. (TC) = Output is controlled by TestBalloon's integrated TeamCity protocol reporting. Notable: There is currently no IDE support for direct display of concurrent, yet test-attributable output. So any test-attributable output is either the result of • sequential test execution, or • TestBalloon doing the proper sequencing (like with its TeamCity reporting), or • extra data (HTML, XML) not shown by the IDE. Thoughts?
c
My long-term goal for Prepared is to have an integration with logging frameworks such that Prepared captures the logging output of regular code and can attribute it back to the current tests and pass that to the underlying framework. Personally I think it's more important to be able to click a test node in the IDE tree and see its actual output than having the interleaved real-time output in the default pane. If it's possible to support that, I don't mind if TestBalloon reorganizes the order of independent tests if it helps IntelliJ display them