Hey <@UL7PXDKL3> I'm interested in enabling click-...

# gsoc

[Akshit Sinha]

Hey @Abduqodiri Qurbonzoda [JB] I'm interested in enabling click-to-run for kotlinx-benchmark benchmarks in IntelliJ IDEA project proposal. I had a small question, whether I should design my proposal as per a plugin, or as a integration into intellij platform itself? I'm suggesting a plugin would be more easy for now as I also have some experience making IntelliJ plugins earlier, and later integrating it into IntelliJ will be much more easier and convenient for both of us.

[Abduqodiri Qurbonzoda [JB]]

Hi!
Starting with a plugin sounds good to me. You can also look at a similar plugin for JMH: https://github.com/artyushov/idea-jmh-plugin

[Abduqodiri Qurbonzoda [JB]]

Also, please make sure to play with and get familiar with the kotlinx-benchmark library. Try writing and benchmarking some Kotlin code on multiple platforms. Experiment with different benchmark configurations. Look into the Gradle tasks available for running benchmarks. Please don’t hesitate to ask questions if you need help at any stage.

[Akshit Sinha]

Hi! Thanks for the reply. I've started experimenting with various benchmark suites. Could you provide any significant project with uses different benchmark configurations for me to look at just to make sure I'm not missing anything. I'm thinking of looking at the JUnit interface available in IntelliJ for unit tests. I believe the benchmarking suite should look/feel/work similar to the JUnit interface already available in IntelliJ. Is that a good idea? Also could you provide a list of points related to the plugin itself that I should be including in my proposal? Thanks!

[Abduqodiri Qurbonzoda [JB]]

You can look at the available benchmark configuration options here:
https://github.com/Kotlin/kotlinx-benchmark/blob/master/docs/configuration-options.md
Here you can see usage examples: https://github.com/Kotlin/kotlinx-benchmark/blob/231ce03739278a5f0f9805dc2760d568dd7fe64e/examples/kotlin-multiplatform/build.gradle#L67

It is also important to read about the tasks the library Gradle plugin creates depending on how benchmarks are configured: https://github.com/Kotlin/kotlinx-benchmark/blob/master/docs/tasks-overview.md

[Abduqodiri Qurbonzoda [JB]]

And of course the library README is essential: https://github.com/Kotlin/kotlinx-benchmark/blob/master/README.md

[Abduqodiri Qurbonzoda [JB]]

I’m thinking of looking at the JUnit interface available in IntelliJ for unit tests.

It’s a good idea to look at different IntelliJ IDEA plugins. For kotlinx-benchmark, I would like the run button to offer multiple options for running the benchmark. For example, it should be possible to select the target platform for the benchmark to run on, similar to how the unit test plugin offers options in Kotlin/Multiplatform projects.

[Abduqodiri Qurbonzoda [JB]]

It might also make sense to be able to set the number of iterations and the duration of each iteration directly from the “run” panel

[Akshit Sinha]

I've looked over the README, and the docs and examples given in the main repo. I was asking is there is any significant project available just to make sure I cover everything. So the given examples/usecases should be enough pertaining to the proposal, is that correct? Anything else I should go over in my proposal?

[Akshit Sinha]

it should be possible to select the target platform for the benchmark to run on, similar to how the unit test plugin offers options in Kotlin/Multiplatform projects.

Got it. How would that work however, would we be having a system to detect the available benchmarks, show those targets only, and execute the respective target gradle task? Is that right?

[Abduqodiri Qurbonzoda [JB]]

Yes, that is how I imagine it. However, it’s worth mentioning that the exact behavior is not decided. During GSoC the mentee should suggest possible approaches and together we will discuss and select one

[Abduqodiri Qurbonzoda [JB]]

So the given examples/usecases should be enough pertaining to the proposal, is that correct?

All configuration options are documented in the repository’s documentation, and they can be utilized for various purposes. For instance, in this project, different configuration profiles are used to filter the benchmarks to run, as seen here: https://github.com/qurbonzoda/MutableMapEntryBenchmarks/blob/master/build.gradle.kts.
It is necessary because the Gradle tasks created by kotlinx-benchmark currently do not offer a CLI to filter which benchmarks to run. In contrast, when running a test Gradle task, one can specify which unit tests to execute using --tests, as detailed in the Gradle documentation: https://docs.gradle.org/current/userguide/java_testing.html#test_filtering.

Implementing such a CLI for kotlinx-benchmark might also be necessary to be able to run only a specific benchmark.

[Abduqodiri Qurbonzoda [JB]]

In this project a “fast” configuration profile is created to reduce the number of values for a benchmark parameter to ensure quick benchmark of essential API of the library: https://github.com/Kotlin/kotlinx.collections.immutable/blob/7fb0d74ea87d1f52e8d30d39d4df066a57f51e50/benchmarks/build.gradle.kts#L68

It allows frequent benchmarking of certain functionalities without the overhead of full-scale testing each time.

[Akshit Sinha]

Noted. Are there any other specific topics you would want me to go through in my proposal? And very much thanks for all the help, really appreciate it!

[Akshit Sinha]

What exactly is the relation between JMH and kotlinx-benchmark library? Does kotlinx-benchmark _harness_(no pun intended) JHM for its benchmarks? Needed to understand this to know how exactly the implementation of the plugin will differ from the idea-jmh-plugin.

[Abduqodiri Qurbonzoda [JB]]

kotlinx-benchmark utilizes JMH behind the scenes to execute benchmarks on JVM.
For more details see https://github.com/Kotlin/kotlinx-benchmark/blob/master/runtime/jvmMain/src/kotlinx/benchmark/jvm/JvmBenchmarkRunner.kt