https://kotlinlang.org logo
Join Slack
Powered by
Is it possible to benchmark some code in a way tha...
# test
m
Is it possible to benchmark some code in a way that's not bound to a specific machine? I'd like to monitor performance regressions in CI but cannot make any assumption about what type of VM runs the code. Is something like this possible?
m
I think the main hurdle is to extract the generated artefacts right? Have you a way to persist them?
m
My main trouble is to find a metric that's not bound to the cpu of the machine running the benchmark
Typically you would get performance benchmark in milliseconds
But that doesn't work if the VM you're running on has very inconsistent capabilities
Ideally I'd use something like 'number of jvm instructions executed' but I have no idea if that's even possible...
m
Ah now I get you. Tough one.
Ideally I'd use something like 'number of jvm instructions executed' but I have no idea if that's even possible...
That sounds like code coverage and there are tools for it, like Jacoco.
m
Does Jacoco count the same "instruction" executed several times though? It only cares about coverage, not total time does it?
m
It does not care about time and collects code paths you walk through, but you are right, I think it does not double count stuff only the total number of instructions.
m
The java-agent + asm approach is interesting though. I'll dig a little bit
m
You mean something like bit-buddy? But think about it, if you probe the instructions you might taint the actual outcome.
Do you need several benchmarks?
m
if you probe the instructions you might taint the actual outcome.
I don't mind if it's slower as long as it's "constantly" slower
bit-buddy is similar to asm, right? It's a way to rewrite the byte code at runtime
So I could intercept each class loaded and rewrite the body of all its methods to introduce an instruction after each instruction 🙃 (to increase a counter)
m
Yes, a connivence tool to build things on runtime
m
This is going to produce a lot of instructions (basically double the size of my bytecode) but maybe I don't care
m
But if you add the code on runtime, why you not add it in the first place (just play devil advocate here)? If you "simply" want to add an additional instructions you could even do it via compiler plugins in Kotlin, I suppose.
m
Ah yes, good point, compiler plugin might work too
Although I'm not so sure. I really want something lowlevel
m
And if you do it correctly you have a KMP right away
This might help
👀 1
I also dig into this topic right now for my little project
m
I'm no expert but my understanding is that the Kotlin compiler produces some intermediate representation (or clang bytecode)
I want to plug in the backend more than the frontend (because clang might still optimize)
Doing it at the bytecode level I'm sure I'm not missing any transformation
m
If you fear optimisation in native - you can always put locks around stuff. Its a promise that compiler do not apply any optimisation on locked code sequences (at least n theory).
And sure with bytecode you can be sure, that it runs in the way you intended it.
With the legacy jvm Compiler I think you can write the bytecode also directly, IR indeed uses a intermediated representation.
m
I think I'll go bytecode. My main use case is the JVM in all cases
(If I ever go 😅 because rewriting all these classes doesn't sound like a trivial task)
Wait, bytecode isn't good... Of course different instructions have different timings which "could" be ok to get a raw estimate of the program complexity.
But then there's JNI and bytecode counters will not help at all in the native code
m
Nope
m
And I'm pretty sure significant parts of the runtime are native 😕
So it has to count the number of CPU cycles not Java bytecodes
m
And then you are back at your starting point
m
Yea kind of...
Damn benchmarking is hard
m
The big question is then if you can turn any potential compiler optimisation off, right?
m
I don't know... There is also JVM JIT, etc...
Maybe instruction count isn't that bad after all...
m
So only measure JVM?
In any case I am very excited to hear with what you end up...sounds like a pretty amazing problem
👍 1
m
THere is some prior art. I bumped into this: https://sdqweb.ipd.kit.edu/publications/pdfs/kuperberg2008a.pdf
m
Thx
In the end you will do something very similar to JaCoCo, at least the paper sounds like it.
➕ 1
Maybe you can borrow from there - how they probe instructions. This might save you a lot of time, and they already have some Kotlin specific stuff pulled in where maybe can benefit from as well.
👀 1
🙏 1
s
I dont think there is any real solution other than keeping the hardware fixed. This is what https://www.techempower.com/benchmarks/#section=environment does.
👀 1
9 Views
Open in Slack
PreviousNext
Powered by