Hi all I was interested in doing some micro benchmarks for K

Question

Hi all I was interested in doing some micro benchmarks for Kotlin vs Java but also Kotlin 1 3 11 vs Kotlin 1 3 20 I found some earlier code based on Kotlin Hidden Costs <https sites google com a athaydes com renato athaydes posts kotlinshiddencosts benchmarks> I ve updated the code to the latest JMH and I ve ran the code for Kotlin 1 3 11 and 1 3 20 I ve found some surprising results but I might also be misinterpreting the results Is there anyone here who has some knowledge on JMH Microbenchmarks and who can a look at some of my results

karelpeeters · Answer

<http sol gfxile net dontask html> kind of applies here

bodiam · Answer

Hi Karel I m sorry you feel that way That was not what I was aiming for I just want to make sure I m not going to publish any nonsense about Kotlin without knowing the details and an extra pair of eyes would be appreciated

karelpeeters · Answer

Ah no I m not trying to put you down sorry it came across that way I d suggest just posting those results and seeing if anyone sees something in them simple smile

bodiam · Answer

Sure I have results here <https github com bodiam kotlin vs java benchmark>

bodiam · Answer

See the results directory

bodiam · Answer

I just didn t want to come across as here are my results please look at them I m more interested in learning how to interpret them myself

karelpeeters · Answer

And which ones are you surprised about

karelpeeters · Answer

A general tip is to look at the generated bytecode but you probably already knew that

bodiam · Answer

Well for one I m not sure if I completely understand the error rate and the significance of that

bodiam · Answer

Secondly if you compare the results they are quite similar except that in benchmark 1 there are 2 tests which show a doubling in throughput One could argue that for that benchmark Kotlin 1 3 20 is now twice as fast

bodiam · Answer

But is that really true

karelpeeters · Answer

The error rate means how much the result differed between runs it s basically the standard deviation from statistics Things that can cause it other OS activity the GC CPU throrlling

karelpeeters · Answer

The doubling in throughput are you talking about between versions 1 3 11 and 1 3 20 I don t see any difference at all

karelpeeters · Answer

for part 1

bodiam · Answer

you re right I was wrong I meant part 2

bodiam · Answer

for example this one `kotlinLocalFunctionWithoutCapturingLocalVariable`

karelpeeters · Answer

Okay there the change is within the margin of error so technically you can t deduce that anything has changed from that

karelpeeters · Answer

The measurements for 1 3 11 have huge margins of errors there something must have gone wrong while measuring

bodiam · Answer

Okay that was what I was wondering about When is an error rate huge

bodiam · Answer

Looking at the sample above score is 502 946 478 and the error is 3 million How bad is that

bodiam · Answer

I m more than happy to rerun both tests see what results they produce

karelpeeters · Answer

Pretty good it means you can be 95 confident that the actual speed is in the interval 500m 506m

karelpeeters · Answer

I don t know the exact JMH uses for their confidence intervals but it s going to be somewhere between 90 and 99

karelpeeters · Answer

But eg for kotlinLocalFunctionCapturingLocalVariable in 1 3 11 it means the actual speed is with 95 confidence in 183m 432m That doesn t tell you a lot does it

bodiam · Answer

it doesn t indeed slightly smiling face

bodiam · Answer

that s a pretty wide gap

karelpeeters · Answer

Were you using your computer when the benchmarks were running Did the OS updater kick in An antivirus Those kinds of things kill benchmarks

bodiam · Answer

btw where did you get the 183m 432m from

karelpeeters · Answer

The score + the error

bodiam · Answer

And no I wasn t using my computer but I might have different programs open in the background I can turn those off

karelpeeters · Answer

309m 125 8m 309 + 125 8m

bodiam · Answer

right I was looking at the 1 3 20 results Sorry

karelpeeters · Answer

Yeah thing is the 1 3 20 result is nearly in that interval so you can t really draw conclusions

bodiam · Answer

ah now I feel like I m getting it I really have to imagine a standard deviation graph in my mind

karelpeeters · Answer

Yup exactly

bodiam · Answer

Mind blown

karelpeeters · Answer

I just took a look at the generated bytecode for the 1 3 11 and 1 3 20 versions of that function and I think it s identical

bodiam · Answer

Right So there shouldn t be any performance difference

bodiam · Answer

Well now I sort of understand what I m looking at I ve loaded my results in JMH visualizer and that black line is that the error range

karelpeeters · Answer

Yes I m on the same site simple smile

bodiam · Answer

ah slightly smiling face

bodiam · Answer

Thanks If I d look at these results I might interpret it as wow Kotlin 1 3 20 has some great enhancements while realistically the result should be Erik your measurement is very wrong try again

karelpeeters · Answer

Yep

bodiam · Answer

This is why I was looking for a peer review to make sure I m not publishing anything which isn t nonsense

bodiam · Answer

is there a way to visualise this information easily into a standard deviation graph

karelpeeters · Answer

Not sure it s hard to find information about JMH I ve found

bodiam · Answer

ah I thought that was just me

bodiam · Answer

Thank for the help Karel much appreciated I ll see what I ll do with these benchmarks for now but I won t publish them that s for sure slightly smiling face

karelpeeters · Answer

My pleasure Be sure to notify me if you eventually publish something I m always interested in performance

bodiam · Answer

sure will do It will be published on <http www jworks io|www jworks io> just like my other blogposts