Does anyone have any tips for decreasing app start times on the JVM? Ktor is fast for a JVM framewor...

# ktor

[spierce7]

Does anyone have any tips for decreasing app start times on the JVM? Ktor is fast for a JVM framework, but the JVM is pretty slow.

[jw]

What's the goal? Start fast but be slower over the lifetime? Or start fast and stay fast? Or start fast and quit after 1 request? Something else?

[Dominik Sandjaja]

You might be interested in GraalVM:
https://ktor.io/docs/graalvm.html

[jw]

You can use epsilon GC if you're short lived. You can use AppCDS to preload classes. You can use CIO instead of Jetty/Netty which doesn't take multiple seconds to start. You can disable C2 if you're short-lived.

[spierce7]

@jw Fast start and stay fast - Running JVM Server in serverless environment.

[spierce7]

@Dominik Sandjaja I paid a lot of attention to graalvm, but I've found maintaining it and getting everything working with it (especially netty) a HUGE pain. Almost worse than proguard.

[Dominik Sandjaja]

[...] stay fast - Running [...] in serverless environment

Isn't this a bit of a contradiction? Isn't the whole idea to only handle a single request and then quickly shut down again?

[spierce7]

@Dominik Sandjaja Take a look at https://cloud.google.com/run

It automatically scales with your traffic. The Goal is to startup and then handle more concurrent requests until the traffic spike subsides.

[spierce7]

You can use CIO instead of Jetty/Netty which doesn't take multiple seconds to start.

@jw This is interesting. I've been hesitant to use CIO, and stayed away from it because I assumed no one else was using it. Plus why would I use it when I could use something like Netty 😛 Maybe Netty is the problem though.

[jw]

I use CIO but not for anything heavy. It much easier to make a minimal JRE with jlink using CIO than Jetty/Netty.

[spierce7]

I guess the reason I'm not using GraalVM is because of Netty now that I think about it

[spierce7]

CIO looks to have worse performance than netty, but I guess that's the tradeoff

[Dominik Sandjaja]

I guess it makes sense to evaluate whether you do benefit from the performance benefits of a longer-running JVM vs. an amazingly quickly starting application.
Depending on your usecase you could maybe even create two application variants, one for the longer running "baseline" and a graalVM based version that handles peak traffic.
But this is now more an architectural discussion than a ktor one.

[spierce7]

For now I'll give CIO a shot and also look into AppCDS.

[Casey Brooks]

JB started working on a ktor-compatible serverless toolkit, Kotless, but it looks like it hasn’t been updated in a while. But it mentions (among other features) AWS Lambda auto-warming

[jw]

that name physically hurts

[psh]

Surely cant be any worse than the (circa 2016) “Gogland” that thankfully was renamed

[Andrew O'Hara]

Here's some tips to keep serverless start times low.

1. Minimize JAR size. Eliminate bloated libraries like jackson, log4j, spring, guava, official aws sdk, and kotlin-reflect
2. Minimize config fetching. If your app needs to make external calls to get secrets and other config params, embed them as env variables instead
3. Optimize JVM AoT JiT compilation for cold-starts. Having fast start and fast run would be having your cake and eating it too. Set this env variable: JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
4. I have no idea why and how you're running Netty in a Lambda to serve your traffic. You would ideally use an adapter to convert Ktor to the serverless programming model, without running an embedded server. This can apparently be done with Kotless (as shown above), but http4k's smaller footprint and modular architecture gives it an an edge in serverless environments. https://www.http4k.org/guide/reference/serverless/
5. Eliminate reflection on app init. No AoP or DI tools (unless done at compile-time)

[jw]

Optimize JVM AoT compilation for cold-starts.

This does not do anything for AOT. This is disabling C2. Both C1 and C2 are JIT compilers since they run at runtime, not ahead-of-time.

[jw]

Both aspect-oriented and DI libraries can be used provided they run at compile-time and not runtime.

[Andrew O'Hara]

This does not do anything for AOT

Yes. I meant JiT. Corrected.

Both aspect-oriented and DI libraries can be used provided they run at compile-time and not runtime.

Good point

[spierce7]

I have no idea why and how you're running Netty in a Lambda to serve your traffic.

lol! I'm not. hahaha I'm using Cloud Run. Cloud run lets you run a docker container, and then it stays running. Cloud Run scales at the docker container level. IMO it's much better than something like functions or lambda.

[spierce7]

Having fast start and fast run would be having your cake and eating it too.

It'd be nice if I could focus on fast starts, and then transition to fast run.

[spierce7]

V8 does a pretty nice job of starting way faster than Java, and then having similar performance.

[jw]

yeah it's unfortunate you can't stop at tier 1 for a while and then enable C2 (tier 4) after some period of time (60 seconds). You could set the tier 2 and 3 thresholds to be equal to tier 4 and then set tier 4 at some level that approximates the amount of time your container would be running at 100% load for some period of time

[spierce7]

It looks like you can change VM parameters at runtime - com.sun.management.HotSpotDiagnosticMXBean#setVMOption(String, String)

I don't know if it would actually have an effect

[spierce7]

You could set the tier 2 and 3 thresholds to be equal to tier 4 and then set tier 4

@jw What thresholds?

[jw]

it's basically the number of invocations before the next level of compilation kicks in

[spierce7]

how do I set those?

[jw]

if you set 1 to 0 (immediately JIT compile all methods with no profiling data) and 2, 3, and 4 to, say, 10,000 (or some number) then C2 which is the most advanced JIT compilation

[jw]

uhhhhh -Xint:Tier2Compilation=10000?

[jw]

i have to google

[spierce7]

I'll google for it

[jw]

this seems to have all the infohttps://www.baeldung.com/jvm-tiered-compilation

[spierce7]

interesting, I remember reading about this a while back

[spierce7]

I've been reading about this for a while. I'm really surprised how I can't find actual documentation for this.

I'm trying to understand the difference between CompileThreshold and Tier2CompileThreshold.

For some reason CompileThreshold default is 10k, but Tier2CompileThreshold default is 0 with Tier3CompileThreshold at 2k.

Tier1CompileThreshold doesn't exist, and based on the defaults, I'd assume that CompileThreshold isn't Tier1.

[spierce7]

@jw So I could do:
Tier2CompileThreshold=10000
Tier3CompileThreshold=10000
Tier4CompileThreshold=10000

But how would I set Tier1 to 0? Based on the defaults, it seems like CompileThreshold is not Tier1.

[jw]

I'm not sure. Maybe it can't be controlled? Or maybe it kicks in right away anyway?

[spierce7]

It looks like Netty finally fixed their graalvm issue yesterday - https://github.com/netty/netty/pull/13158