https://kotlinlang.org logo
Join Slack
Powered by
I'm trying to run ktor server for the first time a...
# server
b
I'm trying to run ktor server for the first time as a jar in the production environment. I package it up as a fatJar and test it on the local environment. It works when given the configuration file. In production it will run and my signup endpoint succeeds in creating a user. My next request to the login handle hangs the server. I only see an error from ctrl-c exiting the process.
Copy code
2024-06-20 02:36:52.013 [main] INFO  ktor.application - Application started in 3.795 seconds.
2024-06-20 02:36:52.477 [DefaultDispatcher-worker-1] INFO  ktor.application - Responding at <http://0.0.0.0:8080>
2024-06-20 02:36:58.192 [eventLoopGroupProxy-4-1] TRACE io.ktor.routing.Routing - Trace for [login]
/, segment:0 -> SUCCESS @ /
  /(RateLimit login), segment:0 -> SUCCESS @ /(RateLimit login)
    /(RateLimit login)/login, segment:1 -> SUCCESS @ /(RateLimit login)/login
      /(RateLimit login)/login/(method:POST), segment:1 -> SUCCESS @ /(RateLimit login)/login/(method:POST)
  /(RateLimit signup), segment:0 -> SUCCESS @ /(RateLimit signup)
    /(RateLimit signup)/signup, segment:0 -> FAILURE "Selector didn't match" @ /(RateLimit signup)/signup
  /(authenticate auth-jwt), segment:0 -> SUCCESS @ /(authenticate auth-jwt)
    /(authenticate auth-jwt)/(RateLimit KTOR_NO_NAME_RATE_LIMITER), segment:0 -> SUCCESS @ /(authenticate auth-jwt)/(RateLimit KTOR_NO_NAME_RATE_LIMITER)
      /(authenticate auth-jwt)/(RateLimit KTOR_NO_NAME_RATE_LIMITER)/dashboard, segment:0 -> FAILURE "Selector didn't match" @ /(authenticate auth-jwt)/(RateLimit KTOR_NO_NAME_RATE_LIMITER)/dashboard
  /.well-known, segment:0 -> FAILURE "Selector didn't match" @ /.well-known
Matched routes:
  "" -> "(RateLimit login)" -> "login" -> "(method:POST)"
Route resolve result:
  SUCCESS @ /(RateLimit login)/login/(method:POST)
2024-06-20 02:36:58.201 [eventLoopGroupProxy-4-1] TRACE i.k.s.plugins.ratelimit.RateLimit - Using rate limit RateLimitName(name=login) for /login
2024-06-20 02:36:58.203 [eventLoopGroupProxy-4-1] TRACE i.k.s.plugins.ratelimit.RateLimit - Using key=kotlin.Unit and weight=1 for /login
2024-06-20 02:36:58.216 [eventLoopGroupProxy-4-1] TRACE i.k.s.plugins.ratelimit.RateLimit - Allowing /login
2024-06-20 02:36:58.315 [eventLoopGroupProxy-4-1] TRACE i.k.server.engine.DefaultTransform - No Default Transformations found for class io.ktor.utils.io.ByteBufferChannel and expected type TypeInfo(type=class com.XXXXXXXXXXX.models.Auth, reifiedType=class com.XXXXXXXXXXX.models.Auth, kotlinType=com.XXXXXXXXXXX.models.Auth) for call /login
^C2024-06-20 02:37:04.047 [eventLoopGroupProxy-4-1] TRACE i.k.s.p.statuspages.StatusPages - Call /login failed with cause com.auth0.jwk.NetworkException: Cannot obtain jwks from url <http://XXX.XX.XX.XXX:8080/.well-known/jwks.json>
2024-06-20 02:37:04.048 [eventLoopGroupProxy-4-1] TRACE i.k.s.p.statuspages.StatusPages - No handler found for exception: com.auth0.jwk.NetworkException: Cannot obtain jwks from url <http://XXX.XX.XX.XXX:8080/.well-known/jwks.json> for call /login
2024-06-20 02:37:04.051 [eventLoopGroupProxy-4-1] ERROR ktor.application - Unhandled: POST - /login
com.auth0.jwk.NetworkException: Cannot obtain jwks from url <http://XXX.XX.XX.XXX:8080/.well-known/jwks.json>
	at com.auth0.jwk.UrlJwkProvider.getJwks(UrlJwkProvider.java:139)
	at com.auth0.jwk.UrlJwkProvider.getAll(UrlJwkProvider.java:145)
	at com.auth0.jwk.UrlJwkProvider.get(UrlJwkProvider.java:163)
	at com.auth0.jwk.RateLimitedJwkProvider.get(RateLimitedJwkProvider.java:28)
	at com.auth0.jwk.GuavaCachedJwkProvider.lambda$get$0(GuavaCachedJwkProvider.java:62)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4925)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3571)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2313)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2190)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2080)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4012)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4920)
	at com.auth0.jwk.GuavaCachedJwkProvider.get(GuavaCachedJwkProvider.java:62)
	at com.XXXXXXXXXXX.routes.LoginKt$login$1.invokeSuspend(Login.kt:39)
	at com.XXXXXXXXXXX.routes.LoginKt$login$1.invoke(Login.kt)
	at com.XXXXXXXXXXX.routes.LoginKt$login$1.invoke(Login.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invokeSuspend(Route.kt:116)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:131)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:89)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:109)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.routing.Routing.executeResult(Routing.kt:190)
	at io.ktor.server.routing.Routing.interceptor(Routing.kt:64)
	at io.ktor.server.routing.Routing$Plugin$install$1.invokeSuspend(Routing.kt:140)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:131)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:89)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invokeSuspend(BaseApplicationEngine.kt:124)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:131)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:89)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invokeSuspend(CommonHooks.kt:45)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:78)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:264)
	at io.ktor.server.application.hooks.CallFailed$install$1.invokeSuspend(CommonHooks.kt:44)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:131)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:89)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:109)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:123)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:131)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:89)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:109)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:140)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:44)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:112)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:126)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:41)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:33)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:425)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion.create$lambda$1$lambda$0(NettyApplicationEngine.kt:296)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/java.net.Socket.connect(Socket.java:583)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:947)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1710)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1611)
	at com.auth0.jwk.UrlJwkProvider.getJwks(UrlJwkProvider.java:135)
	... 87 common frames omitted
🧵 12
I've tried some things so far. Moving from using the 
jwks.json
file inside a certs folder to just a file in the root directory. That doesn't work. I've explicitly chown the directory and files. No change. I'm not sure why this works locally and adding the 
local.properties
file to the same directory as the jar loads that information, but 
jwks.json
won't seem to load. Java versions are identical.
On my local machine I have:
Copy code
jwt_issuer=<http://localhost:8080/>
jwt_audience=<http://localhost:8080/hello>
in 
local.properties
and in production I changed this to the IP address
Screenshot from 2024-06-19 21-57-30.png,Screenshot from 2024-06-19 21-53-39.png
As you can see the server finds the file just fine, so the error seems nonsensical? Signup is successful. Why the hang and error on the login? I have no idea
Copy code
fun Route.login(usersQueries: UserQueries, jwkProvider: JwkProvider, privateKeyString: String,
                audience: String, issuer: String) {
    post("/login") {
        val auth = call.receive<Auth>()
        // Check email exists and return hashed password
        var pwHash = usersQueries.findByEmail(auth.email).executeAsList().toString()

        if(pwHash == "[]") {
            call.respond(HttpStatusCode.Unauthorized, "Email not found")
        }
        // remove leading and trailing brackets from pwHash
        pwHash = pwHash.dropLast(1)
        pwHash = pwHash.drop(1)
        // check password against hash in database
        val validPw = BCrypt.checkpw(auth.password, pwHash)
        // we have found email and verified password
        if (validPw) {
            val publicKey = jwkProvider.get("MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAusmK9bSA4sQDJR5LpR5wL9oEoSke7xfi9MRT7OvzSakOzHm/PvB0vWBW17K7l/wMX/JTHZtjtDARmNq3bNi0JMk+on9ryWyYtaAnOFcQF7xuwS5knO0wpoBraWTFGPH5Iunr+t9nG0J4IdUD+NSl ep6C12JJN5cTe5hlPDe2dKS6RAVo0HsBpw4bmIX+sLBR6kxq70OayI0x0CDN0vfC++yIW+FLUcBU4wjL7R7aGrLiugzNKZX73M7ZplCSG1E9XsMKeAcnXTq+U/eWsdJqKUpEGwkqQoFl9yuhnYtr/E16l18srFxab7RZP5tUuYXLQl22hJEbXnuGKK03e9XFqQIDAQAB").publicKey
            val keySpecPKCS8 = PKCS8EncodedKeySpec(getDecoder().decode(privateKeyString))
            val privateKey = KeyFactory.getInstance("RSA").generatePrivate(keySpecPKCS8)
            val token = JWT.create()
                .withAudience(audience)
                .withIssuer(issuer)
                .withClaim("email", auth.email)
                .withExpiresAt(Date(System.currentTimeMillis() + 57600000)) // 16 hours
                .sign(Algorithm.RSA256(publicKey as RSAPublicKey, privateKey as RSAPrivateKey))
            call.respond(hashMapOf("token" to token))
        } else {
            call.respond(HttpStatusCode.Unauthorized, "Password is invalid")
        }
    }
This is super shitty and I'm wasting a lot of time trying to get ktor to make any sense in a production environment. Just tried to get it to work on a AWS EC2 instance. It's even worse. This was effortless with Spring.
a
Did you redact this URI? 
com.auth0.jwk.NetworkException: Cannot obtain jwks from url <http://XXX.XX.XX.XXX:8080/.well-known/jwks.json>
b
Yes
a
What kind of IP is it? Public or private?
b
It's a standard public IPV4
a
A standard IPv4 can be public or private. Private IPs are physically inaccessible to computers in a different network. To test if this IP is accessible to the server, you can SSH into the machine your ktor server is running on, and try to call that URI with curl.
Copy code
$ curl -v <http://xxx.xx.xx.xxx:8080/.well-known/jwks.json>
There's also EC2 security group rules. Even if the IP is physically accessible, the security group firewall attached to the JWKS server MUST allow inbound traffic to TCP port 8080 from your ktor server's IP or security group.
b
It's accessible. I've allowed port 8080 to all IP (0.0.0.0) for incoming trafficj
Confirmed with curl
When I say an IP is publicly accessible, I mean it. You don't have to reiterate that there's a difference. I started out as sysadmin
💩 5
a
I'll leave this to the ktor experts then.
b
I went line by line commenting out and making 
println()
statements. The route hangs at 
jwkProvider.get
this is bizarre. I can curl the 
jwks.json
file from the ec2 instance terminal and from my local terminal. That's two different compute providers giving the same issue. Digitalocean and Amazon. Not sure how to debug this in production
Is there anything about this guide: https://ktor.io/docs/rsa-keys-generation.html#generating-an-rsa-private-key that would encode information to the original machine? I went through it a second time on the production rig and have the same result. Don't think this keypair should tie down to a rig. Not sure what else to check at this point
I can even hardcode the remote 
jwks.json
file location for my local rig and it'll pull the file locally to build the jwkProvider. This is a really strange bug. I increased read timeout on the builder to 10s. Just stalls until it hits the read timeout and gives the same error that it can't find the file at the location. Why can it be found every which way except inside the program?
At this point think what I'm going to do is copy all the source files, build it remotely, then try to run it as if it was in the IDE. If that works, then something is borked with packaging this as a fatJar and trying to run it remotely. I'll look to package it a different way
Bizarre. Same result when compiling the source files on either remote rig. I'm missing something basic
I moved my configuration file to the standard 
application.conf
hoping that would somehow change things. I'm out of ideas 🤷‍♂️ the error doesn't appear to match what's going on. In every which way except outside of the application, I can access and use the jwks.json file at the server location. Just not by the application at a remote compute instance.
Resolved. Use https endpoints for authorization servers 😛 Ever so slightly less ignorant
e
what was your solution exactly?
7 Views
Open in Slack
PreviousNext
Powered by