the ktor httpclient fails when used concurrently, ...

# ktor

[coder82]

the ktor httpclient fails when used concurrently, returns an incorrect response looks like, really strange

[Deactivated User]

Do you have more information so we can check? Ktor version? How many concurrent requests? Exception thrown, or result returned? And ideally if you have some code to reproduce the problem would help too

[coder82]

well I have to extract the code and create a separate project

[coder82]

I am load testing it with testng parallel execution

[Deactivated User]

Which engine are you using?

[coder82]

applicationEngineEnvironment

[coder82]

do you mean how i start the whole server?

[Deactivated User]

I mean, Apache, Jetty or CIO?

[coder82]

ah sorry, the httpClient engine

[Deactivated User]

when you construct the HttpClient

[coder82]

Apache

[Deactivated User]

okay

[coder82]

I tried all 3 of them, they all fail give the same exception

[coder82]

essentially they return properly but then i check the response returned, the JSON is intermingled

[coder82]

and as soon as I parse it I get this:
java.lang.IllegalArgumentException: JSON at 506: Expected string or non-null literal

[Deactivated User]

aha

[coder82]

indeed the JSON is non well-formed

[coder82]

I isolated it like this:
try {
val ret = httpClient.get<String>(serviceUrl)
unfilteredResultsCache[params.campusId] = JSON.nonstrict.parse(Container.serializer(KlubschuleCalendar.serializer()), ret)
} catch(e:Exception) {
rootLogger.error(e)
}

[coder82]

sometimes it throws the exeption... especially if I run in parallel

[Deactivated User]

the endpoint is a ktor-based endpoint?

[coder82]

yes

[Deactivated User]

are you testing only the HttpClient

[Deactivated User]

or both?

[coder82]

the httpClient calls ktor itself

[Deactivated User]

(to isolate the problem, maybe it is a problem with the server or the client)

[Deactivated User]

have you tried wireshark to see if the generated content is right?

[coder82]

no, should I ?

[Deactivated User]

which engine are you using for the server?

[Deactivated User]

no, just asking 🙂 to delimit the issue

[coder82]

i am http requesting through loopback, will wireshare see it?

[Deactivated User]

not on windows as far as I remember, but I think it works on mac

[Deactivated User]

still, not required at this point

[coder82]

i am on windows

[Deactivated User]

okay

[Deactivated User]

which engine are you using for the server?

[Deactivated User]

Netty, Jetty, CIO, …?

[coder82]

it's really strange the json returned its like this {,{,{,{,{,}

[coder82]

Netty

[Deactivated User]

aha

[Deactivated User]

that’s strange

[Deactivated User]

are you using the ContentNegotiation feature to generate the json response?

[coder82]

no, because I generate only a String, then later I parse it with kotlinx serializer

[Deactivated User]

aha

[coder82]

i get the JSON as a String and then use kotlinx to parse it

[Deactivated User]

the json serializer that are you using is thread-safe?

[coder82]

i asked and yes they said it's state-less

[Deactivated User]

nice

[Deactivated User]

and have you dumped to a file the generated JSON string before responding to the call? (to see if it is malformed at that point and if it is a problem at the server or the client)

[Deactivated User]

if you can reproduce it in a small project, I can try in both a windows machine and a mac machine

[coder82]

yes i validate the json before sending it, and when i send it it's valid

[Deactivated User]

aha

[coder82]

call.respond(validateJson(str))

[coder82]

i also checked that

[coder82]

that's why my attention focused on the httpClient

[coder82]

really really weird

[Deactivated User]

well still, it could be a problem in the server when sending the HTTP content somehow.

[Deactivated User]

Have you tried HttpClient with CIO to see if the problem still reproduces?

[coder82]

could be

[coder82]

yes I tried CIO and problem still happens

[Deactivated User]

that’s really strange

[coder82]

i keep investigating, eventually i isolate the case for u

[Deactivated User]

if you can create a project reproducing it, it would help a lot

[coder82]

sure

[Deactivated User]

fyi @e5l

[coder82]

aha i have to return one thing i said

[coder82]

actually the response is already invalid before call.respond

[coder82]

ok i look deeper and let u know

[coder82]

just making sure it's not me

[coder82]

eheheh

[e5l]

Hi, @coder82. Could you provide an example how you execute the requests in parallel?

[coder82]

ok is the stringify causing the issue

[coder82]

when i call it before calling call.respond

[coder82]

JSON.nonstrict.stringify(Container.serializer(KlubschuleCalendar.serializer()), a)

[coder82]

so essentially there must be an issue in there which i could isolate and make it nicer to debug

[Deactivated User]

call.respond(validateJson(str)) so the validateJson was not working fine?

[coder82]

exactly

[coder82]

i noticed the exception happens way less frequently if I use JSON.nonstrict.stringify instead of just JSON.stringify

[Deactivated User]

Ok. Maybe something needs to be synchronized. And that could be noise because timings are different

[coder82]

inside the JSON class u mean? maybe...

[coder82]

but it's a dataclass should all be on it's own, unless it uses some shared state

[Deactivated User]

I don’t know, in your code in general. If I don’t see the code. Maybe you have something shared that you are not aware of

[coder82]

ok i'll try to reproduce

[Deactivated User]

http://ktor.io/servers/configuration.html#embedded-server

You can try to adjust the connectionGroupSize to maybe 1? But probably it is related to something shared. Or that serializer. Have you tried Jackson or something instead of that JSON class (that I don’t know where it comes from)?

[coder82]

it's inside here:

[coder82]

package kotlinx.serialization.json

[Deactivated User]

oh

[coder82]

it's the official kotlinx JSON class

[coder82]

it works wonders, but as soon as I started loading with test ng... sometimes it stringifies invalid json

[Deactivated User]

I see, that additional information is relevant 🙂 maybe it is a bug from there

[coder82]

could be

[Deactivated User]

I would try with Jackson just to see if the problem reproduces. If it doesn’t reproduce, then I would investigate further in that direction and would report to #serialization . I can help reporting if you manage to reproduce the problem in an isolate project

[coder82]

ok and by the way now i returned back where i asked about serializer thread safety and it seems other ppl have the same issue

[coder82]

https://github.com/Kotlin/kotlinx.serialization/issues/53#issuecomment-395746295

[coder82]

wow looks like the JSON class is not thread-safe

[coder82]

found it..

[coder82]

finally

[coder82]

i run already 1M tests so I am sure

[coder82]

basically JSON object has static fields, if u use it in a projects from multiple threads, like in my case were i fire quite a lot of load to my servers

[coder82]

then it's shared state essentially... u change stuff from one thread while that thing is parsing on the other, then it cracks

[coder82]

the solutions was suggested by sand

[coder82]

when multithreaded, instantiate it locally, like this JSON(nonstrict = true).stringify(...)

[coder82]

and not JSON.nonstrict.stringify(....)

[coder82]

nonstrict is static...

[Deactivated User]

Nice! glad you found the issue 🙂 thanks for the report!

[coder82]

yes basically inside:
package kotlinx.serialization.json

[coder82]

there is a private variable:

[coder82]

modeCache

[coder82]

that's shared if u mean such object as singleton

[coder82]

indeed all threads were using 1 JSON object, but there is one state

[coder82]

I am looking in the master now, apparently they moved the state out to make it thread-safe