https://kotlinlang.org logo
#ktor
Title
# ktor
a

Andrew Steinmetz

07/20/2022, 12:16 AM
Is there a way to configure the ContentNegotiation plugin on ktor to filter out a particular part of response body before trying to parse it? I'm running into an unusual api that decided to add a random string of characters before the actual json object so it keeps throwing an error trying to parse the object since the few characters before the JSON object make it an invalid structure.
Copy code
{}$$ {
    // valid json object
}
essentially trying to figure out how to remove the
{} $$
before the json parsing happens in the plugin
b

Big Chungus

07/20/2022, 12:21 AM
I think you need to write a client plugin that would intersect the body before handing it over to the rest of the pipeline
i.e. no direct link with content negotiation
a

Andrew Steinmetz

07/20/2022, 12:23 AM
I was suspecting I might need to do that, the docs link out to this so I will try use one of these as a reference to configure the client to my use case. Not sure why this was added in the first place to the json, but thanks for the guidance! https://github.com/ktorio/ktor/tree/main/ktor-client/ktor-client-core/common/src/io/ktor/client/plugins
e

ephemient

07/20/2022, 12:25 AM
it looks like XSSI prevention but that's not the typical prefix for it
I would also argue that if the server is sending this with Content-Type: application/json, it is wrong
a

Andrew Steinmetz

07/20/2022, 4:08 AM
I know of cross site scripting, but not an expert. how would injecting some string before json prevent that exactly and what is the typical string?
e

ephemient

07/20/2022, 4:46 AM
suppose you have a JSON endpoint like
my.website/me
which serves up some private data based on the user's cookies. suppose there is some
evil-third-party.website
which some of your users may be tricked into visiting. •
evil-third-party.website
can make a XHR or Fetch request to
my.website/me
and retrieve and read data. ◦ you can stop this with CORS. •
evil-third-party.website
can use
<script src="my.website/me">
to cause the browser to retrieve the content itself. ◦ you might think this wouldn't do anything useful, since it's just a JSON object that disappears as soon as it's done executing. ◦ but you'd be wrong, because
evil-third-party.website
can override the
Object
and
Array
constructors before the script in order to to exfiltrate data while the script is constructing the JSON object. ◦ putting intentionally bad JS in front can prevent this by stopping the script execution before anything else happens. this requires careful thought and awareness of browser/JS features in order to actually make airtight. common prefixes include things like
)]}'
and
for(;;);
. even then, you still need to be careful - avoiding the vulnerability in browsers that implement E4X may require an additional
</x>
or similar to be safe, for example.
🙏 1
a

Andromadus Naruto

07/20/2022, 6:13 AM
I'm bookmarking this... 🦜 🙏
e

ephemient

07/20/2022, 6:24 AM
sure, although there's better resources for learning out there like https://google-gruyere.appspot.com/
a

Aleksei Tirman [JB]

07/20/2022, 8:18 AM
You can solve the problem this way:
Copy code
val client = HttpClient(Apache)

val specialString = "{}\$\$ "
client.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
    // Use `context.request` to narrow this behavior to a specific endpoint or host

    if (body !is ByteReadChannel) return@intercept

    val cleanBody = context.writer {
        val maybeSpecialBytes = ByteArray(specialString.length)
        body.readFully(maybeSpecialBytes) // Tries to read a probable special string from the start of a body
        if (String(maybeSpecialBytes) != specialString) { // Write previously read bytes only if they aren't a special string
            channel.writeFully(maybeSpecialBytes)
        }
        body.copyTo(channel) // Leave the rest of a body as is
    }.channel

    proceedWith(HttpResponseContainer(type, cleanBody))
}
🙏 2
a

Andrew Steinmetz

07/20/2022, 5:16 PM
Thanks for all the info everyone! @Aleksei Tirman [JB] you're solution did help solve the problem for my use case 😄 I tweaked it a little to be a plugin now so it can just be installed on the client, if you have any other suggestions please let me know.
Copy code
class MyJsonPlugin private constructor() {

    @KtorDsl
    public class Config {

    }

    public companion object Plugin : HttpClientPlugin<Config, MyJsonPlugin> {
        override val key: AttributeKey<MyJsonPlugin> = AttributeKey("RFJsonPlugin")

        override fun install(plugin: MyJsonPlugin, scope: HttpClient) {
            scope.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
                if (body !is ByteReadChannel) return@intercept
                val cleanBody = context.writer {
                    val maybeSpecialBytes = ByteArray(SUPER_SECURE_JSON_OBFUSCATOR.length)
                    body.readFully(maybeSpecialBytes)
                    if (io.ktor.utils.io.core.String(maybeSpecialBytes) != SUPER_SECURE_JSON_OBFUSCATOR) {
                        channel.writeFully(maybeSpecialBytes)
                    }
                    body.copyTo(channel)
                }.channel
                proceedWith(HttpResponseContainer(type, cleanBody))
            }
        }

        override fun prepare(block: Config.() -> Unit): MyJsonPlugin =
            MyJsonPlugin()
    }

}
Then can use it with the client like so:
Copy code
private val client = HttpClient(engine) {
        install(MyJsonPlugin)
        install(ContentNegotiation) {
            json()
        }
    }
a

Aleksei Tirman [JB]

07/20/2022, 5:26 PM
I would add a configuration parameter to apply this logic only to specific hosts or endpoints but you can use a separate client instance anyway.
👍 1
6 Views