Is there a way to configure the <ContentNegotiatio...

# ktor

[Andrew Steinmetz]

Is there a way to configure the ContentNegotiation plugin on ktor to filter out a particular part of response body before trying to parse it? I'm running into an unusual api that decided to add a random string of characters before the actual json object so it keeps throwing an error trying to parse the object since the few characters before the JSON object make it an invalid structure.

[Andrew Steinmetz]

{}$$ {
    // valid json object
}

essentially trying to figure out how to remove the {} $$ before the json parsing happens in the plugin

[Big Chungus]

I think you need to write a client plugin that would intersect the body before handing it over to the rest of the pipeline

[Big Chungus]

i.e. no direct link with content negotiation

[Andrew Steinmetz]

I was suspecting I might need to do that, the docs link out to this so I will try use one of these as a reference to configure the client to my use case. Not sure why this was added in the first place to the json, but thanks for the guidance!

https://github.com/ktorio/ktor/tree/main/ktor-client/ktor-client-core/common/src/io/ktor/client/plugins

[ephemient]

it looks like XSSI prevention but that's not the typical prefix for it

[ephemient]

I would also argue that if the server is sending this with Content-Type: application/json, it is wrong

[Andrew Steinmetz]

I know of cross site scripting, but not an expert. how would injecting some string before json prevent that exactly and what is the typical string?

[ephemient]

suppose you have a JSON endpoint like my.website/me which serves up some private data based on the user's cookies.
suppose there is some evil-third-party.website which some of your users may be tricked into visiting.
• evil-third-party.website can make a XHR or Fetch request to my.website/me and retrieve and read data.
◦ you can stop this with CORS.
• evil-third-party.website can use <script src="my.website/me"> to cause the browser to retrieve the content itself.
◦ you might think this wouldn't do anything useful, since it's just a JSON object that disappears as soon as it's done executing.
◦ but you'd be wrong, because evil-third-party.website can override the Object and Array constructors before the script in order to to exfiltrate data while the script is constructing the JSON object.
◦ putting intentionally bad JS in front can prevent this by stopping the script execution before anything else happens. this requires careful thought and awareness of browser/JS features in order to actually make airtight. common prefixes include things like )]}' and for(;;);. even then, you still need to be careful - avoiding the vulnerability in browsers that implement E4X may require an additional </x> or similar to be safe, for example.

[Andromadus Naruto]

I'm bookmarking this... 🦜 🙏

[ephemient]

sure, although there's better resources for learning out there like https://google-gruyere.appspot.com/

[Aleksei Tirman [JB]]

You can solve the problem this way:

val client = HttpClient(Apache)

val specialString = "{}\$\$ "
client.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
    // Use `context.request` to narrow this behavior to a specific endpoint or host

    if (body !is ByteReadChannel) return@intercept

    val cleanBody = context.writer {
        val maybeSpecialBytes = ByteArray(specialString.length)
        body.readFully(maybeSpecialBytes) // Tries to read a probable special string from the start of a body
        if (String(maybeSpecialBytes) != specialString) { // Write previously read bytes only if they aren't a special string
            channel.writeFully(maybeSpecialBytes)
        }
        body.copyTo(channel) // Leave the rest of a body as is
    }.channel

    proceedWith(HttpResponseContainer(type, cleanBody))
}

[Andrew Steinmetz]

Thanks for all the info everyone! @Aleksei Tirman [JB] you're solution did help solve the problem for my use case 😄 I tweaked it a little to be a plugin now so it can just be installed on the client, if you have any other suggestions please let me know.

class MyJsonPlugin private constructor() {

    @KtorDsl
    public class Config {

    }

    public companion object Plugin : HttpClientPlugin<Config, MyJsonPlugin> {
        override val key: AttributeKey<MyJsonPlugin> = AttributeKey("RFJsonPlugin")

        override fun install(plugin: MyJsonPlugin, scope: HttpClient) {
            scope.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
                if (body !is ByteReadChannel) return@intercept
                val cleanBody = context.writer {
                    val maybeSpecialBytes = ByteArray(SUPER_SECURE_JSON_OBFUSCATOR.length)
                    body.readFully(maybeSpecialBytes)
                    if (io.ktor.utils.io.core.String(maybeSpecialBytes) != SUPER_SECURE_JSON_OBFUSCATOR) {
                        channel.writeFully(maybeSpecialBytes)
                    }
                    body.copyTo(channel)
                }.channel
                proceedWith(HttpResponseContainer(type, cleanBody))
            }
        }

        override fun prepare(block: Config.() -> Unit): MyJsonPlugin =
            MyJsonPlugin()
    }

}

Then can use it with the client like so:

private val client = HttpClient(engine) {
        install(MyJsonPlugin)
        install(ContentNegotiation) {
            json()
        }
    }

[Aleksei Tirman [JB]]

I would add a configuration parameter to apply this logic only to specific hosts or endpoints but you can use a separate client instance anyway.