Is there a way to configure the ContentNegotiation plugin on ktor to filter out a particular part of response body before trying to parse it? I'm running into an unusual api that decided to add a random string of characters before the actual json object so it keeps throwing an error trying to parse the object since the few characters before the JSON object make it an invalid structure.

I think you need to write a client plugin that would intersect the body before handing it over to the rest of the pipeline

I was suspecting I might need to do that, the docs link out to this so I will try use one of these as a reference to configure the client to my use case. Not sure why this was added in the first place to the json, but thanks for the guidance! https://github.com/ktorio/ktor/tree/main/ktor-client/ktor-client-core/common/src/io/ktor/client/plugins

it looks like XSSI prevention but that's not the typical prefix for it

I know of cross site scripting, but not an expert. how would injecting some string before json prevent that exactly and what is the typical string?

suppose you have a JSON endpoint like

my.website/me

which serves up some private data based on the user's cookies. suppose there is some

evil-third-party.website

which some of your users may be tricked into visiting. •

evil-third-party.website

can make a XHR or Fetch request to

my.website/me

and retrieve and read data. ◦ you can stop this with CORS. •

evil-third-party.website

can use

<script src="my.website/me">

to cause the browser to retrieve the content itself. ◦ you might think this wouldn't do anything useful, since it's just a JSON object that disappears as soon as it's done executing. ◦ but you'd be wrong, because

evil-third-party.website

can override the

Object

and

Array

constructors before the script in order to to exfiltrate data while the script is constructing the JSON object. ◦ putting intentionally bad JS in front can prevent this by stopping the script execution before anything else happens. this requires careful thought and awareness of browser/JS features in order to actually make airtight. common prefixes include things like

)]}'

and

for(;;);

. even then, you still need to be careful - avoiding the vulnerability in browsers that implement E4X may require an additional

</x>

or similar to be safe, for example.

I'm bookmarking this... :60fps_parrot: :thank-you:

sure, although there's better resources for learning out there like https://google-gruyere.appspot.com/

You can solve the problem this way:

val client = HttpClient(Apache)

val specialString = "{}\$\$ "
client.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
    // Use `context.request` to narrow this behavior to a specific endpoint or host

    if (body !is ByteReadChannel) return@intercept

    val cleanBody = context.writer {
        val maybeSpecialBytes = ByteArray(specialString.length)
        body.readFully(maybeSpecialBytes) // Tries to read a probable special string from the start of a body
        if (String(maybeSpecialBytes) != specialString) { // Write previously read bytes only if they aren't a special string
            channel.writeFully(maybeSpecialBytes)
        }
        body.copyTo(channel) // Leave the rest of a body as is
    }.channel

    proceedWith(HttpResponseContainer(type, cleanBody))
}

Thanks for all the info everyone! @Aleksei Tirman [JB] you're solution did help solve the problem for my use case 😄 I tweaked it a little to be a plugin now so it can just be installed on the client, if you have any other suggestions please let me know.

class MyJsonPlugin private constructor() {

    @KtorDsl
    public class Config {

    }

    public companion object Plugin : HttpClientPlugin<Config, MyJsonPlugin> {
        override val key: AttributeKey<MyJsonPlugin> = AttributeKey("RFJsonPlugin")

        override fun install(plugin: MyJsonPlugin, scope: HttpClient) {
            scope.responsePipeline.intercept(HttpResponsePipeline.Receive) { (type, body) ->
                if (body !is ByteReadChannel) return@intercept
                val cleanBody = context.writer {
                    val maybeSpecialBytes = ByteArray(SUPER_SECURE_JSON_OBFUSCATOR.length)
                    body.readFully(maybeSpecialBytes)
                    if (io.ktor.utils.io.core.String(maybeSpecialBytes) != SUPER_SECURE_JSON_OBFUSCATOR) {
                        channel.writeFully(maybeSpecialBytes)
                    }
                    body.copyTo(channel)
                }.channel
                proceedWith(HttpResponseContainer(type, cleanBody))
            }
        }

        override fun prepare(block: Config.() -> Unit): MyJsonPlugin =
            MyJsonPlugin()
    }

}

Then can use it with the client like so:

private val client = HttpClient(engine) {
        install(MyJsonPlugin)
        install(ContentNegotiation) {
            json()
        }
    }

I would add a configuration parameter to apply this logic only to specific hosts or endpoints but you can use a separate client instance anyway.