Need some API advice. I've added `UTF-8` encoding...

# library-development

[Matt Nelson]

Need some API advice.

I've added UTF-8 encoding/decoding to a library of mine. I have 2 replacement strategies for invalid sequences available from commonMain, and am wondering which I should use for the default.

class UTF8: EncoderDecoder {

    class ReplacementStrategy private constructor {
        companion object {
            // Encodes/decodes UTF-8 the same way Kotlin/JVM does.
            val U_003F // ...

            // Encodes/decodes UTF-8 the same way Kotlin Js/WasmJs/Wasi/Native does.
            val U_FFFD // ...

            // Initalizes to U_003F on Jvm, and U_FFFD on all other platforms.
            val KOTLIN // ...

            // Throw on invalid sequences
            val THROW // ...
        }
    }
}

The current default that I am using is UTF8.ReplacementStrategy.KOTLIN, but wondering if that is the best choice before publishing this feature release.

Kotlin's String.decodeToByteArray() and ByteArray.encodeToString() implementations are different on non-JVM platforms than JVM, producing different results; the above pseudo-code outlines that.

Context: okio and kotlinx-io UTF-8 implementations are the equivalent of UTF8.ReplacementStrategy.U_003F

Thoughts?

[Matt Nelson]

https://github.com/05nelsonm/encoding/blob/master/library/utf8/src/commonMain/kotlin/io/matthewnelson/encoding/utf8/UTF8.kt

[cketti]

People rarely think about edge cases. Replacing invalid UTF-8 sequences could hide the problem and make debugging much harder. So I'd make THROW the default.

[jessewilson]

I disagree. If you throw decoding a string, you've created a very easy mechanism to do an accidental DoS

[jessewilson]

Get a malformed string into one field of your database, and that data becomes unreadable

[jessewilson]

Best to thoroughly reject bad strings when they enter your system (forbid bad data, \0, etc.) and accept that you'll probably not be able to prevent them entirely

[Matt Nelson]

Very solid points, thanks Jesse.

I ended up going with UTF8.ReplacementStrategy.KOTLIN for the UTF8.Default static companion object instance, and also as the default value for UTF8.Builder . Added an additional UTF8.ThrowOnInvalid static object instance configured with the THROW strategy.