Ktor relies upon kotlinx.serialization to work which doesn't support xml out of the box. There is a third party plugin for xml. I haven't used it so I cannot say how good it is. There is probably more that would need to happen at the ktor level to use a custom serializer. I'm not sure.
https://github.com/pdvrieze/xmlutil