Hello. Does kotlinx.serialization support Java's EnumSet or do I need to create custom serializer for them?

If your property type is EnumSet then yes. Otherwise if you're using an EnumSet and assigning to a Set property it will work just fine.

What format will be used for Set type?

It will be serialized as a collection in whatever format you're using

If I were to implement KSerializer for EnumSet, can I write it as generic? I guess not since I wont know max size of ordinal in the Enum inside EnumSet, right?

It should be possible to do it generically for any enum

I am trying to write such KSerializer in a generic way, but I am having trouble creating EnumSet instances in deserialize() when the type of the enum inside of it is erased (ie. when using EnumSet<*> or <T: Enum<T>> with <EnumSet<T>>). Could you please point me into the right direction / suggest an approach?