Someone in the channel mentioned that there is a way to Shadow the jar of the compiler plugin so the same sources can be used for the IDEA one and avoid two different modules. Is that something that makes sense or you'd recommend?

Yes, that’s actually what

kotlin.serialization

plugin does, it shares a lot of logic for IDE and Compiler plugin