What are the security concerns I'd need to take into consideration if I were to use something like JCL or a custom class loader, to dynamically load classes into my Compose Desktop application? My goal is to be able to download my jars dynamically, unload my application's old classes (by ditching the old class loader I assume), and load in my new classes dynamically effectively updating my application without closing the process.

I don't think the security concerns would be any different from any other java application that is loading classes. That is to say, not a compose-specific question. The main (only?) security concern would be if someone could somehow trick your application/classloader into loading untrusted classes instead of the expected classes you are intended to load. For example, suppose you were loading them without TLS or without verifying the remote server's certificate, someone could inject their bad code into the response and thus your application serves as an attack vector. Related, it means your SecurityManager can't be configured to disallow custom classloading (since you want to use classloading), so you aren't able to lock down your application as much as you otherwise could, but that's only an issue if you have other bugs or unaudited code which is exploitable.

Why do you want to invent OSGi again?

I wouldn't assume he is reinventing OSGi, unless you know something about his project that I missed. I'm not sure we have enough information to know what he is building. Also, OSGi is a rather bloated solution, great for companies like SAP that want job security and to charge for support contracts, not necessarily great for a small team. The few times I've encountered OSGi, I've spent (wasted) countless hours on stupid issues getting modules to load as expected. If it works for you, great, but I wouldn't necessarily assume it is a desirable solution for every project.

I looked into OSGi, and tried finding any samples of using it to swap code on the fly in a pure Java environment without having preinstalled some client on the computer. I was unable to any clear examples and ended up giving up on osgi.