What would be the reason behind only allowing public constructors on inline classes?

Because you wouldn't be able to truly restrict Java code from "instantiating" the class.

But that seems similar to internal visibility which is public in the java side

True, keep in mind that inline classes are experimental and subject to change. The KEEP calls them "current limitations": https://github.com/Kotlin/KEEP/blob/master/proposals/inline-classes.md#current-limitations

I forgot about the KEEP, thanks I'm gonna read through it