It should be possible now that binary packages can be distributed. The trickiest part will be creating an xcframework from the various frameworks you need to target. There’s a wwdc video about binary packages you can watch. It’s only 8 minutes.
https://developer.apple.com/wwdc20/10147 Distribute binary frameworks as Swift packages from #wwdc20