Well, for most types of artifacts you could try the Gradle build cache, but that doesn't work for KMP Xcode Frameworks. The Framework output has some kind of soft file linking inside that breaks with Gradle build caching, so for Frameworks specifically, the KMP Gradle plugin disables Gradle build caching.
In GitHub Actions, you can cache the build directory (and I think a few other things), then Gradle should see that you don't need to rebuild the framework, assuming there aren't any changes that would require rebuilding the framework. We experimented with this kind of thing in CI 2-3 years ago. You can get it to work, but we haven't really done it since.