I’m having a bit of trouble with the gradle shadow...

# gradle

[Carter]

I’m having a bit of trouble with the gradle shadow plugin com.github.johnrengelman.shadow

I’m trying to create a fat JAR from various kotlin modules and my attempts to exclude transitive dependencies don’t seem to be working. My output JAR is truly fat at 820mb.

I’ve tried doing this, which is what I think I need:

dependencies {
    implementation(project(":brainLib")) {
        transitive = false
    }
}

But I get the error:

Could not determine the dependencies of task ':brainLib:compileKotlinJvm'.
> Could not resolve all dependencies for configuration ':brainLib:jvmCompileClasspath'.
  > Cannot change dependencies of dependency configuration ':brainLib:commonMainApi' after it has been included in dependency resolution.

I also tried this, which is supposed to be like a “keep” rule in proguard, but that didn’t work either. (I think because the huge dependencies are transitive under brainLib).

shadowJar {
    minimize {
        exclude(project(":brainLib"))
    }
}

Are there other suggestions as to what I could try?

[tapchicoma]

According to documentation excluding project dependency should be look following:

shadowJar {
   dependencies {
       exclude(project(':api'))
   }
}

Also have you considered adding :brainLib as compileOnly dependency?

[Carter]

I want to keep brainLib.

[Carter]

I just don’t want its transitive dependencies.

[Carter]

So I don’t think those options quite work.

[tapchicoma]

then I would propose to create special Gradle configuration, add to it additionally :brainLib with transitive = false and add this custom configuration to shadow jar task:

shadowJar {
  configurations = [project.configurations.compile]
}

[tapchicoma]

and change transitive :brainLib dependency to compileOnly

[Carter]

Changing the dependencies for brainLib to compileOnly sort of breaks the usage with the rest of the application. I could make it work, but it would be a royal pain since I’d have to duplicate the transitive dependencies under my test configuration and by the downstream gradle module consumer of app

brainLib is consumed in my gradle project by an app project, so the transitive dependencies are important there.

brainLib depends on dl4j, kotlin-dl, and smile but hides their implementations from downstream consumers for the most part as an implementation detail.

What I’m really trying to do now is additionally create a fat jar I can upload to Datalore for some analysis as part of debugging some of the ML models.

Kotlin for Jupyer notebooks supports dl4j, kotlin-dl, and smile as libraries so they don’t need to be bundled into the fat JAR.

[Carter]

Maybe we can start with, why is declaring this causing a build failure?

dependencies {
    implementation(project(":brainLib")) {
        transitive = false
    }
}

[Carter]

(And my examples with those ML libraries is just one example, the problem actually spreads across about a dozen different modules in my repo with various different dependencies like exposed and kotlin serialization).

[tapchicoma]

because dependencies :brainLib depends on are not available for :brainLib consumer - meaning some public types may not be available required for compilation.

[Carter]

So maybe creating a separate configuration in brainLib (and all the other modules) with compile dependencies, then consuming that configuration only in my shadow module might be a workaround.

[Carter]

Now I’ll have to figure out the magic syntax to make that work with multiplatform blob scream (since most of the project modules are multiplatform).

[tapchicoma]

or you could try to use this approach and add :brainLib with transitive depdendencies second time to implementation . No idea if it will work 🤔

[tapchicoma]

idea that you adding special configuration that should contain all shadowed depdendencies and then "common" dependencies (implamentation and testImplementation in your case) will extend from it.

[Carter]

I’ll try experimenting with this over the next few days and follow up with what I end up with.

[Carter]

I finally got this working and it ended up being simpler than I expected. The rough kotlin gradle DSL ended up being like this:

plugins {
    id("java")
    id("com.github.johnrengelman.shadow")
}

configurations {
    "implementation" {
        exclude("org.jetbrains.kotlin")
        exclude("org.jetbrains.kotlinx")
        exclude("org.jetbrains.exposed")
        
        ...
    }
}

dependencies {
    implementation(project(":brainLib"))
    implementation(project(":dbLib"))
    ...
}

[tapchicoma]

so you are excluding transitive dependencies for "implementation" configuration and your project still compiles? 🤔

[Carter]

They’re already compiled really.

[Carter]

fatLib just declares dependencies and has no code of its own. So it is consuming the outputs of those other libraries with my code, then dropping the third party dependencies.

[Carter]

Those third party dependencies are needed at runtime, which I include in a later step.

This is ultimately for a Kotlin Jupyter notebook, where I get to declare this to provide those:

%use exposed, serialization, deeplearning4j

[tapchicoma]

seems you have pretty specific case