hello! I got a requirement to create dependency bundle for my project that can be fetched from s3 in...

# gradle

[Dariusz Kuc]

hello! I got a requirement to create dependency bundle for my project that can be fetched from s3 instead of downloading dependencies from internet (i.e. build happens on containers that won’t have shared maven/gradle cache). I’m trying to create archive as

register<Zip>("createS3DependencyBundle") {
        archiveFileName.set("${project.name}-dependencies.zip")
        destinationDirectory.set(file("$buildDir/dist"))

        from(fileTree(File(gradle.gradleHomeDir, "caches/modules-2/files-2.1")))
    }

but no zip archive is created - any ideas?

[octylFractal]

did you depend on the task in any way?

[Dariusz Kuc]

i explicitly run it as gradle createS3DependencyBundle

[Dariusz Kuc]

getting build success but nothing created under build/dist

[octylFractal]

oh, I know what's up -- gradleHomeDir != gradleUserHomeDir

[octylFractal]

the former is wherever the gradle binary is stored, the latter is where things are cached

[Dariusz Kuc]

ah nice! thanks let me try it

[octylFractal]

I would note that this is pretty unreliable, it would probably be better to resolve all of your configurations and take each of the dependencies' artifact files and store that in the zip

[octylFractal]

if gradle ever changes their caching format, this will break, but using the artifact files won't

[Dariusz Kuc]

mind elaborating on how to better to resolve all of your configurations and take each of the dependencies' artifact files and store _that_ in the zip?

[Dariusz Kuc]

tldr i’m migrating our build from maven

[Dariusz Kuc]

so they were just zipping .m2/repository

[Dariusz Kuc]

things get a “little” more complicated on gradle side to do the same thing

[octylFractal]

I suppose it depends on how you're going to use it

[octylFractal]

if you just plan to unzip it on the container, I would capture at a higher level, i.e. take the entire caches folder

[Dariusz Kuc]

was planning to unzip it and drop it to that directory

[Dariusz Kuc]

doesn’t caches folder also contain execution stuff?

[octylFractal]

I think they contain caches for the kotlin DSL, but they don't contain output caches (that's all in the build directory)

[Dariusz Kuc]

I guess I need to filter *.lock files?

[octylFractal]

yes, that's probably a good idea

[Dariusz Kuc]

I’ll try that out thanks

[Rami Baksansky]

you can treat the S3 bucket as a Maven repository and use ‘maven-publish’ plugin to publish your project as a Jar/Aar to it.
https://ryanharter.com/blog/2015/06/18/hosting-a-private-maven-repo-on-amazon-s3/

[octylFractal]

I think that's a fundamentally different question -- this isn't about publishing project output, but sharing the gradle cache among multiple builds

https://docs.gradle.org/6.2/release-notes.html#shared-dependency-cache touches on something similar to this, but it's certainly more complicated

[Rami Baksansky]

oh, sorry I understood it in a different context

[Rami Baksansky]

you can use Gradle Enterprise as remote build cache btw

[Dariusz Kuc]

yeah i need to cache dependencies

[Dariusz Kuc]

pushing out artifacts is much simpler

[Dariusz Kuc]

how does remote cache handles the dependencies? i guess they are downloaded locally on build time?

[octylFractal]

build cache is only for project outputs, it still downloads dependencies to my knowledge

[Dariusz Kuc]

thats what i thought thnx

[octylFractal]

personally I would host a "close" (same host/LAN) maven mirror, rather than trying to play with storing the gradle cache, but at the same time the gradle cache is certainly faster + contains things other than dependencies

[Dariusz Kuc]

well so we do host our artyfactory but for some reason group i’m working with wanted to go with s3 solution anyway

[Dariusz Kuc]

was looking originally into just doing this -> https://discuss.gradle.org/t/need-a-gradle-task-to-copy-all-dependencies-to-a-local-maven-repo/13397/14

[Dariusz Kuc]

so do the copy of the deps and zip the m2