Hi, a bit off topic as it's not directly related t...

# squarelibraries

[Kevin Burgmann]

Hi, a bit off topic as it's not directly related to square libraries but probably still the best place to ask:

I read that Square has a Gradle project with 6000+ modules internally - can you explain how this works? Like how do you open this without IntelliJ crashing and wouldn't build take eternities...?

[Zach Klippenstein (he/him) [MOD]]

@ralf’s talk here has some information: https://www.droidcon.com/2019/11/15/android-at-scale-square/

[Zach Klippenstein (he/him) [MOD]]

Another thing I don’t think that talk mentions is that we use a script and ide plugin to generate Studio configs to only load subsets of modules. So if you’re working on a dev app, you can tell Studio to only load the modules required by that dev app.

[ralf]

Which is similar to https://github.com/dropbox/focus

[trevjones]

Big shoutout to all the dev's working on tooling on these huge projects is in order too. Them feeding info back to the AS tools team and Gradle seems to have really pushed both side of the tool stack into handling the scale better so even us mid to small size project folk still get to see a benefit. (parallel sync, config cache, etc etc)

Not a trivial thing to do all the benchmarking and profiling required to actually get the info to make/design tooling adjustments.

[Josh Friend]

Like @Zach Klippenstein (he/him) [MOD] said we get a lot of mileage out of a tool that trims down settings.gradle to only the projects required to complete sync for a given set of projects. We have an open source version of it here: https://github.com/square/dependency-explorer. This tool makes our average number of projects included in sync be "only" ~2300

[Josh Friend]

we run daily sync benchmarks on a very large project subset (~4500) using gradle-profiler and various build tooling combos of stable/beta/canary IDEs and gradle/kotlin/agp versions. if something weird pops up with that we report that to the studio team or gradle

[Josh Friend]

both google and gradle also have a skeleton project we generated with https://github.com/android/project-replicator a while ago that has worked decently in the past for debugging performance issues in a very large build. I think it is part of gradle's performance test suite in some form?

[baxter]

Also we optimized our Gradle build as much as we could, and basically got some of the beefiest Macbooks I've ever worked on...

[baxter]

I think the more interesting question is how do we handle CI, as that has to build everything... https://developer.squareup.com/blog/supercharging-continuous-integration-with-gradle/

[trevjones]

for your daily benchmarks are you just letting build scans dump the data into develocity for recording? Does your job also do the reporting for you or do you do that manually afterwards?

[trevjones]

would like to try and replicate some of this stuff. Thinking i'll just build out a dashboard using grafana using the athena stuff that DRV added

[Josh Friend]

build scans do not work well for IDE sync benchmarking, they only capture the gradle half of sync. gradle-profiler will measure both gradle duration and IDE duration. we take the csv output from gradle-profiler and send it to datadog. build scans would be additional interesting info sometimes but i havent need it regularly and if i do i just run a few syncs manually.

I also found that in benchmarks with shorter durations (i.e. incremental builds or small syncs) the build scan adds a variable amount of time to the end of the gradle invocation and i think you are also unable to use --offline if you want build scans.

[Josh Friend]

the module structure described at the top of this thread is still in use today and there are some things it does well, like maximizing gradle compilation avoidance and enforcing dependency inversion. I don't really have any objections to it in theory but 5y and 5000 new gradle projects later some practical limitations are being exposed in gradle

[Josh Friend]

about 2y ago i think, android studio started supporting parallel model building which was able to speed up a part of IDE sync pretty significantly https://developer.squareup.com/blog/celebrating-the-release-of-android-studio-electric-eel/

[Josh Friend]

what we have observed is that each gradle project takes a pretty predictable amount of time to configure and the enforced granularity of the module structure tends to keep exploding the gradle project count as time goes on, so IDE sync time keeps getting longer and longer as the project gets bigger

[Josh Friend]

gradle isolated projects promises to make this better by allowing projects to finally configure in parallel, but in my testing so far there is no performance win (yet)

[Josh Friend]

we're currently trying out an approach like this to "artificially" lower the count of gradle projects loaded in the IDE

[hfhbd]

Interesting links!
Do you have any samples about the section regarding AST transformation:

It became evident that our build logic applied dynamic transformations that were not reflected in the static build graph.

[Josh Friend]

ah yeah, so the build-logic plugins would automatically add dependencies to projects according to the module structure rule, like :impl would automatically depend on :public. since that is not explicit in the build.gradle file, statically parsing it would not reveal that dependency and made detecting affected projects impossible without re-implementing the module structure ruleset.

we had actually done some of that implementation in our internal version of dependency-explorer, but it was incomplete and made some conservative "guesses" (including extra modules that might not be needed) to be as functional (ide sync doesn't error on missing module) as possible with a relatively low complexity.

as of today we have mostly removed the implicit rules and require the dependencies to be explicit in the build files. we still haven't gone all the way to using the AST parsing for CI avoidance though because the caching mentioned in the post is very good and the maximum benefit we would get out of it has been lower than other opportunities.