FYI, gradle supports parallel builds: https://docs.gradle.org/current/userguide/multi_project_builds.html#sec:parallel_execution

Ah didn't know that

It doesn't seem to support dependent projects, though

The coupling thing is interesting and one of the main reasons why I picked the plug-in model I did. I think Gradle is being a victim of the fact they let plug-ins modify internal structures, so all the bets are off.

If by dependent projects, you mean sub-projects or modules within a project, it does. However, they must be "decoupled". It seems to work well enough for us at work but agree that Gradle plugin interaction isn't fun

So you’re using the parallel builds with Gradle?

Yep

Have you compared the build times with sequential builds?

A co-worker went through the exercise of testing and enabling it, so I'm not sure but believe it's quite significant. Just having the test suites of different modules run in parallel is a big win. I'll ping him and see if he recalls any numbers

I can also kick off some builds when I'm next a work

when running kobalt-based builds on jenkins, you just have to have jenkins run a script, right?

that works

we just have a script “ci/ci” in every project that we run on jenkins. all jenkins tasks then look the same and its easy to try out locally

Haven't tried jenkins, I have kobalt builds running on Travis which I specify in

.travis.yml

. Same thing for Jenkins I suppose?

script: ./build-travis.sh

and `build-travis.sh`:

java -Xmx2048m -jar $(dirname $0)/kobalt/wrapper/kobalt-wrapper.jar $* assemble

why not just

./kobaltw $* assemble

?

I think I had difficulties getting that to work in the

yml

at some point, might be moot now

Some wildcards/shell stuff I recall

ah

@cedric: I did a couple of runs of one of our larger projects (40 sub-projects) and

parallel

halved the execution time from 16 minutes to 8 minutes on my laptop. The CI has more cores and RAIDed SSDs and seems to run in around 6.5 minutes.

@andrewoma: Nice. I suppose your projects have a high parallelization ratio (big projects that don't depend on each other and that can therefore run in parallel).

I just implemented a first version of a thread safe logger... That thing is hairy but beginning to work well. If all goes well, I think I'll make

--parallel

the default for Kobalt.

Actually not so much a thread safe logger than a project-logging based logger. Hard to explain.

I think I understand ... you don't want to mingle logs from different tasks

Yes, so at any time, there's only one project that's allowed to isssue live logs. All the other project logs are stored for later display.

Then when the current project ends, need to catch up the finished projects and switch to a different "current" project.

Makes sense

Very interesting problem, never faced anything like that before, I couldn't relate it to any existing algorithm or data structure that I know.

The effect is interesting too: you see one project displaying live logs, each line separated by a few seconds. Then suddenly, blam! Five project logs are dumped on the screen. Makes you think they ran instantly, but actually, you are just seeing logs of stuff that happened in the background a few minutes ago.

Hmmm ... you can't just buffer the current tasks and then log the full output from then as they occur? i.e. mingle tasks within projects, but not logs within tasks?