So, I'm working on a monorepo POC, trying to incor...

# http4k

[Andrew O'Hara]

So, I'm working on a monorepo POC, trying to incorporate elements from the testing hyperpyramid and http4k-connect. I'm looking for some suggestions from the community:

Monorepo Build Tools
I'm using Gradle, Bitbucket Pipelines, and hoping to deploy to Amazon ECS.
What tools do you suggest to deploy only affected services? I know Bitbucket Pipelines has a changeSets condition, but it's not smart enough to detect transitive changes, unless I explicitly define them.
I'm hoping for a tool that isn't super over-engineered; something that's a component of the pipeline, rather than a replacement.

Testing Hyperpyramid
Say I have a layout like:

cats/
  contract/
  server/ (depends on cats:contract)
owners/
  contract/
  server/ (depends on owners:contract)
adoptions/
  contract/ (depends on cats:contract and owners:contract)
  server/ (depends on (adoptions:contract)
gateway/
  contract/ (depends on cats:contract, owners:contract, and adoptions:contract)
  server/ (depends on gateway:contract)

The *:contract modules declare the http4k-contract, client, and fake client.

If I have an AdoptionsContractTest that has Fake and Real implementations, and an adopt operation is supposed to fail if an owner or cat doesn't exist, how complete would my FakeAdoptionsClient be?

1. Would it depend on FakeOwnersClient and FakeCatsClient?
2. Would you have to inform the FakeAdoptionsClient that a given cat or owner exists?
3. Would you completely omit this requirement from the contract?
Clearly, Option 1 is the most correct, but comes off as fairly unmaintainable to me. Imagine how complex the interconnected dependencies would be once you add a gateway service that depends on it, or a payment service that depends on owners ?

[s4nchez]

What tools do you suggest to deploy only affected services?

The monorepo projects I see rely on code that can generate a standardised pipeline configuration out of a custom service descriptor.

Dependencies are either defined in this high-level descriptor or generated based on build tool dependencies (e.g. running ./gradlew dependencies), or the build is "flat" (i.e. each service is built only when itself changes, similarly to a multi-repo setup). Each setup has a trade-off, including how much the org is willing to invest in platform engineering.

[s4nchez]

declare the http4k-contract, client, and fake client.

I'm unsure what you mean by client and fake client in this context, but it seems like an unusual setup.

We recommend sharing fakes for services that don't live in the monorepo (i.e. maintained by other teams or companies). Those can be in a single shared module that all services can use, and changing them should trigger the build of all services.

The point here is that fakes should live closer to the clients of a service, and we usually don't expect them to be maintained by the provider.

Again, I may be missing something here, as it's hard to understand what you're trying to achieve by splitting modules in such a fine-grained manner (contract vs server).

[s4nchez]

BTW, if by contract you're referring to the consumer/producer contract, I'd also advise against sharing that code, unless you have a mechanism to exercise the interactions between consumer/producer across multiple versions to ensure backward compatibility at runtime.

[Andrew O'Hara]

Basically the idea is to have each service have a server module that's deployed, and a contract module that includes the client and fake client for that service.

In my current polyrepo, I've found it cumbersome and error-prone to maintain duplicates of clients and fake clients in every service. I was hoping that a shared client could alleviate those issues without devolving into a distributed monolith. Assuming endpoints don't introduce breaking changes, I figured maintaining backwards compatibility wouldn't be any more difficult than in a polyrepo 🤷 .

[Andrew O'Hara]

I'm not sure how relevant these architectural decision are to the channel, but the project was inspired by the testing hyper pyramid.

[s4nchez]

I was hoping that a shared client could alleviate those issues without devolving into a distributed monolith

Yes, sharing clients would alleviate some of this and also introduce different challenges, like potentially unnecessary build/deployments of all consumer services when a producer changes. In continuous deployment scenarios we've worked on the last few years, that'd be tricky, but maybe in manually-triggered deployment scenarios, that'd be acceptable.

It's a trade-off, and you're right that regardless of this setup, one needs to be attention to runtime compatibility.

I don't mind covering this topic here, especially since we've mentioned that approach in our talk. I'd also encourage trying different ways and reporting back, as I'm sure we can all learn from each other 🙂

[Andrew O'Hara]

Yes, in a continuous deployment scenario, you would likely get many unnecessary deployments. Perhaps there are ways to mitigate it with different changeset filters, but there would still be a tradeoff, as you noticed.

[Philipp Mayer]

I enjoyed reading that discussion and hope there will be more of those in the channel :)