https://kotlinlang.org logo
Title
a

Andrew O'Hara

05/03/2023, 5:20 PM
So, I'm working on a monorepo POC, trying to incorporate elements from the testing hyperpyramid and http4k-connect. I'm looking for some suggestions from the community: Monorepo Build Tools I'm using Gradle, Bitbucket Pipelines, and hoping to deploy to Amazon ECS. What tools do you suggest to deploy only affected services? I know Bitbucket Pipelines has a
changeSets
condition, but it's not smart enough to detect transitive changes, unless I explicitly define them. I'm hoping for a tool that isn't super over-engineered; something that's a component of the pipeline, rather than a replacement. Testing Hyperpyramid Say I have a layout like:
cats/
  contract/
  server/ (depends on cats:contract)
owners/
  contract/
  server/ (depends on owners:contract)
adoptions/
  contract/ (depends on cats:contract and owners:contract)
  server/ (depends on (adoptions:contract)
gateway/
  contract/ (depends on cats:contract, owners:contract, and adoptions:contract)
  server/ (depends on gateway:contract)
The
*:contract
modules declare the http4k-contract, client, and fake client. If I have an
AdoptionsContractTest
that has
Fake
and
Real
implementations, and an
adopt
operation is supposed to fail if an owner or cat doesn't exist, how complete would my
FakeAdoptionsClient
be? 1. Would it depend on
FakeOwnersClient
and
FakeCatsClient
? 2. Would you have to inform the
FakeAdoptionsClient
that a given cat or owner exists? 3. Would you completely omit this requirement from the contract? Clearly, Option 1 is the most correct, but comes off as fairly unmaintainable to me. Imagine how complex the interconnected dependencies would be once you add a
gateway
service that depends on it, or a
payment
service that depends on
owners
?
s

s4nchez

05/03/2023, 8:26 PM
What tools do you suggest to deploy only affected services?
The monorepo projects I see rely on code that can generate a standardised pipeline configuration out of a custom service descriptor. Dependencies are either defined in this high-level descriptor or generated based on build tool dependencies (e.g. running
./gradlew dependencies
), or the build is "flat" (i.e. each service is built only when itself changes, similarly to a multi-repo setup). Each setup has a trade-off, including how much the org is willing to invest in platform engineering.
declare the http4k-contract, client, and fake client.
I'm unsure what you mean by client and fake client in this context, but it seems like an unusual setup. We recommend sharing fakes for services that don't live in the monorepo (i.e. maintained by other teams or companies). Those can be in a single shared module that all services can use, and changing them should trigger the build of all services. The point here is that fakes should live closer to the clients of a service, and we usually don't expect them to be maintained by the provider. Again, I may be missing something here, as it's hard to understand what you're trying to achieve by splitting modules in such a fine-grained manner (contract vs server).
BTW, if by
contract
you're referring to the consumer/producer contract, I'd also advise against sharing that code, unless you have a mechanism to exercise the interactions between consumer/producer across multiple versions to ensure backward compatibility at runtime.
a

Andrew O'Hara

05/03/2023, 9:10 PM
Basically the idea is to have each service have a
server
module that's deployed, and a
contract
module that includes the client and fake client for that service. In my current polyrepo, I've found it cumbersome and error-prone to maintain duplicates of clients and fake clients in every service. I was hoping that a shared client could alleviate those issues without devolving into a distributed monolith. Assuming endpoints don't introduce breaking changes, I figured maintaining backwards compatibility wouldn't be any more difficult than in a polyrepo 🤷 .
I'm not sure how relevant these architectural decision are to the channel, but the project was inspired by the testing hyper pyramid.
s

s4nchez

05/04/2023, 7:40 AM
I was hoping that a shared client could alleviate those issues without devolving into a distributed monolith
Yes, sharing clients would alleviate some of this and also introduce different challenges, like potentially unnecessary build/deployments of all consumer services when a producer changes. In continuous deployment scenarios we've worked on the last few years, that'd be tricky, but maybe in manually-triggered deployment scenarios, that'd be acceptable. It's a trade-off, and you're right that regardless of this setup, one needs to be attention to runtime compatibility. I don't mind covering this topic here, especially since we've mentioned that approach in our talk. I'd also encourage trying different ways and reporting back, as I'm sure we can all learn from each other 🙂
a

Andrew O'Hara

05/04/2023, 2:33 PM
Yes, in a continuous deployment scenario, you would likely get many unnecessary deployments. Perhaps there are ways to mitigate it with different changeset filters, but there would still be a tradeoff, as you noticed.
p

Philipp Mayer

05/05/2023, 4:27 PM
I enjoyed reading that discussion and hope there will be more of those in the channel :)