https://kotlinlang.org logo
Join Slack
Powered by
At DoorDash we are one of the largest adopters of ...
# feed
m
At DoorDash we are one of the largest adopters of Kotlin as a backend language. I wrote a blog post with a detailed comparison of all our top choices. It also explains how we overcame some challenges to realize Kotlin as a better version of Java. https://doordash.engineering/2021/05/04/migrating-from-python-to-kotlin-for-our-backend-services/
👍 23
🔥 4
a
I am not sure, why do you need REPL for the backend, but Kotlin does have REPL (and Java has jShell as well, but Kotlin is more powerful in this regard). The REPL is built-in in IDEA and could be run in stand-alone mode, for example in https://github.com/Kotlin/kotlin-jupyter or in a new https://github.com/Kotlin/kotlin-interactive-shell
👍 1
m
Yeah someone on Reddit posted that. The idea is debugging systems in prod by calling functions, etc
With python thats quite easy as we can start the django shell and get a single model, or check the results of an experiment with a single line
The kotlin repl doesn't appear to yet have an easy way to load classpaths, etc and run my code it appears (according to a youtrack bug I found)
a
It is quite easy to load libraries both in Jupyter and in KI, just use 
@file:DependsOn
annotations, you don't even need gradle for that. But Kotlin is a compiled language, so you can't just connect to the running machine and run some arbitrary code in it. It is not safe after all. I think there are a lot of tools on JVM to run online diagnostics though.
m
And we use those as well 😄 We have heavily invested in java agents and even lower level jvm, and ebpf metrics to help us understand our systems. It's just sometimes for those long-tail issues, being able to call our code in the live environment has been useful
I will have to check out easy it would be to integrate either of those into our CI/CD+Dockerfile environment. With python it was 'free' given its nature 🙂
a
You can embedd a kotlin scripting environment in your server and it will have access to the whole classpath. I would strongly discourage doing that for security and stability reasons, but it will work just fine.
m
We don't necessarily care about it running inside the live process, as long as it can pull as the same env vars, etc (or we can fake inject the env vars form the running process if we need to 😉).
a
Well it is actually simpler than in Python because you do not need to set the environment, dependencies are resolved via Ivy.
m
with 
pipenv
thats true in python as well
a
If you can run a fake, the regular debug agent seems to do exactly whay you want. It allows to evaluate expressions in breakpoints. Of course, you can't use it for production environment.
I was also confused by your mention of building 
CoroutineContext
. Are you building it by hand?
m
At times. Before GRPC-Kotlin we had to do a lot of shimming ourself to move things to/from GRPCContext->CoroutineContext
and other similar integrations that weren't kotlin native and instead relied on ThreadLocal
a
Building 
CoroutineContext
by hand is strongly discouraged and what you were writing about is relevant only to one of its keys - Job. I tink they added something for threadLocal. But in general in Kotlin it is possible in most cases to replace it by coroutine-local.
m
Correct, there are some ways to help with that, so we were providing the shims, and sometimes teams accidentally 'cached' the output (which resulted in them trying to used cancelled Contexts).
Thats why I called it out, in case others make the same mistake 🙂
d
@Matt Anger what did you use to replace protokruft btw? Interested that anyone else used it! 😂
m
We are just using grpc-kotlin directly now.
d
Does that get rid of the hideous builder syntax?
m
Not all the way, but we've standardized on using the 
apply {}
syntax to make it easier to read
We are also migrating from grpc-java-netty to armeria as the core serving infra
so we can host GRPC+HTTP on the same port to support easier k8s health checks and prometheus metrics
d
Well glad it's working out for you. We binned grpc because quite frankly it was horrible.
e
Great article! Would love to read about what frameworks you settled on (and why) as well. Do you use any ORM? Http libraries? Dependency injection? 🙂
m
😄 It has its pain points for sure. Particularly around nullable vs notset, etc that matters for some of our endpoints. It has required a bit of re-education in how people think about their services
So we are standardizing on a combination of Guice+Armeria, with R2dbc and Standard JDBC for database connectivity, Lettuce for Redis connectivity
🙌 1
d
Yep - that's what we couldn't get around. Was thinking about wire as a better halfway house
(And testing was a disaster for us with grpc - we could never make it work well enough)
m
Our kotlin-platform team has created a series of 'fake servers' + 'fake clients' to help with testing
basically spin up an embedded grpc server that can host mocked replies. It's working pretty well for standard Req/Rsp style. Still a bit harder for streaming endpoints
d
Sounds similar to the SaaF mechanic we promote for Http4k projects. Testing through the boundary ftw.
r
At DoorDash we are one of the largest adopters of Kotlin as a backend language
Any info on size? LoC or number of services or anything like that?
r
Is not commonly used on the server side, meaning there are fewer samples and examples for our developers to use
Not sure I'd agree with this... yeah its popular on Android but plenty of people use it on the backend too: https://www.jetbrains.com/lp/devecosystem-2019/kotlin/.
@dave
Well glad it's working out for you. We binned grpc because quite frankly it was horrible.
Curious, what did you use instead?
d
We switched to strongly typed JSON marshalling with Jackson (although I'd use Moshi now). It was simpler, much more testable, less error prone, and had lots less hidieous boilerplate (even with Protokruft to take away the builder syntax). The effect on the project was marked and everyone was a lot happier, especially as we were using K8S without a service mesh at the time (which leads to persistent connection issues and you don't have to do client-side LB any more).
r
But that's just the serialization layer. You're comparing JSON with Protobuf, not gRPC.
m
It's just something you have to think of. Client-side load balancing isn't that bad as long as you use headless services in k8s
the default cluster-ip can cause havoc with a stampeding heard overwhelming your grpc (actually all http/2) services
d
I'm comparing the entire experience that we had with grpc and protobuf to a much simpler pattern that worked out better and safer and more reliably. Things like testability probelms were nothing to do with protobuf, and overall we've found a strong correlation with people who like clever things and impending maintenance nightmares. Our developer experience was horrible. YMMV of course.
d
from a perf perspective gRPC will be better (as it is a binary protocol) but unless you are running at a very large scale that perf difference might not be enough to offset some of the "pains" (like already mentioned testing)
d
💯 (needs more 00s!). One of the problems with this industry is that everyone and their dog has been convinced that their scaling requirements are "very large". 😂
➕ 1
😃 1
m
The other benefit of GRPC is that we can use annotations on the protobuf to declare behavior of the client 🙂
EX: we specify timeouts and error handling procedures using protobuf annotations
because service owners know the behavior on their systems better than those calling them
r
The other thing you get from protobuf and similar binary protocols like Avro is explicit rules around schema evolution. This is also a "big system problem" though.
7 Views
Open in Slack
PreviousNext
Powered by