https://kotlinlang.org logo
Join Slack
Powered by
Hey I am really interested in studying the source ...
# compiler
m
Hey I am really interested in studying the source code, but I am finding it very laborious to grasp what's going on by just reading the source at https://github.com/JetBrains/kotlin . I can understand the concepts like IRs, parsing, ..., but I don't grasp the folder structures, filenames and ideas regarding encapsulation. Any ideas on how to proceed? TBH I'm finding it odd how such code can get into big corp's stack, but maybe I'm misled by its organization. I'd particularly like to understand how the devs understand what is where.
d
I'd particularly like to understand how the devs understand what is where
Mostly it's experience sharing We don't have much documentation (well, in most cases it doesn't exists), which is sad, but it's the life
m
Like what is the difference between backend and backend-common. And then things come rough when we come to container, daemon, serialization etc.
I'm very surprised such code gets into a multi-billion $$$ ecosystem of Google's.
d
Well, Kotlin repo includes a lot of different subsystems, including compiler (with two frontends and six backends), number of libraries (like stdlib, reflect, kotlin-test), number of compiler plugins, support for build systems (including maven, gradle and intellij) And each of this big subsystems is split to some parts for better code organisation, useful code sharing between different parts and so on So I don't actually understand with what you are surprized
I'm very surprised such code gets into a multi-billion $$$ ecosystem of Google's
Did you hear that all google products are stored in single repositoy?
m
that when a person like me comes "hey I want to know how this works", then I have no idea where to start even
or how do I know there are two frontends and six backends
since there are three backend folders, rather than two
d
Kotlin is not a teaching project, it is complex production product So there is no main goal to make everything understandable for 5 minutes
m
Or why do I find UnreachableCode, when I look into frontend-common.psi, which I thought was to have the PSI code generation of frontend common to two frontends (Java and JS)
But do you really want to rather sit in Slack explaining every newbie over the same things?
The code itself looks quite clear, but it's just difficult to know how the folders are related and how the classes and functions relate
b
I'm very surprised such code gets into a multi-billion $$$ ecosystem of Google's
lol if you think kotlin repo is messy you probably shouldn't look at aosp šŸ˜€
Took me a better part of three months just to get a basic understanding of how to start working with it. Most of it via word-of-mouth knowledge sharing.
d
but it's just difficult to know how the folders are related and how the classes and functions relate
That's why I suggested you start with some entrypoint and follow the code instead of reading everything by folder
b
There's a point when a project becomes simply too big to expect each single dev to fully understand it. At that stage trying to document it is just not an efficient use of time.
m
yes, but in that case I want to know where are bits where one can begin to work backwards and trust that what's backwards just works
d
two frontends (Java and JS)
There are no Java and JS frontends. There are K1 and K2 frontends and JVM, JS, Native, WASM backends I think the better starting point to understand compiler structure (not project structure) is watch some talk like

thisā–¾

one
m
yeah I understand how compilers in general work
but here are e.g. some very confusing folders under compiler folder:
but here are e.g. some very confusing folders under compiler folder: builtins-serializer/src/org/jetbrains/kotlin/serialization/builtins compiler-runner-unshaded
but here are e.g. some very confusing folders under compiler folder: builtins-serializer/src/org/jetbrains/kotlin/serialization/builtins compiler-runner-unshaded compiler-runner container daemon incremental-compilation-impl ...
d
You want to understand how compiler works or what each folder in project does? It took three years of full-time job for me do get at least briefly familiar with each subsystem
m
how the project is laid out. Yes and that's too much.
But I don't know how are there going to be new devs, if it takes three years to understand what to do.
Are you professional compiler dev though?
I'm not aiming that, but I just wanted to know what I'm using
Where's the entry point after CLI?
d
That's the point of splitting project into different modules For each big subsystem we have separate team which can work independently from other and can more or less don't care about how another subsystem works This simplifies the learning curve for new devs
Are you professional compiler dev though?
Yes, I'm one of main developers of frontend part of Kotlin compiler
m
okay so you just suggest that I pick some part that interest me most, and start with that?
d
Where's the entry point after CLI?
ā€¢ this line triggers frontend to resolve all the files ā€¢ this one converts Frontend IR tree into Backend IR ā€¢ this line triggers backend
m
I was initially interested in getting the JS part
but I don't know how is that not going to require dwelving into backend or something
or perhaps the usage of IR means the parts are truly independent
d
ā€¢ main entrypoint to JS compiler ā€¢ start of JS IR backend
m
is the IR same
d
Yes, basic idea of backend IR is to unify representation of code between all backends, to be able to share common logic between them Also it allows to use klibs, which are basically serializaed IR
m
btw is the IR fixed in design? So that if one implements the JS backend, then the IR that's supported is the same as now?
d
It is more or less fixed in design, but not fixed in implementation (which means that there are no stability guarantees of IR API) But it doesn't affect backedns, because all of them are declared in same repository, and if, for example, dev from JVM team changes IR and it breaks JS backend, he should fix it before pushing changes to 
master
m
what does -common refer to in the repo (backend and frontend)?
BTW I would probably had prefixed the folders so that they would appear in order of usage, not in alphabetic order
now it's odd to see "preloader" at the bottom
a
@Matti Viljamaa Have you seen the attempt at adding a Python backend to Kotlin? While Iā€™m sure JetBrains helped, it wasnā€™t an official attempt, so I think investigating the progress could give a good example (even if the project is not active any more) https://github.com/krzema12/kotlin-python/tree/python-backend/python
p
hi, kotlin-python author here šŸ™‚ yeah, it was a challenge to get started developing the backend because the repo is fairly complex, but trial and error, reading other backendsā€™ code and asking JetBrains super-helpful folks for help did the job! weā€™re on pause with developing the Python backend mostly because the dev iteration loop was taking too long given my experience with the code base (still more trial and error than actual understanding)
regarding all these remarks about naming and structure: while I get the sentiment because I like keeping things in proper order, itā€™s naive to think that Kotlin compiler devs will spend most of their time on polishing the code base. Unfortunately the harsh reality is that thereā€™s often too little time, too small team and generally too little resources to have proper quality in a fast-paced and high-exposure project. Itā€™s always a matter of trade-off to deliver VS keep proper quality
on a positive note regarding Kotlin compilerā€™s quality: ā€¢ I admire the box tests infrastructure - itā€™s great to prevent regressions. We also used them in kotlin-python to quantify our progress ā€¢ ā€œAlmost makes me want to go back to Java just because it seems so amateurish.ā€ - have you seen Javaā€™s code base, and how much garbage and accidental complexity is there?
m
At least the Java compiler is documented
p
there are various videos on Kotlin compiler high-level architecture from its developers, I treat them as documentation to some extent
all other problems besides documentation from the graph: I think Kotlin and its community really shines here
a
I note some irony in you sharing an untitled graph that has no reference or context. Where's the 'documentation' for that graph?
m
^This is a problem of semantic web tbh. https://opensourcesurvey.org/2017/#insights
yeah well, when I was at nim-lang they also had pushy devs that seemed disinterested in newbies
a
Incomplete or outdated documentation is a pervasive problem, observed by 93% of respondents, yet 60% of contributors say they rarely or never contribute to documentation. When you run into documentation issues, help a maintainer out and open a pull request that improves them.
m
I'm happy to realize that it's not intentional though. I can surely appreciate the approach of also providing use value as opposed to doc value.
106 Views
Open in Slack
PreviousNext
Powered by