<@U0RM4EPC7> and I have started a POC for KEEP-87 ...

# kontributors

[raulraja]

@simon.vergauwen and I have started a POC for KEEP-87 https://github.com/Kotlin/KEEP/pull/87 and I’m trying to find my way around the compiler code base. Are there any docs as to how things are architected, organized etc? I have been digging through source files for a couple of days now not making much progress. Any recommendation as to where to look for:
- New modifier extension in grammar. (grammar/src/modifiers.grm) What are other relevant places that need to change for the grammar to be valid Kotlin syntax?
- Function/Class/Interface args scope resolution. Interested as to where it is decided which extension functions apply and what the this and parent scopes refer to.
- Call scope resolution tests: What is an effective way to test my changes to resolution are correct or similar tests I can use as base for the type class definition and resolution tests?

In general I’m also interested if anyone has stories to tell as to how they go about adding a similar feature to the compiler, what is the flow usually looks like when working in this project. Ultimately if anyone wants to help please contact me, thanks!

[mikhail.zarechenskiy]

- New modifier extension in grammar. (grammar/src/modifiers.grm) What are other relevant places that need to change for the grammar to be valid Kotlin syntax?

I believe it's easier to show similar changes in the compiler for other modifiers, please check out these changes
https://github.com/JetBrains/kotlin/commit/6e2ef9b1d2d5834c69ca44567faac77dd5dad476
how to serialize/deserialize modifiers:
https://github.com/JetBrains/kotlin/commit/db1f03958629794b6715663cdc8a743cb1500888

[raulraja]

Thanks!

[dsavvinov]

Are there any docs as to how things are architected, organized etc?

Unfortunately, no, there's no such docs at the moment

Function/Class/Interface args scope resolution. Interested as to where it is decided which extension functions apply and what the this and parent scopes refer to.

Generally, you will need to understand two things:
- how scopes structure is built: nice starting point for that is ClassResolutionScopesSupport
- how this scope structure is then used for names resolution. Entry point for that is NewResolutionOldInference.runResolution, which will lead to large subsystem of resolution.

Call scope resolution tests: What is an effective way to test my changes to resolution are correct or similar tests I can use as base for the type class definition and resolution tests?

There is testData/diagnostics/resovle tests: though that's not all about resolution, I think it's more than enough to begin with. You can use them for two things: a) sanity check that you haven't break existing semantics
b) as source of inspiration for your tests with typeclasses

As for optimal way to dig through sources: I think, step-debugging simple cases is, probably, the most efficient way to grasp what compiler does. Obviously, you'll need to know where to place first breakpoint (trying to step from the very beginning of compiler invocation is quite insane) -- I've already pointed to some crucial parts, but feel free to ask for additional help 🙂

[raulraja]

@dsavvinov thanks so much, this is very helpful

[tomasruizlopez]

Hi @dsavvinov, I've been trying to implement this proposal, but I'd appreciate some additional help/guidance. So far, I've added the two new keywords and the parser is able to compile correctly, but with no semantics given to those keywords. However, running IntelliJ with this new version of the compiler throws an error which, after some investigation, comes from an assertion in ModifierMaskUtils where it is stated that a 32bit Int is used to create a mask to represent keywords for modifiers. With our two new keywords with and extension, the number of keywords is 34, breaking this assertion. Could you let us know the implications of this?

Besides this, I have been trying to run the tests in ResolveTestGenerated, which makes use of multiple .resolve files, but unfortunately they don't run. When I try to run them, I get java.lang.RuntimeException: Could not find installation home path. Please reinstall the software., whereas when I try to run other tests, such as parsing ones in ParsingTestGenerated, I can run them normally. Any hints about this?

Finally, I've been checking the classes you suggested for scope resolution, but I still don't have a clear idea about how to add things into a scope. More precisely, I'd like to implicitly add a value from a parameter to the scope of the body of a function, similar to the with(arg) { ... } function.

I appreciate your help. Thanks in advance!

[tomasruizlopez]

About the tests, I managed to run them now. I just had a misconfigured test runner, so this is now working.

[dsavvinov]

Hi @tomasruizlopez ! Sorry for the delay in response (was on vacation without stable internet access)

1. I think we have some migration plan for the issue with 32 bits, but I don't know its details. Also, it might involve some broad refactorings.

2. Therefore, I propose to focus on resolution part instead of a syntax. The rationale is that changes in resolution are much more complex and there much more open questions. One could also say that they are more risky in the sense that some showstoppers or major design issues may appear. On the other hand, having somehow working resolution, it's purely technical work to implement syntax changes -- it may require some time, but still its only about writing code.
How to prototype changes in resolution without changes in syntax? Well, you indeed need some source of typeclasses, but it's not necessarily some keyword modifiers. E.g., for the purposes of prototype it's absolutely OK to hardcode some typeclasses in test, or maybe use some awkward but easy-to-implement syntax (using some special annotation may be quite feasible here too) -- we do that often when researching/prototyping huge features.

3. As for resolution: it is indeed one of the most complex (if not the most complex) part of the compiler. Because of its complexity, it has been proved that trying to explain it for a non-prepared listener is quite pointless -- it's impossible to explain everything, but it also very hard to decide which parts should be omitted.

Instead, we use another approach, (for onboarding new team members, interns, etc.), and in our experience it's much more productive:
- start with understanding how existing cases work: write some simple code, preferably somehow relevant to desired changes (with(arg) { ... } really a nice example in your case), put breakpoint in some relevant class (it's TowerResolver in your case), step through it trying to understand what compiler does and why
- during this process, you'll inevitably stuck in some questions which you can't answer on your own -- then you come back and ask specific questions, like "what this piece of code does? Why it is necessary?". Needless to say, we'll be happy to answer those questions 🙂
Note that this is the main difference from "lecture-like" approach - we're explaining not some enormously large system in general, but it's specific parts (which will, of course, bring some general concepts too, but now they are tied to the practice and real code).
- at some point you'll think that you understand something - then you can go and actually change something and see if your understanding were correct

I recognize that maybe that wasn't the answer you were expecting (and I'm sorry for that), but we actually believe that it'll be better this way. Hope that helps!

[dsavvinov]

Also, in this specific case, given the complexity of changes and expected amount of work, I highly advise you to ping us with review very-very early, as soon as you achieve first results. It won' be a full-fledged code review, instead we'll just take a look at the general direction of work and give some feedback.

[tomasruizlopez]

Thanks very much @dsavvinov. Since I asked, I have made some progress in the resolution and surprisingly my findings match what you mentioned above, so I am happy to see that I am sort of on the right path. I really appreciate that you are willing to review the implementation of KEEP-87. I hope I can have something for review in the upcoming weeks. In the meantime, I will probably come back to this thread to ask you if I face some challenge that I can't overcome myself.