Currently, we agreed on the following development workflow: 1. New major changes go to a separate branch. 2. A PR should be opened as draft until it's ready for review. 3. When it's ready, remove the draft status and assign a reviewer. 4. When an agreement is achieved, the PR's author merges the PR. GH's "rebase and merge" button is preferred, but in some special cases it's OK to use "create a merge commit" option.

I'm working on translating field declarations (

visitField(declaration: IrField...

, branch translate-field-declarations), however it's tricky because in Python to be able to use some function, it has to be first defined. That's why now all box tests fail with:

Traceback (most recent call last):
  File "/home/piotr/repos/kotlin-python/python/py.translator/testData/out/codegen/irBox/when/noElseInStatement_v5.py4568195347954504164/consumer.py", line 1, in <module>
    from compiled_module import box
  File "/home/piotr/repos/kotlin-python/python/py.translator/testData/out/codegen/irBox/when/noElseInStatement_v5.py4568195347954504164/compiled_module.py", line 1356, in <module>
    UNDEFINED_RESULT = UNDEFINED_RESULT_init_()
NameError: name 'UNDEFINED_RESULT_init_' is not defined

TODO: I think we need to do a similar thing that is done for JavaScript - move field assignments to the bottom, or reorder stuff in some other smart way. Leaving my work-in-progress change on the branch in case someone wants to tackle this problem. If not, I'll get back to it in a few days

@SerVB since I know you're planning to translate some instrincts tomorrow, if I may - I would request handling

jsNot

(probably in favor of Python's

not

)😄

IrComposite

is another thing to cover. Once we have these two, the property initializers should work properly and I expect a nice bump in the number of passed box tests. Details in the PR: Translate property declaration and initialization by krzema12 · Pull Request #12 · krzema12/kotlin-python

By the way, does anybody know a way to make an expression from a statement in Python? I researched this topic a bit today and it seems there is no way. So JS IR won't help much here because JS has a magic comma operator but Python doesn't. I guess we will have to take a look at JVM's IR to avoid reinventing the bicycle for this. Another tricky translation part will be to support labels for continue, break, and return. There are some labels in JVM and JS but there aren't simply any in Python. So looks like the journey is only starting :mind-blown:

We're close to the runtime limit for box tests on GitHub (6 hours, we're at ~5). Worth considering parallelizing them - I'm thinking about it already. AFAIK, GitHub Actions allow parallel execution on multiple machines, the main difficulty will be telling Gradle/JUnit to run a specific subset (map), and then join the results (reduce).

Hi Kotlin team! Does anybody else experience this problem when importing the project? Don't you know a solution? This happens to me on tag

v1.5.21

of the Kotlin repo and since this tag is merged to Kotlin/Py, we now have this same problem too. This doesn't affect code writing though: completion still works normally. Just a button "imprort Gradle changes" is always visible in IDEA. Piotr experiences a similar issue too.

Hi folks, I'm struggling to modify an existing lowering to work with Python. See this PR - 4 box tests complain about e.g.

UnboundLocalError: local variable 'properties_initialized_utf8Encoding_kt' referenced before assignment

The lowering in question is PropertyLazyInitLowering.kt. Why it happens: the lowering produces initialization functions like

def init_properties_utf8Encoding_kt():
    if not (properties_initialized_utf8Encoding_kt):
        properties_initialized_utf8Encoding_kt = True
        tmp0_byteArrayOf_0 = visitConst_other_Byte
        REPLACEMENT_BYTE_SEQUENCE = tmp0_byteArrayOf_0

and before reaching

if not (properties_initialized_utf8Encoding_kt):

, Python has to have

properties_initialized_utf8Encoding_kt

assigned to some value (here

False

would make sense). I wanted to add

properties_initialized_utf8Encoding_kt = False

just before this function definition, but I'm lost in the structure of IR elements hierarchy. I cannot spot a place where the function is added to the IR output. Even if I use some transformers that compile fine, I don't see them do their job in the output. Could you help me figure it out? Side-questions: are there any materials where I could learn more about the internal compiler API regarding IR entities, or only browsing the code is the way to go?

Hey, I've tried to replace logic for AST to text generation from string concatenation to a common stringbuilder, but it gave only about 5% boost. It's available here: https://github.com/krzema12/kotlin-python/tree/2021-07-30-stringbuilder I wasn't able to profile a test run. It seems profiling isn't available in IDEA for test tasks. We can still try profile running profiler manually somehow. Another thing I've tried is to convert test source root to main, but I've stuck with some error related to unavailable IDEA installation required by calls to IDEA logger and other classes. I think I will shelve the task of optimizing tests for now. Will continue with box tests themselves

hi @SerVB! I was wondering if we could change our approach of choosing which features to implement next, and go like this: take some simple Kotlin examples (like in this file), try to run them from Python (see consumer.py), and try to infer which features are missing to make it work. WDYT? It should also naturally lead to having more box tests passing, but the benefit would be that certain real-life Kotlin pieces would already work end to end. Something we could present on some demo!

Today: supported overflowing in conversions and left bit shift; inlined Long values; removed some unneeded parentheses in the generated py file. Next time: • Remove wrappers like toLong and kotlin_Int for some numbers (probably research NumberConversionCallsTransformer). • For Long, there are still calls like

._and

,

.xor

,

.equals

instead of plain operators: research why. • Maybe something else related to signed integers, but for now I don't see other tasks here. • Maybe start with unsigned integers.

Today: opened a PR with initial support for integers (probably there are some hidden bugs but many more things work now) – +52 box tests pass. We still don't support Chars and unsigned integers at all, need to support them. Next time: • I see some integer tests like

binaryOp/call

have lambdas that we can't compile for now. I want to switch my context a bit and try to support lambdas.

FYI, next time I'm planning to prepare an extended report for failing tests. For each failing case, it will contain general reason (failed during compilation or Python execution), details (Kotlin compiler error or Python exception message) and running time. It will allow us to: • see if e. g. lots of tests fail with the same reason. Then we can focus on fixing this, potentially getting more new passing tests with low effort • Troubleshooting long box test running time: see distribution of running times, see how they correlate with failing reason (e. g. infinite loop, stack overflow) It will be a new generated file, checked by CI (i know, one more file to take care of, but we may then somehow merge the failed tests reports).

@SerVB I accidentally fixed the long-running (4-5 h) box tests, even before digging deeper which tests caused it 🤣 they are back to 1-2 h after this PR, see details in the description: Include compilation and running time in box tests exceptions by krzema12 · Pull Request #23 · krzema12/kotlin-python

@Piotr Krzemiński please request my review again when you want me to take a look again: https://github.com/krzema12/kotlin-python/pull/23

We need to pay off some technical debt that now starts to have big impact: name clashes. In e.g. JS target, the transformers use

context.getNameFor...

to calculate a name for a function or a variable. We currently ignore it, which results in such behavior. In our simple , these are seen as

RecursionError: maximum recursion depth exceeded

. The linked code hints me that it also has an indirect influence on issues that occur more frequently, like

NameError: name 'Companion_instance' is not defined

. I think we should work on this next. It's also needed to go forward with my Translate property declaration and initialization by krzema12 · Pull Request #12 · krzema12/kotlin-python. @SerVB do you fancy taking a closer look tomorrow, or you'd rather tackle another problem? My next slot for working on kotlin-python will be in 2 weeks.

Current

box-tests-report.tsv

has a problem having it in the repo: it contains unstable values like execution time and sometimes paths to temporary files with random names (in stacktraces). So we are risking to have many changed lines in each PR... Anyway, it doesn't seem very critical to me

Today: Worked with name clashes. Turned out that many problems were caused by unsupported object instances, so the first PR I made is with some support of those. +192 box tests are passing now. Also, fixed some name clashes, +20 box tests pass. Looked at box tests fail reasons, looks like we have the following major paths: 1. java.lang.ClassCastException happens in compiler when compiling suspend functions. 2. 'Object_create', 'js' is not defined – at least related to arrays creation, maybe to smth else. 3. 'self' is not defined – unsupported extension function and properties. And many others. Next time: I guess number 2 or 3. Number 1 can be shelved for now, I suppose 😅

A random idea about speeding up tests: I've noticed that when tests are starting, all the cores are loaded at 100%, but after some time less and less cores are loaded (see the screenshot). And at the end, only about 1 core or 2 cores are working. If we make all cores work during the whole testing, I guess the tests will take less time

Today: Started with extension functions support but bumped into regressions in enums. Dug into enums and found out that my PR with object instances wasn't enough. So: • Added support of more static fields to PR with object instances, +17 more box tests pass. • Started with support of superclasses that are required for enums: ◦ Resolved the forward declaration stuff. ◦ Did some topology sorting for superclasses in a class declaration to prevent the "Cannot create a consistent method resolution order (MRO)" runtime error ◦ Didn't finish yet so PR will be later. • Extension functions seem to be ready but they are shelved too for now. Also, had a call with @ dmitriy.novozhilov (sorry for pinging) – he suggested to switch to new testing infrastructure that utilizes JUnit 5 and makes parallel tests launching more effective. Some work will be required but I think it's needed for us and also the old testing infrastructure is going to be deprecated. Next time: • Finish what I've started today. • Another idea is to make a script to count box test fail reasons, will do and add CI for it. • Start with switching to the new testing thing.

Next thing I'm going to work on is making

println("Hello world!")

work 🙂 now there's some issue with property initialization,

output

is not defined. In JS version the initializer function is executed at the very end. Maybe we should get back to lazily initiated properties, so I'll try to rebase my still-as-draft PR from a month ago and see if it helps. @SerVB I remember some TODO from your recent PRs that could address this. Is it in scope that you worked on yesterday and didn't send to review yet?

Also, during the call with @ dmitriy.novozhilov, he mentioned that we could make our own DCE that will be based only on used symbols (classes/functions/fields). AFAIU, DCE in K/JS is much more powerful and it can eliminate dead code even inside function bodies, so maybe we don't need such complexity and symbols-level DCE will be quite enough for us. It can speed up our tests since Python will take less time to parse cut file

Today: • Cleaned up README. • Added generated file with stats on failure reasons and counts. • Disabled DCE and PIR invocation, that gave speed up for tests and also +60 tests passing (because there were compilation errors and now that compilation code isn't executed). • Finished support for super classes, +98 tests passing. • Finished support for extension functions, +70 tests passing. • Started to research how new testing infrastructure works, didn't have time to do anything in code. I also see there are still some problems with integers: for example, in IntRange. Also, it seems we don't support default args. So these are next possible steps. Next time: • Continue with the new testing infrastructure.

Piotr and I have some questions regarding so called "PIR": What is it? Is it safe to disable it? I see disabling it doesn't spoil anything visible. Only

out-pir

dir stops to be generated during box testing

Let me try to tackle tomorrow the problem of

NameError: name 'Object_create' is not defined

, it's top 1 of box tests failure reasons

Today: • Finished the PR with removing much of JS stuff. • Continued researching the new and the old testing infrastructures. It seems the main problem of the old one is that it's based on the old JUnit API heavily, for example there are some methods that should be called before and after each test, and they are hidden deep inside the superclasses of our test class. The main problem of the new one is that it uses a specific organization of code that we need to follow (facades, handlers and so on: https://github.com/JetBrains/kotlin/blob/master/compiler/test-infrastructure/ReadMe.md), and I still don't understand it well. We will definitely have to write some code there. Our main task in testing is to start using JUnit5 that will allow running separate test cases in parallel, thus speeding things up. For now, our testing can run only different test classes in parallel, and it's less effective compared to per-case parallelism possible with JUnit5. It can be reasonable to firstly try to get rid of the old JUnit API in our tests, but just invoke the same code manually. I guess this way we could just use JUnit5 per-case parallelism not writing much new code. Or I could continue with switching to the new testing infrastructure with its entirely new DSL right away. This will require more research and rewriting the most of our testing code to that specific organization of code. I'm not sure which way is faster, probably the first one, but both should give the same result that we need: tests should work much faster since true parallelism will be used. In case of any ideas please share 🙂 Next time: • Choose the initial way of switching to JUnit5 and continue with it.

Adapting the old infra for JUnit5. These exceptions happen when I try to run a bit reworked but quite unchanged old testing infra on JUnit5, multiple tests at once. Running only a single test works. My guess is that it can be because these IntelliJ methods can't be called concurrently. If anybody has clues, please share. I don't want to continue research it myself, I will switch to researching new infra :)

Today: • Took a look at secondary constructors PR, left my comments. • Talked to Ivan Kylchik [JB] about new infra in JS (https://github.com/JetBrains/kotlin/commit/0db2ef9a71709bdbe8ef99c99194d83749ce14a7). • Bumped into some problems again when trying to use old infra with JUnit5. • Started to implement new infra but it seems there is much work, decided to shelve this task for some time because about three weeks passed but not much results and it's starting to be tedious. • Switched to implementing IrTry, nothing working yet. Next time: • IrTry.

Today (turns out to be intrinsics day 🙂): • Worked with try-catch a bit, bumped into many problems. • The first one was with unsupported isinstance check, supported it (two intrinsics), +15 box tests pass. • The second one was with secondary constructors. Tried to continue Piotr's PR, added some workarounds, but always bumped into some more problems. The last problem was with not working super and this that some secondary constructors can have at the same time. Started over and searched out that Python supports creating an uninitialized object via

__new__

, so was able to implement Python-compatible variant of

Object_create

intrinsic. Looks like the current state is working, but maybe we could make compilation more python-ish in the future if we want. +47 box tests pass (we have 1999/5787 passing now 😅). Next time: • Maybe some more unimplemented intrinsics. • Continue with IrTry but there can be some more blockers on the way.

still no time to do any real iteration over the compiler, but instead: some eye candy from me today: Add git history plot for box tests by krzema12 · Pull Request #40 · krzema12/kotlin-python (details in PR description)