Running into issues with the hierarchical source s...

# multiplatform

[kpgalligan]

Running into issues with the hierarchical source set config and linux platforms (from mac). I can build code from the command line on my local machine, but Intellij does not recognize "platform.xxx" dependencies for linux source sets, and running in CI (Github actions) fails in the same place as the IDE. I don't know why they will build locally, running the same gradle command as CI does. The CI build fails on trying to compile the source set metadata. compile[source set name]MainKotlinMetadata. If there isn't an obvious issue and setting I should apply that somebody knows offhand, I'll post a simple example a bit later. I'm thinking if I could just set something that tells gradle to ignore non-native hosts, and config CI to also build with a linux host, I might be able to move forward (CI currently runs mac and windows builds). Thoughts?

[kpgalligan]

For a little more detail, in the ide case, the linux source set recognizes kotlin native dependencies, but not posix. For example, this config...

pthreadMain {
            dependsOn nativeCommonMain
        }

        linuxX64Main {
            dependsOn pthreadMain
        }

[kpgalligan]

Produces the above import issues

[kpgalligan]

Hmm. was trying to revert something to show those imports working for darwin targets, but that's not working now. Working on a simpler example...

[russhwolf]

My first guess is that it’s due to having multiple levels of intermediate source-sets. AFAIK that's still not supported, but I'm having trouble finding the tracking issue now

[kpgalligan]

That would be good info (although an unfortunate situation)

[kpgalligan]

So, here's the config for TestHelp

[kpgalligan]

commonMain > nativeCommonMain > nativeDarwinMain > macosX64Main, iosArm32Main (etc). That hierarchy seems to work.

[kpgalligan]

commonMain > nativeCommonMain > nativeLinuxMain > linuxX64Main, linuxArm32HfpMain (etc) For linux, that's the hierarchy.

[kpgalligan]

In the first one, 'platform' and 'posix' dependencies resolve in both the IDE and built. For nativeLinuxMain it'll work in the IDE and build without 'platform/posix'. The command line locally seems to work with 'platform/posix', but neither IDE or CI builds.

[kpgalligan]

Right now I'm testing if the latter works on CI running on linux, without the mac build running

[kpgalligan]

If so, I think maybe if there's a way to tell the multiplatform plugin to not build certain targets, it might work. Mac will try to build linux but not windows. If there's a way to also say "don't build linux on mac", that might be OK, but testing each config takes a while...

[kpgalligan]

Weird. Looks like darwin build is also failing, on a mac cloud machine, but doesn't locally. Uh.

[kpgalligan]

Trying a flattened layout. commonMain > nativeDarwinMain > macosX64Main, iosArm32Main (etc)

[kpgalligan]

commonMain > nativeLinuxMain > linuxX64Main, linuxArm32HfpMain (etc)

[kpgalligan]

Similar for windows

[kpgalligan]

OMG. Trying to trim it down to just mac, and the mac dependency resolution stops working. My general vibe is that hierarchical is still super, super flakey. I've spent hours just slightly tweaking the setup and can't really nail down config patterns that work/don't work. Either it's flakey or the docs need to get way, way more specific about how it works and limitations.

[Arkadii Ivanov]

All my attempts of enabling HMPP were unsuccessful. Decided to not use it for now.

[Arkadii Ivanov]

Compilation works fine without HMPP with any source set structure you can even imagine. For IDE experience I'm just temporary renaming whatever intermediate source set I need to a leaf one.

[kpgalligan]

I've been doing the same. It's frustrating because I get some of it to kind of work, but other parts fail.

[Sebastian Sellmair [JB]]

Hey Kevin 👋
I would be pretty interested in your project to see what is going on. HMPP is still not enabled by default and feedback/use cases like this might be very, very valuable!

Is the project publicly available? I no, is there any chance to see the build scripts e.g.?

[kpgalligan]

I'll be posting examples later tonight and will send them along. Would love to help improve the situation if I can

[kpgalligan]

OK, so I'll need to explain better later, but this build works: https://github.com/touchlab/TestHelp/tree/hmpp_not_enabled

[kpgalligan]

this build does not: https://github.com/touchlab/TestHelp/tree/hmpp_enabled

[kpgalligan]

@Sebastian Sellmair [JB] I invited you as a collaborator so I think you can see the build fail details

[kpgalligan]

This one fails: https://github.com/touchlab/TestHelp/actions/runs/567058273

[russhwolf]

Pretty sure our build logs are public anyway

[kpgalligan]

Essentially, there's a common native source set that includes some common native stuff, then 3 platform source sets that include the same posix code. I know posix is slightly different between platforms, even though these posix sources are 100% the same

[kpgalligan]

@russhwolf Yeah, I thought so, but I tried looking at them in incognito and couldn't get the actual "log". May have been a weird UI thing rather than an access thing, but whatever.

[russhwolf]

Oh I see. The new stuff you’re playing with in Actions is not public. But the old config using Azure was.

[Tijl]

HMPP is still not enabled by default

@Sebastian Sellmair [JB] the KMM plugin enables it by default. This is supposed to newbie friendly I assume. Granted, platform dependencies work there at the start.

[Sebastian Sellmair [JB]]

@kpgalligan Thanks for adding me here! I am planning to get a better picture of what is going on, this week!

@Tijl Fair enough! There are wizards that put that opt-in flag for you, yes. There is active work going on on the commonizer and I hope that we can support all your use-cases soon!

Having access to user projects like this is incredibly valuable. Simply because: We are building HMPP only with Kotlin/JVM technologies! While IntelliJ is developed in IntelliJ, most of Kotlin is done in Kotlin, etc,; MPP is different, because we are not building (H)MPP tech with (H)MPP, right? This makes it much harder to predict use-cases.

[kpgalligan]

To add some more info, I'm developing on a mac. If I run my build locally, ./gradlew build, it succeeds, even with HMPP enabled. The IDE will find the platform/posix dependency in the darwin source src/nativeDarwinMain/kotlin/co/touchlab/testhelp/concurrency/sleep.kt , but not Linux (src/nativeLinuxMain/kotlin/co/touchlab/testhelp/concurrency/sleep.kt), although it'll build Linux. It does seem to skip building linuxMips32, but IIRC, that's expected. Running gradle on the mac does not build windows, which is expected, and intellij can't find platform/posix on windows which makes sense (src/nativeMingwMain/kotlin/co/touchlab/testhelp/concurrency/sleep.kt). To be clear, what's weird is that the gradle build will fail in CI but not locally, and gradle does work after disabling HMPP in CI. Why it works locally but not on CI is definitely confusing.

[russhwolf]

@Sebastian Sellmair [JB] re use-cases, one of the big things I’m missing from HMPP (outside of the inconsistencies Kevin is having) is support for deeper shared hierarchies. While having only a single intermediate source-set will often be sufficient for KMM applications where you just need something to share iosX64 and iosArm64, this is rarely enough for libraries that want to support more platforms. For example Multiplatform Settings has iosArm64 -> apple64 (needed for 64-bit and 32-bit differences) -> apple (ie all darwin targets regardless of bitness) -> native -> multithreaded (native and jvm) -> common. Configured here (but it’s pretty messy because I tried to hack my way out of using the kotlin-dsl plugin) https://github.com/russhwolf/multiplatform-settings/blob/master/buildSrc/src/main/kotlin/BuildHelpers.kt#L99

[Sebastian Sellmair [JB]]

@russhwolf Absolutely.
This is also planned. As is the commonization of custom c-interops. (Currently work in progress)
I hope that we can ship a preview in the next few months!

[Sebastian Sellmair [JB]]

@kpgalligan Thanks for reporting the issue.
After short investigation, I found what is going on.
During commonization the incompatible linuxMips32 target is ignored and only the other linux platforms get commonized. This should not be a huuuge problem, since most symbols will still be similar enough.

However, the resulting commonized klib’s manifest mentions only the native targets that it was commonized for. The compiler seems to not load this klib for analysis and therefore the red code.

You can fix this manually, be navigating into the klib and fixing the manfiest:
the native targets shall be native_targets=linux_arm32_hfp linux_mips32 linux_x64

[Sebastian Sellmair [JB]]

I am creating a ticket for you to track today. And will start working on a fix tomorrow! 👍

[Sebastian Sellmair [JB]]

https://youtrack.jetbrains.com/issue/KT-45163

[Javier]

I am having a similar problem too, iosMain source set is working perfectly in local, IDE and CLI, but GitHub actions fails (build task).

[Javier]

As mentioned here, that fixed the issue. @Sebastian Sellmair [JB] do you know if Kotlin 1.5 will fix this issue?

[Sebastian Sellmair [JB]]

Do you mean if Kotlin 1.5 will enable HMPP by default? No, this won’t happen (at leat for 1.5.0).
But we would love to finally flipp it on by default and we’re working on it heavily!

[Tijl]

I noticed in the upcoming KMM release HMPP also won’t be on in the template anymore (youtrack issue somewhere)

[Sebastian Sellmair [JB]]

Yes, this is correct! This was done for multiple reasons. The one that would justify this to me is that we feel like still not having the necessary test coverage to be confident about wizards producing this kind of projects. There are still edge cases, that went over our heads in some recent releases. We fixed those and pushed many tests to prevent this from happening again. I, personally, would still use HMPP in my personal projects and even when adapting mpp commercially. However, users now have to opt in explicitly.

[Javier]

I have had to disable it in a project because with it, the project fails on GitHub Actions