Matt Nelson
09/15/2023, 6:24 PMxerial/sqlite-jdbc .
Now I'm tackling linking things with SQLiter, but am sort of at a loss with cinterop.
Do I have to compile the amalgamation (.c/.h) files into a static lib for each native platform so they're included with the .klib?Matt Nelson
09/15/2023, 6:25 PMMatt Nelson
09/15/2023, 6:36 PMkpgalligan
09/20/2023, 6:52 PMMatt Nelson
09/25/2023, 8:56 PMMatt Nelson
09/27/2023, 10:01 AMkotlinc-native compiler plugin build everything, but CI keeps failing (even though all tests pass locally on my linux machine, windows vm, and macOS vm).
What's crazy, is the ubuntu-latest runner is based on Ubuntu 22.04 LTS and it's failing, while my local machine is Pop!_OS 22.04 LTS (based off of the latest Ubuntu build).
So, unsure wtf is going on. Any suggestions would be much appreciated.
.def file:
package = co.touchlab.sqliter.sqlite3
---
// all sqlite3mc.c amalgamations
build.gradle.kts cinterop function for compiling
fun KotlinNativeTarget.sqlite3mcInterop() {
    val libs = projectDir.resolve("libs")
    compilations["main"].apply {
        cinterops.create("sqlite3mc") {
            defFile(libs.resolve("sqlite3mc.def"))
            compilerOpts += "-O3"
            when (konanTarget.architecture) {
                Architecture.X64,
                Architecture.X86 -> compilerOpts("-msse4.2", "-maes")
                else -> {}
            }
            compilerOpts(
                "-I${libs.resolve("include")}",
                "-DSQLITE_HAVE_ISNAN=1",
                "-DHAVE_USLEEP=1",
                "-DSQLITE_ENABLE_COLUMN_METADATA=1",
                "-DSQLITE_CORE=1",
                "-DSQLITE_ENABLE_FTS3=1",
                "-DSQLITE_ENABLE_FTS3_PARENTHESIS=1",
                "-DSQLITE_ENABLE_FTS5=1",
                "-DSQLITE_ENABLE_RTREE=1",
                "-DSQLITE_ENABLE_STAT4=1",
                "-DSQLITE_ENABLE_DBSTAT_VTAB=1",
                "-DSQLITE_ENABLE_MATH_FUNCTIONS=1",
                "-DSQLITE_THREADSAFE=1",
                "-DSQLITE_DEFAULT_MEMSTATUS=0",
                "-DSQLITE_DEFAULT_FILE_PERMISSIONS=0666",
                "-DSQLITE_MAX_VARIABLE_NUMBER=250000",
                "-DSQLITE_MAX_MMAP_SIZE=1099511627776",
                "-DSQLITE_MAX_LENGTH=2147483647",
                "-DSQLITE_MAX_COLUMN=32767",
                "-DSQLITE_MAX_SQL_LENGTH=1073741824",
                "-DSQLITE_MAX_FUNCTION_ARG=127",
                "-DSQLITE_MAX_ATTACHED=125",
                "-DSQLITE_MAX_PAGE_COUNT=4294967294",
                "-DSQLITE_DQS=0",
                "-DCODEC_TYPE=CODEC_TYPE_CHACHA20",
                "-DSQLITE_ENABLE_EXTFUNC=1",
                "-DSQLITE_ENABLE_REGEXP=1",
                "-DSQLITE_TEMP_STORE=2",
                "-DSQLITE_USE_URI=1",
            )
        }
    }
}Filip Dolník
09/27/2023, 10:06 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:10 AMFilip Dolník
09/27/2023, 10:11 AMkotlin.IllegalStateException at null:-1 😄
I’d recommend setting up an Ubuntu 22.04 LTS VM and run it manually. Debugging this in a CI won’t be easy.Matt Nelson
09/27/2023, 11:30 AMMatt Nelson
09/27/2023, 11:31 AMMatt Nelson
09/27/2023, 1:47 PMorg.touchlab.sqliter.interop.dbOpen function's call to sqlite3_open_v2
It's not an issue with using the /tmp directory for the tests, b/c JVM tests work fine. Gotta be something like you suggested; an environment issue like a missing library.
io.toxicity.sqlite.mc.driver.test.GeneralNativeTest.givenDriver_whenClose_thenCredentialsCleared[linuxX64] STANDARD_OUT
    kotlin.IllegalStateException: Failed to create DatabaseManager
        at 0   test.kexe                           0x42633e           kfun:kotlin.Exception#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 1   test.kexe                           0x42666e           kfun:kotlin.RuntimeException#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 2   test.kexe                           0x426ade           kfun:kotlin.IllegalStateException#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 3   test.kexe                           0x695ff4           kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 2324 
        at 4   test.kexe                           0x66b663           kfun:io.toxicity.sqlite.mc.driver.SQLiteMCDriver.Factory.$createActual$lambda$4COROUTINE$5.invokeSuspend#internal + 1619 
        at 5   test.kexe                           0x4312c4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
        at 6   test.kexe                           0x636cd0           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504 
        at 7   test.kexe                           0x63aae4           kfun:kotlinx.coroutines.internal.LimitedDispatcher.Worker.run#internal + 388 
        at 8   test.kexe                           0x663085           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$2COROUTINE$0.invokeSuspend#internal + 3077 
        at 9   test.kexe                           0x4312c4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
        at 10  test.kexe                           0x636cd0           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504 
        at 11  test.kexe                           0x5ddfc1           kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1585 
        at 12  test.kexe                           0x659692           kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 674 
        at 13  test.kexe                           0x658d46           kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2134 
        at 14  test.kexe                           0x659008           kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 312 
        at 15  test.kexe                           0x660538           kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 200 
        at 16  test.kexe                           0x66205c           kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$1$lambda$0#internal + 76 
        at 17  test.kexe                           0x664000           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.invoke#internal + 80 
        at 18  test.kexe                           0x6640e0           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.$<bridge-UNN>invoke(){}#internal + 80 
        at 19  test.kexe                           0x43cd8a           WorkerLaunchpad + 170 
        at 20  test.kexe                           0x779a40           _ZN6Worker19processQueueElementEb + 4624 
        at 21  test.kexe                           0x77876e           _ZN12_GLOBAL__N_113workerRoutineEPv + 142 
        at 22  libc.so.6                           0x7f0215894b42     0x0 + 139646927981378 
        at 23  libc.so.6                           0x7f02159269ff     0x0 + 139646928579071 
    Caused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
        at 0   test.kexe                           0x42c5f1           kfun:kotlin.Throwable#<init>(kotlin.String?){} + 113 
        at 1   test.kexe                           0x42628d           kfun:kotlin.Exception#<init>(kotlin.String?){} + 109 
        at 2   test.kexe                           0x58d2d7           kfun:co.touchlab.sqliter.interop.SQLiteException#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig){} + 135 
        at 3   test.kexe                           0x58d401           kfun:co.touchlab.sqliter.interop.SQLiteExceptionErrorCode#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig;kotlin.Int){} + 257 
        at 4   test.kexe                           0x596749           kfun:co.touchlab.sqliter.interop#dbOpen(kotlin.String;kotlin.collections.List<co.touchlab.sqliter.interop.OpenFlags>;kotlin.String;kotlin.Boolean;kotlin.Boolean;kotlin.Int;kotlin.Int;kotlin.Int;co.touchlab.sqliter.interop.Logger;kotlin.Boolean){}co.touchlab.sqliter.interop.SqliteDatabase + 4697 
        at 5   test.kexe                           0x5a62e2           kfun:co.touchlab.sqliter.native.NativeDatabaseManager.createConnection#internal + 2466 
        at 6   test.kexe                           0x5a55ea           kfun:co.touchlab.sqliter.native.NativeDatabaseManager#createMultiThreadedConnection(){}co.touchlab.sqliter.DatabaseConnection + 234 
        at 7   test.kexe                           0x57c7fd           kfun:co.touchlab.sqliter#withConnection__at__co.touchlab.sqliter.DatabaseManager(kotlin.Function1<co.touchlab.sqliter.DatabaseConnection,0:0>){0§<kotlin.Any?>}0:0 + 365 
        at 8   test.kexe                           0x695dbc           kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 1756 
        ... and 20 more common stack frames skippedFilip Dolník
09/27/2023, 1:52 PMCaused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
So I’d start by looking for the file it tries to open and the version of the database (and compare those to what you get in Pop!_OS)Matt Nelson
09/27/2023, 1:56 PMsqlite3mc.c amalgamations sad pandaMatt Nelson
09/27/2023, 1:58 PMMatt Nelson
09/27/2023, 2:26 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:30 PMMatt Nelson
09/27/2023, 2:30 PMkpgalligan
09/27/2023, 2:31 PMMatt Nelson
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:35 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:37 PMMatt Nelson
09/27/2023, 2:50 PMkpgalligan
09/27/2023, 2:51 PMMatt Nelson
09/27/2023, 2:53 PMsqlite-mc is a part of another project I'm working on under toxicity-io that embeds Tor + KTor server, and allows apps to register with it (like firebase messaging) and will act as a reverse proxy exposing that app as a Tor Hidden Service.
A viable open source replacement for google push notifications 😄Matt Nelson
09/27/2023, 7:14 PMcklib I think I found the underlying issue that I was having earlier
Seems that libtinfo is causing the issues
From CI (ubuntu-latest 22.04 LTS)
> Task :library:android-unit-test:test NO-SOURCE
> Task :library:android-unit-test:check
> Task :library:driver:jvmProcessResources NO-SOURCE
> Task :library:driver:jvmTestProcessResources NO-SOURCE
/home/runner/.cklib/clang-llvm-8.0.0-linux-x86-64/bin/clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
> Task :library:driver:linuxX64Libsqlite3mc FAILED
From my machine (Pop!_OS 22.04 LTS)
$ ldd $(which info)
        linux-vdso.so.1 (0x00007ffd39bf4000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc193842000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc193600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc1938d7000)
EDIT: actually, this is an issue with the version of llvm that cklib is using.Matt Nelson
09/28/2023, 3:58 PM.so or .dylib (like how it's being done in xerial/sqlite-jdbc ) and somehow package that shared lib inside the .klib for library consumers, then link to it dynamically using relative paths...
Might just only support ios for the time being as one can get away with compiling a static lib .a and including it via cinterop...Matt Nelson
10/02/2023, 5:28 PMcklib for iOS, tvOS, and watchOS with 1 exception...
> Task :library:driver:iosSimulatorArm64Test SKIPPED
e: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
> Task :library:driver-test:linkDebugTestIosSimulatorArm64 FAILED
The /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
ld: warning: object file (/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/konan_temp130378032671[85](<https://github.com/toxicity-io/sqlite-mc/actions/runs/6381376828/job/17317720607?pr=43#step:9:85)517018/result.o)> was built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
  "_SecRandomCopyBytes", referenced from:
      _entropy in result.o
  "_kSecRandomDefault", referenced from:
      _entropy in result.o
ld: symbol(s) not found for architecture arm64
> Task :library:driver:iosX64Sqlite3mc
FAILURE: Build failed with an exception.
So, module :driver compiles just fine, but when :driver-test goes to link against it, this warning keeps popping up and the build fails.
This is only occurring on CI (using macos-latest), and does not happen on my vm of macOS 11
Any thoughts?
I've tried a plethora of things w/o success:
• Adding to each Darwin target freeCompilerArgs += listOf("-linker-options", "-framework Security")
• Adding to compilerArgs for each Darwin target  -isysroot </path/to/platform/specific/sdk> -iframeworkwithsysroot Security
•  Adding to linkerArgs the --suppress-warnings flagFilip Dolník
10/02/2023, 5:30 PMwas built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:Matt Nelson
10/02/2023, 5:31 PM--target declared?Matt Nelson
10/02/2023, 5:32 PMamd64-apple-iphoneos14.0-simulatorFilip Dolník
10/02/2023, 5:33 PMThis is only occurring onThese environments will likely have a different version of Xcode installed(usingCI), and does not happen on my vm ofmacos-latestmacOS 11
Matt Nelson
10/02/2023, 5:36 PMcompilerArgs for cklib , does that do nothing?
if (kt.family.isAppleFamily && xcode != null) {
            when (kt) {
                IOS_ARM64 -> listOf(
                    "-mios-version-min=9.0",
                    "-isysroot",
                    xcode.iphoneosSdk,
                )
                IOS_SIMULATOR_ARM64 -> listOf(
                    "-mios-simulator-version-min=9.0",
                    "-isysroot",
                    xcode.iphonesimulatorSdk,
                )
                IOS_X64 -> listOf(
                    "-mios-version-min=9.0",
                    "-isysroot",
                    xcode.iphoneosSdk,
                )
                MACOS_ARM64 -> listOf(
                    "-mmacosx-version-min=10.9",
                    "-isysroot",
                    xcode.macosxSdk,
                )
                MACOS_X64 -> listOf(
                    "-mmacosx-version-min=10.7",
                    "-isysroot",
                    xcode.macosxSdk,
                )
                TVOS_ARM64 -> listOf(
                    "-mtvos-version-min=9.0",
                    "-isysroot",
                    xcode.appletvosSdk,
                )
                TVOS_SIMULATOR_ARM64 -> listOf(
                    "-mtvos-simulator-version-min=9.0",
                    "-isysroot",
                    xcode.appletvsimulatorSdk,
                )
                TVOS_X64 -> listOf(
                    "-mtvos-version-min=9.0",
                    "-isysroot",
                    xcode.appletvosSdk,
                )
                WATCHOS_ARM32 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_ARM64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_DEVICE_ARM64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_SIMULATOR_ARM64 -> listOf(
                    "-mwatchos-simulator-version-min=3.0",
                    "-isysroot",
                    xcode.watchsimulatorSdk,
                )
                WATCHOS_X64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                else -> null
            }?.let { compilerArgs.addAll(it) }
        }Filip Dolník
10/02/2023, 5:38 PMMatt Nelson
10/02/2023, 5:40 PM1.8.22 -> 1.9.10, and CI is using macos 12 where locally, my machine is running macOS 11. So.
Why the shit is min version 14? is that just the simulator?Filip Dolník
10/02/2023, 5:41 PMFilip Dolník
10/02/2023, 5:42 PMFilip Dolník
10/02/2023, 5:42 PMMatt Nelson
10/02/2023, 5:43 PMiOS 14 of September 16, 2020 👀russhwolf
10/02/2023, 5:44 PMrusshwolf
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:47 PMiOS 14 of September 16, 2020
Depending on your use case it might not be a big deal. Most iOS users update their OS. Plus if you target KMP projects, then everyone will have the same limit.Filip Dolník
10/02/2023, 5:49 PMMatt Nelson
10/02/2023, 5:50 PMMatt Nelson
10/02/2023, 6:40 PMiosSimulatorArm64, F it...14.0, but Xcode 13 and 14 default to min version 9.0, so linking against it is impossible.
Did I get that right? They fixed it for Xcode 15+, but broke it for Xcode not 15+
EDIT: That didn't fix it, as it's for all iOS targets (even though the ticket expressed only simulator). Fix was to change CI runner to macos-11  temporarily and figure it out later.Matt Nelson
10/02/2023, 6:48 PMversion-min from freeCompilerArgs,
If the resulting .klib contains bitcode that was compiled with -m<platform>-version-min=<version> , does that propagate to consumers of the .klib?
Should I set both cklib compiled code and freeCompilerArgs for the module?Matt Nelson
10/03/2023, 3:11 AM-framework Security link option to module :driver-testFilip Dolník
10/03/2023, 1:15 PMThat didn’t fix it, as it’s for allYeah, it’s for all iOS targetstargetsiOS
The unerlying problem is that Kotlin 1.9.10 is compiled with min versionYes, you need to link with the same min version (or higher)., but Xcode14.0and13default to min version14, so linking against it is impossible.9.0
If the resultingYes, it’s the same problem - you need to link against the highest min version.contains bitcode that was compiled with.klib, does that propagate to consumers of the-m<platform>-version-min=<version>?.klib
Wow, I am regarded... The fix was addingBut I assume the warning about min versions didn’t go away right? If no then it’s still a problem (I’d recommend to address it because it can blow up elsewhere)link option to module-framework Security
Matt Nelson
10/03/2023, 2:17 PMsqlite3).Filip Dolník
10/03/2023, 2:20 PMNope, warning went away completely. Everything compiled and tests ran.Even on
macos-latest ? (you mentioned that at one point you changed CI to use macos-11 )Matt Nelson
10/03/2023, 2:21 PMFilip Dolník
10/03/2023, 2:22 PMMatt Nelson
10/13/2023, 6:28 PM