Matt Nelson
09/15/2023, 6:24 PMxerial/sqlite-jdbc .
Now I'm tackling linking things with SQLiter, but am sort of at a loss with cinterop.
Do I have to compile the amalgamation (.c/.h) files into a static lib for each native platform so they're included with the .klib?Matt Nelson
09/15/2023, 6:25 PMMatt Nelson
09/15/2023, 6:36 PMkpgalligan
09/20/2023, 6:52 PMMatt Nelson
09/25/2023, 8:56 PMMatt Nelson
09/27/2023, 10:01 AMkotlinc-native compiler plugin build everything, but CI keeps failing (even though all tests pass locally on my linux machine, windows vm, and macOS vm).
What's crazy, is the ubuntu-latest runner is based on Ubuntu 22.04 LTS and it's failing, while my local machine is Pop!_OS 22.04 LTS (based off of the latest Ubuntu build).
So, unsure wtf is going on. Any suggestions would be much appreciated.
.def file:
package = co.touchlab.sqliter.sqlite3
---
// all sqlite3mc.c amalgamations
build.gradle.kts cinterop function for compiling
fun KotlinNativeTarget.sqlite3mcInterop() {
val libs = projectDir.resolve("libs")
compilations["main"].apply {
cinterops.create("sqlite3mc") {
defFile(libs.resolve("sqlite3mc.def"))
compilerOpts += "-O3"
when (konanTarget.architecture) {
Architecture.X64,
Architecture.X86 -> compilerOpts("-msse4.2", "-maes")
else -> {}
}
compilerOpts(
"-I${libs.resolve("include")}",
"-DSQLITE_HAVE_ISNAN=1",
"-DHAVE_USLEEP=1",
"-DSQLITE_ENABLE_COLUMN_METADATA=1",
"-DSQLITE_CORE=1",
"-DSQLITE_ENABLE_FTS3=1",
"-DSQLITE_ENABLE_FTS3_PARENTHESIS=1",
"-DSQLITE_ENABLE_FTS5=1",
"-DSQLITE_ENABLE_RTREE=1",
"-DSQLITE_ENABLE_STAT4=1",
"-DSQLITE_ENABLE_DBSTAT_VTAB=1",
"-DSQLITE_ENABLE_MATH_FUNCTIONS=1",
"-DSQLITE_THREADSAFE=1",
"-DSQLITE_DEFAULT_MEMSTATUS=0",
"-DSQLITE_DEFAULT_FILE_PERMISSIONS=0666",
"-DSQLITE_MAX_VARIABLE_NUMBER=250000",
"-DSQLITE_MAX_MMAP_SIZE=1099511627776",
"-DSQLITE_MAX_LENGTH=2147483647",
"-DSQLITE_MAX_COLUMN=32767",
"-DSQLITE_MAX_SQL_LENGTH=1073741824",
"-DSQLITE_MAX_FUNCTION_ARG=127",
"-DSQLITE_MAX_ATTACHED=125",
"-DSQLITE_MAX_PAGE_COUNT=4294967294",
"-DSQLITE_DQS=0",
"-DCODEC_TYPE=CODEC_TYPE_CHACHA20",
"-DSQLITE_ENABLE_EXTFUNC=1",
"-DSQLITE_ENABLE_REGEXP=1",
"-DSQLITE_TEMP_STORE=2",
"-DSQLITE_USE_URI=1",
)
}
}
}Filip Dolník
09/27/2023, 10:06 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:10 AMFilip Dolník
09/27/2023, 10:11 AMkotlin.IllegalStateException at null:-1 😄
I’d recommend setting up an Ubuntu 22.04 LTS VM and run it manually. Debugging this in a CI won’t be easy.Matt Nelson
09/27/2023, 11:30 AMMatt Nelson
09/27/2023, 11:31 AMMatt Nelson
09/27/2023, 1:47 PMorg.touchlab.sqliter.interop.dbOpen function's call to sqlite3_open_v2
It's not an issue with using the /tmp directory for the tests, b/c JVM tests work fine. Gotta be something like you suggested; an environment issue like a missing library.
io.toxicity.sqlite.mc.driver.test.GeneralNativeTest.givenDriver_whenClose_thenCredentialsCleared[linuxX64] STANDARD_OUT
kotlin.IllegalStateException: Failed to create DatabaseManager
at 0 test.kexe 0x42633e kfun:kotlin.Exception#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 1 test.kexe 0x42666e kfun:kotlin.RuntimeException#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 2 test.kexe 0x426ade kfun:kotlin.IllegalStateException#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 3 test.kexe 0x695ff4 kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 2324
at 4 test.kexe 0x66b663 kfun:io.toxicity.sqlite.mc.driver.SQLiteMCDriver.Factory.$createActual$lambda$4COROUTINE$5.invokeSuspend#internal + 1619
at 5 test.kexe 0x4312c4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 6 test.kexe 0x636cd0 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504
at 7 test.kexe 0x63aae4 kfun:kotlinx.coroutines.internal.LimitedDispatcher.Worker.run#internal + 388
at 8 test.kexe 0x663085 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$2COROUTINE$0.invokeSuspend#internal + 3077
at 9 test.kexe 0x4312c4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 10 test.kexe 0x636cd0 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504
at 11 test.kexe 0x5ddfc1 kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1585
at 12 test.kexe 0x659692 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 674
at 13 test.kexe 0x658d46 kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2134
at 14 test.kexe 0x659008 kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 312
at 15 test.kexe 0x660538 kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 200
at 16 test.kexe 0x66205c kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$1$lambda$0#internal + 76
at 17 test.kexe 0x664000 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.invoke#internal + 80
at 18 test.kexe 0x6640e0 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.$<bridge-UNN>invoke(){}#internal + 80
at 19 test.kexe 0x43cd8a WorkerLaunchpad + 170
at 20 test.kexe 0x779a40 _ZN6Worker19processQueueElementEb + 4624
at 21 test.kexe 0x77876e _ZN12_GLOBAL__N_113workerRoutineEPv + 142
at 22 libc.so.6 0x7f0215894b42 0x0 + 139646927981378
at 23 libc.so.6 0x7f02159269ff 0x0 + 139646928579071
Caused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
at 0 test.kexe 0x42c5f1 kfun:kotlin.Throwable#<init>(kotlin.String?){} + 113
at 1 test.kexe 0x42628d kfun:kotlin.Exception#<init>(kotlin.String?){} + 109
at 2 test.kexe 0x58d2d7 kfun:co.touchlab.sqliter.interop.SQLiteException#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig){} + 135
at 3 test.kexe 0x58d401 kfun:co.touchlab.sqliter.interop.SQLiteExceptionErrorCode#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig;kotlin.Int){} + 257
at 4 test.kexe 0x596749 kfun:co.touchlab.sqliter.interop#dbOpen(kotlin.String;kotlin.collections.List<co.touchlab.sqliter.interop.OpenFlags>;kotlin.String;kotlin.Boolean;kotlin.Boolean;kotlin.Int;kotlin.Int;kotlin.Int;co.touchlab.sqliter.interop.Logger;kotlin.Boolean){}co.touchlab.sqliter.interop.SqliteDatabase + 4697
at 5 test.kexe 0x5a62e2 kfun:co.touchlab.sqliter.native.NativeDatabaseManager.createConnection#internal + 2466
at 6 test.kexe 0x5a55ea kfun:co.touchlab.sqliter.native.NativeDatabaseManager#createMultiThreadedConnection(){}co.touchlab.sqliter.DatabaseConnection + 234
at 7 test.kexe 0x57c7fd kfun:co.touchlab.sqliter#withConnection__at__co.touchlab.sqliter.DatabaseManager(kotlin.Function1<co.touchlab.sqliter.DatabaseConnection,0:0>){0§<kotlin.Any?>}0:0 + 365
at 8 test.kexe 0x695dbc kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 1756
... and 20 more common stack frames skippedFilip Dolník
09/27/2023, 1:52 PMCaused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
So I’d start by looking for the file it tries to open and the version of the database (and compare those to what you get in Pop!_OS)Matt Nelson
09/27/2023, 1:56 PMsqlite3mc.c amalgamations sad pandaMatt Nelson
09/27/2023, 1:58 PMMatt Nelson
09/27/2023, 2:26 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:30 PMMatt Nelson
09/27/2023, 2:30 PMkpgalligan
09/27/2023, 2:31 PMMatt Nelson
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:35 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:37 PMMatt Nelson
09/27/2023, 2:50 PMkpgalligan
09/27/2023, 2:51 PMMatt Nelson
09/27/2023, 2:53 PMsqlite-mc is a part of another project I'm working on under toxicity-io that embeds Tor + KTor server, and allows apps to register with it (like firebase messaging) and will act as a reverse proxy exposing that app as a Tor Hidden Service.
A viable open source replacement for google push notifications 😄Matt Nelson
09/27/2023, 7:14 PMcklib I think I found the underlying issue that I was having earlier
Seems that libtinfo is causing the issues
From CI (ubuntu-latest 22.04 LTS)
> Task :library:android-unit-test:test NO-SOURCE
> Task :library:android-unit-test:check
> Task :library:driver:jvmProcessResources NO-SOURCE
> Task :library:driver:jvmTestProcessResources NO-SOURCE
/home/runner/.cklib/clang-llvm-8.0.0-linux-x86-64/bin/clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
> Task :library:driver:linuxX64Libsqlite3mc FAILED
From my machine (Pop!_OS 22.04 LTS)
$ ldd $(which info)
linux-vdso.so.1 (0x00007ffd39bf4000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc193842000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc193600000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1938d7000)
EDIT: actually, this is an issue with the version of llvm that cklib is using.Matt Nelson
09/28/2023, 3:58 PM.so or .dylib (like how it's being done in xerial/sqlite-jdbc ) and somehow package that shared lib inside the .klib for library consumers, then link to it dynamically using relative paths...
Might just only support ios for the time being as one can get away with compiling a static lib .a and including it via cinterop...Matt Nelson
10/02/2023, 5:28 PMcklib for iOS, tvOS, and watchOS with 1 exception...
> Task :library:driver:iosSimulatorArm64Test SKIPPED
e: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
> Task :library:driver-test:linkDebugTestIosSimulatorArm64 FAILED
The /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
ld: warning: object file (/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/konan_temp130378032671[85](<https://github.com/toxicity-io/sqlite-mc/actions/runs/6381376828/job/17317720607?pr=43#step:9:85)517018/result.o)> was built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
"_SecRandomCopyBytes", referenced from:
_entropy in result.o
"_kSecRandomDefault", referenced from:
_entropy in result.o
ld: symbol(s) not found for architecture arm64
> Task :library:driver:iosX64Sqlite3mc
FAILURE: Build failed with an exception.
So, module :driver compiles just fine, but when :driver-test goes to link against it, this warning keeps popping up and the build fails.
This is only occurring on CI (using macos-latest), and does not happen on my vm of macOS 11
Any thoughts?
I've tried a plethora of things w/o success:
• Adding to each Darwin target freeCompilerArgs += listOf("-linker-options", "-framework Security")
• Adding to compilerArgs for each Darwin target -isysroot </path/to/platform/specific/sdk> -iframeworkwithsysroot Security
• Adding to linkerArgs the --suppress-warnings flagFilip Dolník
10/02/2023, 5:30 PMwas built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:Matt Nelson
10/02/2023, 5:31 PM--target declared?Matt Nelson
10/02/2023, 5:32 PMamd64-apple-iphoneos14.0-simulatorFilip Dolník
10/02/2023, 5:33 PMThis is only occurring onThese environments will likely have a different version of Xcode installed(usingCI), and does not happen on my vm ofmacos-latestmacOS 11
Matt Nelson
10/02/2023, 5:36 PMcompilerArgs for cklib , does that do nothing?
if (kt.family.isAppleFamily && xcode != null) {
when (kt) {
IOS_ARM64 -> listOf(
"-mios-version-min=9.0",
"-isysroot",
xcode.iphoneosSdk,
)
IOS_SIMULATOR_ARM64 -> listOf(
"-mios-simulator-version-min=9.0",
"-isysroot",
xcode.iphonesimulatorSdk,
)
IOS_X64 -> listOf(
"-mios-version-min=9.0",
"-isysroot",
xcode.iphoneosSdk,
)
MACOS_ARM64 -> listOf(
"-mmacosx-version-min=10.9",
"-isysroot",
xcode.macosxSdk,
)
MACOS_X64 -> listOf(
"-mmacosx-version-min=10.7",
"-isysroot",
xcode.macosxSdk,
)
TVOS_ARM64 -> listOf(
"-mtvos-version-min=9.0",
"-isysroot",
xcode.appletvosSdk,
)
TVOS_SIMULATOR_ARM64 -> listOf(
"-mtvos-simulator-version-min=9.0",
"-isysroot",
xcode.appletvsimulatorSdk,
)
TVOS_X64 -> listOf(
"-mtvos-version-min=9.0",
"-isysroot",
xcode.appletvosSdk,
)
WATCHOS_ARM32 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_ARM64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_DEVICE_ARM64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_SIMULATOR_ARM64 -> listOf(
"-mwatchos-simulator-version-min=3.0",
"-isysroot",
xcode.watchsimulatorSdk,
)
WATCHOS_X64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
else -> null
}?.let { compilerArgs.addAll(it) }
}Filip Dolník
10/02/2023, 5:38 PMMatt Nelson
10/02/2023, 5:40 PM1.8.22 -> 1.9.10, and CI is using macos 12 where locally, my machine is running macOS 11. So.
Why the shit is min version 14? is that just the simulator?Filip Dolník
10/02/2023, 5:41 PMFilip Dolník
10/02/2023, 5:42 PMFilip Dolník
10/02/2023, 5:42 PMMatt Nelson
10/02/2023, 5:43 PMiOS 14 of September 16, 2020 👀russhwolf
10/02/2023, 5:44 PMrusshwolf
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:47 PMiOS 14 of September 16, 2020
Depending on your use case it might not be a big deal. Most iOS users update their OS. Plus if you target KMP projects, then everyone will have the same limit.Filip Dolník
10/02/2023, 5:49 PMMatt Nelson
10/02/2023, 5:50 PMMatt Nelson
10/02/2023, 6:40 PMiosSimulatorArm64, F it...14.0, but Xcode 13 and 14 default to min version 9.0, so linking against it is impossible.
Did I get that right? They fixed it for Xcode 15+, but broke it for Xcode not 15+
EDIT: That didn't fix it, as it's for all iOS targets (even though the ticket expressed only simulator). Fix was to change CI runner to macos-11 temporarily and figure it out later.Matt Nelson
10/02/2023, 6:48 PMversion-min from freeCompilerArgs,
If the resulting .klib contains bitcode that was compiled with -m<platform>-version-min=<version> , does that propagate to consumers of the .klib?
Should I set both cklib compiled code and freeCompilerArgs for the module?Matt Nelson
10/03/2023, 3:11 AM-framework Security link option to module :driver-testFilip Dolník
10/03/2023, 1:15 PMThat didn’t fix it, as it’s for allYeah, it’s for all iOS targetstargetsiOS
The unerlying problem is that Kotlin 1.9.10 is compiled with min versionYes, you need to link with the same min version (or higher)., but Xcode14.0and13default to min version14, so linking against it is impossible.9.0
If the resultingYes, it’s the same problem - you need to link against the highest min version.contains bitcode that was compiled with.klib, does that propagate to consumers of the-m<platform>-version-min=<version>?.klib
Wow, I am regarded... The fix was addingBut I assume the warning about min versions didn’t go away right? If no then it’s still a problem (I’d recommend to address it because it can blow up elsewhere)link option to module-framework Security
Matt Nelson
10/03/2023, 2:17 PMsqlite3).Filip Dolník
10/03/2023, 2:20 PMNope, warning went away completely. Everything compiled and tests ran.Even on
macos-latest ? (you mentioned that at one point you changed CI to use macos-11 )Matt Nelson
10/03/2023, 2:21 PMFilip Dolník
10/03/2023, 2:22 PMMatt Nelson
10/13/2023, 6:28 PM