Matt Nelson
09/15/2023, 6:24 PMxerial/sqlite-jdbc
.
Now I'm tackling linking things with SQLiter
, but am sort of at a loss with cinterop.
Do I have to compile the amalgamation (.c/.h) files into a static lib for each native platform so they're included with the .klib
?Matt Nelson
09/15/2023, 6:25 PMMatt Nelson
09/15/2023, 6:36 PMkpgalligan
09/20/2023, 6:52 PMMatt Nelson
09/25/2023, 8:56 PMMatt Nelson
09/27/2023, 10:01 AMkotlinc-native
compiler plugin build everything, but CI keeps failing (even though all tests pass locally on my linux machine, windows vm, and macOS vm).
What's crazy, is the ubuntu-latest
runner is based on Ubuntu 22.04 LTS and it's failing, while my local machine is Pop!_OS 22.04 LTS (based off of the latest Ubuntu build).
So, unsure wtf is going on. Any suggestions would be much appreciated.
.def
file:
package = co.touchlab.sqliter.sqlite3
---
// all sqlite3mc.c amalgamations
build.gradle.kts
cinterop function for compiling
fun KotlinNativeTarget.sqlite3mcInterop() {
val libs = projectDir.resolve("libs")
compilations["main"].apply {
cinterops.create("sqlite3mc") {
defFile(libs.resolve("sqlite3mc.def"))
compilerOpts += "-O3"
when (konanTarget.architecture) {
Architecture.X64,
Architecture.X86 -> compilerOpts("-msse4.2", "-maes")
else -> {}
}
compilerOpts(
"-I${libs.resolve("include")}",
"-DSQLITE_HAVE_ISNAN=1",
"-DHAVE_USLEEP=1",
"-DSQLITE_ENABLE_COLUMN_METADATA=1",
"-DSQLITE_CORE=1",
"-DSQLITE_ENABLE_FTS3=1",
"-DSQLITE_ENABLE_FTS3_PARENTHESIS=1",
"-DSQLITE_ENABLE_FTS5=1",
"-DSQLITE_ENABLE_RTREE=1",
"-DSQLITE_ENABLE_STAT4=1",
"-DSQLITE_ENABLE_DBSTAT_VTAB=1",
"-DSQLITE_ENABLE_MATH_FUNCTIONS=1",
"-DSQLITE_THREADSAFE=1",
"-DSQLITE_DEFAULT_MEMSTATUS=0",
"-DSQLITE_DEFAULT_FILE_PERMISSIONS=0666",
"-DSQLITE_MAX_VARIABLE_NUMBER=250000",
"-DSQLITE_MAX_MMAP_SIZE=1099511627776",
"-DSQLITE_MAX_LENGTH=2147483647",
"-DSQLITE_MAX_COLUMN=32767",
"-DSQLITE_MAX_SQL_LENGTH=1073741824",
"-DSQLITE_MAX_FUNCTION_ARG=127",
"-DSQLITE_MAX_ATTACHED=125",
"-DSQLITE_MAX_PAGE_COUNT=4294967294",
"-DSQLITE_DQS=0",
"-DCODEC_TYPE=CODEC_TYPE_CHACHA20",
"-DSQLITE_ENABLE_EXTFUNC=1",
"-DSQLITE_ENABLE_REGEXP=1",
"-DSQLITE_TEMP_STORE=2",
"-DSQLITE_USE_URI=1",
)
}
}
}
Filip Dolník
09/27/2023, 10:06 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:08 AMMatt Nelson
09/27/2023, 10:10 AMFilip Dolník
09/27/2023, 10:11 AMkotlin.IllegalStateException at null:-1
😄
I’d recommend setting up an Ubuntu 22.04 LTS VM and run it manually. Debugging this in a CI won’t be easy.Matt Nelson
09/27/2023, 11:30 AMMatt Nelson
09/27/2023, 11:31 AMMatt Nelson
09/27/2023, 1:47 PMorg.touchlab.sqliter.interop.dbOpen
function's call to sqlite3_open_v2
It's not an issue with using the /tmp
directory for the tests, b/c JVM tests work fine. Gotta be something like you suggested; an environment issue like a missing library.
io.toxicity.sqlite.mc.driver.test.GeneralNativeTest.givenDriver_whenClose_thenCredentialsCleared[linuxX64] STANDARD_OUT
kotlin.IllegalStateException: Failed to create DatabaseManager
at 0 test.kexe 0x42633e kfun:kotlin.Exception#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 1 test.kexe 0x42666e kfun:kotlin.RuntimeException#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 2 test.kexe 0x426ade kfun:kotlin.IllegalStateException#<init>(kotlin.String?;kotlin.Throwable?){} + 142
at 3 test.kexe 0x695ff4 kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 2324
at 4 test.kexe 0x66b663 kfun:io.toxicity.sqlite.mc.driver.SQLiteMCDriver.Factory.$createActual$lambda$4COROUTINE$5.invokeSuspend#internal + 1619
at 5 test.kexe 0x4312c4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 6 test.kexe 0x636cd0 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504
at 7 test.kexe 0x63aae4 kfun:kotlinx.coroutines.internal.LimitedDispatcher.Worker.run#internal + 388
at 8 test.kexe 0x663085 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$2COROUTINE$0.invokeSuspend#internal + 3077
at 9 test.kexe 0x4312c4 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916
at 10 test.kexe 0x636cd0 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504
at 11 test.kexe 0x5ddfc1 kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1585
at 12 test.kexe 0x659692 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 674
at 13 test.kexe 0x658d46 kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2134
at 14 test.kexe 0x659008 kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 312
at 15 test.kexe 0x660538 kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 200
at 16 test.kexe 0x66205c kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$1$lambda$0#internal + 76
at 17 test.kexe 0x664000 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.invoke#internal + 80
at 18 test.kexe 0x6640e0 kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.$<bridge-UNN>invoke(){}#internal + 80
at 19 test.kexe 0x43cd8a WorkerLaunchpad + 170
at 20 test.kexe 0x779a40 _ZN6Worker19processQueueElementEb + 4624
at 21 test.kexe 0x77876e _ZN12_GLOBAL__N_113workerRoutineEPv + 142
at 22 libc.so.6 0x7f0215894b42 0x0 + 139646927981378
at 23 libc.so.6 0x7f02159269ff 0x0 + 139646928579071
Caused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
at 0 test.kexe 0x42c5f1 kfun:kotlin.Throwable#<init>(kotlin.String?){} + 113
at 1 test.kexe 0x42628d kfun:kotlin.Exception#<init>(kotlin.String?){} + 109
at 2 test.kexe 0x58d2d7 kfun:co.touchlab.sqliter.interop.SQLiteException#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig){} + 135
at 3 test.kexe 0x58d401 kfun:co.touchlab.sqliter.interop.SQLiteExceptionErrorCode#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig;kotlin.Int){} + 257
at 4 test.kexe 0x596749 kfun:co.touchlab.sqliter.interop#dbOpen(kotlin.String;kotlin.collections.List<co.touchlab.sqliter.interop.OpenFlags>;kotlin.String;kotlin.Boolean;kotlin.Boolean;kotlin.Int;kotlin.Int;kotlin.Int;co.touchlab.sqliter.interop.Logger;kotlin.Boolean){}co.touchlab.sqliter.interop.SqliteDatabase + 4697
at 5 test.kexe 0x5a62e2 kfun:co.touchlab.sqliter.native.NativeDatabaseManager.createConnection#internal + 2466
at 6 test.kexe 0x5a55ea kfun:co.touchlab.sqliter.native.NativeDatabaseManager#createMultiThreadedConnection(){}co.touchlab.sqliter.DatabaseConnection + 234
at 7 test.kexe 0x57c7fd kfun:co.touchlab.sqliter#withConnection__at__co.touchlab.sqliter.DatabaseManager(kotlin.Function1<co.touchlab.sqliter.DatabaseConnection,0:0>){0§<kotlin.Any?>}0:0 + 365
at 8 test.kexe 0x695dbc kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 1756
... and 20 more common stack frames skipped
Filip Dolník
09/27/2023, 1:52 PMCaused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
So I’d start by looking for the file it tries to open and the version of the database (and compare those to what you get in Pop!_OS)Matt Nelson
09/27/2023, 1:56 PMsqlite3mc.c
amalgamations sad pandaMatt Nelson
09/27/2023, 1:58 PMMatt Nelson
09/27/2023, 2:26 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:29 PMkpgalligan
09/27/2023, 2:30 PMMatt Nelson
09/27/2023, 2:30 PMkpgalligan
09/27/2023, 2:31 PMMatt Nelson
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:32 PMkpgalligan
09/27/2023, 2:35 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:36 PMkpgalligan
09/27/2023, 2:37 PMMatt Nelson
09/27/2023, 2:50 PMkpgalligan
09/27/2023, 2:51 PMMatt Nelson
09/27/2023, 2:53 PMsqlite-mc
is a part of another project I'm working on under toxicity-io
that embeds Tor + KTor server, and allows apps to register with it (like firebase messaging) and will act as a reverse proxy exposing that app as a Tor Hidden Service.
A viable open source replacement for google push notifications 😄Matt Nelson
09/27/2023, 7:14 PMcklib
I think I found the underlying issue that I was having earlier
Seems that libtinfo
is causing the issues
From CI (ubuntu-latest 22.04 LTS)
> Task :library:android-unit-test:test NO-SOURCE
> Task :library:android-unit-test:check
> Task :library:driver:jvmProcessResources NO-SOURCE
> Task :library:driver:jvmTestProcessResources NO-SOURCE
/home/runner/.cklib/clang-llvm-8.0.0-linux-x86-64/bin/clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory
> Task :library:driver:linuxX64Libsqlite3mc FAILED
From my machine (Pop!_OS 22.04 LTS)
$ ldd $(which info)
linux-vdso.so.1 (0x00007ffd39bf4000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc193842000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc193600000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1938d7000)
EDIT: actually, this is an issue with the version of llvm that cklib
is using.Matt Nelson
09/28/2023, 3:58 PM.so or .dylib
(like how it's being done in xerial/sqlite-jdbc
) and somehow package that shared lib inside the .klib for library consumers, then link to it dynamically using relative paths...
Might just only support ios
for the time being as one can get away with compiling a static lib .a
and including it via cinterop...Matt Nelson
10/02/2023, 5:28 PMcklib
for iOS
, tvOS
, and watchOS
with 1 exception...
> Task :library:driver:iosSimulatorArm64Test SKIPPED
e: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
> Task :library:driver-test:linkDebugTestIosSimulatorArm64 FAILED
The /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
ld: warning: object file (/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/konan_temp130378032671[85](<https://github.com/toxicity-io/sqlite-mc/actions/runs/6381376828/job/17317720607?pr=43#step:9:85)517018/result.o)> was built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
"_SecRandomCopyBytes", referenced from:
_entropy in result.o
"_kSecRandomDefault", referenced from:
_entropy in result.o
ld: symbol(s) not found for architecture arm64
> Task :library:driver:iosX64Sqlite3mc
FAILURE: Build failed with an exception.
So, module :driver
compiles just fine, but when :driver-test
goes to link against it, this warning keeps popping up and the build fails.
This is only occurring on CI
(using macos-latest
), and does not happen on my vm of macOS 11
Any thoughts?
I've tried a plethora of things w/o success:
• Adding to each Darwin target freeCompilerArgs += listOf("-linker-options", "-framework Security")
• Adding to compilerArgs
for each Darwin target -isysroot </path/to/platform/specific/sdk> -iframeworkwithsysroot Security
• Adding to linkerArgs
the --suppress-warnings
flagFilip Dolník
10/02/2023, 5:30 PMwas built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
Matt Nelson
10/02/2023, 5:31 PM--target
declared?Matt Nelson
10/02/2023, 5:32 PMamd64-apple-iphoneos14.0-simulator
Filip Dolník
10/02/2023, 5:33 PMThis is only occurring onThese environments will likely have a different version of Xcode installed(usingCI
), and does not happen on my vm ofmacos-latest
macOS 11
Matt Nelson
10/02/2023, 5:36 PMcompilerArgs
for cklib
, does that do nothing?
if (kt.family.isAppleFamily && xcode != null) {
when (kt) {
IOS_ARM64 -> listOf(
"-mios-version-min=9.0",
"-isysroot",
xcode.iphoneosSdk,
)
IOS_SIMULATOR_ARM64 -> listOf(
"-mios-simulator-version-min=9.0",
"-isysroot",
xcode.iphonesimulatorSdk,
)
IOS_X64 -> listOf(
"-mios-version-min=9.0",
"-isysroot",
xcode.iphoneosSdk,
)
MACOS_ARM64 -> listOf(
"-mmacosx-version-min=10.9",
"-isysroot",
xcode.macosxSdk,
)
MACOS_X64 -> listOf(
"-mmacosx-version-min=10.7",
"-isysroot",
xcode.macosxSdk,
)
TVOS_ARM64 -> listOf(
"-mtvos-version-min=9.0",
"-isysroot",
xcode.appletvosSdk,
)
TVOS_SIMULATOR_ARM64 -> listOf(
"-mtvos-simulator-version-min=9.0",
"-isysroot",
xcode.appletvsimulatorSdk,
)
TVOS_X64 -> listOf(
"-mtvos-version-min=9.0",
"-isysroot",
xcode.appletvosSdk,
)
WATCHOS_ARM32 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_ARM64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_DEVICE_ARM64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
WATCHOS_SIMULATOR_ARM64 -> listOf(
"-mwatchos-simulator-version-min=3.0",
"-isysroot",
xcode.watchsimulatorSdk,
)
WATCHOS_X64 -> listOf(
"-mwatchos-version-min=3.0",
"-isysroot",
xcode.watchosSdk,
)
else -> null
}?.let { compilerArgs.addAll(it) }
}
Filip Dolník
10/02/2023, 5:38 PMMatt Nelson
10/02/2023, 5:40 PM1.8.22
-> 1.9.10
, and CI
is using macos 12
where locally, my machine is running macOS 11
. So.
Why the shit is min version 14? is that just the simulator?Filip Dolník
10/02/2023, 5:41 PMFilip Dolník
10/02/2023, 5:42 PMFilip Dolník
10/02/2023, 5:42 PMMatt Nelson
10/02/2023, 5:43 PMiOS 14
of September 16, 2020
👀russhwolf
10/02/2023, 5:44 PMrusshwolf
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:45 PMFilip Dolník
10/02/2023, 5:47 PMiOS 14
of September 16, 2020
Depending on your use case it might not be a big deal. Most iOS users update their OS. Plus if you target KMP projects, then everyone will have the same limit.Filip Dolník
10/02/2023, 5:49 PMMatt Nelson
10/02/2023, 5:50 PMMatt Nelson
10/02/2023, 6:40 PMiosSimulatorArm64
, F it...14.0
, but Xcode 13
and 14
default to min version 9.0
, so linking against it is impossible.
Did I get that right? They fixed it for Xcode 15+, but broke it for Xcode not 15+
EDIT: That didn't fix it, as it's for all iOS
targets (even though the ticket expressed only simulator). Fix was to change CI
runner to macos-11
temporarily and figure it out later.Matt Nelson
10/02/2023, 6:48 PMversion-min
from freeCompilerArgs
,
If the resulting .klib
contains bitcode that was compiled with -m<platform>-version-min=<version>
, does that propagate to consumers of the .klib
?
Should I set both cklib
compiled code and freeCompilerArgs
for the module?Matt Nelson
10/03/2023, 3:11 AM-framework Security
link option to module :driver-test
Filip Dolník
10/03/2023, 1:15 PMThat didn’t fix it, as it’s for allYeah, it’s for all iOS targetstargetsiOS
The unerlying problem is that Kotlin 1.9.10 is compiled with min versionYes, you need to link with the same min version (or higher)., but Xcode14.0
and13
default to min version14
, so linking against it is impossible.9.0
If the resultingYes, it’s the same problem - you need to link against the highest min version.contains bitcode that was compiled with.klib
, does that propagate to consumers of the-m<platform>-version-min=<version>
?.klib
Wow, I am regarded... The fix was addingBut I assume the warning about min versions didn’t go away right? If no then it’s still a problem (I’d recommend to address it because it can blow up elsewhere)link option to module-framework Security
Matt Nelson
10/03/2023, 2:17 PMsqlite3
).Filip Dolník
10/03/2023, 2:20 PMNope, warning went away completely. Everything compiled and tests ran.Even on
macos-latest
? (you mentioned that at one point you changed CI to use macos-11
)Matt Nelson
10/03/2023, 2:21 PMFilip Dolník
10/03/2023, 2:22 PMMatt Nelson
10/13/2023, 6:28 PM