So, creating a library that uses <SQLite3MultipleC...
# touchlab-tools
m
So, creating a library that uses SQLite3MultipleCiphers for db encryption. Have everything compiling and working on Java and Android via
xerial/sqlite-jdbc
. Now I'm tackling linking things with
SQLiter
, but am sort of at a loss with cinterop. Do I have to compile the amalgamation (.c/.h) files into a static lib for each native platform so they're included with the
.klib
?
I guess I'm wondering how best to go about compiling static libs for each target; use dockcross or something?
k
Hi! This came in while we were at Droidcon last week and got lost in the shuffle. Adding to the todo list to dig in and reply.
m
Heyo, all good Kevin. Got it all figured out and have my scripts setup (because I prefer sh/bash over Make). Things are shaping up with the lib, should be a solid replacement for SQLCipher for all platforms (Jvm, Android, Native).
So, reworked things to have
kotlinc-native
compiler plugin build everything, but CI keeps failing (even though all tests pass locally on my linux machine, windows vm, and macOS vm). What's crazy, is the
ubuntu-latest
runner is based on Ubuntu 22.04 LTS and it's failing, while my local machine is Pop!_OS 22.04 LTS (based off of the latest Ubuntu build). So, unsure wtf is going on. Any suggestions would be much appreciated.
.def
file:
Copy code
package = co.touchlab.sqliter.sqlite3
---
// all sqlite3mc.c amalgamations
build.gradle.kts
cinterop function for compiling
Copy code
fun KotlinNativeTarget.sqlite3mcInterop() {
    val libs = projectDir.resolve("libs")

    compilations["main"].apply {
        cinterops.create("sqlite3mc") {
            defFile(libs.resolve("sqlite3mc.def"))

            compilerOpts += "-O3"

            when (konanTarget.architecture) {
                Architecture.X64,
                Architecture.X86 -> compilerOpts("-msse4.2", "-maes")
                else -> {}
            }

            compilerOpts(
                "-I${libs.resolve("include")}",
                "-DSQLITE_HAVE_ISNAN=1",
                "-DHAVE_USLEEP=1",
                "-DSQLITE_ENABLE_COLUMN_METADATA=1",
                "-DSQLITE_CORE=1",
                "-DSQLITE_ENABLE_FTS3=1",
                "-DSQLITE_ENABLE_FTS3_PARENTHESIS=1",
                "-DSQLITE_ENABLE_FTS5=1",
                "-DSQLITE_ENABLE_RTREE=1",
                "-DSQLITE_ENABLE_STAT4=1",
                "-DSQLITE_ENABLE_DBSTAT_VTAB=1",
                "-DSQLITE_ENABLE_MATH_FUNCTIONS=1",
                "-DSQLITE_THREADSAFE=1",
                "-DSQLITE_DEFAULT_MEMSTATUS=0",
                "-DSQLITE_DEFAULT_FILE_PERMISSIONS=0666",
                "-DSQLITE_MAX_VARIABLE_NUMBER=250000",
                "-DSQLITE_MAX_MMAP_SIZE=1099511627776",
                "-DSQLITE_MAX_LENGTH=2147483647",
                "-DSQLITE_MAX_COLUMN=32767",
                "-DSQLITE_MAX_SQL_LENGTH=1073741824",
                "-DSQLITE_MAX_FUNCTION_ARG=127",
                "-DSQLITE_MAX_ATTACHED=125",
                "-DSQLITE_MAX_PAGE_COUNT=4294967294",
                "-DSQLITE_DQS=0",
                "-DCODEC_TYPE=CODEC_TYPE_CHACHA20",
                "-DSQLITE_ENABLE_EXTFUNC=1",
                "-DSQLITE_ENABLE_REGEXP=1",
                "-DSQLITE_TEMP_STORE=2",
                "-DSQLITE_USE_URI=1",
            )
        }
    }
}
f
Hi! Do you have the error / stack trace from the failing build? This looks like an environment issue (given the project works on another Linux distro). It can be things like missing library or a different version of a library, or some files being in a different folder, etc.
m
Unfortunately, CI ouptut for test failure is awful...
Tests for ephemeral dbs (temporary, in-memory, named) pass, so.
f
The CI log doesn’t contain the error, or more precisely it says:
kotlin.IllegalStateException at null:-1
😄 I’d recommend setting up an Ubuntu 22.04 LTS VM and run it manually. Debugging this in a CI won’t be easy.
sad panda 1
m
Wish there was a way to see terminal output across the llvm bridge when running native tests sad panda
Unless there is and I am simply unaware of it
Here's a stack trace. Seems like it's happening even before trying to decrypt, in the
org.touchlab.sqliter.interop.dbOpen
function's call to
sqlite3_open_v2
It's not an issue with using the
/tmp
directory for the tests, b/c JVM tests work fine. Gotta be something like you suggested; an environment issue like a missing library.
Copy code
io.toxicity.sqlite.mc.driver.test.GeneralNativeTest.givenDriver_whenClose_thenCredentialsCleared[linuxX64] STANDARD_OUT
    kotlin.IllegalStateException: Failed to create DatabaseManager
        at 0   test.kexe                           0x42633e           kfun:kotlin.Exception#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 1   test.kexe                           0x42666e           kfun:kotlin.RuntimeException#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 2   test.kexe                           0x426ade           kfun:kotlin.IllegalStateException#<init>(kotlin.String?;kotlin.Throwable?){} + 142 
        at 3   test.kexe                           0x695ff4           kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 2324 
        at 4   test.kexe                           0x66b663           kfun:io.toxicity.sqlite.mc.driver.SQLiteMCDriver.Factory.$createActual$lambda$4COROUTINE$5.invokeSuspend#internal + 1619 
        at 5   test.kexe                           0x4312c4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
        at 6   test.kexe                           0x636cd0           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504 
        at 7   test.kexe                           0x63aae4           kfun:kotlinx.coroutines.internal.LimitedDispatcher.Worker.run#internal + 388 
        at 8   test.kexe                           0x663085           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$2COROUTINE$0.invokeSuspend#internal + 3077 
        at 9   test.kexe                           0x4312c4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
        at 10  test.kexe                           0x636cd0           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3504 
        at 11  test.kexe                           0x5ddfc1           kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1585 
        at 12  test.kexe                           0x659692           kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 674 
        at 13  test.kexe                           0x658d46           kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2134 
        at 14  test.kexe                           0x659008           kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 312 
        at 15  test.kexe                           0x660538           kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 200 
        at 16  test.kexe                           0x66205c           kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$1$lambda$0#internal + 76 
        at 17  test.kexe                           0x664000           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.invoke#internal + 80 
        at 18  test.kexe                           0x6640e0           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$1$lambda$0$FUNCTION_REFERENCE$3.$<bridge-UNN>invoke(){}#internal + 80 
        at 19  test.kexe                           0x43cd8a           WorkerLaunchpad + 170 
        at 20  test.kexe                           0x779a40           _ZN6Worker19processQueueElementEb + 4624 
        at 21  test.kexe                           0x77876e           _ZN12_GLOBAL__N_113workerRoutineEPv + 142 
        at 22  libc.so.6                           0x7f0215894b42     0x0 + 139646927981378 
        at 23  libc.so.6                           0x7f02159269ff     0x0 + 139646928579071 
    Caused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
        at 0   test.kexe                           0x42c5f1           kfun:kotlin.Throwable#<init>(kotlin.String?){} + 113 
        at 1   test.kexe                           0x42628d           kfun:kotlin.Exception#<init>(kotlin.String?){} + 109 
        at 2   test.kexe                           0x58d2d7           kfun:co.touchlab.sqliter.interop.SQLiteException#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig){} + 135 
        at 3   test.kexe                           0x58d401           kfun:co.touchlab.sqliter.interop.SQLiteExceptionErrorCode#<init>(kotlin.String;co.touchlab.sqliter.interop.SqliteDatabaseConfig;kotlin.Int){} + 257 
        at 4   test.kexe                           0x596749           kfun:co.touchlab.sqliter.interop#dbOpen(kotlin.String;kotlin.collections.List<co.touchlab.sqliter.interop.OpenFlags>;kotlin.String;kotlin.Boolean;kotlin.Boolean;kotlin.Int;kotlin.Int;kotlin.Int;co.touchlab.sqliter.interop.Logger;kotlin.Boolean){}co.touchlab.sqliter.interop.SqliteDatabase + 4697 
        at 5   test.kexe                           0x5a62e2           kfun:co.touchlab.sqliter.native.NativeDatabaseManager.createConnection#internal + 2466 
        at 6   test.kexe                           0x5a55ea           kfun:co.touchlab.sqliter.native.NativeDatabaseManager#createMultiThreadedConnection(){}co.touchlab.sqliter.DatabaseConnection + 234 
        at 7   test.kexe                           0x57c7fd           kfun:co.touchlab.sqliter#withConnection__at__co.touchlab.sqliter.DatabaseManager(kotlin.Function1<co.touchlab.sqliter.DatabaseConnection,0:0>){0§<kotlin.Any?>}0:0 + 365 
        at 8   test.kexe                           0x695dbc           kfun:io.toxicity.sqlite.mc.driver.PlatformDriver.Companion#create__at__io.toxicity.sqlite.mc.driver.config.FactoryConfig(kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>;kotlin.collections.MutableMap<io.toxicity.sqlite.mc.driver.config.Pragma<*>,kotlin.String>?){}io.toxicity.sqlite.mc.driver.PlatformDriver.Companion.Args + 1756 
        ... and 20 more common stack frames skipped
f
the important part is probably this:
Copy code
Caused by: co.touchlab.sqliter.interop.SQLiteExceptionErrorCode: unable to open database file
So I’d start by looking for the file it tries to open and the version of the database (and compare those to what you get in Pop!_OS)
m
File does not exist yet. All tests run with a random 64 character hex string for db name (and are subsequently deleted after test runner completes). Think it's something within the
sqlite3mc.c
amalgamations sad panda
Well, or something with the compilation of them.
wish I could compile a shared lib and distribute it like the JVM/Android side....
k
They build C code and stuff it in a klib, which can then be sent out as a Kotlin/Native dependency.
m
👀
k
Disclaimer: we very much do not support cklib. Mostly because the vast majority of the questions have nothing to do with the library and are mostly from people who don't quite understand the native/llvm build chain. However, since you're doing an open source thing, time permitting, I'll see what I can do 🙂
m
Well I appreciate it very much! Know you're a busy guy, so will dig as deep as I can before asking
k
This talk is labeled wrong on the site, but it was a talk that did a deep dive on cinterop, etc: https://www.droidcon.com/2022/06/28/sdk-design-and-publishing-for-kotlin-multiplatform-mobile/
It's conceptually very simple. cklib gets the clang flags for each target directly from the Kotlin Gradle plugin jars and uses those to configure the C build per-target. It then takes the binary, and packages it with the klib in such a way that the Kotlin compiler will see it and add it automatically. The Cash project I liked to builds it's own JS interpreter for distribution from C code, but you (the developer) don't need to deal with that.
On the down side, because cklib uses those jars, and accesses them with reflection, it is brittle and prone to issues with Kotlin version changes.
But, it seems to continue to work with zipline, so you'll be OK 🙂 Eventually the plan is to just grab those settings and put them in cklib directly. It's not like clang really changes all that much.
But, calling cklib a "niche library" would be putting it mildly, so it hasn't been a priority.
m
damn... I am pumped about this now b/c I was racking my brain on how best to support native targets in another project of mine https://github.com/05nelsonm/kmp-tor
k
Embedding tor in an app? Interesting.
m
yup.
sqlite-mc
is a part of another project I'm working on under
toxicity-io
that embeds Tor + KTor server, and allows apps to register with it (like firebase messaging) and will act as a reverse proxy exposing that app as a Tor Hidden Service. A viable open source replacement for google push notifications 😄
Well, thanks to
cklib
I think I found the underlying issue that I was having earlier Seems that
libtinfo
is causing the issues From CI (ubuntu-latest 22.04 LTS)
Copy code
> Task :library:android-unit-test:test NO-SOURCE
> Task :library:android-unit-test:check
> Task :library:driver:jvmProcessResources NO-SOURCE
> Task :library:driver:jvmTestProcessResources NO-SOURCE
/home/runner/.cklib/clang-llvm-8.0.0-linux-x86-64/bin/clang: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

> Task :library:driver:linuxX64Libsqlite3mc FAILED
From my machine (Pop!_OS 22.04 LTS)
Copy code
$ ldd $(which info)
        linux-vdso.so.1 (0x00007ffd39bf4000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc193842000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc193600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc1938d7000)
EDIT: actually, this is an issue with the version of llvm that
cklib
is using.
native side of this is wild, especially for linux. Need to compile it as a shared lib
.so or .dylib
(like how it's being done in
xerial/sqlite-jdbc
) and somehow package that shared lib inside the .klib for library consumers, then link to it dynamically using relative paths... Might just only support
ios
for the time being as one can get away with compiling a static lib
.a
and including it via cinterop...
Alright, got everything compiling (almost) using
cklib
for
iOS
,
tvOS
, and
watchOS
with 1 exception...
Copy code
> Task :library:driver:iosSimulatorArm64Test SKIPPED
e: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
> Task :library:driver-test:linkDebugTestIosSimulatorArm64 FAILED
The /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
ld: warning: object file (/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/konan_temp130378032671[85](<https://github.com/toxicity-io/sqlite-mc/actions/runs/6381376828/job/17317720607?pr=43#step:9:85)517018/result.o)> was built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
  "_SecRandomCopyBytes", referenced from:
      _entropy in result.o
  "_kSecRandomDefault", referenced from:
      _entropy in result.o
ld: symbol(s) not found for architecture arm64
> Task :library:driver:iosX64Sqlite3mc
FAILURE: Build failed with an exception.
So, module
:driver
compiles just fine, but when
:driver-test
goes to link against it, this warning keeps popping up and the build fails. This is only occurring on
CI
(using
macos-latest
), and does not happen on my vm of
macOS 11
Any thoughts? I've tried a plethora of things w/o success: • Adding to each Darwin target
freeCompilerArgs += listOf("-linker-options", "-framework Security")
• Adding to
compilerArgs
for each Darwin target
-isysroot </path/to/platform/specific/sdk> -iframeworkwithsysroot Security
• Adding to
linkerArgs
the
--suppress-warnings
flag
f
Try fixing this warning because this version mismatch can often cause missing symbols and similar types of issues:
Copy code
was built for newer iOS Simulator version (14.0) than being linked (9.0)
Undefined symbols for architecture arm64:
m
Is there a way to override the
--target
declared?
b/c that's declaring the xcode version in the triple cross compilation target: e.g.
amd64-apple-iphoneos14.0-simulator
f
yes one option: ``freeCompilerArgs += listOf("-Xoverride-konan-properties=osVersionMin.ios_arm32=7;osVersionMin.ios_arm64=7;osVersionMin.ios_x64=7")`` but there might some other API
This is only occurring on
CI
(using
macos-latest
), and does not happen on my vm of
macOS 11
These environments will likely have a different version of Xcode installed
m
So I've declared version-min in the
compilerArgs
for
cklib
, does that do nothing?
Copy code
if (kt.family.isAppleFamily && xcode != null) {
            when (kt) {
                IOS_ARM64 -> listOf(
                    "-mios-version-min=9.0",
                    "-isysroot",
                    xcode.iphoneosSdk,
                )
                IOS_SIMULATOR_ARM64 -> listOf(
                    "-mios-simulator-version-min=9.0",
                    "-isysroot",
                    xcode.iphonesimulatorSdk,
                )
                IOS_X64 -> listOf(
                    "-mios-version-min=9.0",
                    "-isysroot",
                    xcode.iphoneosSdk,
                )

                MACOS_ARM64 -> listOf(
                    "-mmacosx-version-min=10.9",
                    "-isysroot",
                    xcode.macosxSdk,
                )
                MACOS_X64 -> listOf(
                    "-mmacosx-version-min=10.7",
                    "-isysroot",
                    xcode.macosxSdk,
                )

                TVOS_ARM64 -> listOf(
                    "-mtvos-version-min=9.0",
                    "-isysroot",
                    xcode.appletvosSdk,
                )
                TVOS_SIMULATOR_ARM64 -> listOf(
                    "-mtvos-simulator-version-min=9.0",
                    "-isysroot",
                    xcode.appletvsimulatorSdk,
                )
                TVOS_X64 -> listOf(
                    "-mtvos-version-min=9.0",
                    "-isysroot",
                    xcode.appletvosSdk,
                )

                WATCHOS_ARM32 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_ARM64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_DEVICE_ARM64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                WATCHOS_SIMULATOR_ARM64 -> listOf(
                    "-mwatchos-simulator-version-min=3.0",
                    "-isysroot",
                    xcode.watchsimulatorSdk,
                )
                WATCHOS_X64 -> listOf(
                    "-mwatchos-version-min=3.0",
                    "-isysroot",
                    xcode.watchosSdk,
                )
                else -> null
            }?.let { compilerArgs.addAll(it) }
        }
f
Oh, I think I know what’s going on. The minimum iOS version for Kotlin 1.9 and Xcode 15 is iOS 14. You cannot set it lower. That works only with older version of Xcode
m
Ah ok, I've just updated from kotlin
1.8.22
->
1.9.10
, and
CI
is using
macos 12
where locally, my machine is running
macOS 11
. So. Why the shit is min version 14? is that just the simulator?
f
yeah this was changed in 1.9.0. I think it’s for all iOS targets
I don’t know why they changed it, but I would assume it was to fix some issues in the compiler
Actually, it was changed in 1.9.10
m
Release date for
iOS 14
of
September 16, 2020
👀
r
1.9.10 fixed some Xcode 15 incompatibilities. I'm guessing Xcode is what drove the min ios requirement.
But also I'm not sure there's many good reasons to be below 14 at this point
f
(this was the issue)
> Release date for
iOS 14
of
September 16, 2020
Depending on your use case it might not be a big deal. Most iOS users update their OS. Plus if you target KMP projects, then everyone will have the same limit.
And if you want to support older OS versions, you can stay on Xcode 14 (you can change which Xcode the CI uses) for a little bit longer.
m
Well, thank you for having all of the answers this afternoon, lol. Saved me bundles of time 🙏
I just disabled
iosSimulatorArm64
, F it...
The unerlying problem is that Kotlin 1.9.10 is compiled with min version
14.0
, but Xcode
13
and
14
default to min version
9.0
, so linking against it is impossible. Did I get that right? They fixed it for Xcode 15+, but broke it for Xcode not 15+ EDIT: That didn't fix it, as it's for all
iOS
targets (even though the ticket expressed only simulator). Fix was to change
CI
runner to
macos-11
temporarily and figure it out later.
So regarding the ability to set
version-min
from
freeCompilerArgs
, If the resulting
.klib
contains bitcode that was compiled with
-m<platform>-version-min=<version>
, does that propagate to consumers of the
.klib
? Should I set both
cklib
compiled code and
freeCompilerArgs
for the module?
Wow, I am regarded... The fix was adding
-framework Security
link option to module
:driver-test
f
That didn’t fix it, as it’s for all
iOS
targets
Yeah, it’s for all iOS targets
The unerlying problem is that Kotlin 1.9.10 is compiled with min version
14.0
, but Xcode
13
and
14
default to min version
9.0
, so linking against it is impossible.
Yes, you need to link with the same min version (or higher).
If the resulting
.klib
contains bitcode that was compiled with
-m<platform>-version-min=<version>
, does that propagate to consumers of the
.klib
?
Yes, it’s the same problem - you need to link against the highest min version.
Wow, I am regarded... The fix was adding
-framework Security
link option to module
But I assume the warning about min versions didn’t go away right? If no then it’s still a problem (I’d recommend to address it because it can blow up elsewhere)
m
> But I assume the warning about min versions didn’t go away right? Nope, warning went away completely. Everything compiled and tests ran. Building a gradle plugin now that extends the SQLDelightPlugin which'll handle all the dependency, dialects, and linking stuff (b/c no longer linking agains
sqlite3
).
f
Nope, warning went away completely. Everything compiled and tests ran.
Even on
macos-latest
? (you mentioned that at one point you changed CI to use
macos-11
)
m
Yup, CI ran just fine, too.
f
Interesting. But at least it works.
m
Thanks for all your help, gents. Was able to publish an alpha release today 🦜
👍 1