Is using <CacheStorage> supposed to work offline? ...

# ktor

[Luca]

Is using CacheStorage supposed to work offline? It does work offline for a cached request on android (with the android engine). But returns a connection error for all jvm engines and the native darwin engine? I’m and trying to fix loading cached images offline for kamel on native and jvm desktop builds

[Aleksei Tirman [JB]]

The FileStorage interacts only with the file system so it works offline. Can you share the code where you test this behavior?

[Luca]

Actually android fails as well.

You can run kamel-samples (which opens to the gallery page that loads remote images that are cached) once online to save the data then again offline to try and load the gallery.

It will fail to load the cached image data after calling an api offline:

jvm desktop for example will fail with:

java.nio.channels.UnresolvedAddressException
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:137)
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:145)
	at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:842)
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:865)
	at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:44)
	at io.ktor.network.sockets.ConnectUtilsJvmKt.connect(ConnectUtilsJvm.kt:21)
	at io.ktor.network.sockets.TcpSocketBuilder.connect(TcpSocketBuilder.kt:37)
	at io.ktor.client.engine.cio.ConnectionFactory.connect(ConnectionFactory.kt:30)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invokeSuspend(Endpoint.kt:207)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invoke(Endpoint.kt)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invoke(Endpoint.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:72)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:148)
	at kotlinx.coroutines.TimeoutKt.withTimeoutOrNull(Timeout.kt:104)
	at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:215)
	at io.ktor.client.engine.cio.Endpoint.makeDedicatedRequest(Endpoint.kt:100)
	at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:64)
	at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:79)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:99)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)

[Luca]

This is our multiplatform CacheStorage implementation.

https://github.com/Kamel-Media/Kamel/blob/013f42e1c6bff04e99890b9bc6e8ca09601d4015[…]/commonMain/kotlin/io/kamel/core/cache/disk/DiskCacheStorage.kt

and it is installed in the client here:

https://github.com/Kamel-Media/Kamel/blob/b5bb34d86b9e65e448451b20412b5ed275ad60fa[…]rc/commonMain/kotlin/io/kamel/core/config/KamelConfigBuilder.kt

This is how requests are made:

https://github.com/Kamel-Media/Kamel/blob/013f42e1c6bff04e99890b9bc6e8ca09601d4015[…]core/src/commonMain/kotlin/io/kamel/core/fetcher/HttpFetcher.kt

CacheStorage.find is not called at all when offline

[Aleksei Tirman [JB]]

The problem may be in the DNS resolver, which tries to resolve the domain name using the network. Can you share a reproducer where only Ktor is used (without Kamel dependency)?

[Luca]

Probably don’t have time today. But, i’ll try to make a sample soon, I’ll let you know.

[Luca]

@Aleksei Tirman [JB] https://github.com/luca992/ktor-documentation/blob/db044e5837b946846a26eaf2e917349[…]ppets/client-caching/src/main/kotlin/com/example/Application.kt

here I just modified the sample to get a remote image. If you run it twice once online, then offline running offline will fail:

Exception in thread "main" java.nio.channels.UnresolvedAddressException
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
	at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
	at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:44)
	at io.ktor.network.sockets.ConnectUtilsJvmKt.connect(ConnectUtilsJvm.kt:21)
	at io.ktor.network.sockets.TcpSocketBuilder.connect(TcpSocketBuilder.kt:37)
	at io.ktor.client.engine.cio.ConnectionFactory.connect(ConnectionFactory.kt:30)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invokeSuspend(Endpoint.kt:207)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invoke(Endpoint.kt)
	at io.ktor.client.engine.cio.Endpoint$connect$2$connect$1.invoke(Endpoint.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:89)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:151)
	at kotlinx.coroutines.TimeoutKt.withTimeoutOrNull(Timeout.kt:107)
	at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:215)
	at io.ktor.client.engine.cio.Endpoint.makeDedicatedRequest(Endpoint.kt:100)
	at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:64)
	at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:79)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:99)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

FAILURE: Build failed with an exception.

[Aleksei Tirman [JB]]

There are two problems here. The first one is DNS resolution, which requires a network connection. The second one is a redirect from the picsum.photos to the fastly.picsum.photos , which requires a connection, too. So, if I add the following mappings into the hosts file:

104.26.4.30     picsum.photos
146.75.117.91   fastly.picsum.photos

And replace the URLs with the ones located by the redirect:

client.get("<https://fastly.picsum.photos/id/134/500/500.jpg?hmac=B9KyHkmMllXNwIDo64ywJL9MAt8E-paC3WhMGHTfcy4>")
client.get("<https://fastly.picsum.photos/id/134/500/500.jpg?hmac=B9KyHkmMllXNwIDo64ywJL9MAt8E-paC3WhMGHTfcy4>")

Then, the second code run works without a connection to the network.

[Luca]

@Aleksei Tirman [JB] so basically it's not possible to use it offline without modifying the system hosts file? Since there's no way to have ktor persist DNS resolution and fallback to it when the Internet is unavailable.

[Luca]

Also modifying the hosts file isn't really possible on mobile afaik.

[Aleksei Tirman [JB]]

You can either use the OkHttp engine, where the OkHttp client builder supports setting up a custom DNS service or set up a local DNS server, which could cache resolutions locally.

[Luca]

This is for a multiplatform image caching library. So just I don’t think that will really be helpful. Is there a way I can intercept redirects so that I can get the actual endpoint and persist those mappings to disk so then I can check if those mappings exist offline and then try and call CacheStorage instance directly to get the cached data?

[Aleksei Tirman [JB]]

You can write a plugin that adds a handler for the Send hook. Here is an example:

val plugin = createClientPlugin("my plugin") {
    on(Send) { request ->
        val call = proceed(request)
        if (listOf(301, 302, 307, 308).contains(call.response.status.value)) {
            println(call.response.headers[HttpHeaders.Location])
        }
        call
    }
}