Hi guys, Ran into a problem with request timeout with ktor client when downloading big files with s...
s

Sergei

over 2 years ago
Hi guys, Ran into a problem with request timeout with ktor client when downloading big files with slow internet speed. I have these timeouts set:
val httpClient = HttpClient(OkHttp) {
    install(HttpTimeout) {
        requestTimeoutMillis = 15000 // 15 seconds
        connectTimeoutMillis = 15000 // 15 seconds
        socketTimeoutMillis = 300000 // 5 minutes
    }
}
The way I receive it is pretty much the same as described in ktor client documentation here:
logger.debug("Downloading `$resourcePath` of size $length")

val output = FileOutputStream(fileToUpdate, true)
httpClient.prepareGet(url) {
    header("Authorization", "Bearer $SLAVE_TOKEN")
}.execute { httpResponse ->
    val channel: ByteReadChannel = httpResponse.body()
    while (!channel.isClosedForRead) {
        val packet = channel.readRemaining(DEFAULT_BUFFER_SIZE.toLong())
        while (!packet.isEmpty) {
            val bytes = packet.readBytes()
            output.write(bytes)
        }
    }
}
With big files I get this:
[DEBUG] 2023-06-25 17:51:54.678 [DefaultDispatcher-worker-83]: Downloading `clients/techno_magic/mods/netherlicious-3.0.2.jar` of size 98916964
[ERROR] 2023-06-25 17:52:09.799 [DefaultDispatcher-worker-31]: Failed to download `clients/techno_magic/mods/netherlicious-3.0.2.jar`
io.ktor.client.plugins.HttpRequestTimeoutException: Request timeout has expired [url=<http://example.com/api/resource/clients/techno_magic/mods/netherlicious-3.0.2.jar>, request_timeout=15000 ms]
I may be wrong or miss understand something, but shouldn't sequential reading prevent the request timeout? For me it feels like this kind of timeout should come out when a server is not responding at all, but if you are actively receiving bytes, then why is it considered a timeout? In theory, I could slap a huge number and call it a day, but this feels wrong and unreliable. Is there a more appropriate way to solve that? p.s. I have tried "range" requests but with too big blocks they don't solve anything, as a user with 20kb/s speed would still fail to download 1MB block, and with smaller blocks(like 64kb) application is making a whole lot more requests(like +1500 requests for a 100MB file) and for those with good bandwidth will download files for 5 minutes instead of 20 seconds as before.
I'm having trouble accessing optional columns from multiple left joins. Calling `ResultRow.hasValue...
a

Andrew O'Hara

about 3 years ago
I'm having trouble accessing optional columns from multiple left joins. Calling
ResultRow.hasValue
for the joined columns always returns
true
, which makes it difficult for me to tell which optional columns are actually present. Below is a contrived example, followed by a passing and a failing test. It should illustrate my problem better than my explanation. Am I doing this completely wrong?
object BaseUser: IntIdTable("base_users")
object SuperUser: IntIdTable("super_users") {
    val baseId = reference("base_id", BaseUser)
    val name = varchar("name", 128)
}
object NormalUser: IntIdTable("normal_users") {
    val baseId = reference("base_id", BaseUser)
    val name = varchar("name", 128)
}

fun Transaction.getUserName(baseId: EntityID<Int>): String? {
    val row = BaseUser
        .leftJoin(SuperUser, {SuperUser.baseId}, {BaseUser.id})
        .leftJoin(NormalUser, {NormalUser.baseId}, {BaseUser.id})
        .slice(NormalUser.name, SuperUser.name)
        .select { BaseUser.id eq baseId }
        .firstOrNull()

    return when {
        row == null -> null
        row.hasValue(NormalUser.name) -> row[NormalUser.name]  // Still enters here even for SuperUser
        row.hasValue(SuperUser.name) -> row[SuperUser.name]
        else -> null
    }
}

@Test
fun `navigate subtype relationship`() {
    transaction {
        SchemaUtils.create(BaseUser, SuperUser, NormalUser)

        val baseUserId1 = BaseUser.insertAndGetId {}
        val baseUserId2 = BaseUser.insertAndGetId {}

        NormalUser.insert {
            it[baseId] = baseUserId1
            it[name] = "normaldude"
        }
        SuperUser.insert {
            it[baseId] = baseUserId2
            it[name] = "superdude"
        }

        getUserName(baseUserId1) shouldBe "normaldude" // PASS
        getUserName(baseUserId2) shouldBe "superdude" // FAIL; was actually null
    }
}