https://kotlinlang.org logo
Join Slack
Powered by
When I'm trying to download a "larger" file (355k)...
# ktor
m
When I'm trying to download a "larger" file (355k) with HttpClient, it seems that it does not let me read more than 4096 bytes. What I could I be doing wrong? I am also not seeing the response logging in Logcat, just the request
e
Hi @Maurice Wingbermuhle, could you tell me what engine you’re using and how you download the file? We test in on ~900Mb in integration tests.
m
My environment is Kotlin MPP, and I'm never really setting the engine explicitly, so my guess is that on Android OKHttp is being used.
Copy code
suspend fun downloadFile(url: String): ByteArray {
    val httpClient = HttpClient{
        install(Logging) {
            this.logger = Logger.SIMPLE
            this.level = LogLevel.ALL
        }
    }
    val statement = httpClient.request<HttpStatement>(url) {
        range?.let {
            header(HttpHeaders.Range, range)
        }
    }
    return statement.execute {
        val contentLength = it.contentLength()?.lowInt ?: 0
        val byteArray = ByteArray(contentLength)

        var offset = 0
        do {
            val currentRead = it.content.readAvailable(byteArray, offset, byteArray.size)
            offset += currentRead
            logger.logDebug("Download in progress, offset: ${offset}, current read ${currentRead} / ${contentLength}")
        } while (offset < contentLength)
        logger.logDebug("Download done")

        return@execute byteArray
    }
}
e
Could you try to lookup engine in the gradle dependencies block? What version of ktor are you using?
m
Ktor version 1.3.0 Main dependencies
Copy code
implementation "io.ktor:ktor-client-core:$ktor_version"
                implementation "io.ktor:ktor-client-json:$ktor_version"
                implementation "io.ktor:ktor-client-logging:$ktor_version"
                implementation "io.ktor:ktor-client-serialization:$ktor_version"
JVM common depdendencies:
Copy code
implementation "io.ktor:ktor-client-json-jvm:$ktor_version"
                implementation "io.ktor:ktor-client-logging-jvm:$ktor_version"
                implementation "io.ktor:ktor-client-serialization-jvm:$ktor_version"
Android dependencies:
Copy code
implementation "io.ktor:ktor-client-android:$ktor_version"
Android depends on jvm common, and jvm common depends on main
e
Could you try using 
1.3.1
there was an issue fixed there?
m
Nope, same result
Copy code
Download in progress, offset: 4088, current read 4088 / 355835
Download in progress, offset: 4096, current read 8 / 355835
👀 1
Since we're doing multiplatform, I ran the same code on iOS: Also same result, except that it immediatly jumps to 4096 bytes.
Copy code
Download in progress, offset: 4096, current read 4096 / 355835
@e5l No clue?
e
I’m trying to figure out
m
Ok. I'm signing out for today, work continues on monday. In the mean time I managed to download the entire file by keeping under the 4096 bytes limit, and using a loop and the Range header to fetch chunks... Dirty but it works
@e5l Still looking into the issue? Should I file a bug for this?
e
Yep, it’ll be great
m
Meaning this might really be an issue?
e
We don’t know, we should investigate it at least
m
Hi @e5l I saw you closed this issue a while ago, with the message that it had been fixed on master. Does that mean that it should be fixed in the latest ktor 1.3.2 release?
e
Yep
The iOS engine added support for the response streaming and it should work in 1.3.2
m
Do you have a code sample that illustrates correct usage in that case? Because my (restored) code still has the same issue
both on Android and iOS by the way
Basically I'm trying to download a file from Amazon AWS file storage to a kotlin bytearray in a kotlin MPP environment, so I cannot use File, or any Java-specific stuff here :S
e
Could you try changing
Copy code
httpClient.request<HttpStatement> { ... } -> httpClient.request<String> { ... }
and drop the execute part
I’m talking about the reproducer here: https://github.com/ktorio/ktor/issues/1636
m
Will try it.
Sadly, no improvement
Well, there is a minor improvement, because I can now see the logging of the response (headers) in Logcat and Xcode, which previously was only the request
@e5l If I use a requestTimeout, it will expire after 30 seconds on a 2MB file. I do have a faster connection than that...
e
@Elena Lepilkina could you take a look?
m
@Elena Lepilkina @e5l Can you update / reopen issue 1636? I'm still subscribed there and can provide more info if required.
e
Sure
Could you share the project with the configuration? we need to know targets and compilation configuration
e
I have a couple of questions, what target exactly do you use, where you have time problems. Is it ios_x64 or ioa_arm64? DO you run on simulator? And what build do you use to produce framework? Debug or release?
e
Minified snipped would be enough
e
And also is it problem reproduced if you are trying to download from any server? Because the easiest way for me to reproduce is to use bintray/artifactory.
m
I'm currently running on both Android and iOS simulators, using these targets:
Copy code
targets {
        jvm()
        android()
        iosX64()
        iosArm64()

        configure([iosX64, iosArm64]) {
            binaries.framework {
                baseName = frameworkName
            }
        }
    }
I will try to make a isolated reproduction project which can zip entirely
e
So you face problems on iosSimulator? What framework version do you use? Debug or release? If you send us a small project to reproduce this problem it would be great
m
I face the same problem on all platforms, not specifically for iOS only
e
All platforms from your script? Same behaviour with JVM target?
m
yes
e
@e5l so then it can be eighter ktor problem or some AWS specific (I still think it's useful to download from some other server to check). Then it isn't K/N problem. At least for now. Leonid, could you start researching and if you face K/N performance problem ping me.
e
Sure. Thanks!
m
@e5l Been trying to reproduce the issue in a small project - the good news is that I can reproduce, the bad news is that I can only reproduce with URLs that come from a secured backend. They are signed with tokens and last for only 5 minutes
I even have a small kotlin test that produces the issue in jvmTest context
e
Could you try 
time curl
this url in terminal, just to make sure that url is fine
m
let me check
yes, curl output:
Copy code
curl   0.03s user 0.06s system 1% cpu 4.800 total
with a lot of other noise on the screen
e
thanks
m
I can give you a url in DM for you to test with?
e
Sure
Open in Slack
PreviousNext
Powered by