https://kotlinlang.org logo
#ktor
Title
# ktor
m

Maurice Wingbermuhle

02/07/2020, 12:33 PM
When I'm trying to download a "larger" file (355k) with HttpClient, it seems that it does not let me read more than 4096 bytes. What I could I be doing wrong? I am also not seeing the response logging in Logcat, just the request
e

e5l

02/07/2020, 12:46 PM
Hi @Maurice Wingbermuhle, could you tell me what engine you’re using and how you download the file? We test in on ~900Mb in integration tests.
m

Maurice Wingbermuhle

02/07/2020, 12:51 PM
My environment is Kotlin MPP, and I'm never really setting the engine explicitly, so my guess is that on Android OKHttp is being used.
Copy code
suspend fun downloadFile(url: String): ByteArray {
    val httpClient = HttpClient{
        install(Logging) {
            this.logger = Logger.SIMPLE
            this.level = LogLevel.ALL
        }
    }
    val statement = httpClient.request<HttpStatement>(url) {
        range?.let {
            header(HttpHeaders.Range, range)
        }
    }
    return statement.execute {
        val contentLength = it.contentLength()?.lowInt ?: 0
        val byteArray = ByteArray(contentLength)

        var offset = 0
        do {
            val currentRead = it.content.readAvailable(byteArray, offset, byteArray.size)
            offset += currentRead
            logger.logDebug("Download in progress, offset: ${offset}, current read ${currentRead} / ${contentLength}")
        } while (offset < contentLength)
        logger.logDebug("Download done")

        return@execute byteArray
    }
}
e

e5l

02/07/2020, 12:58 PM
Could you try to lookup engine in the gradle dependencies block? What version of ktor are you using?
m

Maurice Wingbermuhle

02/07/2020, 1:02 PM
Ktor version 1.3.0 Main dependencies
Copy code
implementation "io.ktor:ktor-client-core:$ktor_version"
                implementation "io.ktor:ktor-client-json:$ktor_version"
                implementation "io.ktor:ktor-client-logging:$ktor_version"
                implementation "io.ktor:ktor-client-serialization:$ktor_version"
JVM common depdendencies:
Copy code
implementation "io.ktor:ktor-client-json-jvm:$ktor_version"
                implementation "io.ktor:ktor-client-logging-jvm:$ktor_version"
                implementation "io.ktor:ktor-client-serialization-jvm:$ktor_version"
Android dependencies:
Copy code
implementation "io.ktor:ktor-client-android:$ktor_version"
Android depends on jvm common, and jvm common depends on main
e

e5l

02/07/2020, 1:03 PM
Could you try using
1.3.1
there was an issue fixed there?
m

Maurice Wingbermuhle

02/07/2020, 1:07 PM
Nope, same result
Copy code
Download in progress, offset: 4088, current read 4088 / 355835
Download in progress, offset: 4096, current read 8 / 355835
👀 1
Since we're doing multiplatform, I ran the same code on iOS: Also same result, except that it immediatly jumps to 4096 bytes.
Copy code
Download in progress, offset: 4096, current read 4096 / 355835
@e5l No clue?
e

e5l

02/07/2020, 4:42 PM
I’m trying to figure out
m

Maurice Wingbermuhle

02/07/2020, 4:45 PM
Ok. I'm signing out for today, work continues on monday. In the mean time I managed to download the entire file by keeping under the 4096 bytes limit, and using a loop and the Range header to fetch chunks... Dirty but it works
@e5l Still looking into the issue? Should I file a bug for this?
e

e5l

02/11/2020, 10:30 AM
Yep, it’ll be great
m

Maurice Wingbermuhle

02/11/2020, 10:31 AM
Meaning this might really be an issue?
e

e5l

02/11/2020, 10:32 AM
We don’t know, we should investigate it at least
m

Maurice Wingbermuhle

02/11/2020, 12:34 PM
Hi @e5l I saw you closed this issue a while ago, with the message that it had been fixed on master. Does that mean that it should be fixed in the latest ktor 1.3.2 release?
e

e5l

03/18/2020, 11:13 AM
Yep
The iOS engine added support for the response streaming and it should work in 1.3.2
m

Maurice Wingbermuhle

03/18/2020, 11:14 AM
Do you have a code sample that illustrates correct usage in that case? Because my (restored) code still has the same issue
both on Android and iOS by the way
Basically I'm trying to download a file from Amazon AWS file storage to a kotlin bytearray in a kotlin MPP environment, so I cannot use File, or any Java-specific stuff here :S
e

e5l

03/18/2020, 11:26 AM
Could you try changing
Copy code
httpClient.request<HttpStatement> { ... } -> httpClient.request<String> { ... }
and drop the execute part
I’m talking about the reproducer here: https://github.com/ktorio/ktor/issues/1636
m

Maurice Wingbermuhle

03/18/2020, 11:28 AM
Will try it.
Sadly, no improvement
Well, there is a minor improvement, because I can now see the logging of the response (headers) in Logcat and Xcode, which previously was only the request
@e5l If I use a requestTimeout, it will expire after 30 seconds on a 2MB file. I do have a faster connection than that...
e

e5l

03/18/2020, 12:06 PM
@Elena Lepilkina could you take a look?
m

Maurice Wingbermuhle

03/18/2020, 1:24 PM
@Elena Lepilkina @e5l Can you update / reopen issue 1636? I'm still subscribed there and can provide more info if required.
e

e5l

03/18/2020, 1:26 PM
Sure
Could you share the project with the configuration? we need to know targets and compilation configuration
e

Elena Lepilkina

03/18/2020, 1:28 PM
I have a couple of questions, what target exactly do you use, where you have time problems. Is it ios_x64 or ioa_arm64? DO you run on simulator? And what build do you use to produce framework? Debug or release?
e

e5l

03/18/2020, 1:28 PM
Minified snipped would be enough
e

Elena Lepilkina

03/18/2020, 1:30 PM
And also is it problem reproduced if you are trying to download from any server? Because the easiest way for me to reproduce is to use bintray/artifactory.
m

Maurice Wingbermuhle

03/18/2020, 1:31 PM
I'm currently running on both Android and iOS simulators, using these targets:
Copy code
targets {
        jvm()
        android()
        iosX64()
        iosArm64()

        configure([iosX64, iosArm64]) {
            binaries.framework {
                baseName = frameworkName
            }
        }
    }
I will try to make a isolated reproduction project which can zip entirely
e

Elena Lepilkina

03/18/2020, 1:34 PM
So you face problems on iosSimulator? What framework version do you use? Debug or release? If you send us a small project to reproduce this problem it would be great
m

Maurice Wingbermuhle

03/18/2020, 1:36 PM
I face the same problem on all platforms, not specifically for iOS only
e

Elena Lepilkina

03/18/2020, 1:38 PM
All platforms from your script? Same behaviour with JVM target?
m

Maurice Wingbermuhle

03/18/2020, 1:39 PM
yes
e

Elena Lepilkina

03/18/2020, 1:43 PM
@e5l so then it can be eighter ktor problem or some AWS specific (I still think it's useful to download from some other server to check). Then it isn't K/N problem. At least for now. Leonid, could you start researching and if you face K/N performance problem ping me.
e

e5l

03/18/2020, 1:46 PM
Sure. Thanks!
m

Maurice Wingbermuhle

03/18/2020, 3:02 PM
@e5l Been trying to reproduce the issue in a small project - the good news is that I can reproduce, the bad news is that I can only reproduce with URLs that come from a secured backend. They are signed with tokens and last for only 5 minutes
I even have a small kotlin test that produces the issue in jvmTest context
e

e5l

03/18/2020, 3:04 PM
Could you try
time curl
this url in terminal, just to make sure that url is fine
m

Maurice Wingbermuhle

03/18/2020, 3:04 PM
let me check
yes, curl output:
Copy code
curl   0.03s user 0.06s system 1% cpu 4.800 total
with a lot of other noise on the screen
e

e5l

03/18/2020, 3:06 PM
thanks
m

Maurice Wingbermuhle

03/18/2020, 3:07 PM
I can give you a url in DM for you to test with?
e

e5l

03/18/2020, 3:07 PM
Sure