Hi guys, I am in a situation that our backend team is unable to provide us with a paginated endpoint...

# android

[Ernestas]

Hi guys, I am in a situation that our backend team is unable to provide us with a paginated endpoint for a large dataset. Any tips/articles on how to handle huge json responses with coroutines/flow? (I’m using retrofit for networking)

[Alexander af Trolle]

How big responses are we talking?

[Ernestas]

Up to ~200 mb, enough to get out of memory exceptions

[Alexander af Trolle]

My I/O knowledge isn't the best, but I think you could get the response as a stream Response<ResponseBody>
https://stackoverflow.com/questions/33030137/retrofit-2-0b2-how-to-get-inputstream-from-the-response

Then you'll pipe the stream into a serialiser like Gson or Moshi to parse out objects. Then I would store each object into Room or SQLDelight and do pagination from that.

This article looks promising for the serialization.
https://www.amitph.com/java-parse-large-json-files/

[Ernestas]

Thank you! definitely looks promising, I’ll try it out

[Vitaliy Zarubin]

https://proandroiddev.com/retrofitting-and-rxjaving-heavy-jsons-2c1fcfa6383c

[Alexander af Trolle]

One day though we might be able to use- kotlinx.serialization, instead of Gson or Moshi https://github.com/Kotlin/kotlinx.serialization/issues/204
https://github.com/Kotlin/kotlinx.serialization/issues/1073