hi, I have to paginate some web api and stop after a request returns “more”: false. I’m currently lo...

# announcements

[Bernhard]

hi, I have to paginate some web api and stop after a request returns “more”: false. I’m currently looking at sequences and takeWhile stops immediately after the predicate returns false; is there a way to say: take from this sequence until you encounter something including that thing?

Only solution I can think of is using a mutable local variable and set that to false after the takeWhile

[Ryan]

is the client requesting a list of elements?

[Ryan]

results.takeWhile { it.more }

[Bernhard]

right, but the last result has it.more as false and includes elements as well

[Ryan]

not sure what your data looks like but that code i posted will break once condition is not met

[Bernhard]

so if I paginated by 10 elements and had 18 elements in total, the last request would return more: false and would include 8 elements

[Bernhard]

results.takeWhile { it.more } would skip the remaining 8 elements right?

[kqr]

how are you getting more items from api?

[Bernhard]

by making another request with an offset

[kqr]

this takeWhile you want to achieve "should" be bound to take/do more requests

[kqr]

yes but how does the pipeline look like?

[Bernhard]

you start at offset 0 and send a size as well, e.g. 10

[Bernhard]

you get the 10 first items with more: true

[Bernhard]

then you make another request with offset 10 and size 10

[Bernhard]

then you get the remaining 8 items and more: false

[kqr]

yes I understand that

[kqr]

but do you have some concrete code?

[Bernhard]

yep, but it’s a bit abstracted, sec

[Bernhard]

val hasMore = true
generateSequence(0, { it + maximumPageSize })
        .takeWhile { hasMore }
        .map { offset ->
            val end = offset + maximumPageSize
            val targetFileName = "${offset}-${end}__${download.targetFileName}"
            val paginatedDownload = download.copy(
                    targetFileName = targetFileName,
                    urlParameters = paginateUrlParameters(offset)
            )
            val importer = HttpImporter(paginatedDownload)
            importer.run(targetDirectory)
        }

[Bernhard]

the next map will parse the json and set the hasMore variable

[Bernhard]

have to implement that

[Bernhard]

and maybe there’s an issue with the generateSequence, have to check if the seed is sent as the initial element or if the lambda is executed first

[Bernhard]

ideally I could just use something like: .endSequenceAfter { it.hasMore }

[Bernhard]

and that would keep the first element where the predicate is false

[kqr]

I can't come up with other solution than having mutable variable as you have

[Rob Elliot]

Would this be useful?

data class Page<T>(
  val elements: List<T>,
  val offset: Int,
  val hasMore: Boolean
)

val allThings: Sequence<Thing> = 
  generateSequence(getPage(0)) { page ->
    if (page.hasMore)
      getPage(page.offset + page.elements.size)
    else null
  }
  .flatMap { it.elements.asSequence() }