Daniel
04/14/2023, 10:38 AMselectBatched
will split the query across multiple SQL queries? If that's the case, is there a way to use a single SQL query, but retrieve its results in batches? Or is this achievable transparently using the Iterable.chunked
called on the returned Query
?Alexey Soshin
04/17/2023, 10:29 AMchunked
would work, but it will read the entire query result set into memoryDaniel
04/17/2023, 3:36 PMHASHBYTES
function and then in my kotlin code concatenate those hashes to compute the resulting final hash. I want to conservatively limit the memory usage in case of tables with lots of entries by computing intermediate hashes for each chunk of 10000 results and then compute the resulting hash from that. I found that Query
has fetchSize
parameter, but I don't know if that's what I want.
This is code I have for now
const val TABLE_HASH_BATCH_SIZE = 10_000
suspend inline fun <reified T> T.tableHash(): ByteArray where T : Table, T : Hashable {
val dbHashes = slice(hash).selectAll().orderBy(hash).fetchSize(TABLE_HASH_BATCH_SIZE)
val count = dbHashes.count()
val digester = Digest(hashBytesAlgo.algoName)
if (count == 0L) return ByteArray(digester.build().size) // bytearray of zeroes with correct size
return if (count <= TABLE_HASH_BATCH_SIZE) {
dbHashes.map { it[hash].bytes }.forEach(digester::plusAssign)
digester.build()
} else {
val digesterOuter = Digest(hashBytesAlgo.algoName)
for (chunk in dbHashes.chunked(TABLE_HASH_BATCH_SIZE)) {
chunk.map { it[hash].bytes }.forEach(digester::plusAssign)
digesterOuter.plusAssign(digester.build())
digester.reset()
}
digesterOuter.build()
}
}