A modern programming language that makes developers happier.

kotlinlang

What's best practise for processing a lot of SQL rows without loading them all into memory at once? spring-data will return a (java8) Stream but I'm not sure if anything (mysql driver, hibernate, one of kotlin's composable operators) might pull on the Stream and effectively turn it into a List?
Alternatively spring-data's pageable can return the data in chunks but I can't see a nice way of iterating around each chunk in a composable manner?

<@U7GRDKKFY> I usually use `JdbcTemplate` and write prepared-statement SQL directly or create a StoredProcedure and call it from my app.  That way the rows stay on the SQL instance and do not transit the network

You could take a look at `fetch-size` property - <https://stackoverflow.com/questions/60372230/setting-hibernate-fetch-size-in-spring-boot#comment106796746_60372230|stackoverflow thread>.
I am not sure how it works for hibernate, but for jdbc it should load data in chunks :slightly_smiling_face:
Don’t turn it into List because Stream is lazy and List isn’t. If you want to perform some logic on this stream, for example smth like this:
```val users: Stream&lt;User&gt; = repository.users() // or even kotlin Sequence would be better
users.forEach { 
  // do something with user
}```
the data would be loaded in chunks (as you would specify it in fetch size).
So for example, if you set fetch size to 100, you would take 100 users, perform logic in `forEach` block, and then fetch another chunk of data (100users) till the end of stream (or sequence which is kotlin equivalent of lazy collection)

Thanks!
I wasn't sure if Sequence is supported in spring-data, should be an easy change.

I vaguely remember that something in Hibernate might convert from List and so making it eager, but I can't find mention of it so hopefully I remember incorrectly.

Last time I checked in data-mongo, repository methods that returned a stream were backed by a list. I suspect data-jpa behaves the same. I  would use jdbctemplate directly for this sort of thing.

I might be wrong. <https://docs.spring.io/spring-data/jpa/docs/2.5.1/reference/html/#repositories.query-streaming|https://docs.spring.io/spring-data/jpa/docs/2.5.1/reference/html/#repositories.query-streaming>

<@UG7GAS1QV> That doc does have this disclaimer:
```Not all Spring Data modules currently support Stream&lt;T&gt; as a return type```
In the end, <@U7GRDKKFY>, if you do not _need_ the rows in your app, then JdbcTemplate is a better solution.  If you require them as a Stream (rather than List), it appears that is also possible (under certain circumstances)

Thanks.

Stream would be much preferred as the conversion of column to typesafe property is handled automatically.

Yes, I think I read the below post about Hibernate not implementing Stream fully and defaulting to go via List, but there is discussion in the comments that says it is actually ok:

<https://stackoverflow.com/questions/60166850/hibernate-getresultstream-not-streaming|https://stackoverflow.com/questions/60166850/hibernate-getresultstream-not-streaming>