Hello! This is a bit of a long shot, but trying my...

# apollo-kotlin

[Eduard Boloș]

Hello! This is a bit of a long shot, but trying my luck. If you had a long list that is paginated, let's say of tens of thousands of items, that is cached locally in a normalized SQLite db, how would you go about searching for specific items in that list (e.g. where a field matches a partial query)? Would you have to load the whole result set and do the filtering in memory, or does Apollo have some facilities to do such filtering at a lower level? (I am guessing that the answer is no.)
Loading the whole list in memory for filtering raises some concerns from a performance point of view, and I am afraid that this would mean saving duplicate data in a separate SQLite DB where we can filter directly from the SQL query. But maybe someone has a better pointer 🙏

[bod]

Hi! 👋

does Apollo have some facilities to do such filtering at a lower level

I'm afraid not, the cache APIs (ApolloStore and NormalizedCache ) are GraphQL oriented.

The sqlite db itself does use text for the data so in theory you could do some text queries in it... however, this is an implementation detail that shouldn't be relied on, and also I'm not sure how you would reconstruct the data easily - maybe that depends on the kind of searches you were thinking of (also I'm not sure several connections to the same db are OK with SQLite)

[bod]

if the list is paginated you could do a page by page search maybe, to avoid loading everything into memory (unless you are using the incubating paginated APIs?)

[Eduard Boloș]

We are not using the incubating paginated APIs. I was playing with a prototype doing exactly that, but I noticed a couple of things:
• The search is slow, if what you are looking for is something that is at the end of the list (page size right now is 10, maybe making it larger would make things somewhat faster 🤷 )
• It still creates a lot of memory pressure, there are GC runs very frequently (multiple per second), and eventually the VM dies because it fails to allocate memory for some reason.
But yeah, the data in GQL normalized form is not easy to filter, as the object type is an union with multiple possible types, and each type has different fields to filter by 🙈 If I were to save it in a separate DB, probably I would have in one table only the object IDs, and in another one a row for each searchable field of each object, or something like that.

[bod]

yeah maybe there are good ideas I'm not thinking of to use the normalized cache directly, but for now my initial thought is that saving the data in a separate db to index/search is probably your best bet.

[Eduard Boloș]

thanks a lot for your input!