I'm using okhttp to download a big text file from the web (think a list of curse words that are sepa...

# squarelibraries

[Colton Idle]

I'm using okhttp to download a big text file from the web (think a list of curse words that are separated by a new line) and my goal is for my app to prevent a user from setting their name to a curse word.

Currently (just as a POC) I'm doing

state.curseWords = result.body()?.string()?.lines()

I'm predicting that this is bad for 2 reasons and wanted to sanity check if my reasoning is sound

1. string() could crash if the file is large enough? Best to use a "stream" (still learning about bytestreams, etc) but it makes sense that I'd want to "stream" the info into a list before checking if the list
2. If the list is extremely large, it might not even fit into a List<String> and so it might be best to stream the file and somehow on each line (while streaming) check if the username == currentWord in list.
Are my two assumptions above correct?

[Tolriq]

Yes but this sounds like a server side job no? Downloading large file to device for that is not really eco friendly.

[Colton Idle]

Agree! This is mainly a proof of concept because I'm trying to get myself more familiar with files/IO/streams etc.

Glad to hear my two assumptions are correct... 😄

[Tolriq]

If it can be big and you don't control the size then it will always explode at some point 🙂

[yschimke]

Also relevant, how often it changes.

[yschimke]

But if stable, and small like these, I don't think it's an issue.

[yschimke]

https://github.com/coffee-and-fun/google-profanity-words/blob/main/data/list.txt

[yschimke]

https://github.com/snguyenthanh/better_profanity/blob/master/better_profanity/profanity_wordlist.txt

[jessewilson]

If I wanted to super over-engineer this (of course I do), I’d get the server to format the data as a SQLite DB file with an index, and download that whole DB to the device, and query it.

[jessewilson]

Avoid putting all those strings into memory when you only really need to test presence

[jessewilson]

The drawback of this is dramatically more complexity, and it’s annoying operationally

[Colton Idle]

Oooh. I like the idea that if you needed a local approach, you could go with a db with an index. Thanks @jessewilson

Thank you everyone! You all make me feel like I'm not an imposter!

[Tolriq]

You can also expose as json then steam parse with moshi and store in a local database with SQLDelight, even better to learn things 😛

[Colton Idle]

Just riffing off of that. There's no reason why I couldn't stream this file currently, and store into a local database, right? Or does json+moshi give me some kind of additional opportunity here?

[jessewilson]

Nope, that’d work

[yschimke]

Or formats like flatbuffers https://google.github.io/flatbuffers/

[yschimke]

An interesting paper if this was for a massive dataset https://aclanthology.org/W09-1505.pdf, overkill or suboptimal for a 10kb file.
Memory mapping in an encoded trie. But I guess a database probably does similarly optimised implementations depending on the table structure.