Trying to pinpoint an issue where I'm searching fo...

# squarelibraries

[Colton Idle]

Trying to pinpoint an issue where I'm searching for an id that I know exists in my database, but sometimes I see our internal logs saying that the id was not found.

I think I mostly chalked it up to the fact that the database is updated once a day (from a network call), and maybe during that daily update it fails/is malformed/truncated for some reason, and that causes me not being able to find it.

This is me doing the network request and putting it into the db

try {
  client.newCall(request).execute().use { response ->
    if (!response.isSuccessful) return@withContext

    val bufferedReader = BufferedReader(InputStreamReader(response.body!!.byteStream()))
    persistence.deleteAllThenInsertAll(bufferedReader)
    appStateHolder.updateLastDbSync(System.currentTimeMillis())
  }
} catch (e: Exception) {
  // Network issue
}

Does that look sus to anyone?

[Colton Idle]

suspend fun deleteAllThenInsertAll(bufferedReader: BufferedReader) =
      withContext(<http://Dispatchers.IO|Dispatchers.IO>) {
        idQueries.transaction {
          idQueries.deleteAll()
          var currentLine: String? = null
          try {
            while (bufferedReader.readLine().also { currentLine = it } != null) {
              val (cardId, date) = currentLine!!.split(",")
              idQueries.insert(cardId, date)
            }
          } catch (e: Exception) {
            //log to crashlytics
          }
        }
      }

[Colton Idle]

Maybe I'm completely misunderstanding the concept of a try/catch in a transaction?

[Colton Idle]

Alright I think I was able to finally write a test that proces that my try catch is actually in the wrong spot and should actually be outside of the transaction. I think that makes sense. not sure what i was initially thinking.

[Tolriq]

That's why you should always be explicit about marking a transaction successful or failed.

[Colton Idle]

Hm. How do I do that? let me look up that docs in a bit.

the way you're saying it makes me think that I can return true or false in a transaction? if so, TIL!

[Colton Idle]

transaction docs are good. should have read them before. feel like an idiot putting the try catch in the transaction lol.

tolriq, if you could though, could you clarify what you mean about explicitly marking it success or failed?

[Tolriq]

I have not yet migrated to sqldelight but the concept in SQL are the same everywhere, you start a transaction do a bunch of stuff then before ending the transaction you tell the engine if the transaction is successful or not so that it apply or not everything you did during the transaction.

[Tolriq]

From your extract I guess sqldelight mark the transaction successful by default if there's no throw hence your problem. But it's something to remember when dealing with transactions.

[Colton Idle]

got it. yeah this is also a perfect spot for a unit test, but i didn't know how to best create a bufferedReader that would blow up in the middle. lol. which seems to be happening from this very large file that i'm download -> streaming into my table.

[Tolriq]

You should probably mock the response not the bufferedreader.

[Colton Idle]

Yeah, but I'm not sure (in this case) whats up with the response that just causes it to blow up.

[Colton Idle]

it's supposed to be a static file with just 100k lines of data

[Tolriq]

That's a lot of data, you probably want to do streamparsing.

[Colton Idle]

So I do have a network try catch surrounding this transaction and that does seem to work fine. but anyway. im going to add logs. fix this bug (because the rollback doesn't actually happen) and then see if the issue happens again in the wild.

[Colton Idle]

streamparsing? hm. will look that up. thanks

[Tolriq]

forget that I did not read again the second snippet that's already what you do.

[Colton Idle]

oh cool. well thanks for the rubberducking. plenty of things i learned with this feature/bug.