https://kotlinlang.org logo
#android-architecture
Title
# android-architecture
u

ursus

03/12/2019, 9:17 PM
tldr; how to figure out what to delete from locals, if remotes are paginated
g

ghedeon

03/12/2019, 9:19 PM
can't you fetch like a range of messages or a specific one? You've got a push that a specific range of messages has changed, you check if it's within your local messages and if true, than fetch this chunk.
u

ursus

03/12/2019, 9:22 PM
if im foreground = websocket is active, then its all good,
but after I start the app / come back from foreground , I wont receive pushes "of what I have missed"
or should I ?
g

ghedeon

03/12/2019, 9:31 PM
aren't things like firebase are designed for this kind of case?
do you have a particular issue with reliability of your push service?
it's a bit superficial, but "Grokking the System Design Interview" course contains some examples of Messenger services and explains how WhatsApp or Twitter is made. Proper synchronization is a quite challenging task 🙂.
u

ursus

03/12/2019, 9:44 PM
I dont think slack has that automatic push all missed events
I think its regular https call from client
i.e. pull
yea I know its challenging, just wondering what would be the mechanism to update the off-pages
unless only keeping 1 page of data locally, but that is lame
honestly this is a client thing I think, I have push of changes, and pull to get all history, but its paginated, so im wondering how or if I should update the off pages, hopefully you know what i mean
I mean If I were to sync all local data, then its no problem, but this wont scale..
g

ghedeon

03/12/2019, 10:21 PM
Not sure about slack, but Facebook Messanger and WhatsApp are using push model (long polling/websockets). It makes even more sense once you want to display the delivery status of the message. Anyhow, with pull model I guess you'd probably need to have two sections in your server response:
edited
and
latest
. So you can patch your history.
u

ursus

03/12/2019, 10:29 PM
but neither of then for sure has websocket on background -- so you need to "pull" to synchronize, when coming back from background
g

gildor

03/12/2019, 11:08 PM
Yes, you need to pull and to do that properly you need some API that returns diff of data from your last connection
u

ursus

03/13/2019, 4:59 AM
yes, but, how to do that? -- meaning, sending every message id I have to server.. that cant scale
g

gildor

03/13/2019, 5:12 AM
No, you can have special token/cursor (for simplest, but not very robust solution, it may be just timestamp of last message) on each diff request, so server know snapshot version of messages on your device
u

ursus

03/13/2019, 5:32 AM
hm not sure if I udnerstand, if I tell server what my last message is, how can it tell that I for example need to update some n-th message I have locally because it was edited? (meaning that message is from before my last message)
g

gildor

03/13/2019, 5:36 AM
I used this example just to illustrate how this cursor for diff request may work
of course real syncronization implementation may require much more sophisticated implementation, depending on your case and required features
This is why I mentioned token/cursor, something like you do initial syncronization by requesting all the data (which is additional challenge) together with last chunk of data you getting token/cursor that may be used to request changes happened after receiving this data
u

ursus

03/13/2019, 5:39 AM
okay so the calculation is a backend responsibility, based on my token / cursor?
g

gildor

03/13/2019, 5:40 AM
there is no easy solution and trade-offs between simple/featureful implementation are everywhere, for example you may concider sync each conversation separately
u

ursus

03/13/2019, 5:40 AM
yea Im doing that, but still that can not scale, if I query all of history in that conversation ever
g

gildor

03/13/2019, 5:41 AM
what do you mean?
u

ursus

03/13/2019, 5:41 AM
Im thinking only about this one use: I come back from offline, and I need to know what changed -- somehow -- other than fetching all of convo history to figure it out
basically the trouble is that message can be edited and deleted, so I need to look backwards also on all messages
g

gildor

03/13/2019, 5:44 AM
I come back from offline, and I need to know what changed -- somehow -- other than fetching all of convo history to figure it out
Globally you have only 2 choices, download everything (or just first page) or use some diff/sync API, but how those diffs will be implemented highly depends on your use case, complexity of your server etc
basically the trouble is that message can be edited and deleted, so I need to look backwards also on all messages
No, if you have diffs
u

ursus

03/13/2019, 5:45 AM
okay so, lets say I want the diffs, since I dont want to download everything
g

gildor

03/13/2019, 5:45 AM
okay
u

ursus

03/13/2019, 5:46 AM
that means I need to tell the server about every message id I have locally for that conversation?
or is there some othe way
g

gildor

03/13/2019, 5:46 AM
it one of possible options and your server will create diff info (added, deleted, updated messages) based on this, but it’s not very scalable, you right
u

ursus

03/13/2019, 5:47 AM
yea, given huge conversation that will suck
g

gildor

03/13/2019, 5:47 AM
there are other ways, when server remembers your initial sync session and knows which messages added/updated/deleted from your last sync
u

ursus

03/13/2019, 5:48 AM
im thinking those etags you mentioned, cant that be used?
i.e. every conversation.history pull id get some fingerprint, .. but not sure how would that work
g

gildor

03/13/2019, 5:49 AM
server just should keep snapshots for every client, and to identify this snapshot you need some sort id/tag/cursor/etc, it also just may be auth token
u

ursus

03/13/2019, 5:50 AM
hm, what snapshot do you mean -- snapshot = what server thinks my local data is?
btw Im thiking im missign some kind of thery, do you have something to recommend for me to read maybe?
im feeling like im reinventing the wheel
g

gildor

03/13/2019, 5:53 AM
snapshot = what server thinks my local data is
yes
u

ursus

03/13/2019, 5:54 AM
okay -- maybe I'll ask different, If I can only calculate the diffs on client, my only options is to fetch all history, right?
g

gildor

03/13/2019, 5:55 AM
there are probably some materials about it, but I don’t know any actual “theory” about this, just engineering solution, if you know date of modification of every message on client’s device you can just request all the messages changed after this date, actual implementation may be different of course
calculate the diffs on client
Yes, request the data from server about all changes 😄
to calculate diffs you need both versions of data, old and new, I don’t think that there is any solution that may solve it
u

ursus

03/13/2019, 5:56 AM
yea, thats super stupid, and all data from first message I have locally,. thats only stupid
g

gildor

03/13/2019, 5:57 AM
I fixed my message, it may be not “All” data, but only data that required to create diff, like list of all message ids with last update date (or something like version id)
u

ursus

03/13/2019, 5:58 AM
yea ..would that me durable against breaking my database locally? meaning someone with root -- or more likely developer via bug, causing a message to be deleted ... or is that overkill?
g

gildor

03/13/2019, 5:59 AM
but because by definition server has much more knowledge than client, because has much more resources, all the data is persistent, it much easier and more efficient (in terms of client) to do on server side
not sure what you mean
u

ursus

03/13/2019, 6:00 AM
yes, im now looking for ideal solution, only after that ill try to bring it to reality and how flexible my api team is 😄
what I mean is that if I send the id - timestamp map to server, id only get .. okay nevermind, server would tell that it is missing the one message
im still wondering how would the server keep track of my data -- what if I delete something locally in a bug, and therefore server would still keep it as I have it ..idk, I feel like that will be very britle
g

gildor

03/13/2019, 6:02 AM
I think it’s overkill, if you cannot trust your local data it’s really strange and it looks like overkill to “validate” your local DB on each sync SQL is very robust solution, supports transactions, so not sure what is “developer via bug, causing a message to be deleted”
u

ursus

03/13/2019, 6:03 AM
ofc I trust sql, I dont trust developers to not cause a bug and deleting some unrelated id or something
- this would be handlable given a full sync, as youd see something in remote data not presend in local = insert it
g

gildor

03/13/2019, 6:04 AM
im still wondering how would the server keep track of my data
depens on strategy, but for conversation it may have some database of operations on particular conversation and then when you request it again just return all new operations add/delete/update
If you know about some existing problem you may just trigger full sync
u

ursus

03/13/2019, 6:04 AM
ok I can see that, but thats still not resilient agains the bug
hm, how would I tell there is a problem?
g

gildor

03/13/2019, 6:05 AM
Your user complains, you QA found it, your developer discovered it in code
u

ursus

03/13/2019, 6:06 AM
I mean this will manifest as runtime thing -- how will user tell he is missing a message, when he doesnt know it should be there
g

gildor

03/13/2019, 6:06 AM
you may do periodical validateion with server using special api, like check amount of conersations and amount of messages
u

ursus

03/13/2019, 6:07 AM
😄 what if he not deletes, but updates it with foobar data, and wont update the lastUpdated timestamp hh
g

gildor

03/13/2019, 6:07 AM
when he doesnt know it should be there
i don’t think this is somehow related to you problem, how to write program without bugs, how to write system that detects bugs, I don’t know, it’s even more broad question than “how to write real time chat messenger”
u

ursus

03/13/2019, 6:08 AM
hm not really because full diff would fix it, the problem is pagination
g

gildor

03/13/2019, 6:08 AM
no, it’s not related to pagination too
there are bugs in program and any kind API cannot solve all of them
u

ursus

03/13/2019, 6:09 AM
I think so, because if you diff against a page, you dont know about the rest of the data in next pages
g

gildor

03/13/2019, 6:09 AM
You asking questions that don’t have any good answer, each solution is trade-off between of big amout of factors, choose which are more important for you, perfection solution doesn’t exist
u

ursus

03/13/2019, 6:09 AM
basically yea, I was jsut curious about state of art
g

gildor

03/13/2019, 6:09 AM
I think so, because if you diff against a page, you dont know about the rest of the data in next pages
I never suggest to use this approach, you right, it’s imo bad solution
u

ursus

03/13/2019, 6:10 AM
whatsapp doesnt give a damn, they dont even have a remote database right? they only relay messages
g

gildor

03/13/2019, 6:10 AM
Yes
each client has own set of messages
they have backups, but it’s just a file with history that can be used to restore them
u

ursus

03/13/2019, 6:11 AM
hm but then you can delete messages, and you can do that while the other is offline so they still have this problem
g

gildor

03/13/2019, 6:11 AM
also, as user I hate WhatsApp solutions, but if I would developer, maybe I would do the same, just depending on current situation, product requirements etc
u

ursus

03/13/2019, 6:11 AM
okay guess the only 100% solution is to track the disconnects
g

gildor

03/13/2019, 6:12 AM
not sure what you mean about track the disconnect and why it’s 100%
u

ursus

03/13/2019, 6:12 AM
really? I feel whatsapp is the most solid of messengers, all other crap out on me from time to time
or were you thinking about them not keeping history if you reisntall the app?
g

gildor

03/13/2019, 6:13 AM
hm but then you can delete messages, and you can do that while the other is offline so they still have this problem
there are different strategies about message deleting, sometimes you can delete only local message, sometimes you delete message also on devices of all your receipients, later require deliver delete intent same way as you deliver information about new message or message status
u

ursus

03/13/2019, 6:14 AM
yes, its the same issue = meaning you miss the websocket push when offline / background
and need to pull upon online / foreground
and or "server tell me what I have missed since timestamp = x"
g

gildor

03/13/2019, 6:15 AM
really? I feel whatsapp is the most solid of messengers, all other crap out on me from time to time
They support only 1 client at the same time, super bad solution, because of their limitation of this architecture even cannot have proper standalone web client You don’t have real history of messages, no proper syncronization There is no way to use it without phone
u

ursus

03/13/2019, 6:15 AM
yea true
I meant in terms of reliability of push notifs
g

gildor

03/13/2019, 6:16 AM
It’s not related to push notifications
It’s important for messanger, but it’s just one of tools
u

ursus

03/13/2019, 6:16 AM
just saying, since we are giving our feelings on whatsapp 😄
g

gildor

03/13/2019, 6:17 AM
You can check Telegram or Signal or some XMPP messanger implementation, they are pretty different, but open source
u

ursus

03/13/2019, 6:17 AM
im wondering, is there a way to see websocket traffic via say Charles? id just copy messenger / slack
g

gildor

03/13/2019, 6:18 AM
Yes, I believe it’s possible, but never tried
u

ursus

03/13/2019, 6:21 AM
sounds the best, okay thank you!
g

gildor

03/13/2019, 6:24 AM
there are open source solutions, not sure that attempt to do reverse engineering for such huge, complicated, proprietary solutions is good for the first step
also all of them are encrypted, so you can use Charles, but you should replace certificates on device to allow to read them
also as I know FB messenger uses MQTT which is binary so you probably anyway cannot read it easily, especially understand how it works
you can also use Chrome Dev tools to see request at least on web
But sync on web may be pretty different from mobile app
Just checked, slack uses “cursors” word and has properties like “has_more” on message update request
g

ghedeon

03/13/2019, 6:56 AM
The server can probably keep a history of edit versions and once you send your
lastUpdate
timestamp, the
now - lastUpdate
diff is calculated for you. In a way it's similar to DB migrations or git.
b

BMG

03/14/2019, 10:35 AM
This is exactly what Couchbase Sync gateway solves by having
revision
documents. For ex, say you have documents 1 to 10 in remote and in local device. And some
change
happened in document #1 while local device was offline. Now, remote
has to
create a new revision document #11. (Here it need not be a new document itself, it can just be a
revision
attached to it). When local comes online, it will ask changes since #10. Remote can send Revision #11 which is actually a modified document #1.
Without explicit
revision
, there is no way (other than full sync) for remote server to know that the local device documents are in sync with remote.
This can scale really well as server does not need to
track
anything explicitly per device. They are all document revisions and each one is created for any change (modification, deletion, creation)
2 Views