tldr; how to figure out what to delete from locals, if remotes are paginated

can't you fetch like a range of messages or a specific one? You've got a push that a specific range of messages has changed, you check if it's within your local messages and if true, than fetch this chunk.

if im foreground = websocket is active, then its all good,

aren't things like firebase are designed for this kind of case?

I dont think slack has that automatic push all missed events

Not sure about slack, but Facebook Messanger and WhatsApp are using push model (long polling/websockets). It makes even more sense once you want to display the delivery status of the message. Anyhow, with pull model I guess you'd probably need to have two sections in your server response:

edited

and

latest

. So you can patch your history.

but neither of then for sure has websocket on background -- so you need to "pull" to synchronize, when coming back from background

Yes, you need to pull and to do that properly you need some API that returns diff of data from your last connection

yes, but, how to do that? -- meaning, sending every message id I have to server.. that cant scale

No, you can have special token/cursor (for simplest, but not very robust solution, it may be just timestamp of last message) on each diff request, so server know snapshot version of messages on your device

hm not sure if I udnerstand, if I tell server what my last message is, how can it tell that I for example need to update some n-th message I have locally because it was edited? (meaning that message is from before my last message)

I used this example just to illustrate how this cursor for diff request may work

okay so the calculation is a backend responsibility, based on my token / cursor?

there is no easy solution and trade-offs between simple/featureful implementation are everywhere, for example you may concider sync each conversation separately

yea Im doing that, but still that can not scale, if I query all of history in that conversation ever

what do you mean?

Im thinking only about this one use: I come back from offline, and I need to know what changed -- somehow -- other than fetching all of convo history to figure it out

I come back from offline, and I need to know what changed -- somehow -- other than fetching all of convo history to figure it out

Globally you have only 2 choices, download everything (or just first page) or use some diff/sync API, but how those diffs will be implemented highly depends on your use case, complexity of your server etc

okay so, lets say I want the diffs, since I dont want to download everything

okay

that means I need to tell the server about every message id I have locally for that conversation?

it one of possible options and your server will create diff info (added, deleted, updated messages) based on this, but it’s not very scalable, you right

yea, given huge conversation that will suck

there are other ways, when server remembers your initial sync session and knows which messages added/updated/deleted from your last sync

im thinking those etags you mentioned, cant that be used?

server just should keep snapshots for every client, and to identify this snapshot you need some sort id/tag/cursor/etc, it also just may be auth token

hm, what snapshot do you mean -- snapshot = what server thinks my local data is?

snapshot = what server thinks my local data is

yes

okay -- maybe I'll ask different, If I can only calculate the diffs on client, my only options is to fetch all history, right?

there are probably some materials about it, but I don’t know any actual “theory” about this, just engineering solution, if you know date of modification of every message on client’s device you can just request all the messages changed after this date, actual implementation may be different of course

yea, thats super stupid, and all data from first message I have locally,. thats only stupid

I fixed my message, it may be not “All” data, but only data that required to create diff, like list of all message ids with last update date (or something like version id)

yea ..would that me durable against breaking my database locally? meaning someone with root -- or more likely developer via bug, causing a message to be deleted ... or is that overkill?

but because by definition server has much more knowledge than client, because has much more resources, all the data is persistent, it much easier and more efficient (in terms of client) to do on server side

yes, im now looking for ideal solution, only after that ill try to bring it to reality and how flexible my api team is 😄

I think it’s overkill, if you cannot trust your local data it’s really strange and it looks like overkill to “validate” your local DB on each sync SQL is very robust solution, supports transactions, so not sure what is “developer via bug, causing a message to be deleted”

ofc I trust sql, I dont trust developers to not cause a bug and deleting some unrelated id or something

im still wondering how would the server keep track of my data

depens on strategy, but for conversation it may have some database of operations on particular conversation and then when you request it again just return all new operations add/delete/update

ok I can see that, but thats still not resilient agains the bug

Your user complains, you QA found it, your developer discovered it in code

I mean this will manifest as runtime thing -- how will user tell he is missing a message, when he doesnt know it should be there

you may do periodical validateion with server using special api, like check amount of conersations and amount of messages

😄 what if he not deletes, but updates it with foobar data, and wont update the lastUpdated timestamp hh

when he doesnt know it should be there

i don’t think this is somehow related to you problem, how to write program without bugs, how to write system that detects bugs, I don’t know, it’s even more broad question than “how to write real time chat messenger”

hm not really because full diff would fix it, the problem is pagination

no, it’s not related to pagination too

I think so, because if you diff against a page, you dont know about the rest of the data in next pages

You asking questions that don’t have any good answer, each solution is trade-off between of big amout of factors, choose which are more important for you, perfection solution doesn’t exist

basically yea, I was jsut curious about state of art

I think so, because if you diff against a page, you dont know about the rest of the data in next pages

I never suggest to use this approach, you right, it’s imo bad solution

whatsapp doesnt give a damn, they dont even have a remote database right? they only relay messages

Yes

hm but then you can delete messages, and you can do that while the other is offline so they still have this problem

also, as user I hate WhatsApp solutions, but if I would developer, maybe I would do the same, just depending on current situation, product requirements etc

okay guess the only 100% solution is to track the disconnects

not sure what you mean about track the disconnect and why it’s 100%

really? I feel whatsapp is the most solid of messengers, all other crap out on me from time to time

hm but then you can delete messages, and you can do that while the other is offline so they still have this problem

there are different strategies about message deleting, sometimes you can delete only local message, sometimes you delete message also on devices of all your receipients, later require deliver delete intent same way as you deliver information about new message or message status

yes, its the same issue = meaning you miss the websocket push when offline / background

really? I feel whatsapp is the most solid of messengers, all other crap out on me from time to time

They support only 1 client at the same time, super bad solution, because of their limitation of this architecture even cannot have proper standalone web client You don’t have real history of messages, no proper syncronization There is no way to use it without phone

yea true

It’s not related to push notifications

just saying, since we are giving our feelings on whatsapp 😄

You can check Telegram or Signal or some XMPP messanger implementation, they are pretty different, but open source

im wondering, is there a way to see websocket traffic via say Charles? id just copy messenger / slack

Yes, I believe it’s possible, but never tried

sounds the best, okay thank you!

there are open source solutions, not sure that attempt to do reverse engineering for such huge, complicated, proprietary solutions is good for the first step

The server can probably keep a history of edit versions and once you send your

lastUpdate

timestamp, the

now - lastUpdate

diff is calculated for you. In a way it's similar to DB migrations or git.

This is exactly what Couchbase Sync gateway solves by having

revision

documents. For ex, say you have documents 1 to 10 in remote and in local device. And some

change

happened in document #1 while local device was offline. Now, remote

has to

create a new revision document #11. (Here it need not be a new document itself, it can just be a

revision

attached to it). When local comes online, it will ask changes since #10. Remote can send Revision #11 which is actually a modified document #1.