Reactions received before the message reacted to are trashed

[link2xt]

It is possible that Bob's reaction to Alice's message is received before Alice's message. One common reason for this is that Alice and Bob use different sending relays (SMTP servers) and these relays may not be able to deliver messages to some of your relays. Eventually all messages are received, but when you come online, it is possible that you first fetch the messages from a relay that has only Bob's messages delivered, and then fetch from another relay that has Alice's messages.

I just got this problem that my Desktop has some reactions missing because they are trashed this way. There is a related issue #8365, i had both problems in the same chat. I now have a message that has 9 reactions on Android, but only four reactions on Desktop, from just a day ago.

This can be reproduced with a straightforward Rust test. What to do is not that straighforward, i don't have a concrete plan and have not looked at the SQL schema, but whatever is done, the end result should be Alice's message with Bob's reaction even though Bob's reaction is received first and Alice's message is received later. If Alice's message never arrives, orphan reactions should be cleaned up eventually, can probably be done somewhere in housekeeping.

[Hocuri]

Right now, the recactions table has three columns msg_id, contact_id, reaction.

I basically see two proper solutions:

• Add a table pending_reactions with a column rfc724_mid. When receiving a reaction without a corresponding reaction, add it there. When receiving a message, check if it has any pending reactions.
• Deprecate the reactions table, and instead start using a new table reactions_new, which references the message by rfc724_mid. Then we don't need to know the message in order to add the reaction.

One piece of context that is relevant here: Throughout the core code, msg_id means the integer row ID in the msgs SQL table, while rfc724_mid means the Message-Id header.

FTR, there are two previous PRs #8225 and #8181, but they only solve part of the problem.

[iequidoo]

Eventually all messages are received, but when you come online, it is possible that you first fetch the messages from a relay that has only Bob's messages delivered, and then fetch from another relay that has Alice's messages.

If the 2nd relay has Bob's messages as well then it's fixed in #8181 / #8225 . If the 2nd relay only has Alice's messages and we want to support this case, we should close the mentioned PRs / remove the corresponding commits from them.

Deprecate the reactions table, and instead start using a new table reactions_new, which references the message by rfc724_mid. Then we don't need to know the message in order to add the reaction.

+1, i also was thinking of such a solution.

EDIT: See #8367 (comment) -- pending_reactions is a likely better solution, and one more reason is that there can be multiple message bubbles with the same rfc724_mid. We shouldn't display the same reactions on multiple bubbles if so.

[link2xt]

I think the issue should not be seen as a problem with IMAP fetching, just two receive_imf calls, one for Bob's reaction and then another for Alice's message, should already result in a message with a reaction. Which means it should work even if Bob's messages arrive only to the first relay, and Alice's messages arrive only to the second relay.

[iequidoo]

Then one more case to test is when Alice adds Bob to a group and Bob reacts to Alice's message, but Bob's reaction is received earlier.

[link2xt]

I looked up, on my largest profilereactions table is small:

 reactions:              624 KiB,  30313 rows

So migrating all reactions to a new table while keeping the deprecated reactions around for compatibility is possible, but I think adding pending_reactions is better, you will not get lost reactions on downgrade, no old table around etc.

[iequidoo]

The problem isn't specific to reactions, see #8372 for other cases (not sure i found all of them, and there will be more in the future). I made a crutchy fix for realtime advertisements (#8373 ), but now i think it's better to add some pending_msgs (or msgs_pending) table to solve all those cases by rerunning receive_imf for "orphaned" messages, this doesn't look more complex than the pending_reactions solution.

However, for sync messages it can't be solved this way because sync messages contain multiple sync items and every item may reference a message, but it's better to be discussed in #7984 .