gmail: retry messages after body fetch failures

[fallintoplace]

Problem

Gmail search and body retrieval happen separately. When gog gmail get failed or timed out, the source still created a record from the search metadata with an empty body and advanced the shared timestamp cursor. The recovered message was then behind the cursor, leaving only the header-only record permanently stored.

Fix

• represent an unsuccessful body retrieval separately from a valid empty body
• skip records whose body could not be retrieved
• keep the previous cursor when any message in the batch fails
• continue returning successfully retrieved messages and rely on inbox ID deduplication for unchanged retries

Verification

Regression coverage exercises a partial failure followed by recovery, an exception during initial sync, and a nonzero body-fetch command.

• .venv/bin/python -m pytest -q tests/test_gmail_source.py tests/test_source_filters.py tests/test_db.py: 134 passed
• broader suite: 2941 passed

[alex-clickhouse]

One issue with this approach: suppose there is a message that for some reason fails gog gmail get deterministically. Then the cursor is going to be stuck at that point, until it falls out of the batch size window (and, since any failure means no cursor progress, it would take a bunch of other messages with it).

One option would be to retry in place for 2-3 times with exponential backoff. That still doesn't save us from longer-lived failures though.

More complex would be to do something similar to the Telegram one, which stores json, and then we could keep track of individual ids and their status. But that's probably a bit too much.

[mc806888-png]

समस्या

जीमेल सर्च और बॉडी रिट्रीवल अलग से होता है। कब gog gmail getअसफल या समयबद्ध, स्रोत ने अभी भी एक खाली शरीर के साथ खोज मेटाडेटा से एक रिकॉर्ड बनाया और साझा टाइमस्टैम्प कर्सर को उन्नत किया। बरामद संदेश तब कर्सर के पीछे था, केवल हेडर-केवल रिकॉर्ड को स्थायी रूप से संग्रहीत किया गया था।

फिक्स करें

* एक वैध खाली शरीर से अलग एक असफल शरीर पुनर्प्राप्ति का प्रतिनिधित्व करें

* रिकॉर्ड छोड़ें जिनके शरीर को पुनः प्राप्त नहीं किया जा सका

* बैच में कोई भी संदेश विफल होने पर पिछला कर्सर रखें

* सफलतापूर्वक पुनर्प्राप्त किए गए संदेशों को वापस करना जारी रखें और अपरिवर्तित रिट्री के लिए इनबॉक्स आईडी डिड्युप्लीकेशन पर भरोसा करें

सत्यापन

प्रतिगमन कवरेज वसूली के बाद आंशिक विफलता, प्रारंभिक सिंक के दौरान एक अपवाद और एक गैर-शून्य शरीर-पंछ लेने वाली कमांड का प्रयोग करता है।

* `.venv/bin/python -m pytest -q tests/test_gmail_source.py tests/test_source_filters.py tests/test_db.py`: 134 पास

* व्यापक सुइट: 2941 पारित

One issue with this approach: suppose there is a message that for some reason fails gog gmail get deterministically. Then the cursor is going to be stuck at that point, until it falls out of the batch size window (and, since any failure means no cursor progress, it would take a bunch of other messages with it).

One option would be to retry in place for 2-3 times with exponential backoff. That still doesn't save us from longer-lived failures though.

More complex would be to do something similar to the Telegram one, which stores json, and then we could keep track of individual ids and their status. But that's probably a bit too much.