Repository navigation
fix: bound how long content/search waits on Meilisearch document writes - #39240
blarghmatey wants to merge 1 commit into
Conversation
_wait_for_meili_task polls until Meilisearch finishes a task, with no deadline. Meilisearch applies tasks one at a time in enqueue order across every index, so a one-document write waits behind whatever is queued ahead of it. The library handlers run these writes with .apply(), inside the Studio request, so a backed-up queue turns V2 library component creation into a gateway timeout for a component that was created. This adds MEILISEARCH_WRITE_WAIT_TIMEOUT (default None, which keeps the current behavior) and a timeout argument on _wait_for_meili_task. The incremental write helpers pass the setting through; when it runs out they log and return, and Meilisearch still applies the write. Index creation, rebuilds and settings updates keep waiting until done. Claude-Session: https://claude.ai/code/session_01M1juEUTm73Bd97rWKX7aWG
openedx-webhooks
added
the
open-source-contribution
|
Thanks for the pull request, @blarghmatey! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
This branch has not been deployed
2 participants
Description
_wait_for_meili_taskpolls Meilisearch until a task finishes, with no deadline. Meilisearch applies tasks one at a time in the order they were enqueued, across every index on the instance, so a one-document write waits behind whatever is ahead of it in the queue. Several library handlers inhandlers.pyrun their search task with.apply(), so that wait happens inside the Studio request. When the queue is backed up, creating a Libraries V2 component outlives the gateway timeout and the author gets a 504 for a component that was in fact created.This adds a
MEILISEARCH_WRITE_WAIT_TIMEOUTsetting (seconds, defaultNone) and atimeoutargument on_wait_for_meili_task. The three helpers used for incremental document writes (_update_index_docs,_delete_index_doc,_delete_documents) pass the setting through. If the task is still enqueued or processing when the timeout runs out, the helper logs a warning and returns instead of raising. The write stays in Meilisearch's queue and is applied when Meilisearch reaches it; only the caller stops waiting. Index creation, rebuilds and settings updates call_wait_for_meili_taskdirectly and keep waiting until done.The default keeps current behavior. The limit is per write, not per request. A library change with fewer than five affected entities runs
send_change_events_for_modified_entitiesinline, and each entity's handler makes its own write, so an operator has to keep the timeout times that count under the gateway timeout (the setting's description says so). I went with a per-write limit because a shared per-request budget needs request-scoped state threaded through the Celery tasks, and the per-write version is enough to stop the 504s.The cost of setting it: a write that is not waited for is not visible to the Authoring MFE's refetch until Meilisearch applies it, so a new component can be missing from the library listing for that long, and a write that later fails is logged by Meilisearch but not raised. The helpers are shared with the
.delay()paths, so in a worker that also meansautoretry_fordoes not see a failure that happens after the timeout. Both are better than a 504 on a write that succeeded, and an operator who doesn't want either keeps the default.The index split in #39103 does not remove this wait. It separates documents, but
studio_contentandstudio_library_contentshare one Meilisearch task queue ("Enqueued tasks are processed one at a time, following the order in which they were requested", and a batch only groups tasks on the same index, per https://www.meilisearch.com/docs/learn/async/asynchronous_operations), so library writes still queue behind course writes.This is narrower than the async API discussed in #38993, which would remove the polling altogether. It bounds the existing behavior until that lands. It does not overlap #39042, which bounds Celery retries and time limits but cannot bound the eager path, since
soft_time_limitis not enforced under.apply().Roles affected: Course Author (no 504 on library component creation when the setting is used), Operator (new optional setting).
Supporting information
What we measured on courses.learn.mit.edu (Meilisearch 1.53.1, single instance, after #39103):
studio_contentholds 1,588,735 documents andstudio_library_content1,226.POST /api/libraries/v2/<lib>/blocks/requests got a 504 from our CDN at 60.0s. The traces for two of them show the CMS returned 200 after 140.4s and 141.9s, and 140.37s of the first was the eagerupsert_library_block_index_docspan.meilisearch_task_queue_latency_secondswas 76-201s for that whole window and peaked at 439s at 15:12.Testing instructions
Unit tests in
TestWaitForMeiliTaskcover: waiting until done without a timeout, returning with a warning once the timeout runs out, still raising when a task fails inside the timeout, and the three write helpers passing the setting through (andNonewhen unset).To reproduce against a running Studio, the queue has to be backed up, not unreachable:
MEILISEARCH_WRITE_WAIT_TIMEOUT = 5../manage.py cms reindex_studioon a course-heavy instance), so the queue is busy for minutes.Stopped waiting for Meilisearch task <uid> on index <prefix>studio_library_content after 5s. Without the setting, the same request blocks until the batch ahead of it finishes.Deadline
None.
https://claude.ai/code/session_01M1juEUTm73Bd97rWKX7aWG