[abhalsod-sonata]

Description

Very large assessment units, for example a 90-question mock exam where every question comes from a randomized question bank, can take longer to render in a single response than the request timeout allows, so the unit never loads.

This PR adds incremental loading for those units, behind a new course waffle flag, courseware.incremental_assessment_load (opt-in per course). With the flag on, a unit with more than INCREMENTAL_LOAD_PROBLEM_THRESHOLD (20) problems:

1. Renders its first INCREMENTAL_LOAD_EAGER_COUNT (5) children inline and leaves placeholders for the rest.
2. Fills the placeholders from the new GET /api/courseware/xblock_children/{usage_key}?child_usage_keys=... endpoint, in sequential batches of at most XBLOCK_CHILDREN_BATCH_MAX (10). The page shows "Loading question X of Y...", retries transient failures with backoff, halves a batch that times out, and offers a Retry button for any child that fails.

Units at or below the threshold, and requests from the mobile apps, always render in full.

The PR is seven commits. The first three are the downstream changes as they were merged in edx/edx-platform, so they can be compared with the originals. The rest adapt them to master:

1. feat: incrementally load large assessment units (feat: incrementally load large assessment xblocks edx/edx-platform#445): the flag, settings, endpoint, vertical shell and page script.
2. fix: hydrate learner state for incrementally loaded children (feat: hydrate learner state for paginated assessment children edx/edx-platform#465): lazily loaded problems now show the learner's saved answer, score and attempts. Without this, an answered problem rendered as a fresh attempt and resubmitting it failed with "The state of this problem has changed since you loaded this page". One FieldDataCache is populated per batch (new FieldDataCache.add_block_descendents_batch), so a batch costs a single learner-state lookup.
3. fix: hydrate item bank problems and staff debug in lazily loaded units (fix: LP-1397 hydrate item bank problems and staff debug in lazily loaded units edx/edx-platform#529): the same fix for problems inside randomized banks (item_bank / library_content), which are loaded through the shared runtime the parent bound. It also makes Staff Debug Info and Submission History work for lazily loaded children, by inserting them with edx.HtmlUtils.setHtml so their inline setup scripts run, the same as an inline render.
4. chore: adapt the incremental assessment load port to master: the hunks that did not apply to master (imports, the vertical render loop, the batch cache loop, the batch test) and the capa test factory path on master.
5. style: fix ruff findings in the incremental assessment load port.
6. docs: annotate incremental assessment load settings.
7. docs: address review feedback on comments and annotations.

Commits 1 to 4 do not pass CI on their own; only the full PR does, so it is meant to be squash-merged.

User roles impacted:

• Learner: large exams in courses with the flag on load reliably; the rest of the questions appear a moment after the page opens.
• Operator: a new opt-in course waffle flag and three settings, all annotated (lms/djangoapps/courseware/toggles.py, lms/envs/common.py).
• Developer: a new courseware API endpoint, xblock_children.

Before vs after

Measured on a local devstack with a 90-question unit, each question an item bank holding one problem, warm cache. The absolute numbers depend on the machine; the comparison is what matters.

	Before: flag off (current master behavior)	After: flag on
First response	6.0 to 6.3 s server time, 2.9 MB HTML, all 90 problems	2.6 to 2.7 s server time, 241 KB HTML, 5 problems and 85 placeholders
Follow-up requests	None	9 batch requests of up to 10 children, each 1.1 s or less
All 90 questions on screen	About 8 s	About 10 s

The total time to show every question stays about the same. The difference is that no single request has to render the whole unit, and the learner can start right after the first response.

1. Before (flag off): the whole 90-question unit arrives in a single 2.9 MB response.

2. After (flag on): the page arrives with the first 5 questions; questions 6 to 90 are placeholders.

3. After: the rest load in batches while the status line reports progress. The batch responses were slowed down in the browser to capture this state.

4. After: Staff Debug Info on question 7, a lazily loaded problem inside an item bank, opens.

5. After: question 7 answered and the page reloaded: the saved answer, the score (1/1 point) and the attempts (1 of 1) are kept, and Submit stays disabled.

Supporting information

• feat: incrementally load large assessment xblocks edx/edx-platform#445: the original incremental load.
• feat: hydrate learner state for paginated assessment children edx/edx-platform#465: learner-state hydration for lazily loaded children.
• fix: LP-1397 hydrate item bank problems and staff debug in lazily loaded units edx/edx-platform#529: hydration for problems inside randomized banks, plus Staff Debug Info and Submission History.

The original feature and the first fix were written by Santhosh Kumar, who is credited as co-author on those two commits. This PR ports them, with the follow-up fix, onto master.

Testing instructions

1. Create a course with a unit that has more than 20 problems. To cover the bank case, make each question a randomized content block (item bank) that picks one problem.
2. With the flag off, open the unit as a learner: it renders in full, as before.
3. In Django admin, add a waffle flag course override for courseware.incremental_assessment_load, forced on, for that course.
4. Reload the unit:
   • The first 5 questions arrive with the page.
   • The rest show "Loading question X of Y..." and fill in.
   • The network tab shows GET /api/courseware/xblock_children/<unit key>/?child_usage_keys=... requests with at most 10 keys each.
5. Answer a question after Q5 and refresh the page: the answer, score and attempts are still shown, and submitting again does not show "The state of this problem has changed...".
6. As course staff, click Staff Debug Info and Submission history under a question after Q5: both open.
7. Units with 20 or fewer problems, and mobile app requests, still render in full with the flag on.

Automated tests:

• lms/djangoapps/courseware/tests/test_incremental_render.py: the threshold and flag decision, the shell split, the endpoint helpers.
• openedx/core/djangoapps/courseware_api/tests/test_views.py (XBlockChildrenApiTestViews):
  • batch rendering, auth, 400 cases and partial failure;
  • the security boundary: only the parent's own, learner-visible children are served;
  • persisted state for plain problems and for problems inside an item bank.
• lms/djangoapps/courseware/tests/test_model_data.py: add_block_descendents_batch populates the cache once per batch.

Deadline

None.

Other information

• No migrations. The new endpoint uses JWT or session authentication and requires an enrolled learner with access to the parent unit. It only serves the parent's own children that are visible to that learner. For a randomized bank that means only the learner's assigned problems, never the whole pool.
• Security: child HTML comes from this LMS's own render of the same blocks the eager path inlines. The page script inserts it with edx.HtmlUtils.setHtml(el, edx.HtmlUtils.HTML(html)), the same idiom capa's display.js uses for server-rendered problem HTML.
• Not included: the downstream prefetch narrowing for randomized banks (edx/edx-platform@04e7ed024f). Without it, a batch prefetches each bank's whole pool, which is what master does today. It is independent of this feature and can be proposed separately.
• Locally verified with Python 3.12 and the uv.lock dependencies:
  • Tests: test_incremental_render.py, courseware_api/tests/test_views.py, test_model_data.py, xmodule/tests/test_vertical.py,
    lms/djangoapps/courseware/tests/test_block_render.py and lms/djangoapps/courseware/tests/test_views.py (496 passed).
  • Checks: ruff check ., make xsslint, make pii_check, make check_keywords, make lint-imports, make check-types, and pylint on the changed files.

AI use

I used Claude Code (Anthropic Claude) for this PR:

• Porting the three downstream PRs onto master and adapting them (imports, the capa test factory path, the setting annotations).
• Writing the item bank and Staff Debug Info fix in commit 3, and its tests.
• Drafting this description.

I reviewed every change and verified it with the tests and checks listed above. Commits 3 to 7 have a Co-authored-by trailer for Claude. Commits 1 and 2 are Santhosh Kumar's downstream code and credit him as co-author.

🤖 Generated with Claude Code

[robrap]

Some quick initial comments.

Also, Claude noted the following, and you should correct the first issue at a minimum. Thank you:

Findings (all minor, none block correctness)

1. Dangling reference: three comments in block_render.py (~1120, ~1185, ~1343) cite FieldDataCache._children_to_prefetch, which exists only downstream (added by excluded 04e7ed0). Not in master. Reword to cite has_dynamic_children()/get_child_blocks() / VerticalBlock.block_has_access_error only.
2. Attribution: description credits Santhosh Kumar (original feature + first fix) but commits are all authored by abhalsod-sonata with no Co-authored-by: trailers. Consider adding trailers so history credits the original author.

[robrap]

This should be left out for permanent toggles.

Suggested change

# .. toggle_target_removal_date: None

[abhalsod-sonata]

Done in b066cbd, removed toggle_target_removal_date since this is a permanent toggle.

[robrap]

Does this make sense? I think it may be more clear, if it is correct. Thoughts?

Suggested change

	# .. setting_description: Maximum children accepted by a single request to
	# /api/courseware/xblock_children/. Requests above this are rejected with a 400.
	# .. setting_description: Maximum children expected and accepted by a single request to
	# /api/courseware/xblock_children/. Unexpected requests above this are rejected with a 400.

[abhalsod-sonata]

Yes, that's correct, so I applied it in b066cbd. The page script batches using the same XBLOCK_CHILDREN_BATCH_MAX, so only an unexpected client would send more.

[abhalsod-sonata]

Some quick initial comments.

Also, Claude noted the following, and you should correct the first issue at a minimum. Thank you:

Findings (all minor, none block correctness)

1. Dangling reference: three comments in block_render.py (~1120, ~1185, ~1343) cite FieldDataCache._children_to_prefetch, which exists only downstream (added by excluded 04e7ed0). Not in master. Reword to cite has_dynamic_children()/get_child_blocks() / VerticalBlock.block_has_access_error only.
2. Attribution: description credits Santhosh Kumar (original feature + first fix) but commits are all authored by abhalsod-sonata with no Co-authored-by: trailers. Consider adding trailers so history credits the original author.

Thanks! Both are addressed:

1. Fixed in b066cbd. block_has_access_error on master only walks get_children(), and the dynamic-children branch is downstream-only, so citing it would also be wrong. The comments now describe the has_dynamic_children() / get_child_blocks() API directly.

2. Commits 1 and 2 now carry Co-authored-by for Santhosh Kumar. I also split the history so commits 1 to 3 are the downstream changes as merged, and the later commits adapt them to master.