feat: implement authz schema loading pipeline

[rodmgwgu]

Closes #410

Problem

Authorization schema definitions need to be discovered, compiled, validated, rendered, and loaded into the database, with a management command and persistence models to support it.

Approach

Implements the full pipeline and supporting code:

• openedx_authz/engine/schema/ — discovery, compilation, validation, loading, pipeline, types, exceptions
• openedx_authz/engine/renderer.py — schema rendering
• openedx_authz/management/commands/load_authz_schema.py — management command
• openedx_authz/models/schema.py + migration 0011_authz_schema_definitions.py
• MANIFEST.in / setup.py packaging updates for the YAML data files
• tests under openedx_authz/tests/schema/

Manual testing instructions

Run in a tutor dev environment with this repo mounted. Every command below is shown for cms; lms works identically.

Apply the new migration first:

tutor dev run cms ./manage.py cms migrate openedx_authz

1. Dry run reports the change without writing

tutor dev run cms ./manage.py cms load_authz_schema --dry-run

Expect Rows to add (132), Rows to remove (0), and definition sections for 17 categories, 48 permissions, 11 roles, and 132 role-permission grants. Confirm nothing was written:

tutor dev run cms ./manage.py cms shell -c "
from openedx_authz.models.schema import AuthzRoleDefinition
print(AuthzRoleDefinition.objects.count())"   # 0

2. Apply, then confirm it is idempotent

tutor dev run cms ./manage.py cms load_authz_schema
tutor dev run cms ./manage.py cms load_authz_schema

The first run prints 132 row(s) added, 0 removed. The second prints Authz schema unchanged; nothing would be written. — no duplicate rows, no churn (ADR 0018 §2).

3. Enforcement is unchanged relative to authz.policy

The schema files render exactly the 132 p rows that authz.policy defines, so a deployment that already ran load_policies sees no behavior change. Assign yourself a role and spot-check both a granted and a non-granted permission:

tutor dev run cms ./manage.py cms enforcement

At the prompt (format: subject action scope):

<your-username> content_libraries.view_library lib:OpenedX:CSPROB
✓ ALLOWED
<your-username> content_libraries.delete_library lib:OpenedX:CSPROB
✗ DENIED     # unless your role grants it

4. Provenance is recorded and queryable (ADR 0025)

tutor dev run cms ./manage.py cms shell -c "
from openedx_authz.models.schema import origins_for_role, origin_for_role_permission
print(origins_for_role('course_admin'))
print(origin_for_role_permission('course_admin', 'courses.view_course'))"

Both print ['openedx-authz'], since openedx-authz is the contributing distribution for the shipped schema.

5. A role extension changes an existing role without copying it (ADR 0023)

Create an override directory inside an importable package — this is the same YAML a Tutor patch will supply once the plugin lands:

mkdir -p openedx_authz/authz/site_overrides
cat > openedx_authz/authz/site_overrides/overrides.yaml <<'YAML'
schema_version: "1.0"
priority: 200
role_extensions:
  - role: course_editor
    add_permissions:
      - courses.export_course
    remove_permissions:
      - courses.manage_tags
    display_name: Course author
  - role: course_staff
    hidden: true
YAML

tutor dev run cms ./manage.py cms load_authz_schema --dry-run \
    --dir openedx_authz/authz/site_overrides

Expect exactly one row added (courses.export_course), one removed (courses.manage_tags), ~ course_editor and ~ course_staff under role definition changes, and the two matching grant changes. Apply it and verify the result:

tutor dev run cms ./manage.py cms load_authz_schema --dir openedx_authz/authz/site_overrides

tutor dev run cms ./manage.py cms shell -c "
from openedx_authz.models.schema import AuthzRoleDefinition, AuthzRolePermissionSource
print(AuthzRoleDefinition.objects.get(role_id='course_editor').display_name)   # Course author
print(AuthzRoleDefinition.objects.get(role_id='course_staff').hidden)          # True
link = AuthzRolePermissionSource.objects.get(
    role_permission__role__role_id='course_editor',
    role_permission__permission__name='export_course')
print(link.origin_kind, link.priority, link.source.source_id)                  # extension 200 ...site_overrides"

Then confirm enforcement followed, via ./manage.py cms enforcement with a user holding course_editor: courses.export_course is now ALLOWED and courses.manage_tags is DENIED.

Clean up and re-apply the base schema to restore the original state:

rm -rf openedx_authz/authz/site_overrides
tutor dev run cms ./manage.py cms load_authz_schema

6. Removing an assigned role is blocked without --force (ADR 0018 §6)

Assign a static role to a test user through the Admin Console or the assignment API, then remove that role from the schema (comment it out of openedx_authz/authz/schema/course_roles.yaml) and apply:

tutor dev run cms ./manage.py cms load_authz_schema

Expect a CommandError naming the role and the assigned subject, ending with "Re-run with force to remove them together with their assignments", and no change to the database: the role, its p rows, and the assignment are all still present and the user still enforces as allowed.

Re-run with the flag:

tutor dev run cms ./manage.py cms load_authz_schema --force

The role, its p rows, its assignment, and its definition rows are gone, the user is now denied, and one audit record exists per removed assignment:

tutor dev run cms ./manage.py cms shell -c "
from openedx_authz.models.core import RoleAssignmentAudit
print(RoleAssignmentAudit.objects.filter(operation='deleted').values_list('subject','role','scope'))"

Restore course_roles.yaml, re-apply, and re-create the assignment if you want the environment back as it was.

7. Pre-existing policy rows are adopted, unmanaged rows are left alone (ADR 0025 §6)

On a database where load_policies has already run but the schema has never been applied, load_authz_schema adds 0 rows and still creates all the definition and source records — the existing rows are adopted rather than rewritten. A p row that no schema declares stays in place and enforceable:

tutor dev run cms ./manage.py cms shell -c "
from openedx_authz.engine.enforcer import AuthzEnforcer
e = AuthzEnforcer.get_enforcer()
e.add_policy('role^legacy_thing', 'act^courses.view_course', 'course-v1^*', 'allow')"

tutor dev run cms ./manage.py cms load_authz_schema --dry-run

Rows to remove must stay at 0, and the legacy_thing row must still be in the policy after a full apply.

8. An invalid schema stops deployment before any write

Introduce an error in a schema file — for example point a role at a permission that does not exist, or give a permission a Namespace in CamelCase — and run the command. It exits with a CommandError, the log lists every validation error with its source file, and the database is untouched. Remove the error and confirm the command succeeds again.

Automated coverage

openedx_authz/tests/schema/ (216 tests) covers the discover → load → validate → compile → render → plan/apply path. openedx_authz/tests/integration/test_schema_apply.py (13 tests) exercises the real Casbin enforcer and ORM, including enforce() assertions for permission removal, force-removal, adoption, and failure rollback. The integration module is excluded from the default pytest run and needs an edx-platform environment:

tutor dev run cms pytest -p no:randomly --create-db --ds=cms.envs.test \
    /mnt/openedx-authz/openedx_authz/tests/integration/test_schema_apply.py

Rollback plan

Revert this PR to remove the pipeline; the schema YAML files (PR 2) and ADR (PR 1) remain but are inert without this code. The new migration is additive; roll back by reverting and running the reverse migration if applied.

Retro compatibility

At this point, existing loading of authz.policy works as previously, so there is no change on the existing functionality.

The new policy loading is not executed automatically, only manually by calling the new load_authz_schema management command.

No authorization behavior changed.

Follow up work required

After this lands, the following is required to complete the schema functionality:

1. Once docs: add authorization schema reference #431 is merged, extend the validation to use that schema to cover all cases defined in the ADR.
2. Implement the API layer to gather the new roles and permissions information
3. Create tutor plugin to implement auto running the management command on deploy
4. In the same tutor plugin, implement the patch support to add schemas via tutor.
5. Deprectate authz.policy file - Remove any role and permission definition from there that is already defined in the new schema, we will need to keep the g2 definitions there for now until we decide if we should support those in the schema in the future.

AI Usage

Kiro was used to assist on feature planning and implementation. Implementation was done step by step with human guidance and validation, based on the ADRs.

---

Stack (3/3):

1. docs: add ADR for authorization schema source tracking #444 — ADR doc
2. feat: add authz schema definition files #445 — authz schema YAML files
3. This PR — schema loading pipeline (base: rod/authz-schema-files)

Base is the PR 2 branch; review only the incremental diff. Merge last, after #444 and #445.

Merge checklist:

• Version bumped — 1.23.0 -> 1.24.0
• Changelog record added
• Documentation updated (not only docstrings)
• Fixup commits are squashed away
• Unit tests added/updated
• Manual testing instructions provided
• Noted any: Concerns, dependencies, migration issues, deadlines, tickets — depends on feat: add authz schema definition files #445 and docs: add ADR for authorization schema source tracking #444; contains a DB migration

[openedx-webhooks]

Thanks for the pull request, @rodmgwgu!

This repository is currently maintained by @openedx/committers-openedx-authz.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.