Merge 5 17 26

.gitignore

@@ -39,4 +39,4 @@ learning_observer/learning_observer/static_data/google/
 learning_observer/learning_observer/static_data/admins.yaml
 .ipynb_checkpoints/
 .eggs/
-.next/
+.next/

Dockerfile

@@ -2,6 +2,15 @@ FROM python:3.10
 RUN git config --global --add safe.directory /app
 WORKDIR /app
 
+# Future work:
+# - This image currently uses an older Python base image version.
+# - Update the Python version to align with current LO/WO support targets.
+# - Some configurations may also require extra roster files mounted/provided
+#   at runtime (for example: admins.yaml or teachers.yaml).
+# - Consider a layered setup:
+#   1) maintain a shared "base LO" Docker image
+#   2) extend it with a WO-specific image (or other module-set images)
+
 # TODO start redis in here
 # see about docker loopback
 RUN apt-get update && \

Makefile

@@ -46,8 +46,8 @@ install: install-pre-commit-hook
 	# recent version. We would like to directly fetch `lodrc-current`,
 	# however, the fetch only returns the name of the file it's
 	# linked to. We do an additional fetch for the linked file.
-	@LODRC_CURRENT=$$(curl -s https://raw.githubusercontent.com/ETS-Next-Gen/lo_assets/main/lo_dash_react_components/lo_dash_react_components-current.tar.gz); \
-	pip install https://raw.githubusercontent.com/ETS-Next-Gen/lo_assets/main/lo_dash_react_components/$${LODRC_CURRENT}
+	@LODRC_CURRENT=$$(curl -s https://raw.githubusercontent.com/ArgLab/lo_assets/main/lo_dash_react_components/lo_dash_react_components-current.tar.gz); \
+	pip install https://raw.githubusercontent.com/ArgLab/lo_assets/main/lo_dash_react_components/$${LODRC_CURRENT}
 
 install-dev:
 	# TODO create a dev requirements file

README.md

@@ -7,7 +7,7 @@ learning analytics dashboard to help instructors be able to manage
 student learning processes, and in particular, student writing
 processes.
 
-![linting](https://github.com/ETS-Next-Gen/writing_observer/actions/workflows/pycodestyle.yml/badge.svg)
+![linting](https://github.com/ArgLab/writing_observer/actions/workflows/pycodestyle.yml/badge.svg)
 
 ## Learning Observer
 
@@ -17,6 +17,14 @@ per-student writing data, and aggegators to make dashboards. We've
 tested this in math and writing, but our focus is on writing process
 data.
 
+At a high level, Learning Observer functions as an application platform.
+The primary `learning_observer` module bootstraps the system: it loads
+configuration, connects to storage and messaging back ends, and brokers
+communication with data sources. Other modules plug into that
+infrastructure to define the specific reducers, dashboards, and other
+items that users interact with, letting teams experiment with new
+features without having to reimplement the platform core.
+
 It's not finished, but it's moving along quickly.
 
 ## Writing Observer
@@ -54,9 +62,10 @@ that the core approach and APIs are correct.
 
 ## Getting Started
 
-We have a short guide to [getting started](docs/workshop.md). Getting
-the base system working is pretty easy. The guide is pretty
-comprehensive (including how to develop your first module).
+We have a short guide to [installing the system](docs/tutorials/install.md).
+Getting the base system working is pretty easy. To create a new module
+for the system to use, check out our [cookiecutter module guide](docs/tutorials/cookiecutter-module.md).
+For a current package/module overview, see the [module inventory](docs/reference/module_inventory.md).
 
 ### System requirements
 

VERSION

@@ -1 +1 @@
-0.1.0+2025.10.01T21.16.15.146Z.8b205d0c.master
+0.1.0+2026.06.08T14.57.09.3NZ.c99f98d.refactor.portfolio.diff

autodocs/how-to.rst

@@ -4,22 +4,29 @@ How-to
 Practical instructions for achieving specific goals within Learning Observer. Use these guides when you know what outcome you need and want a proven recipe to follow:
 
 - :doc:`Communication Protocol <docs/how-to/communication_protocol>` - How to query data from reducers or system endpoints for dashboards.
-- :doc:`Configure Learning Observer <docs/how-to/config>` - Set up credentials, environment variables, and other configuration details required for a smooth deployment.
 - :doc:`Build Dashboards <docs/how-to/dashboards>` - Walk through creating dashboards from reducer outputs, including layout choices and data wiring.
-- :doc:`LTI <docs/how-to/lti>` - Cover how to install Learning Observer as an LTI application.
+- :doc:`Offline Reducer Replay <docs/how-to/offline_replay>` - Explain how to repopulate reducer content with study logs.
+- :doc:`Serve as LTI application<docs/how-to/lti>` - Cover how to install Learning Observer as an LTI application.
+- :doc:`Connect LO Blocks to Canvas via Learning Observer<docs/how-to/connect_lo_blocks_to_canvas>` - Show how to connect launch LO Blocks through Learning Observer from within Canvas.
+- :doc:`Configure Multiple Roster Sources<docs/how-to/multiple_roster_sources>` - Allow the system to dynamically choose a roster source given a user's context.
 - :doc:`Run with Docker <docs/how-to/docker>` - Learn how to containerize the stack, manage images, and operate the project using Docker Compose.
 - :doc:`Writing Observer Extension <docs/how-to/extension>` - Install, configure, and validate the Writing Observer browser extension for capturing events.
 - :doc:`Interactive Environments <docs/how-to/interactive_environments>` - Connect Learning Observer to Jupyter and other live coding setups for iterative development.
+- :doc:`Impersonate Users <docs/how-to/impersonation>` - Start and stop acting as another user while keeping dashboards informed.
+
 
 .. toctree::
    :hidden:
    :maxdepth: 1
    :titlesonly:
 
    docs/how-to/communication_protocol.md
-   docs/how-to/config.md
    docs/how-to/dashboards.md
+   docs/how-to/offline_replay.md
    docs/how-to/lti.md
+   docs/how-to/connect_lo_blocks_to_canvas.md
+   docs/how-to/multiple_roster_sources.md
    docs/how-to/docker.md
    docs/how-to/extension.md
    docs/how-to/interactive_environments.md
+   docs/how-to/impersonation.md

autodocs/index.rst

@@ -12,6 +12,12 @@ per-student writing data, and aggegators to make dashboards. We've
 tested this in math and writing, but our focus is on writing process
 data.
 
+At a high level, Learning Observer operates as an application platform:
+the core :mod:`learning_observer` package boots the system, loads
+configured modules, and manages shared data services, while each module
+provides the specific dashboards, reducers, and other artifacts that
+users interact with.
+
 Our documentation is organized into four main categories, each serving a different purpose. You can explore them below:
 
 - :doc:`Tutorials <tutorials>` - Step-by-step guides to help you learn by doing.

autodocs/reference.rst

@@ -9,6 +9,8 @@ Detailed, structured information about APIs, configurations, and technical detai
 - :doc:`Linting Rules <docs/reference/linting>` - Review the automated checks that keep the codebase healthy and how to run them locally.
 - :doc:`Testing Strategy <docs/reference/testing>` - Explore the testing layers we rely on and guidelines for writing reliable tests.
 - :doc:`Versioning and Releases <docs/reference/versioning>` - See how we tag releases, manage dependencies, and maintain backward compatibility.
+- :doc:`lo_assets Built Packages <docs/reference/lo_assets>` - Understand how prebuilt package artifacts are published and consumed.
+- :doc:`Module Inventory <docs/reference/module_inventory>` - Get a quick at-a-glance list of current monorepo modules and their roles.
 - :doc:`Module Reference <modules>` - Dive into the autogenerated API reference for Python modules within Learning Observer.
 - :doc:`API Reference <api>` - Inspect the internal functionality of the system.
 
@@ -23,5 +25,7 @@ Detailed, structured information about APIs, configurations, and technical detai
    docs/reference/linting.md
    docs/reference/testing.md
    docs/reference/versioning.md
+   docs/reference/lo_assets.md
+   docs/reference/module_inventory.md
    modules
    api

devops/README.md

@@ -10,4 +10,65 @@ We would like to be cross-platform, and evenually support both
 Debian-based distros and RPM-based distros. We're not there yet
 either. We'd also like to support multiple cloud providers. We're not
 there yet either. However, we probably won't accept PRs which move us
-away from this goal.
+away from this goal.
+
+## Create an AWS account and an EC2 instance.
+* Select Ubuntu, nano AMI. The cost should be 0.5 cents per hour.
+* In security groups, add HTTP, HTTPS rules. Open up only to your computer's IP address (the client).
+* Launch instance.
+* Suppose your instance public DNS is {ec2_ip}, e.g., ec2-18-223-122-172.us-east-2.compute.amazonaws.com.
+* Create aSSH key pair, save PEM file say under ~/.ssh, chmod to u+r.
+
+## Set up the EC2 instance.
+* SSH into the machine: `bash ssh -i {pem_file} ubuntu@{ec2_ip}`.
+* (Optional) Create a user account: sudo useradd {user}
+* Download the server code (same repository as the extension):
+```bash
+cd
+git clone https://github.com/ETS-Next-Gen/writing_analysis.git writing_analysis
+```
+* Install Ansible.
+```bash
+sudo apt-get update
+sudo apt-get upgrade
+sudo apt-get install ansible
+````
+* Configure Ansible.
+sudo pico /etc/ansible/hosts
+Add
+```
+[localhost]
+127.0.0.1
+```
+* `cd ~/writing_analysis/configuration`.
+* Run `sudo ansible-playbook local.yaml`. This may take a while on an EC2 nanon machine.
+If all goes well, you should see an output with no errors, like this:
+```
+bash
+...
+PLAY RECAP ******************************************************************************************
+127.0.0.1                  : ok=5    changed=4    unreachable=0    failed=0   
+```
+* Navigate to http://{ec2_ip}; you should see the message "Welcome to nginx!" if it's working.
+
+## Obtain a free domain name
+* Go to noip.com.
+* Sign up 
+
+## Obtain a free SSL Certificate Using Certbot
+* Run the following commands:
+```bash
+sudo apt-get install software-properties-common
+Sudo add-apt-repository universe
+sudo add-apt-repository ppa:certbot/certbot
+sudo apt-get update
+sudo apt-get install certbot python-certbot-nginx
+```
+```
+bash
+sudo certbot --nginx
+```
+-- Put in your {mydomain}.hopto.org address.
+-- Choose 1 - no redirect.
+
+## Stand up a backend server on the EC2 instance.

devops/tasks/remote_scripts/gitpaths.py

@@ -10,7 +10,7 @@ def gitpath_to_name(packagepath):
     '''
     Convert a git path to the name of the repo. For example:
 
-    `https://github.com/ETS-Next-Gen/writing_observer.git` ==> `writing_observer`
+    `https://github.com/ArgLab/writing_observer.git` ==> `writing_observer`
     '''
     package = os.path.split(packagepath)[1]
     if package.endswith(".git"):

devops/tasks/scripts/baseline_packages.fab

@@ -1,4 +1,4 @@
 cd
 sudo apt-get -y install git ansible awscli
-git clone https://github.com/ETS-Next-Gen/writing_observer.git
+git clone https://github.com/ArgLab/writing_observer.git
 cd writing_observer/devops/ansible ; sudo ansible-playbook local.yaml

docs/concepts/architecture.md

@@ -5,7 +5,7 @@ Piotr Mitros
 ## Introduction
 
 Like all such documents, this document should be taken with a grain of
-salt. It my be out-of-date, or not fully implemented. 
+salt. It my be out-of-date, or not fully implemented.
 
 ## Overview
 
@@ -19,6 +19,17 @@ salt. It my be out-of-date, or not fully implemented.
 5. Consumers can aggregate these notifications, inspect the external state,
    and make a dashboard.
 
+## Application platform structure
+
+Learning Observer acts as the shared platform that hosts and coordinates
+modules. The core `learning_observer` package owns the boot process: it
+loads configuration, establishes connections to databases and pub/sub
+systems, and exposes the APIs modules use to register reducers,
+dashboards, and other artifacts. Individual modules focus on defining
+those artifacts, relying on the platform to handle data ingestion and
+communication so new functionality can be added without duplicating the
+runtime infrastructure.
+
 ### Technology choices
 
 1. Generic student information (e.g. names, auth, etc.) cn live in

docs/concepts/events.md

@@ -129,3 +129,34 @@ authentication, safety checks, and analytics while keeping the event format
 itself lightweight. Clients only need to agree on the JSON structure of events,
 while the server handles durability and routing responsibilities on their
 behalf.
+
+## Configuring domain-based event blacklisting
+
+Incoming events can be blacklisted through PMSS rules so that specific domains
+either continue streaming, are told to retry later, or drop events entirely.
+The `blacklist_event_action` setting controls the action and defaults to
+`TRANSMIT`. Define rules under the `incoming_events` namespace and include a
+`domain` attribute to scope the behavior per organization. When the action is
+`MAINTAIN`, the `blacklist_time_limit` setting controls whether the client
+should wait a short time or stop sending forever.
+
+```pmss
+incoming_events {
+  blacklist_event_action: TRANSMIT;
+}
+
+incoming_events[domain="example.org"] {
+  blacklist_event_action: DROP;
+}
+
+incoming_events[domain="pilot.example.edu"] {
+  blacklist_event_action: MAINTAIN;
+  blacklist_time_limit: DAYS;
+}
+```
+
+When a client connects, the server extracts a candidate domain from the event
+payload and uses it to resolve the `blacklist_event_action` setting. If
+a rule returns `DROP`, the client is instructed to stop sending events.
+`MAINTAIN` asks the client to retain events and retry after a delay (as defined
+by `blacklist_time_limit`), while `TRANSMIT` streams events normally.

docs/concepts/key_value_store.md

@@ -0,0 +1,107 @@
+# Key-value store
+
+Learning Observer reducers and dashboards communicate through a key-value store (KVS).
+Reducers write internal state and dashboard-facing summaries to the store, while
+queries and presentation layers read those JSON blobs back. The
+[`learning_observer.kvs` module](../../learning_observer/learning_observer/kvs.py)
+wraps the different storage backends behind a common async API.
+
+## Router and lifecycle
+
+The `KVSRouter` constructed during startup owns every configured backend. When
+`learning_observer.prestartup` runs it reads the `kvs` section from system
+settings, instantiates the requested implementations, and exposes them through
+the module-level `KVS` callable. Most code imports `learning_observer.kvs.KVS`
+and invokes it to obtain the default backend:
+
+```python
+from learning_observer import kvs
+
+store = kvs.KVS()  # returns the default backend configured in settings
+value = await store[key]
+```
+
+Reducers obtain the store implicitly through decorators such as
+`kvs_pipeline` and `student_event_reducer`. Those helpers capture the module
+context, derive the reducer keys, and persist the reducer's internal and
+external state back to the configured KVS.
+
+The router also exposes named backends as attributes. If a module needs to
+store data in a non-default backend it can call `kvs.KVS.<name>()` where
+`<name>` matches the identifier from configuration.
+
+## Configuring backends
+
+The `kvs` block in `creds.yaml` (or an equivalent PMSS overlay) declares each
+backend. Settings accept either a mapping or an array of key/value tuples.
+Every entry must provide a `type` that matches one of the built-in
+implementations:
+
+```yaml
+kvs:
+  default:
+    type: filesystem
+    path: .lo_kvs
+  redis_cache:
+    type: redis_ephemeral
+    expiry: 30
+```
+
+During startup the router validates the configuration and raises a
+`StartupCheck` error if a backend is missing required parameters or references
+an unknown type. Once the process finishes booting, the resulting callable is
+available to the rest of the system as `learning_observer.kvs.KVS`.
+
+### Supported types
+
+| Type             | Class                       | Persistence behavior                                                  | Required settings                             |
+|------------------|-----------------------------|------------------------------------------------------------------------|-----------------------------------------------|
+| `stub`           | `InMemoryKVS`               | Data lives only in process memory and disappears on restart.          | None                                          |
+| `redis_ephemeral`| `EphemeralRedisKVS`         | Uses Redis with a per-key TTL for temporary caches.                    | `expiry` (seconds)                            |
+| `redis`          | `PersistentRedisKVS`        | Stores data in Redis without an expiry; persistence depends on Redis. | Redis connection parameters in system config. |
+| `filesystem`     | `FilesystemKVS`             | Serializes each key to JSON on disk for simple local persistence.      | `path`; optional `subdirs` boolean            |
+
+All backends share the async API: `await store[key]`, `await store.set(key, value)`,
+`await store.keys()`, and `await store.dump()` for debugging.
+
+### Filesystem layout
+
+The filesystem implementation writes JSON documents under the configured path.
+If `subdirs` is true it mirrors slash-separated key prefixes into nested
+folders while prefixing directory names with underscores to avoid collisions.
+This backend is convenient for workshops or debugging because state survives
+restarts as long as the directory remains intact, but it is not designed for
+large-scale deployments.
+
+### Redis variants
+
+Both Redis implementations rely on the shared connection utilities in
+`learning_observer.redis_connection`. The ephemeral variant requires an
+`expiry` value so it can set a TTL when calling `SET`, making it suitable for
+integration tests or scratch environments. The persistent variant omits the TTL
+so keys remain until explicitly deleted or until the Redis server evicts them.
+Ensure the Redis instance has persistence enabled (`appendonly` or RDB
+snapshots) if the deployment expects reducer state to survive reboots.
+
+### In-memory stub
+
+The stub backend keeps a Python dictionary in memory. It is useful for unit
+tests or prototype scripts but should not be used when the process restarts or
+scales across workers. The module exposes a `clear()` helper to wipe the store
+between tests.
+
+## Working with reducer state
+
+Reducer keys follow the pattern `<scope>,<module>,<selector>` where the scope
+captures whether the state is internal or external, the module identifies the
+producer, and the selector encodes the entity (for example, a student ID). When
+reducers process new events they read the previous state from the KVS, compute
+an updated value, and call `set()` to write it back. Dashboards and protocol
+adapters then fetch the external state by constructing the same key or by using
+higher-level query helpers that wrap the KVS API.
+
+If a dashboard appears empty after restarting the server, confirm which backend
+is active. In-memory and ephemeral Redis stores start empty on boot, so the
+system needs a fresh stream of events to repopulate reducer state. Filesystem
+and persistent Redis backends retain data unless their underlying storage was
+cleared.