Skip to content

docs(development): add a code map for one observation's journey - #947

Merged
r0ssing merged 4 commits into
OpenDataEnsemble:devfrom
kingmakeruix:docs/observation-journey
Oct 1, 2026
Merged

r0ssing merged 4 commits into
OpenDataEnsemble:devfrom
kingmakeruix:docs/observation-journey

Conversation

@kingmakeruix

Copy link
Copy Markdown
Contributor

What

A contributor-facing code map that follows one observation — a tree measured in the rain — from the form on a device, through local storage and push, into server storage, and back out through pull and export.

New page: docs/docs/development/observation-journey.md, registered in docs/sidebars.ts next to development/architecture.

It contains:

  • a Mermaid sequence diagram of the happy path;
  • a table of ten steps, each naming the responsible project and linking to the symbol that implements it;
  • what changes when the device is offline (the write path makes no network call, and pending is derived from synced_at/updated_at rather than stored);
  • why attachments are a separate pipeline (observation JSON holds a GUID-shaped basename, binaries move over /api/attachments and are tracked on their own cursor);
  • five existing tests that exercise the path;
  • a short list of what the map deliberately does not cover.

Verification

Every source link was opened and confirmed before it was written, and every symbol named in the table was checked to exist at the cited path. The Mermaid diagram was parsed with the same Mermaid version the docs site uses.

From docs/, which is exactly what the docs workflow runs:

npm ci
npm run test    # No critical errors
npm run build   # Generated static files

The validator's warnings are all pre-existing anchor warnings in other pages; this page adds none. The page is 619 words excluding code blocks.

One thing I did not resolve

The issue says to ask in the issue when two sources disagree, so I am flagging rather than silently picking a story.

docs/docs/reference/form-specifications.md L784 states:

New observations use the latest form version

The Formulus write path cannot currently do that:

  1. PersistObservationInput has formType, finalData, observationId — no version field. submitObservation(formType, finalData) is positional, also with no version.
  2. persistObservationWithAttachments L232 calls saveObservation({ formType, data: committedData }).
  3. WatermelonDBRepo.saveObservation L225 then stores record.formVersion = input.formVersion || '1.0'.

So every locally created observation is persisted as '1.0', while formplayer does know the version — it reads formSchema.version for drafts and sticky fields, but never sends it across the bridge. This looks like the same root cause as #909, and I did not want to rewrite that documentation or claim a behaviour the code does not implement, so the page simply states that choosing a form's schema version is a separate question.

Closes #912

A contributor-facing trail map that follows a single observation from the
form on a device to server storage and back out through export, so a newcomer
can see where rendering, local persistence, push, server storage and
pull/export each happen.

- Mermaid sequence diagram of the happy path
- a table of the ten steps, each naming the responsible project and linking
  to the source symbol that implements it
- what changes when the device is offline: the write path makes no network
  call, and "pending" is derived from synced_at/updated_at rather than
  stored
- why attachments are a separate pipeline: observation JSON stores a
  GUID-shaped basename, binaries move over /api/attachments and are tracked
  on their own cursor
- five existing tests that exercise the path, two of which need PostgreSQL
- a short list of what the map deliberately does not cover

Every source link was opened and confirmed before writing, and the Mermaid
diagram parses cleanly.

Refs OpenDataEnsemble#912
@najuna-brian

Copy link
Copy Markdown
Member

Please comment on the issue so that it can be assigned to you

@kingmakeruix

Copy link
Copy Markdown
Contributor Author

@najuna-brian Thanks — done. I have commented on #912 asking to be assigned, with a short summary of what the draft already contains and what I deliberately left out.

I will not push anything further to this branch until the issue is assigned to me, so the diff stays as reviewed.

@r0ssing r0ssing left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sequence diagram shows Push then Pull, but both Formulus and ODE Desktop actually sync in Pull then Push order:

  • Formulus — syncObservationsImpl in formulus/src/api/synkronus/index.ts (L2013–2015):

    const pulled = await this.pullObservations(includeAttachments, options);
    throwIfSyncCancelled(options?.isCancelled);
    const pushed = await this.pushObservations(includeAttachments, options);
  • ODE Desktop — pull and push are separate Rust jobs dispatched individually from the store (synkPull at L696 and synkPush at L735 in useCustodianStore.ts), but the UI runs pull first.

The diagram arrows should be reordered: FM->>SK: POST /api/sync/pull before FM->>SK: POST /api/sync/push, and step 9 in the table ("Pull, then apply") should come before step 7/8 (Push).

r0ssing and others added 2 commits September 30, 2026 19:27
`r0ssing` pointed out that the sequence diagram showed push before pull, which
is not what the code does: `syncObservationsImpl` awaits `pullObservations`
and only then `pushObservations`.

- reorder the diagram so `/api/sync/pull` precedes `/api/sync/push`, and show
  `applyServerChanges` where it actually happens
- reorder the steps table so the pull steps come before the push steps
- add a short section on why the order matters: a device reconciles against
  fresh server state before offering its own changes
- note that ODE Desktop drives the two as separate operations
  (`synkPull`, `synkPush` in useCustodianStore.ts)

Verified before changing: `syncObservationsImpl` in
`formulus/src/api/synkronus/index.ts` pulls first, and both Desktop
operations exist in `desktop/src/store/useCustodianStore.ts`.

Refs OpenDataEnsemble#947
@kingmakeruix

Copy link
Copy Markdown
Contributor Author

@r0ssing Thanks — you were right, and I had checked it before changing anything.

I confirmed both claims in the code rather than taking the diagram's word for it:

  • syncObservationsImpl in formulus/src/api/synkronus/index.ts awaits pullObservations (L2013) and only then pushObservations (L2015), with throwIfSyncCancelled in between
  • Desktop does expose the two as separate operations — synkPull at L686 and synkPush at L735 in desktop/src/store/useCustodianStore.ts (small correction: the directory is store, not stores)

What I changed in b78b180:

  • the diagram now runs /api/sync/pull → applyServerChanges → /api/sync/push, with markObservationsAsSynced last, since that is the order the calls happen in
  • the steps table is reordered to match, so the pull steps come before the push steps
  • added a short "A sync pulls before it pushes" section, because the order itself is worth explaining rather than only drawing: a device reconciles against fresh server state before it offers its own changes
  • the Desktop sentence points at useCustodianStore.ts and states only that the two are separate operations, which is what I verified — I did not trace what the UI does with that freedom, so I did not claim it

Also thank you for merging dev into the branch. I rebased on top of that commit rather than over it, so your merge is still in the history.

Re-verified after the change: 686 words excluding code blocks, all 20 source links resolve, and the Mermaid block still parses with the version the docs site uses. npm run test and npm run build in docs/ are clean.

@r0ssing
r0ssing self-requested a review October 1, 2026 15:09

@r0ssing r0ssing left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kingmakeruix ! This is a great first contribution - I will merge it now, so that it gets published with the docs for v1.4.0 that will be released later today. Welcome to ODE 🦒

@r0ssing
r0ssing merged commit cbff59e into OpenDataEnsemble:dev Oct 1, 2026
10 checks passed
@kingmakeruix

Copy link
Copy Markdown
Contributor Author

Thank you @kingmakeruix ! This is a great first contribution - I will merge it now, so that it gets published with the docs for v1.4.0 that will be released later today. Welcome to ODE 🦒

Thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: trace one observation from a form to Synkronus and back

3 participants