Create more elaborate base class for curation step

[zyzzyxdonta]

Proposed substeps for the curate step:

1. Prepare (e.g. initialize, check config, check token permissions, load policies)
2. Validate (runs checks on the data model using the loaded policies)
3. Create report (human-readable, machine-readable)
4. Process decision (using data model and human-made decisions, either fail the pipeline or exit the curation step successfully)

The execution stops after step 3. If hermes curate --continue is run (i.e. with --continue flag, only step 4 is run.

The first concrete implementation of this will be in the (to be developed) Software CaRD curation plugin for HERMES.

[sdruskat]

Thanks, this generally makes sense at first glance, but I'm wondering about potential side effects.

2. Validate (runs checks on the data model using the loaded policies)

This sounds as if policies were required, which - if I understand correctly - they aren't at the moment, as they have a SoftwareCARD scope, not a hermes scope. I.e., validation should be possible without policies, and your proposed implementation in #441 doesn't mention them.

However, issues such as #127 (mentioned by @SKernchen today as discussion point of the SC meeting) remain unresolved, and require the mapping of existing values to values required by the deposit target.

If the validate substep is a suitable place for checking against SC policies (representing institutional policies for software publication), wouldn't it also be a good place to at least make it possible to validate against deposition requirements?

Policies are one way in which deposition requirements could be exchanged with curation plugins. 💡

Where would these policies come from?

1. Directly via deposition target APIs (one can dream).
2. Deposit plugin devs have the option to include policies for their platforms (e.g., Invenio plugin would provide "last, first" name format requirement mentioned in JSON data query and recombination during metadata mapping #127).

Deposit plugins would still have to solve the mapping issue in #127 separately from potentially catching it in curation for NOP and automated curation workflows.

The execution stops after step 3. If hermes curate --continue is run (i.e. with --continue flag, only step 4 is run.

How would non-active curation be handled on the workflow level, e.g., in cases of NOP (e.g., AcceptCuratePlugin) or fully automated curation?

Have we had an ADR, description or implementation of --continue flag somewhere already?

[zyzzyxdonta]

This sounds as if policies were required, which - if I understand correctly - they aren't at the moment, as they have a SoftwareCARD scope, not a hermes scope.

You're right. In 1., loading the policies was mentioned as an example. In 2., I forgot to indicate that this is also meant as an example.

If the validate substep is a suitable place for checking against SC policies (representing institutional policies for software publication), wouldn't it also be a good place to at least make it possible to validate against deposition requirements?

Would you consider this "curation"? To me this feels more like a precondition check in the deposit plugin. Also, at the Software CaRD meeting, @poikilotherm was in favour of having only one curate plugin in place at a time. I would agree with him because it would be strange to have two curation processes that could disagree with each other.

Policies are one way in which deposition requirements could be exchanged with curation plugins. 💡

Right. But as you mentioned, "policies" are a Software CaRD concept. This would get us into a weird circular dependency situation where Software CaRD requires HERMES for its interfaces and HERMES requires Software CaRD for the policy checks.

How would non-active curation be handled on the workflow level, e.g., in cases of NOP (e.g., AcceptCuratePlugin) or fully automated curation?

Good question. The flag is useless in this case. Maybe it should be an option specific to the plugin then, similarly to how it works for auth tokens?

Have we had an ADR, description or implementation of --continue flag somewhere already?

No.

---

I think it's pretty clear that this issue was based on what we're planning to do in Software CaRD. 😅 Maybe it could also be a good idea to have only the __call__ function in the base plugin and everything else is up to the plugin implementations? We were also discussing this for the deposit plugin because it turned out that most functions that were based on how things work in Invenio weren't all that useful when @nheeb implemented Dataverse support...