Support for detached crates by simleo · Pull Request #250 · ResearchObject/ro-crate-py

simleo · 2026-02-04T16:41:37Z

Closes #231.
Closes #232.

Adds several new features to support detached crates:

More testing for reading a crate from a JSON dictionary, which was already supported. This is one of the ways to read a detached crate
Much simpler algorithm to find the metadata descriptor and root data entity -- the previous one was based on a version of the 1.2 draft that does not exist anymore (see Update docs on finding the root dataset ro-crate#198).
Support for creating a detached crate from scratch: the ROCrate initializer now accepts a root_dataset_id argument to specify an arbitrary absolute URI
Support for reading a crate from a remote or local (file://) URL
Support for reading a crate from a local metadata file
Support for writing an RO-Crate as detached (write only the metadata file, with a custom name)
Support referencing detached crates as "subcrates"

Note I didn't add any is_detached or similar property: the distinction between attached and detached is quite subtle, and being one or the other depends on the current state of the crate in memory. For instance, if you add a data entity with a relative URI to a crate that had only web-based data entities, it won't be detached anymore.

elichad · 2026-02-11T12:06:39Z

@simleo just letting you know I've seen this and I'm looking at it today

elichad

Thanks @simleo, I made a few comments. Looks like this will resolve #231 and #232.

This review also got me thinking: do you think it would be useful to implement specific helper functions for converting between attached & detached crates, which follow the processes described in the spec appendix:

Not suggesting we do that in this PR (it's a bunch more work), but could make another issue for it if you think it would be a good idea.

elichad · 2026-02-11T11:52:12Z

    def write(self, base_path):
-        out_file_path = Path(base_path) / unquote(self.id)
+        local_path = self.get("localPath")
+        out_file_path = Path(base_path) / unquote(local_path or self.id)


If self.id is an absolute URI and localPath is not set, then the out_file_path becomes convoluted, for example:
/tmp/crate/https:/raw.githubusercontent.com/ResearchObject/ro-crate-py/master/test/test-data/sample_file.txt

It'd be good to error/warn here if self.id is an absolute URI and fetch_remote is true, and remind the user to set localPath. I would favour an error, but don't know if there are use cases where this would be the desired behavior?

Alternatively, could implement some default behavior where if fetch_remote is true and self.id begins with the RDE @id as a base, then that base can be stripped off and the remaining relative path can be used as the local path. This is the kind of behaviour I would expect.

Though checking the Converting from Detached to Attached RO-Crate Package section in the spec made me think about how it does get a bit more complex when nested Datasets/Files are involved - need to consider the case where a Dataset has localPath but the Files in its hasPart don't - and probably make the Files follow the Dataset's localPath.

Alternatively, could implement some default behavior where if fetch_remote is true and self.id begins with the RDE @id as a base, then that base can be stripped off and the remaining relative path can be used as the local path.

Done in 8c16c88

it does get a bit more complex when nested Datasets/Files are involved - need to consider the case where a Dataset has localPath but the Files in its hasPart don't - and probably make the Files follow the Dataset's localPath

Done in commits deda277 to fc61747, which also include fixes for the way remote Datasets are handled in general.

elichad · 2026-02-11T13:28:38Z

+    rcrate = ROCrate(out_path)
+    assert rcrate.get(f"{base_uri}sample_file.txt")
+    assert rcrate.get(f"{base_uri}test_file_galaxy.txt")
+    # now set fetch_remote to True


same as above - this test is quite long which makes it harder to follow - perhaps the part from here onward could be split into a different test

elichad · 2026-02-11T13:37:15Z

+    rp = crate.get(orcid)
+    assert rp["name"] == name
+
+    detached_md_path = tmpdir / "example-ro-crate-metadata.json"


This section could be a separate test

elichad · 2026-02-11T13:39:08Z

+    with pytest.raises(ValueError):
+        ROCrate(root_dataset_id="foo/bar")
+    with pytest.raises(ValueError):
+        ROCrate(root_dataset_id="/foo/bar")


not clear what these are testing - that you can't make a crate with a relative path as the RDE id? Would be clearer as a separate test since these lines don't seem to relate to the stuff above

Co-authored-by: Eli Chadwick <eli.chadwick@manchester.ac.uk>

…te is True

simleo · 2026-04-13T08:24:16Z

Thanks @simleo, I made a few comments. Looks like this will resolve #231 and #232.

This review also got me thinking: do you think it would be useful to implement specific helper functions for converting between attached & detached crates, which follow the processes described in the spec appendix:

attached to detached

detached to attached

Not suggesting we do that in this PR (it's a bunch more work), but could make another issue for it if you think it would be a good idea.

I think all comments have been addressed at this point. I'm going to merge, then open a new issue for conversion functions. The main difference with respect to using crate.write / crate.write_detached is that entity ids cannot be changed for an existing crate, so conversion functions will likely need to create and return a new crate. Some of the recommendations in the spec appendix don't seem to be very machine actionable (e.g. things like"Decide ... if this file is acceptable to archive"), but we could aim for a best effort and try to provide useful functionality.

simleo added 12 commits February 3, 2026 11:31

add test_detached

9b4fa8f

test_detached: more checks

815542b

find RDE id as prescribed by the spec

fc025ec

support specifying RDE id when creating a new crate

c2dc3f3

expand test_detached_creation

f72e7f5

add support for reading a crate from a remote URL

3e3a079

add test_from_uri_detached

26a2071

add test sections to check the written crates

8fff3ab

support for reading crates from file: URIs

8e36253

update branch to master in test data URI

6f0cb47

test_read.py: check reading from file: URI

60be6d5

support for reading crates from local metadata file

190186c

simleo marked this pull request as draft February 4, 2026 16:42

simleo added 6 commits February 5, 2026 10:38

fix test_read on Windows

137e106

add write_detached

779762d

support referencing detached crates

4d5f6bd

remove non-json content type warning when reading from url

567f169

add a section on detached crates to the docs

37a53d8

fix code highlighting in the docs

5ca8aef

simleo marked this pull request as ready for review February 10, 2026 15:33

simleo requested a review from elichad February 10, 2026 15:41

elichad reviewed Feb 11, 2026

View reviewed changes

simleo and others added 7 commits February 12, 2026 12:19

Apply suggestions from code review

8382d29

Co-authored-by: Eli Chadwick <eli.chadwick@manchester.ac.uk>

clarify some bits in the docs

0955c89

file.write: cut out path to the basename if it's a url and fetch_remo…

312a2a5

…te is True

file write: set localPath when dest path is set to basename

4e46b2c

docs: clarify localPath usage

dbd5e33

file.write: strip rde id to get relative path when applicable

8c16c88

override dest with localPath only when downloading a remote file

ff6a267

simleo added 8 commits March 27, 2026 12:59

fix remote dir handling

deda277

merge master into detached_crates

9d52960

remove docs on writing a detached crate "as attached"

5eeccdc

don't override a relative dest_path with localPath (create mode)

35cb0b8

override dataset localPath with part localPath if set

6b84395

fetch_remote on Dataset: skip Dataset parts

5975361

don't auto-set localPath after a download

fc61747

split long tests into smaller ones

ba531a6

simleo merged commit a084f18 into ResearchObject:master Apr 13, 2026
13 checks passed

simleo deleted the detached_crates branch April 13, 2026 08:25

simleo mentioned this pull request Apr 13, 2026

Functions for converting between attached & detached crates #255

Open

Conversation

simleo commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elichad commented Feb 11, 2026

Uh oh!

elichad left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elichad Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elichad Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

simleo Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

simleo Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

elichad Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elichad Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elichad Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

simleo commented Apr 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simleo commented Feb 4, 2026 •

edited

Loading

elichad Feb 11, 2026 •

edited

Loading