Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
392 commits
Select commit Hold shift + click to select a range
5513e64
update nextflow version
antoniasaracco Jul 6, 2026
70f8209
Merge branch 'nf-core:implement_rclone' into implement_rclone
antoniasaracco Jul 6, 2026
3a98fc6
move rclone to local folder
antoniasaracco Jul 6, 2026
80b09ca
run fix lint
antoniasaracco Jul 6, 2026
03a79a3
fix
antoniasaracco Jul 6, 2026
db581e8
Update CHANGELOG with new features and fixes
antoniasaracco Jul 7, 2026
dad2dc2
Merge pull request #38 from antoniasaracco/implement_rclone
antoniasaracco Jul 7, 2026
4968204
Merge remote-tracking branch 'upstream/dev' into report
delfiterradas Jul 7, 2026
7da1d5a
Do not expert plots for multiqc and update snapshots
delfiterradas Jul 7, 2026
b550108
Ignore linting for multiqc_config file
delfiterradas Jul 7, 2026
018e1af
Remove notes from report
delfiterradas Jul 7, 2026
c889174
Update multiqc module
delfiterradas Jul 7, 2026
a7d6b58
Update citations
delfiterradas Jul 7, 2026
27c8a9e
Update CHANGELOG
delfiterradas Jul 8, 2026
9fee02b
Merge pull request #41 from delfiterradas/report
delfiterradas Jul 8, 2026
4553595
import rclone_copy from nf core
antoniasaracco Jul 14, 2026
23d9d02
update changelog
antoniasaracco Jul 14, 2026
69b94fa
Add local rclone modules
delfiterradas Jul 15, 2026
2f48fbf
Switch to use rclone modules
delfiterradas Jul 15, 2026
35e2f22
Use rclone outputs in multiqc report
delfiterradas Jul 15, 2026
e7ef374
Update snapshots
delfiterradas Jul 15, 2026
272e80c
Uninstall md5 and checkusm modules and delete local comparechecksum m…
delfiterradas Jul 15, 2026
5be9584
Fix linting
delfiterradas Jul 15, 2026
6d6f22d
improve samplesheet parsing
Jul 16, 2026
c244677
Install `rclone` modules from nf-core
delfiterradas Jul 16, 2026
6b2e34b
Revert changes made to conf containers
delfiterradas Jul 16, 2026
1e0b28d
Remove whitespace
delfiterradas Jul 16, 2026
e9456f2
update snapshots
antoniasaracco Jul 16, 2026
9ad475c
update rclone copy
antoniasaracco Jul 16, 2026
f3a4727
update containers files
antoniasaracco Jul 16, 2026
e294dee
remove withespace
antoniasaracco Jul 16, 2026
684bdfd
Update CHANGELOG.md
delfiterradas Jul 16, 2026
5d1da84
Merge pull request #46 from antoniasaracco/import_rclone_copy
delfiterradas Jul 16, 2026
27eed24
Update snapshots
delfiterradas Jul 16, 2026
55ac0bc
Merge remote-tracking branch 'upstream/dev' into rclone_check
delfiterradas Jul 16, 2026
ce56247
Fix container config files
delfiterradas Jul 16, 2026
6624c41
Update CHANGELOG.md
delfiterradas Jul 16, 2026
36491b2
Update CHANGELOG.md
delfiterradas Jul 16, 2026
a185ae8
Make report descriptions clearer
delfiterradas Jul 16, 2026
97909c7
Update methods description
delfiterradas Jul 16, 2026
5981d68
Create helper function to parse rclone output
delfiterradas Jul 16, 2026
4b960ac
Fix typo
delfiterradas Jul 16, 2026
e91d1f9
Merge pull request #49 from delfiterradas/rclone_check
atrigila Jul 17, 2026
25c08ec
add color to multiqc report
atrigila Jul 17, 2026
656582d
Merge pull request #54 from atrigila/multiqc_report
atrigila Jul 17, 2026
f678fee
update multiqc report
atrigila Jul 17, 2026
1fa3a48
docs: document datasync pipeline
antoniasaracco Jul 17, 2026
447fa14
docs: expand rclone remote configuration guidance
antoniasaracco Jul 17, 2026
ce6bd4a
Merge pull request #55 from atrigila/multiqc_report
delfiterradas Jul 17, 2026
0506dc7
add edge test cases
atrigila Jul 20, 2026
4b9c262
update changelog
atrigila Jul 20, 2026
b1120ce
Merge pull request #57 from atrigila/add_nf-tests
atrigila Jul 20, 2026
1946fbe
docs: clarify tested rclone configuration scope
antoniasaracco Jul 21, 2026
9483a25
Merge pull request #2 from antoniasaracco/codex/create-documentation-…
antoniasaracco Jul 21, 2026
be56cb8
Update README.md
antoniasaracco Jul 22, 2026
ca5f192
Update usage.md
antoniasaracco Jul 22, 2026
ceb7d63
Update CHANGELOG.md
antoniasaracco Jul 22, 2026
f10bc2c
Update README.md
antoniasaracco Jul 22, 2026
c0cb403
Update CHANGELOG.md
antoniasaracco Jul 22, 2026
7dea7d8
Update README.md
antoniasaracco Jul 22, 2026
6ff8774
Update output.md
antoniasaracco Jul 22, 2026
790a648
Update rclone modules and inputs in workflow
delfiterradas Jul 22, 2026
eb934c8
Update README.md
antoniasaracco Jul 22, 2026
4f16da8
Add files via upload
antoniasaracco Jul 22, 2026
40af01f
Update README.md
antoniasaracco Jul 22, 2026
fdb0a8d
Update README.md
antoniasaracco Jul 22, 2026
51b9fdf
Update README.md
antoniasaracco Jul 22, 2026
b558b3c
Update README.md
antoniasaracco Jul 22, 2026
c3c93c9
Update README.md
antoniasaracco Jul 22, 2026
c6d12e3
Update snapshots
delfiterradas Jul 22, 2026
fd9b00c
Update README.md
antoniasaracco Jul 22, 2026
110eaf8
Update docs/usage.md
antoniasaracco Jul 22, 2026
cfd37ed
Update docs/usage.md
antoniasaracco Jul 22, 2026
fa3c7e8
Apply suggestion
delfiterradas Jul 23, 2026
fbf1c85
Merge pull request #61 from delfiterradas/update_rclone
delfiterradas Jul 23, 2026
65d43d6
Update usage.md
antoniasaracco Jul 23, 2026
673cd47
Update usage.md
antoniasaracco Jul 23, 2026
f59d297
Update usage.md
antoniasaracco Jul 23, 2026
13c6314
Update usage.md
antoniasaracco Jul 23, 2026
e4b7ddd
Update usage.md
antoniasaracco Jul 23, 2026
90155e6
Add files via upload
antoniasaracco Jul 23, 2026
dd4a552
Delete docs/images/metromap_style_pipeline_workflow_components.drawio…
antoniasaracco Jul 23, 2026
64a41d6
Update README.md
antoniasaracco Jul 23, 2026
3baecb6
Update output.md
antoniasaracco Jul 23, 2026
18e1b6b
Update usage.md
antoniasaracco Jul 24, 2026
98c775e
Add files via upload
antoniasaracco Jul 24, 2026
331046b
Update output.md
antoniasaracco Jul 24, 2026
c184be4
Update output.md
antoniasaracco Jul 24, 2026
10631f6
Update usage.md
antoniasaracco Jul 24, 2026
77738d5
Template update for nf-core/tools version 4.0.3
nf-core-bot Jul 24, 2026
d5daf90
Update docs/output.md
antoniasaracco Jul 24, 2026
db62912
Update docs/output.md
antoniasaracco Jul 24, 2026
01f6cda
Update docs/output.md
antoniasaracco Jul 24, 2026
679d210
Update docs/usage.md
antoniasaracco Jul 24, 2026
083f023
Update docs/output.md
antoniasaracco Jul 24, 2026
70519cc
Update docs/output.md
antoniasaracco Jul 24, 2026
1983a4f
Update docs/output.md
antoniasaracco Jul 24, 2026
e9a292a
Update docs/usage.md
antoniasaracco Jul 24, 2026
e5dd017
Delete docs/images/datasync-metromap.drawio.png
antoniasaracco Jul 24, 2026
359a1b0
Add files via upload
antoniasaracco Jul 24, 2026
2b553c3
Delete docs/images/datasync-metromap.drawio.png
antoniasaracco Jul 24, 2026
c6dcf3d
Add files via upload
antoniasaracco Jul 24, 2026
2ff8c91
Delete docs/images/datasync-metromap.drawio.png
antoniasaracco Jul 24, 2026
d11002a
Add files via upload
antoniasaracco Jul 24, 2026
a720a7b
Delete docs/images/datasync-metromap.drawio.png
antoniasaracco Jul 24, 2026
c38ce42
Add files via upload
antoniasaracco Jul 24, 2026
316efd3
Delete docs/images/datasync-metromap.drawio.png
antoniasaracco Jul 24, 2026
1a69b5e
Add files via upload
antoniasaracco Jul 24, 2026
589fb05
Update README.md
antoniasaracco Jul 24, 2026
3fe2a48
Update output.md
antoniasaracco Jul 24, 2026
1da035b
Update output.md
antoniasaracco Jul 24, 2026
d881783
Implement only copying files that matched their checksum
delfiterradas Jul 27, 2026
999dca5
Compute siz to avoid silent data loss
delfiterradas Jul 27, 2026
32dcb74
Refactor to only use rclone_copy once
delfiterradas Jul 27, 2026
e0d1df9
Bump nf-schema to 2.7.2 to fix errors in validation of boolean parame…
delfiterradas Jul 27, 2026
556e2ba
Remove vulnerable PR-comment artifact pattern
nf-core-bot Jul 28, 2026
8f34b11
Merge pull request #68 from nf-core/patch
mashehu Jul 28, 2026
426b319
Apply suggestions from code review
apeltzer Jul 28, 2026
06a59a6
Merge branch 'nf-core:dev' into create_documentation
antoniasaracco Jul 28, 2026
35778f5
Update README.md
antoniasaracco Jul 28, 2026
3c365be
Update README.md
antoniasaracco Jul 28, 2026
71318c9
Update usage.md
antoniasaracco Jul 28, 2026
f892c20
Add test scenario copying only matching files
delfiterradas Jul 28, 2026
fe97a5c
Remove vulnerable PR-comment artifact pattern
nf-core-bot Jul 28, 2026
cac46dd
Update changelog.md
delfiterradas Jul 28, 2026
71a8dc6
Merge pull request #59 from antoniasaracco/create_documentation
apeltzer Jul 28, 2026
6cb7cac
Update rclone_copy
delfiterradas Jul 29, 2026
ff96d7d
Merge pull request #67 from delfiterradas/only_copy_validated
delfiterradas Jul 29, 2026
e1d29fa
Merge dev into nf-core-template-merge-4.0.3
apeltzer Jul 30, 2026
371ae90
Fix linting
apeltzer Jul 30, 2026
65b7dfe
Create temp file avoiding errors with write permissions in workdir
delfiterradas Jul 30, 2026
790b8fe
Update CHANGELOG
delfiterradas Jul 30, 2026
05b139e
Fix linting
delfiterradas Jul 30, 2026
54b293e
Merge pull request #73 from delfiterradas/only_copy_validated
delfiterradas Jul 30, 2026
4a705aa
Add download parameter when using shasums
delfiterradas Jul 30, 2026
fa6dd32
Add validation when using shasums
delfiterradas Jul 30, 2026
0ed68fc
Checksums cannot be csvs and accept additional formats
delfiterradas Jul 30, 2026
447fc7b
Update tests and snapshots
delfiterradas Jul 30, 2026
7261126
Update CHANGELOG
delfiterradas Jul 30, 2026
caad856
Merge remote-tracking branch 'upstream/dev' into shasum_handling
delfiterradas Jul 31, 2026
e3e47d2
Update usage documentation
delfiterradas Jul 31, 2026
97e6f57
Update checksum verification documentation to SHA256
delfiterradas Jul 31, 2026
3621357
Update docs/usage.md
delfiterradas Jul 31, 2026
b91df99
Update workflows/datasync.nf
delfiterradas Jul 31, 2026
0bb34e3
Update CHANGELOG.md
delfiterradas Jul 31, 2026
fff74fc
Update workflows/datasync.nf
delfiterradas Jul 31, 2026
a7c7f08
Update nextflow_schema.json
delfiterradas Jul 31, 2026
d5b06b4
Update nextflow_schema.json
delfiterradas Jul 31, 2026
8cbd838
Add failing test scenario
delfiterradas Jul 31, 2026
c802695
Clean up edge.nf.test by removing unused variables
delfiterradas Jul 31, 2026
d29ad7e
Merge pull request #74 from delfiterradas/shasum_handling
delfiterradas Jul 31, 2026
c6ea9c6
fix linting
atrigila Aug 3, 2026
39eb943
fix container config linting
atrigila Aug 3, 2026
6949fb1
Revert linting fixes and container issues
atrigila Aug 3, 2026
080bef8
fix pull request template
atrigila Aug 3, 2026
03e0139
Merge pull request #66 from nf-core/nf-core-template-merge-4.0.3
atrigila Aug 3, 2026
66f5df3
remove todos
atrigila Aug 4, 2026
292d673
add missing subworkflow meta.yml
atrigila Aug 4, 2026
fd4e5a2
fix nextflow lint issues
atrigila Aug 4, 2026
ff429db
fix linting warnings
atrigila Aug 4, 2026
4cb4feb
remove todos
atrigila Aug 4, 2026
b2b267b
update subworkflows to latest versions
atrigila Aug 4, 2026
91d2264
fix rocrate
atrigila Aug 4, 2026
f03a010
fix nextflow lint
atrigila Aug 4, 2026
3110f6a
delete unused modules
atrigila Aug 4, 2026
fffea3c
update rclone modules
atrigila Aug 4, 2026
6fb5387
update snapshots
atrigila Aug 4, 2026
ef3f45f
update changelog
atrigila Aug 4, 2026
71af415
Bugfix: Rclone checksum should check if any files are present in sour…
delfiterradas Aug 5, 2026
16acce4
Update snapshot
delfiterradas Aug 5, 2026
1967cd2
Update changelog
delfiterradas Aug 5, 2026
71440e4
Merge pull request #76 from delfiterradas/dev
delfiterradas Aug 5, 2026
e14ff6e
Merge branch 'dev' into fix_warnings_linting
atrigila Aug 6, 2026
807a647
fix container config
atrigila Aug 6, 2026
7f992a0
Merge branch 'fix_warnings_linting' of https://github.com/atrigila/da…
atrigila Aug 6, 2026
fd10349
Merge pull request #78 from atrigila/fix_warnings_linting
atrigila Aug 6, 2026
800f6d4
Add exit code summary to report
delfiterradas Aug 7, 2026
ad82719
Update snapshots
delfiterradas Aug 7, 2026
4e15bfe
Fix minor format issues in readme file
delfiterradas Aug 7, 2026
17d50d8
Update changlelog
delfiterradas Aug 7, 2026
b41718e
Add module name to prefix and update tests
delfiterradas Aug 7, 2026
442e237
Bugfix: Add hash to module name in exit code summary
delfiterradas Aug 7, 2026
16eade4
Update snapshots
delfiterradas Aug 7, 2026
7d1f26d
Add validation summary section to output documentation
delfiterradas Aug 7, 2026
5ce0a10
Merge pull request #79 from delfiterradas/dev
delfiterradas Aug 7, 2026
296469f
Bump version
delfiterradas Aug 18, 2026
b17c85c
Fix linting
delfiterradas Aug 18, 2026
220c860
Add release dat to changlelog
delfiterradas Aug 18, 2026
bdf1f34
Update tests
delfiterradas Aug 18, 2026
23ae865
Fix linting
delfiterradas Aug 18, 2026
3d99943
Update PR template links to use master branch
delfiterradas Aug 18, 2026
54988aa
Merge pull request #81 from delfiterradas/dev
delfiterradas Aug 19, 2026
fb32bbf
Add apptainer version to avoid error in CI test
delfiterradas Aug 20, 2026
7bc7797
Update CHANGELOG.md
delfiterradas Aug 20, 2026
d2d46b6
Merge pull request #83 from delfiterradas/dev
delfiterradas Aug 20, 2026
6711148
Make sample column a unique entry
delfiterradas Aug 25, 2026
229af6b
Avoid silenlty dropping samples
delfiterradas Aug 25, 2026
c14cd93
Move samplesheet validation to utils subworkflow and get a single war…
delfiterradas Aug 25, 2026
26f7d59
Skip RCLONE_COPY if there are no files in common in files_to_copy
delfiterradas Aug 25, 2026
2194875
Modify regex to capture inputs with ":"
delfiterradas Aug 25, 2026
cb6bd9f
Capture 0 exit code status
delfiterradas Aug 25, 2026
1720b85
Update rclone_copy version
delfiterradas Aug 31, 2026
dc2ee38
Template update for nf-core/tools version 4.1.0
nf-core-bot Sep 1, 2026
2b98f35
Apply literal match for rclone_coopy whn requested
delfiterradas Sep 1, 2026
50b86cb
Touch all output files to avoid losing error code
delfiterradas Sep 1, 2026
2a50d9b
Raise clear error message
delfiterradas Sep 1, 2026
31cb76f
Add local module to generate files_to_copy for rclone_copy
delfiterradas Sep 1, 2026
18d85ed
Update docs to accept only local and URI paths
delfiterradas Sep 1, 2026
33bd607
Add normalization for http paths in rclone check and checksum
delfiterradas Sep 2, 2026
78bcc98
Do not accept `http` paths as source or destination paths
delfiterradas Sep 2, 2026
5a945af
Normalize input and destination paths
delfiterradas Sep 2, 2026
2088f3f
Update snapshots
delfiterradas Sep 2, 2026
9178433
Update rclone modules
delfiterradas Sep 3, 2026
29f8ee9
Run prek
delfiterradas Sep 3, 2026
46d4636
Update CHANGELOG
delfiterradas Sep 3, 2026
23f779f
Merge pull request #84 from delfiterradas/review_dev
delfiterradas Sep 3, 2026
412a67d
Add local test profile
delfiterradas Sep 4, 2026
4cf6105
Fix linting
delfiterradas Sep 4, 2026
d68d8bf
Update CHANGELOG.md
delfiterradas Sep 4, 2026
23161e1
Addexplanation for real data test profile
delfiterradas Sep 4, 2026
366487c
Merge pull request #87 from delfiterradas/review_dev
delfiterradas Sep 7, 2026
5cd483d
Implement review suggestions
delfiterradas Sep 8, 2026
e4372d9
Template update for nf-core/tools version 4.1.0
delfiterradas Sep 8, 2026
5619860
Merge branch 'TEMPLATE' into dev
delfiterradas Sep 8, 2026
a3bfda8
Change default branch and update rclone checksum
delfiterradas Sep 8, 2026
5117470
Fix linting
delfiterradas Sep 8, 2026
ba00df5
Skip linting for PULL_REQUEST_TEMPLATE.md
delfiterradas Sep 8, 2026
0bdf8b9
Merge pull request #90 from delfiterradas/dev
delfiterradas Sep 8, 2026
bcfdc6a
Update changelog for first release
delfiterradas Sep 8, 2026
69ad859
Merge remote-tracking branch 'upstream/dev' into review_dev
delfiterradas Sep 8, 2026
4cbf146
Add nf-metro plot
delfiterradas Sep 8, 2026
c43ec12
Fix linting
delfiterradas Sep 8, 2026
89905cb
Clean up CHANGELOG by removing empty sections
delfiterradas Sep 9, 2026
d2afa32
Merge pull request #89 from delfiterradas/review_dev
delfiterradas Sep 9, 2026
d6056fc
Add memory definition to avoid failing on seqera platform
delfiterradas Sep 9, 2026
d25e58a
Merge pull request #92 from delfiterradas/review_dev
delfiterradas Sep 9, 2026
4af74f2
Implement suggestions to documentation and minor changes in outputs
delfiterradas Sep 23, 2026
fc37342
Add documentation to create md5 and shasums
delfiterradas Sep 24, 2026
3024ac0
Create prepareSamplesheet function
delfiterradas Sep 24, 2026
520599b
Add test assertions
delfiterradas Sep 24, 2026
b3b9581
Move warning checks to edge test
delfiterradas Sep 25, 2026
e713808
Fix linting
delfiterradas Sep 25, 2026
29a2987
Fix linting
delfiterradas Sep 25, 2026
fc53e82
Update release date
delfiterradas Sep 25, 2026
b6a189d
Use nft-csv for assertion of line counts for rclone output files
delfiterradas Sep 25, 2026
bd709bf
Add back md5 snapshots
delfiterradas Sep 25, 2026
ba46be6
Clarify output_path usage in documentation
delfiterradas Sep 28, 2026
485314f
Merge pull request #93 from delfiterradas/dev
delfiterradas Sep 28, 2026
e686912
Change release date for v1.0.0
delfiterradas Sep 28, 2026
c67b1f9
Merge pull request #95 from delfiterradas/dev
delfiterradas Sep 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 15 additions & 22 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,28 +1,21 @@
{
"$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
"name": "nfcore",
"image": "nfcore/gitpod:latest",
"remoteUser": "gitpod",
"runArgs": ["--privileged"],
"image": "nfcore/devcontainer:latest",

// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/opt/conda/bin/python",
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
"python.formatting.yapfPath": "/opt/conda/bin/yapf",
"python.linting.flake8Path": "/opt/conda/bin/flake8",
"python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
"python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
"python.linting.pylintPath": "/opt/conda/bin/pylint"
},
"remoteUser": "root",
"privileged": true,

// Add the IDs of extensions you want installed when the container is created.
"extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack"]
}
"remoteEnv": {
// Workspace path on the host for mounting with docker-outside-of-docker
"LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}"
},

"onCreateCommand": "./.devcontainer/setup.sh",

"hostRequirements": {
"cpus": 4,
"memory": "16gb",
"storage": "32gb"
}
}
13 changes: 13 additions & 0 deletions .devcontainer/setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/usr/bin/env bash

# Customise the terminal command prompt
echo "export PROMPT_DIRTRIM=2" >> $HOME/.bashrc
echo "export PS1='\[\e[3;36m\]\w ->\[\e[0m\\] '" >> $HOME/.bashrc
export PROMPT_DIRTRIM=2
export PS1='\[\e[3;36m\]\w ->\[\e[0m\\] '

# Update Nextflow
nextflow self-update

# Update welcome message
echo "Welcome to the nf-core/datasync devcontainer!" > /usr/local/etc/vscode-dev-containers/first-run-notice.txt
37 changes: 0 additions & 37 deletions .editorconfig

This file was deleted.

123 changes: 0 additions & 123 deletions .github/CONTRIBUTING.md

This file was deleted.

1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ body:

- [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
- [nf-core/datasync pipeline documentation](https://nf-co.re/datasync/usage)

- type: textarea
id: description
attributes:
Expand Down
6 changes: 3 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ These are the most common things requested on pull requests (PRs).

Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.

Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/datasync/tree/master/.github/CONTRIBUTING.md)
Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/datasync/tree/main/docs/CONTRIBUTING.md)
-->

## PR checklist

- [ ] This comment contains a description of changes (with reason).
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/datasync/tree/master/.github/CONTRIBUTING.md)
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/datasync/tree/main/docs/CONTRIBUTING.md)
- [ ] If necessary, also make a PR on the nf-core/datasync _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
- [ ] Make sure your code lints (`nf-core lint`).
- [ ] Make sure your code lints (`nf-core pipelines lint`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
Expand Down
69 changes: 69 additions & 0 deletions .github/actions/get-shards/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: "Get number of shards"
description: "Get the number of nf-test shards for the current CI job"
inputs:
max_shards:
description: "Maximum number of shards allowed"
required: true
paths:
description: "Component paths to test"
required: false
tags:
description: "Tags to pass as argument for nf-test --tag parameter"
required: false
outputs:
shard:
description: "Array of shard numbers"
value: ${{ steps.shards.outputs.shard }}
total_shards:
description: "Total number of shards"
value: ${{ steps.shards.outputs.total_shards }}
runs:
using: "composite"
steps:
- name: Install nf-test
uses: nf-core/setup-nf-test@4069fbbaabe94c08faba4ad261bfa88225ba133f # v2
with:
version: ${{ env.NFT_VER }}
- name: Get number of shards
id: shards
shell: bash
run: |
# Run nf-test with dynamic parameter
nftest_output=$(nf-test test \
--profile +docker \
$(if [ -n "${{ inputs.tags }}" ]; then echo "--tag ${{ inputs.tags }}"; fi) \
--dry-run \
--ci \
--changed-since HEAD^) || {
echo "nf-test command failed with exit code $?"
echo "Full output: $nftest_output"
exit 1
}
echo "nf-test dry-run output: $nftest_output"

# Default values for shard and total_shards
shard="[]"
total_shards=0

# Check if there are related tests
if echo "$nftest_output" | grep -q 'No tests to execute'; then
echo "No related tests found."
else
# Extract the number of related tests
number_of_shards=$(echo "$nftest_output" | sed -n 's|.*Executed \([0-9]*\) tests.*|\1|p')
if [[ -n "$number_of_shards" && "$number_of_shards" -gt 0 ]]; then
shards_to_run=$(( $number_of_shards < ${{ inputs.max_shards }} ? $number_of_shards : ${{ inputs.max_shards }} ))
shard=$(seq 1 "$shards_to_run" | jq -R . | jq -c -s .)
total_shards="$shards_to_run"
else
echo "Unexpected output format. Falling back to default values."
fi
fi

# Write to GitHub Actions outputs
echo "shard=$shard" >> $GITHUB_OUTPUT
echo "total_shards=$total_shards" >> $GITHUB_OUTPUT

# Debugging output
echo "Final shard array: $shard"
echo "Total number of shards: $total_shards"
Loading
Loading