Nextflow

[dimalvovs]

This pull request introduces several improvements to the Nextflow pipeline for LLM-based interpretation, focusing on better GPU support, more flexible model caching, and simplified configuration. The main changes include enhanced documentation, more robust handling of the Ollama model cache directory, improved resource and container management for GPU execution, and streamlined output paths for intermediate and final results.

Pipeline and Execution Improvements:

• Added a new section in README.md detailing how the Nextflow workflow manages the Ollama model cache (OLLAMA_MODELS), including default paths, auto-pulling behavior, and instructions for overriding the cache location.
• Updated the INTERPRET process in main.nf to support GPU execution with appropriate resource limits, dynamic container selection, and explicit environment setup for the Ollama model cache. [1] [2] [3]

Configuration and Resource Management:

• Refactored nextflow.config to add ollama_models_dir and container as parameters, improved GPU resource allocation under the igs profile, and ensured proper mounting of the model cache directory in Apptainer containers.
• Removed the old Docker-specific process block and made container and cache directory configuration more flexible and profile-driven. [1] [2]

Output Path Simplification:

• Changed the default output and intermediate file paths in pipeline.py to use the current directory instead of a hardcoded data directory, making outputs more predictable and compatible with containerized execution. [1] [2] [3]

These updates make the pipeline more portable, cluster-friendly, and easier to configure for different environments.

[Copilot]

Pull request overview

This PR adds/updates a Nextflow-based execution path for running llmize (including Slurm/Apptainer support) and introduces a pipeline.py option intended to control where outputs are written.

Changes:

• Document Nextflow execution and model-cache behavior in README.md.
• Add --work-dir to pipeline.py and pass it from the Nextflow process.
• Extend Nextflow configuration for an igs Slurm/Apptainer GPU profile, including report/timeline output paths.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
README.md	Adds a “Nextflow execution” section with usage examples and cache notes.
pipeline.py	Adds --work-dir and uses it to choose a default output path when --output is not provided.
nextflow.config	Introduces an igs Slurm/Apptainer profile and redirects report/timeline outputs under params.outdir.
main.nf	Updates the INTERPRET process to use a GPU label, set resource limits, configure OLLAMA_MODELS, and run pipeline.py in a container.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

nextflow.config:50

• In the GPU label block, both clusterOptions and containerOptions can resolve to strings containing null when slurm_account or ollama_models_dir are unset (e.g. --gres=gpu:1 -A null / -B null), which will fail at runtime. Make these options conditional (or set a non-null default for the profile).

                // Allocate a gpu
                clusterOptions = "--gres=gpu:1 -A ${params.slurm_account?.toString()?.trim()}"
                // Enable Nvidia in apptainer and mount ollama dir
                containerOptions = "--nv -B ${params.ollama_models_dir}"
            

README.md:63

• The docs mention an igs_cpu profile and claim the igs/igs_cpu profiles default OLLAMA_MODELS to /usr/local/scratch/$USER/ollama/models, but nextflow.config currently defines only igs and does not set a default ollama_models_dir. Either add/define those profiles + defaults, or adjust the README so it matches the actual configuration/required flags.

The Nextflow module uses an Ollama model cache directory via `OLLAMA_MODELS`.

- In the `igs` and `igs_cpu` profiles, the default cache path is user-specific scratch:
   `/usr/local/scratch/$USER/ollama/models`
- On the first run, if the model is missing, the workflow auto-pulls it.
- On later runs, the same model is reused from scratch and pull is skipped.

pipeline.py:70

• The --output help text says the default output ends with .txt, but run_pipeline() actually writes a .md file (and the README examples also show .md). This makes the CLI usage misleading.

    parser.add_argument(
        "--output", "-o",
        default=None,
        help="Path to save the final interpreted report text. Defaults to ./<input_stem>_interpretation_<timestamp>.txt.",
    )

pipeline.py:178

• default_output_name() is being called with ".", which produces an odd stem (e.g. extracted_.) and can overwrite files across runs. It should derive the stem from the actual input JSON path (input_path).

    if save_intermediates:
        save_json(reduced, ".", extracted_filename or default_output_name(".", prefix="extracted_"))
        save_json(report, ".", annotated_filename or "annotated_report.json")