This repository contains curated reference datasets, validation manifests, supporting metadata, reproducible scripts, and documentation for evaluating Mycobacterium tuberculosis (MTB) bioinformatics workflows.
The work forms part of the advISO project, which aims to support the development, validation, and implementation of quality-assured bioinformatics workflows in clinical and public health laboratories.
Clinical bioinformatics workflows require documented evidence that they are suitable for their intended use. International standards, including ISO 15189, provide a framework for evaluating analytical performance, reproducibility, traceability, and the management of changes to validated methods in laboratories.
This repository supports these requirements by developing structured, traceable validation datasets for MTB genomic analysis.
The emphasis is on establishing reliable expected results, documenting dataset selection and processing, and providing reusable resources against which workflow outputs can be evaluated.
The repository provides validation resources; it does not, by itself, establish ISO accreditation or demonstrate that a particular workflow has met all validation acceptance criteria.
The current work is organized around four validation objectives.
- Antimicrobial resistance (AMR) prediction
- Lineage assignment
- Variant calling
- Species determination
- Detection of mixed or contaminated samples
- End-to-end workflow reproducibility
Each objective has its own data sources, selection methodology, reference datasets, scripts, and supporting documentation.
validation_data_MTB/
├── contamination/
│ ├── metadata/
│ ├── reference_dataset/
│ ├── reports/
│ └── README.md
│
├── lineage/
│ ├── metadata/
│ │ └── fastlin/
│ ├── pilot/
│ └── README.md
│
├── mixed_infection/
│ ├── metadata/
│ ├── reference_dataset/
│ ├── reports/
│ └── README.md
│
├── tbtamr/
│ └── reference_dataset/
│
├── scripts/
│ ├── amr/
│ ├── contamination/
│ ├── lineage/
│ └── mixed_infection/
│
├── references/
├── docs/
├── env/
├── workflow/
├── .gitignore
└── README.md
The objective-specific directories contain validation resources and supporting evidence. The scripts/ directory contains utilities for dataset preparation and processing, while workflow/ contains workflow-related resources.
Primary source: tbtAMR
This objective focuses on evaluating antimicrobial resistance predictions against reference resistance information derived from published MTB datasets.
Phenotypic DST results serve as the ground truth, against which resistance predictions derived from WGS data are evaluated.The repository contains curated validation results and sample manifests supporting the selection and processing of MTB sequencing datasets.
The validation approach involves:
- Identifying samples with paired WGS and phenotypic DST results.
- Curating reference datasets and generating validation manifests.
- Running the bioinformatics workflow to obtain genotypic resistance predictions.
- Comparing predicted resistance profiles against phenotypic DST results.
- Assessing concordance and relevant performance metrics.
Primary source: Fastlin
This objective evaluates the ability of MTB bioinformatics workflows to assign the expected lineage and sub-lineage.
Fastlin and TB-Profiler classifications were used to identify candidate samples. Rather than accepting these classifications as ground truth, selected lineage assignments were further confirmed using sequencing evidence at established lineage-defining SNP positions.
The preparation process includes:
- Integrating Fastlin and TB-Profiler classification information.
- Identifying and selecting suitable single-lineage candidates.
- Preparing lineage-defining SNP barcode references and expected markers.
- Aligning sequencing reads and examining SNP-level evidence to confirm expected lineage assignments.
- Refining the reference panel based on SNP confirmation results.
- Evaluating lineage-supporting observations.
Supporting metadata, candidate selection tables, pilot manifests, reports, and scripts are maintained in the lineage/ and scripts/lineage/ directories.
See lineage/README.md for the detailed methodology.
Primary source: Sobkowiak et al. (2018)
This objective evaluates whether MTB workflows can identify samples containing more than one MTB strains.
The reference dataset includes known mixed-sample compositions and expected major and minor lineage information derived from the published study.
The dataset preparation process includes:
- Extracting reference sample information.
- Recording expected mixed-infection status.
- Documenting major and minor lineage assignments.
- Capturing expected mixture proportions.
- Generating a validation manifest.
- Recording data provenance and integrity checks.
These resources support subsequent comparisons between workflow predictions and known mixed-sample compositions.
See mixed_infection/README.md for further documentation.
Primary source: Heupink et al. (2021)
This objective evaluates workflow behaviour when MTB sequencing data contain reads originating from contaminating organisms or simulated contamination scenarios.
The reference dataset is constructed from published contamination simulation data and associated sequencing metadata.
The current preparation process includes:
- Extracting expected contamination information from the published dataset.
- Applying documented sample selection criteria.
- Linking selected samples to sequencing accessions.
- Generating a validation manifest.
- Selecting a balanced pilot dataset across contamination levels.
- Preparing FASTQ download information for subsequent workflow evaluation.
The current reference panel contains 100 samples at the selected coverage level, with a 30-sample pilot panel comprising five samples at each contamination level from 0% to 5%.
The repository includes the truth set, validation manifest, pilot manifest, selection policy, metadata, and preparation reports.
See contamination/README.md for the detailed selection methodology and supporting evidence.
Although each objective has its own reference dataset and performance characteristics, the overall validation approach follows a common structure.
Reference data identification
Identify published datasets with suitable expected results and sufficient supporting metadata.
Dataset preparation
Extract, standardize, and curate reference information while retaining links to the original sources.
Sample selection
Apply objective-specific inclusion and exclusion criteria, documenting selection decisions and dataset limitations.
Validation manifest generation
Create structured sample manifests linking identifiers, expected results, and sequencing resources.
Workflow evaluation
Execute the workflow under evaluation using the selected datasets and collect the relevant outputs.
Performance assessment
Compare observed outputs with the defined reference results and calculate appropriate objective-specific performance measures.
Documentation and traceability
Maintain scripts, reports, source references, dataset versions, and integrity information needed to reproduce and review the validation process.
Not every objective has completed every stage of this process. The objective-specific documentation should be consulted for current progress and outstanding work.
The repository is designed to support transparent and reproducible dataset preparation.
Where applicable, this includes:
- Public source references and accession identifiers.
- Dataset selection criteria and supporting metadata.
- Version-controlled scripts.
- Structured reference datasets and manifests.
- Data preparation reports.
- Checksums for integrity verification.
- Documentation of assumptions and limitations.
Large sequencing files, such as raw FASTQs and alignment BAMs, are excluded from routine Git version control. They should be obtained from the referenced data sources using the appropriate manifests and preparation procedures.
Local legacy files and temporary analysis outputs are also excluded where specified in .gitignore.
| Validation objective | Current status |
|---|---|
| AMR genotype-to-phenotype | Reference dataset and supporting validation resources prepared |
| Lineage assignment | Reference dataset and supporting validation resources prepared |
| Mixed-infection detection | Reference dataset and supporting validation resources prepared |
| Contamination detection | Reference dataset and supporting validation resources prepared |
Additional validation objectives, including speciation, SNP and indel detection, will be incorporated as the framework develops. Current objectives are also continously refined.
To work with the validation resources:
- Clone the repository.
- Review the README and metadata for the relevant validation objective.
- Identify the required source data and software dependencies.
- Obtain sequencing data using the supplied accessions and manifests.
- Run the relevant dataset preparation or workflow evaluation procedures.
- Compare observed results with the reference data.
- Document performance results, limitations, and any deviations from the defined methodology.
The scripts and datasets are intended to be used with their corresponding documentation. Users should verify dependencies, file paths, and required inputs before running individual scripts.
The validation resources are derived from publicly available research datasets.
Key sources include:
- tbtAMR: MTB antimicrobial resistance validation data.
- Fastlin: MTB lineage assignment and classification resources.
- Sobkowiak et al. (2018): Mixed MTB infection reference data.
- Heupink et al. (2021): MTB contamination simulation datasets.
Full source information, accession references, and methodological details should be obtained from the relevant objective-specific documentation.
Users should cite the original publications and datasets when reusing these resources.
The advISO project develops practical resources to support the implementation of quality-assured bioinformatics workflows and the adoption of ISO-aligned validation practices.
Project information:
https://www.cardiff.ac.uk/adviso-bioinformatics-accreditation
If you use resources from this repository, please acknowledge the advISO project and cite the original publications and datasets relevant to the validation objectives used.
This repository is under active development. Dataset preparation, pilot analyses, workflow evaluation, and supporting documentation may be updated as validation activities progress.