The next phase of BioFSharp.INSDC is a SQLite-backed store for the INSDC entity hierarchy (BioProject > Study > BioSample > Experiment > Run), intended to feed a crawler for these metadata records.
A prior iteration left behind a SQL dump at insdc_schema.sql (~80 tables) — that is the only file currently in the new BioFSharp.INSDC.SQLite/ project. Meanwhile, the F# side has moved on: types are now auto-generated from XSDs via dotnet xscgen into BioFSharp.FileFormats.INSDC/Generated/, with thin F# I/O wrappers in BioFSharp.IO.INSDC/ for XML round-tripping.
Two things need to happen:
- Reconcile the schema with the freshly generated types — fix obvious bugs in the dump, and decide table-by-table whether to keep, rework, or drop.
- Build an F# layer that deconstructs an INSDC record into the many flat SQLite rows that represent it, and reconstructs it on the way back out.
| F# type (Object-derived) | SQLite core table | Match notes |
|---|---|---|
BioProject (BioProject.cs:22) |
project (insdc_schema.sql:21) |
Core columns line up (accession, alias, broker_name, center_name, title, name, description, first_public). Snake_case consistent here. |
Study (Study.cs:29) |
study + study_descriptor + study_type |
F# Study is thin; almost everything sits under Descriptor. SQLite splits the descriptor into its own table — sane. study_type is two columns (existing / new) and could be inlined. |
BioSample (BioSample.cs:29) |
sample + sample_name |
OK shape. Column casing inconsistent (brokerName/centerName vs broker_name/center_name elsewhere). sample_name.displayName Text has a typo (Text not TEXT). |
Experiment (Experiment.cs:30) |
experiment + ~24 detail tables |
Largest entity. Design/Library/LibraryDescriptor/SpotDescriptor/Platform/Processing map roughly. Platform discriminated union is not captured — only experiment_platform.instrumentModel exists; per-platform parameters are lost. |
Run (Run.cs:26) |
run + ~19 detail tables |
Same shape as experiment. Same platform-DU gap. Schema has two syntax errors here (see "Bugs" below). |
| F# type | SQLite shape | Match notes |
|---|---|---|
Object base (Identifiers, Alias, CenterName, BrokerName, Accession) — Object.cs:34 |
Each entity's core table repeats the columns | Fine. No need to share a parent table. |
RefObject base (Refname, Refcenter, Accession, Identifiers) — RefObject.cs:34 |
experiment_studyRef, experiment_sampleDescriptor, run_experimentRef each get their own table plus a parallel set of nested ID tables |
Big duplication. Each reference re-creates the entire 5-table ID family (*RefPrimaryID, *RefSecondaryID, *RefExternalID, *RefSubmitterID, *RefUUID). |
Identifier composite (Identifier.cs:25) — PrimaryId, SecondaryId[], ExternalId[], SubmitterId, Uuid[] |
5 separate tables per owner | Same shape repeated for every owner and reference. Inconsistent column orderings between siblings. |
Attribute (Tag, Value, Units) — Attribute.cs:25 |
{entity}_attributes (tag, value, units) × 5 entities |
Clean and consistent. Keep. |
Link (DU: UrlLink | XrefLink | EntrezLink) — Link.cs:26 |
{entity}_links (db, id, label, url, item, itemElementName) × 5 |
item + itemElementName are the DU discriminator. Works but opaque. |
Platform (DU over 11 sequencing techs, each with own config) — Platform.cs:27 |
experiment_platform.instrumentModel, run_platform.platformName — single TEXT field |
Gap: only the instrument string is stored. Platform-specific parameters are dropped. |
BioSampleName (TaxonId, ScientificName, CommonName, DisplayName) |
sample_name |
Matches, modulo the Text typo. |
Library / LibraryDescriptor / SpotDescriptor / Processing |
experiment_design, experiment_spotDescriptor, experiment_processingDirectives, experiment_pipeline, experiment_targetedLoci (and run_* analogues) |
Coverage exists but denormalized; e.g. LibraryStrategy/Source/Selection enums live as free TEXT. |
- F# uses typed reference objects:
Experiment.StudyRef,Run.ExperimentRef,Library.SampleDescriptor. Each carries fullRefname/Refcenter/Accession/Identifiers. - SQLite has two parallel mechanisms that aren't reconciled:
accession_map (project_accession, study_accession, sample_accession, experiment_accession, run_accession)— one row flattens the whole tree.experiment_studyRef,experiment_sampleDescriptor,run_experimentRef(+ their nested ID tables) — per-reference detail.
- Neither side enforces FKs to the target entity.
accession_maphas no PK.
BioProject.SubmissionProjectandBioProject.UmbrellaProject(nested project compositions).- Platform-DU per-platform parameters.
Run.RunProperty(RunRunType) and fullRun.Processing(only partial coverage today).LibraryDescriptor.LibraryLayoutsingle vs paired-end distinction.
project_collaborators— F#BioProject.CollaboratorsisCollection<string>directly on the project; side-table is correct but column is one untyped TEXT.run_inner (assembly, sequence)— unclear purpose; possibly leftover.accession_map— convenience denormalization; redundant once proper FKs exist.
Grouped by severity. The dump has ~80 tables; problems cluster into five categories.
- A1 — Missing column type. insdc_schema.sql:34 —
run_dataBlockType.member_namehas no type declaration. - A2 — Invalid SQL type. insdc_schema.sql:43 —
run_experimentRef.submitterIDNamespace LaTeXString—LaTeXStringisn't a SQLite type; looks like a botched search-and-replace. - A3 — Stray column name. insdc_schema.sql:76 —
accessionStudydoesn't fit surrounding naming.
- B1 — Mixed casing on duplicated columns.
project.broker_name/sample.brokerName;project.center_name/sample.centerName/experiment.centerName. - B2 — Field-order inconsistencies.
sample_secondaryIDis(label, value)whileproject_secondaryIDis(value, label). - B3 — Same concept, different names.
run_secIDvsexperiment_secondaryIDvssample_secondaryID.run_experimentRefExIDvsexperiment_studyRefExternalID. - B4 — Inconsistent FK actions. Most tables
ON UPDATE CASCADE;study_linksandexperiment_designuseON UPDATE NO ACTION; many specify noON DELETE. Reads like accidents. - B5 —
TextvsTEXTat insdc_schema.sql:16. Harmless to SQLite but a smell.
- C1 — Identifier fragmentation. Five sibling tables (
*_primaryID,*_secondaryID,*_externalID,*_submitterID,*_uuid) per entity. F# uses oneIdentifiercomposite. - C2 — Reference fragmentation, squared. Every typed reference re-creates the entire 5-table identifier family (
*RefPrimaryID, etc.) — ~15 extra tables for three references. - C3 — Two competing hierarchy mechanisms.
accession_mapvs per-edge*_studyRef/*_experimentRef. No source of truth. - C4 — No cross-entity FKs. Reference fields are plain TEXT — an experiment can point at a nonexistent study with no complaint.
- C5 —
accession_maphas no primary key. Cardinality undefined. - C6 — Sparse tables that should be columns.
experiment_platform,run_inner,run_processingDirectives,experiment_processingDirectivesare 1–2-column tables only because they're optional.
- D1 — Platform variants. F#
Platformis a DU over 11 techs each with config; schema stores onlyinstrumentModel/platformName. - D2 — Nested project compositions.
BioProject.SubmissionProject,BioProject.UmbrellaProjecthave no SQLite home. - D3 — Library layout single vs paired-end.
- D4 —
Run.RunPropertyand fullRun.Processing.
- E1 — Zero indexes (even on owner-FK columns).
- E2 — No views, no triggers.
- E3 — Detail tables have no PKs — duplicates accepted freely.
- Schema strategy — Hybrid. Keep what works (per-entity core tables, the
*_attributespattern, the descriptor / sample_name / spot_descriptor splits). Rebuild identifiers, references/hierarchy, platform. Fix all A/B bugs along the way. The existing dump becomes reference, not constraint. - F# ↔ SQLite —
Microsoft.Data.Sqlite+ hand-written SQL. No ORM. Connection helpers and parameter binding live inInternal/. - Identifier modeling — one identifier table per owner, kind-discriminated.
- Scope this phase — schema + deconstruction/reconstruction only. No crawler entry points yet.
- Table naming — match F# type names:
bioproject,study,biosample,experiment,run(avoidsBioProjectvsProjectandBioSamplevsSampleambiguity).
Single SQL file at src/BioFSharp.INSDC.SQLite/schema/insdc_schema.sql, rewritten in place. Conventions:
- All identifiers
snake_case. All types uppercase (TEXT,INTEGER). - Every table has an explicit PRIMARY KEY.
- Every FK uses
ON DELETE CASCADE ON UPDATE CASCADEunless documented otherwise. - Indexes on every FK column.
PRAGMA foreign_keys = ONdocumented at top of schema; enforced by the F# connection helper on every open.- Entity table names:
bioproject,study,biosample,experiment,run.
PK on accession for each. Parent FKs live on the entity itself, replacing accession_map:
study.bioproject_accession→bioproject.accession(NULLable)biosamplehas no parent FKexperiment.study_accession→study.accession(NOT NULL)run.experiment_accession→experiment.accession(NOT NULL)
study_descriptor (study_accession PK/FK, study_title, study_type, study_abstract, center_project_name, study_description, ...)— collapses oldstudy_descriptor+study_type.biosample_name (biosample_accession PK/FK, taxon_id, scientific_name, common_name, display_name)— fixes theTexttypo.experiment_design (experiment_accession PK/FK, design_description, library_strategy, library_source, library_selection, library_layout_kind, library_layout_nominal_length, library_layout_nominal_sdev)—library_layout_kindcaptures D3.experiment_spot_descriptor/run_spot_descriptor— PK on(owner_accession, read_index).experiment_targeted_loci— PK on(experiment_accession, locus_name).run_data_block (run_accession PK/FK, files, member_name TEXT, ...)— fixes A1.
One per owner. Replaces ~25 entity tables + ~15 reference-nested ID tables.
CREATE TABLE bioproject_identifiers (
bioproject_accession TEXT NOT NULL REFERENCES bioproject(accession) ON DELETE CASCADE ON UPDATE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('PRIMARY', 'SECONDARY', 'EXTERNAL', 'SUBMITTER', 'UUID')),
ordinal INTEGER NOT NULL DEFAULT 0,
value TEXT NOT NULL,
label TEXT,
namespace TEXT,
PRIMARY KEY (bioproject_accession, kind, ordinal)
);
CREATE INDEX ix_bioproject_identifiers_kind ON bioproject_identifiers (bioproject_accession, kind);Analogous: study_identifiers, biosample_identifiers, experiment_identifiers, run_identifiers.
Each typed reference gets one row in a reference table plus rows in a sibling identifier table:
CREATE TABLE experiment_study_ref (
experiment_accession TEXT PRIMARY KEY REFERENCES experiment(accession) ON DELETE CASCADE ON UPDATE CASCADE,
accession TEXT REFERENCES study(accession) ON UPDATE CASCADE, -- soft FK; NULL ok when only refname given
refname TEXT,
refcenter TEXT
);
CREATE TABLE experiment_study_ref_identifiers (
experiment_accession TEXT NOT NULL REFERENCES experiment_study_ref(experiment_accession) ON DELETE CASCADE ON UPDATE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('PRIMARY', 'SECONDARY', 'EXTERNAL', 'SUBMITTER', 'UUID')),
ordinal INTEGER NOT NULL DEFAULT 0,
value TEXT NOT NULL,
label TEXT,
namespace TEXT,
PRIMARY KEY (experiment_accession, kind, ordinal)
);Same shape for experiment_sample_descriptor / experiment_sample_descriptor_identifiers and run_experiment_ref / run_experiment_ref_identifiers.
{entity}_attributes (entity_accession FK, ordinal INTEGER, tag TEXT NOT NULL, value TEXT, units TEXT, PRIMARY KEY(entity_accession, ordinal))— five tables (bioproject_attributes, etc.).{entity}_links (entity_accession FK, ordinal INTEGER, link_kind TEXT CHECK ('URL'|'XREF'|'ENTREZ'), label TEXT, url TEXT, db TEXT, id TEXT, query TEXT, PRIMARY KEY(entity_accession, ordinal))— replaces opaqueitem/itemElementNamewith explicitlink_kind.
CREATE TABLE experiment_platform (
experiment_accession TEXT PRIMARY KEY REFERENCES experiment(accession) ON DELETE CASCADE ON UPDATE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('LS454','ILLUMINA','HELICOS','ABI_SOLID','COMPLETE_GENOMICS','BGISEQ','OXFORD_NANOPORE','PACBIO_SMRT','CAPILLARY','DNBSEQ','AVITI')),
instrument_model TEXT
);
CREATE TABLE experiment_platform_params (
experiment_accession TEXT NOT NULL REFERENCES experiment_platform(experiment_accession) ON DELETE CASCADE ON UPDATE CASCADE,
key TEXT NOT NULL,
value TEXT,
PRIMARY KEY (experiment_accession, key)
);Same shape for run_platform / run_platform_params. Per-platform fields go in the key/value bag — avoids 11 sub-tables; deconstruction layer serializes per-DU-case fields into params.
accession_map— replaced by parent FKs.run_inner— unclear purpose, no F# counterpart.- Fragmented
*_primaryID/*_secondaryID/*_externalID/*_submitterID/*_uuidtables (per entity and per reference) — replaced by unified*_identifierstables.
BioProject.SubmissionProjectandBioProject.UmbrellaProject(D2).- Full
Run.Processing/Run.RunProperty(D4) — minimal coverage retained.
New project src/BioFSharp.INSDC.SQLite/BioFSharp.INSDC.SQLite.fsproj, netstandard2.0, mirroring BioFSharp.IO.INSDC.fsproj.
src/BioFSharp.INSDC.SQLite/
├── BioFSharp.INSDC.SQLite.fsproj
├── schema/
│ └── insdc_schema.sql (rewritten, embedded resource)
├── Internal/
│ ├── Sql.fs connection + transaction helpers; PRAGMA foreign_keys=ON
│ ├── Schema.fs loads embedded schema, applies to a fresh DB
│ ├── Identifiers.fs writeIdentifiers / readIdentifiers — generic over owner-table name
│ ├── Attributes.fs same pattern for Attribute collections
│ ├── Links.fs same pattern, with link_kind DU handling
│ └── References.fs writeRefObject / readRefObject — covers StudyRef, SampleDescriptor, ExperimentRef
├── Schema.fs public: Schema.init (conn: SqliteConnection) → unit
├── BioProject.fs public: insert, tryGet, delete, listAccessions
├── Study.fs
├── BioSample.fs
├── Experiment.fs
└── Run.fs
module BioProject =
/// Deconstruct a BioProject into rows across bioproject, bioproject_identifiers,
/// bioproject_links, bioproject_attributes, bioproject_collaborators
/// and insert in a single transaction. Throws on FK or constraint violations.
val insert : SqliteConnection -> BioProject -> unit
/// Reconstruct a BioProject by joining all owner tables for the given accession.
val tryGet : SqliteConnection -> accession:string -> BioProject option
val delete : SqliteConnection -> accession:string -> unit
/// List all accessions present in the bioproject table (for crawler bookkeeping).
val listAccessions : SqliteConnection -> string seqPer-entity transaction:
- INSERT into
bioproject. - For each
(kind, items)inproject.Identifiers→ INSERT intobioproject_identifierswithkindandordinal. - For each item in
project.ProjectAttributes→ INSERT intobioproject_attributes. - For each item in
project.ProjectLinks→ resolve DU case → INSERT intobioproject_linkswithlink_kindset. - Experiment/Run additionally call
References.writeRefObjectfor their outgoing references.
Helper signature in Internal/Identifiers.fs:
type IdentifierOwner = { Table: string; AccessionColumn: string; Accession: string }
val writeIdentifiers : SqliteConnection -> IdentifierOwner -> Identifier -> unit
val readIdentifiers : SqliteConnection -> IdentifierOwner -> IdentifierThe same helper serves all five entities and all three reference contexts — just with a different Table. Payoff of unifying the identifier schema.
tryGet runs one query per owner-table family, then assembles:
- Single-row SELECT on
bioproject. - SELECT all
bioproject_identifiers; group bykindto rebuildIdentifiercomposite. - SELECT all
bioproject_attributes,bioproject_links,bioproject_collaborators. - Experiment/Run: SELECT the reference row + its identifier rows, build
RefObjectvalues, wire into the parent. - Construct the C# type via parameterless constructor; populate Collections via
.Add(...).
O(entity_count) queries per reconstruction; with FK indexes this stays cheap.
- Rewrite src/BioFSharp.INSDC.SQLite/schema/insdc_schema.sql per the "Reference — proposed schema shape" section above
- Top of file:
PRAGMA foreign_keys = ON;and a banner comment listing conventions - Core entity tables:
bioproject,study(withbioproject_accessionFK),biosample,experiment(withstudy_accessionFK),run(withexperiment_accessionFK) - Nested composite tables:
study_descriptor,biosample_name,experiment_design,experiment_spot_descriptor,run_spot_descriptor,experiment_targeted_loci,run_data_block - Unified identifier tables × 5:
bioproject_identifiers,study_identifiers,biosample_identifiers,experiment_identifiers,run_identifiers - Reference tables × 3:
experiment_study_ref(+_identifiers),experiment_sample_descriptor(+_identifiers),run_experiment_ref(+_identifiers) - Attribute tables × 5 (
{entity}_attributes) - Link tables × 5 (
{entity}_links) withlink_kindCHECK constraint - Platform tables × 2:
experiment_platform(+_params),run_platform(+_params) -
bioproject_collaboratorstable forCollection<string>field - Indexes on every FK column
- Smoke test:
sqlite3 :memory: < src/BioFSharp.INSDC.SQLite/schema/insdc_schema.sqlexits clean - FK enforcement verified: inserting an experiment with a missing
study_accessionis rejected withFOREIGN KEY constraint failed
Result: 42 tables (down from 80 in the dump), 11 indexes. Identifier fragmentation collapsed (5 entity tables × 5 ID kinds → 5 unified identifier tables; 15 reference-nested ID tables → 3 reference identifier tables). accession_map and run_inner dropped; sparse processing-directive tables inlined into experiment/run.
- Create src/BioFSharp.INSDC.SQLite/BioFSharp.INSDC.SQLite.fsproj —
netstandard2.0, matches existingBioFSharp.IO.INSDC.fsprojstyle - Add
PackageReference Include="Microsoft.Data.Sqlite"(8.0.10 — last line that still targets netstandard2.0; 9.x dropped it) - Add
ProjectReferencetoBioFSharp.FileFormats.INSDC.csprojandBioFSharp.IO.INSDC.fsproj - Add
<EmbeddedResource Include="schema/insdc_schema.sql" /> - Add
<PackageProjectUrl>,<Description>,<PackageTags>etc. matching repo conventions - Register the new project in
BioFSharp.INSDC.slnx -
dotnet build src/BioFSharp.INSDC.SQLitesucceeds (empty project compiles)
Result: Empty F# library compiles; full solution build is clean. Pre-existing NU1903 warnings on FAKE build tooling are unchanged.
-
Internal/Sql.fs—openConnection : string -> SqliteConnection(issuesPRAGMA foreign_keys = ONafter open);withTransactionwrapper; small param-binding helpers (execNonQuery,queryAll,tryQueryOne,readStringOrNull) -
Internal/Schema.fs— loadsinsdc_schema.sqlfrom embedded resources and executes against a connection -
Internal/Identifiers.fs—IdentifierOwnerrecord +write/read(generic over owner table, accession column, accession value) -
Internal/Attributes.fs—write/read(same shape) -
Internal/Links.fs—write/readwithlink_kindresolution against theLinkDU (UrlLink / XrefLink / EntrezLink) -
Internal/References.fs—write/readfor anyRefObjectsubtype, including its sibling identifier table (new()-constrained generic so the right concrete subclass —ExperimentStudyRef,BioSampleDescriptor,RunExperimentRef— is returned)
Result: Clean build, zero warnings. Smoke test through dotnet fsi: schema loads (42 tables), all 5 identifier kinds round-trip (PRIMARY/SECONDARY/EXTERNAL/SUBMITTER/UUID) with label/namespace preserved, attributes round-trip with NULL value/units preserved, all 3 Link DU cases (URL/XREF/ENTREZ with nullable int64 Id) round-trip cleanly.
-
Schema.fs— publicSchema.init : SqliteConnection -> unit -
BioProject.fs—insert,tryGet,delete,listAccessions; coversCollaboratorsandRelatedProjects(PARENT/CHILD/PEER DU) -
Study.fs— same surface; reconstructs the fullStudyDescriptor(existing-type enum, NullableProjectId, RelatedStudies with XRef). Takes parentbioProjectAccession(NULL ok) since the F# type doesn't model the parent FK. -
BioSample.fs— same surface; handlesbiosample_name -
Experiment.fs— same surface; usesReferencesforStudyRefandLibrary.SampleDescriptor; serializesPlatformDU via the new sharedInternal/Platforms.fshelper; round-trips library strategy/source/selection enums, paired-end nominal length/sdev, targeted loci, spot descriptor read specs -
Run.fs— same surface; usesReferencesforExperimentRef; reusesPlatformshelper; persistsRunDate(ISO-8601),RunCenter,DataBlock.MemberName, spot descriptor
Extras landed:
- Schema's platform
CHECK (kind IN (...))constraint expanded from 11 to all 18 DU cases (the F#Platformtype has 18, the original constraint would have rejected 7). Internal/Platforms.fsextracted as a sibling toIdentifiers/Attributes/Links/References— both Experiment and Run share it.
Result: fsi round-trip test exercises every entity (and through them every helper). Outputs match inputs for: identifiers (all 5 kinds), attributes, URL/XREF/ENTREZ links, collaborators, related-projects (PARENT case), FirstPublic date, StudyDescriptor with enum + nullable long + RelatedStudies, library strategy/source/selection enums, paired layout, Platform ILLUMINA → IlluminaNovaSeq6000, RunDate timestamp, ExperimentRef. FK enforcement verified (bogus parent rejected). listAccessions works for all five entities. Cascade delete behaves correctly with bioproject → study being ON DELETE SET NULL (so dropping a BioProject leaves orphan studies, by design).
- Add
ProjectReferenceto the new SQLite project in tests/BioFSharp.INSDC.Tests/BioFSharp.INSDC.Tests.fsproj - Schema init smoke test — open in-memory connection, call
Schema.init, assert expected table set viasqlite_master - Round-trip BioProject — fixture XML →
BioFSharp.IO.INSDC.BioProject.read→BioProject.insert→BioProject.tryGet→ structural compare - Round-trip Study
- Round-trip BioSample
- Round-trip Experiment (exercises StudyRef, SampleDescriptor, Platform)
- Round-trip Run (exercises ExperimentRef)
- FK enforcement test — insert Experiment with bogus
study_accession; assert SQLite raises FK violation - Identifier round-trip — populate all five kinds (Primary, Secondary, External, Submitter, UUID); verify bit-for-bit
-
dotnet buildof the solution succeeds -
./build.sh runtests(ordotnet test tests/BioFSharp.INSDC.Tests) — all tests green - Append a
RELEASE_NOTES.mdentry for the newBioFSharp.INSDC.SQLitepackage