Homoset & MIXLE curation multisource dataset comparison#8
Open
thegodone wants to merge 18 commits into
Open
Conversation
Harmonize the heterogeneous GuthrieSolv dump (53,895 records, 172 unit strings) into 12,027 Ben-Naim hydration free energies over 2,473 molecules, temperature-corrected and keyed by InChIKey, validated per-route against FreeSolv (median MAE 0.40 kcal/mol, R=0.95). Compare two multi-source reconciliation methods on the redundant per-molecule observations, scored on the 556-molecule FreeSolv overlap: - Homoset: proportional-similarity (PS) gate with median source-offset alignment + chi-square critical threshold; noise scale L swept (adaptive vs fixed physical 0.6) and outlier threshold swept. - mixle: hierarchical Normal(Normal(mu0,tau),sigma_source) partial pooling with per-source bias (EM), plus a robust Student-t variant. Homoset (fixed L) is the best transparent curator (MAE 0.283); mixle is the best reconciler (RMSE 0.866, bias +0.07) and uniquely rescues molecules with no clean reference source. Includes harmonizer, comparison, validation scripts, an arXiv-style preprint (PAPER.tex), a technical report, a visual HTML report, and output tables.
Add the mixle GitHub reference (gmboquet/mixle) to the preprint bibliography and methods, note the mixle model is a numpy reproduction of its Normal(Normal(mu0,tau),sigma) partial-pooling dialect, and reference the full per-unit conversion code (harmonize_guthrie.py) explicitly in the paper, report, and README.
…ndardization The Homoset (homogeneous set) / proportional-similarity procedure was developed in the 1970s-80s to standardize scattered literature values of human olfactory detection thresholds (Devos, Patte, Rouault, Laffort & Van Gemert, Standardized Human Olfactory Thresholds, IRL Press / Oxford University Press, 1990), and is applied here unchanged to hydration free energies. Add the citation to the preprint abstract, methods and bibliography, and to the report and README.
…ion L
Homoset is governed by exactly two parameters -- the noise level (max tolerated
RMS/L for a homogeneous set) and the dimension value L -- not an alpha/chi2
threshold plus a separate outlier cutoff. Reimplement the gate as
'admit iff RMS/L <= noise_level', sweep the 2-D grid (L in {adaptive, 0.6} x
noise in {0.25, 0.50}), and anchor the per-molecule conflict flag to the
physical dimension L=0.6.
The grid exposes the mechanism: per-source noise KWG 0.14 / PAIR 0.18 /
KGW 0.34 at adaptive L; noise=0.25 admits KWG+PAIR, 0.50 admits all; L=0.6
rejects every converted source (noise 1.4-5.0). The source Homoset most
resists (KGW) is the one mixle assigns the largest bias (+1.18) -- independent
agreement on the worst source.
Update paper, report, README, and HTML with the two-parameter framing and
refreshed numbers (conflict set now the 278 molecules failing the homogeneity
test).
The physics switchboard leaves ~12k rows across the tail of the 172 unit
strings unconverted. calibrate_units_loop.py recovers many without hand-coding
each one, reusing the Homoset noise gate as the acceptance test:
anchors = molecules already given dG_hyd by the physics routes; for each
unconverted (unit,process) fit dG ~ a*f(value)+b (f in {log10,-log10,ln,id},
robust trimmed LS) against shared anchors; ADMIT iff |R|>=0.85 AND RMS/L<=eta;
admitted units convert all their rows -> new anchors -> next round reaches
further; loop until dry.
Admits 22 units in 2 rounds (12 high-confidence Henry/free-energy), adding
5,244 observations and 268 previously-unreachable molecules (2,473 -> 2,741).
Self-validating: re-derives the air/water Henry sign (log(M/M) slope -0.99
gas/water vs +1.00 water/gas) and cracks opaque strings (MPv/(RTCw) R0.99,
'logK=y/x at 1 atm' R0.99). Improves accuracy: FreeSolv median-MAE 0.399->0.328,
R 0.948->0.961 (FreeSolv never an anchor -> held-out). Written up in PAPER.tex,
REPORT.md, README, and the HTML report.
…nealing The single-pass loop went dry in 2 rounds. Wrap it in an outer fixed-point co-training loop (calibrate_units_iterative.py): after each admission pass, reconcile all obs to a per-molecule consensus, feed that back as cleaner/larger anchors, and anneal the gate strict->loose ((eta,R)=(0.25,0.90)->(0.65,0.78)). Continue until a pass at the loosest gate admits nothing. Converges in 6 outer steps: 26 units (vs 22 single-pass), +5,446 obs, +290 molecules (2,473 -> 2,763). The strict first step does the accuracy lifting (held-out FreeSolv MAE 0.399 -> 0.321, R 0.948 -> 0.962); later annealed steps add coverage at flat ~0.32 MAE, then self-terminate -- a coverage/accuracy Pareto. Adds trajectory figure; updates paper/report/README/HTML.
…math
Answering 'why can't we convert more units -- it's just math': the Henry and
free-energy units ARE molecule-independent conversions of dG_hyd, and two bugs
were hiding thousands of them:
- a one-char regex fault (^log10? needs 'log1') silently dropped every
log(M)/log(mol/L)/log(atm)/log(mmHg) spelling from the switchboard;
- the dimensionless Henry ratio direction is process-dependent (KWG stores the
air/water constant -> invert; KGW stores water/gas directly -> keep), the
labels being effectively swapped in the source.
Also add mole-fraction solubility and the kN/m^2 spelling.
Fixing these (pure deterministic math, no learning) grows the physics route to
16,357 obs / 2,675 molecules and IMPROVES FreeSolv agreement: raw-median MAE
0.40 -> 0.30, R 0.948 -> 0.965. It also dissolves the earlier 'both methods
distrust KGW' story -- that was the direction bug (KGW mixle bias +1.18 -> +0.21);
on clean data PAIR is the least-trusted source (+0.91).
The remaining ~35k unconverted rows are vapour pressure + solubility, which are
DIFFERENT observables, not units of dG_hyd; they convert only by pairing
(1,722 molecules). What's left -- 750 VP-only + 1,666 solubility-only molecules --
is a single-observable DATA gap no math can close (pair_vp_solubility.py reports
this). Refresh comparison + iterative loop (19 units) + paper/report/README/HTML
on the corrected base; mixle-robust is now best overall (MAE 0.249).
MobleyLab#3 Rule system: unit_rules.py is a declarative dimensional-analysis engine -- de-wrap log/ln/-log, tokenise into <prefix><base>^power factors, reduce to a net dimension signature (pressure/amount/volume/mass/mole-fraction/energy), classify into a physical quantity, convert. Adding a unit is one table row, not a regex. harmonize_rules.py runs it over GuthrieSolv. MobleyLab#2 More curation: converts 65 (unit,process) combos vs 57 for the switchboard, resolves mass-fraction-vs-Henry-ratio by process (AQSOL fraction vs KWG/KGW ratio), and IMPROVES FreeSolv agreement to MAE 0.275 / R 0.972 (vs 0.30 / 0.965). The ~10 remaining exotic named strings (MPv/(RTCw), logK=y/x at 1 atm) are exactly what the active-learning loop calibrates -- deterministic rules + data-driven fallback. MobleyLab#1 Figures: embed_figures.py inlines the calibration-trajectory and Homoset-vs-mixle scatter as base64 into report.html (CSP blocks external images); PAPER.tex now includes both via graphicx. Update paper/report/README/HTML accordingly.
Mine the Excel columns the flat CSV pipeline dropped (guthrie_metadata.py): - final: Guthrie's own converted dG_hyd (737 rows). Three-way vs FreeSolv on the 176 shared molecules -- OUR rule-engine+median beats his own values (MAE 0.153 R 0.988 vs his 0.267 / 0.943): pooling redundant measurements out- curates a single hand-converted number. - error1: his per-measurement trust flag (1.93 kcal 'not trustworthy', kJ twin 8.08). Using it as a HARD FILTER hurts (drops 2,884 obs, MAE 0.195 -> 0.240); redundancy beats the trust signal, so it belongs as a soft mixle weight. - pH conditions in comments (92 ionizable-drug rows) to tag AQSOL. Documented in PAPER.tex (sec:meta), REPORT.md (1d), README.
The HTML report was missing two things present in REPORT.md: a dedicated 'Cleaning the units: a rule engine, not a regex' section (declarative dimensional-analysis engine, 65 vs 57 unit combos, MAE 0.275/R 0.972, the four parsing bugs) and the 'Guthrie's own curation -- and we beat it' three-way comparison (his final ΔG 0.267 vs ours 0.153 vs FreeSolv; trust-flag-as-filter hurts). Section flow + numbers now match REPORT.md.
plot_unit_coverage.py: disposition of all 53,895 measurements -> 38,539 (71%) convert to dG_hyd (free-energy 3,160 / Henry 11,582 / VP-paired 12,697 / solubility-paired 11,100); only 186 rows unparsed. The unconverted are single- observable VP-only (7,881) / solubility-only (4,654) -- physically un-pairable -- plus non-hydration (2,635). Right panel: rule engine 65 (unit,process) combos vs switchboard 57. Embedded in report.html.
Member
|
Would need some more detail on this; provenance/explanation? Is this just someone running a bot to try and standardize/extract data, or is there expertise and human intelligence behind it? How are the results being checked/tested? How can we know this is correct? |
…_VP) Extend the FreeSolv-for-dG validation to the two Henry inputs: - AQSOL solubility vs AqSolDB (mcsorkun, 9,982 mols): 2,726 overlap, MAE 0.153 log, R 0.980, zero bias -- GuthrieSolv's solubility extraction matches the dedicated reference DB. - VP vs unified_VP: 1,582 overlap, MAE 1.056 log, R 0.817, bias +0.36, with an artifact band near ~1 atm (boiling-point/1-atm entries). VP is the noisy input. So for Henry (=VP-WS): WS is solid, VP is the noise source. compare_aqsoldb_vp.py + scatter plot; REPORT.md 1e.
…37 BP) meta37 BP proves the VP artifact band: the 1,506 VP rows pinned near 1 atm were measured within a median 12 K of the boiling point (vs 85 K for other rows) -- i.e. VP-at-BP entries taken uncorrected as if at 25 C. Fix: restricting to VP measured near 25 C drops MAE vs unified_VP 1.03->0.67, R 0.83->0.90; meta37 deltaHvap_kJmol enables a Clausius-Clapeyron correction instead of filtering. REPORT.md 1e updated.
log10 P(298) = log10 P(T) - (dHvap/R)/ln10 * (1/298 - 1/T), dHvap from meta37. On 759 correctable molecules, VP vs unified_VP jumps MAE 0.687->0.262, R 0.848->0.931 (62% error cut) -- VP from noisiest input to near solubility grade. dHvap covers 45.5% of VP rows. vp_temperature_correction.py; REPORT.md 1e.
harmonize_guthrie.py VP route now T-corrects each vapour pressure to 25 C via a cached meta37 dHvap-by-InChIKey table (meta37_dhvap_by_inchikey.csv, 3,406 mols) before VP x solubility pairing. Fixes the boiling-point/1-atm VP artifact at source. Effect: vp_sol_pair per-obs residual median -> 0.02, overall harmonized- median FreeSolv MAE 0.299 -> 0.290 (R 0.965), no coverage loss.
… + CC fix) Append the physchem-input validation to the HTML report (ODT excluded): AQSOL solubility vs AqSolDB (R 0.98), VP vs unified_VP with the boiling-point artifact (confirmed via meta37 BP), the Clausius-Clapeyron fix (VP MAE 0.69->0.26; pairing 1.08->0.33; harmonizer overall 0.299->0.290), and a note that meta37 is a sparse master table (logWS 10.6% / logVP 7.5% / logHenrycc 4.4% / dHvap 3.6% / BP 12.4%). Embed guthrie_vs_aqsoldb_vp.png. Verified zero ODT text in the report.
…racy meta37_dghyd.py generates dG_hyd from meta37 logHenrycc (4,186 mols; calib slope -1.38 ~ RT.ln10, physics recovered) + VP x WS pairing (2,721) = 5,020 union molecules, ~2x GuthrieSolv's 2,675. On 545 molecules with FreeSolv truth: GuthrieSolv mixle 0.186 < Homoset 0.205 < meta37 Henry 0.373 -- curated reconciliation ~2x more accurate than model-derived dG, mixle edges Homoset. Coverage vs accuracy trade-off; meta37 = a third source to fold in. Report section 1f + figure (ODT excluded, verified).
Since Henry <-> dG_hyd (x1.364), GuthrieSolv's reconciled dG extends meta37's sparse logHenrycc: converting mixle dG -> logHenrycc adds 733 molecules meta37 lacks (union 4,919, +18%), and supplies the more accurate value for the 2,675 it covers (dG validated 0.19 vs meta37 0.37). Best-of-both extended_logHenrycc.csv = GuthrieSolv-reconciled where literature exists (2,675) + meta37-model elsewhere (2,244). The two datasets mutually extend across dG_hyd<->Henry<->VP/WS. REPORT 1f.
Author
|
There are 2 documents to read report.md and report.html and my comparison to Freesolv. I also include my own Database (of Physical properties) for comparison. So there is a human in the loop. And the mixle is really improving the statistical Homoset results without any questions from https://github.com/gmboquet/mixle. We can discuss the pipeline and docs (guillaume.godin at gmail.com). |
Author
|
I do not have access to the Minesota database from https://comp.chem.umn.edu/mnsol/ unfortunately at work, so I cannot make a comparison on more molecules. That would be great to extend to it, for this project I could use it I think. |
Member
|
Will take me a bit to get back to this (possibly quite a while) as I'm dealing with some grant proposals and upcoming travel. Please ping me again if you need a response/review before I get to it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Goal is to review the Large GuthrieSolv dataset after proper unit conversion
Two approaches are review and compare the Homoset from the 1970/1980 combining Homoset and rejecting Heteroset and the very recent MIXLE leveraging bayesian approaches without rejection.