diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5fa993a..a5fd0d0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -114,7 +114,7 @@ script for a new application appears, not for the first. The test is what the script *serves*, not what it mentions. `tools/universal_evaluator.py` names KEGG and DOCCI but evaluates every -benchmark, so it is general. `applications/bioreason/submit_bioreason_sft.sh` +benchmark, so it is general. `applications/bioreason/scripts/training/submit_bioreason_sft.sh` runs one training job for one application, so it is not. Note `tools/` and `scripts/` are not cleanly separated from each other — diff --git a/applications/bioreason/bioreason_demo.ipynb b/applications/bioreason/bioreason_demo.ipynb new file mode 100644 index 0000000..62b56ca --- /dev/null +++ b/applications/bioreason/bioreason_demo.ipynb @@ -0,0 +1,3127 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "arch-diagram", + "metadata": {}, + "source": [ + "# PRISM-BioReason Model Architecture\n", + "\n", + "![PRISM-BioReason model architecture](flow-diag.png)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-000", + "metadata": {}, + "source": [ + "# BioReason on Aurora: training a DNA + language model with PRISM\n", + "\n", + "This notebook trains **BioReason-style DNA reasoning** with PRISM: an LLM reads a reference and a variant DNA sequence, then answers *\"which disease does this variant contribute to?\"*. It runs on **one Aurora interactive node** (12 PVC tiles).\n", + "\n", + "Key stages of training and respective files:\n", + "| Stage | Config (`src/conf/training/`) | Trains | Frozen | Script |\n", + "|---|---|---|---|---|\n", + "| 1. Projector alignment | `bioreason_projector.yaml` | DNA projector | NT encoder, OLMo | `scripts/training/submit_bioreason_projector.sh` |\n", + "| 2. SFT | `bioreason_sft.yaml` | projector + LoRA (r=32) | NT encoder, OLMo base | `scripts/training/submit_bioreason_sft.sh` |\n", + "| 3. GRPO | `bioreason_grpo.yaml` | LoRA (r=16, on top of merged SFT LoRA) | everything else | `scripts/training/submit_bioreason_grpo_aurora.sh` |\n", + "| 4. Eval | — | — | — | `scripts/eval/submit_eval_kegg_aurora.sh` |\n", + "\n", + "**Sections**\n", + "0. Environment installation (terminal, once)\n", + "1. Settings and environment variables (HF cache)\n", + "2. Prerequisites\n", + "3. Download models and data from Hugging Face\n", + "4. Visualize the data\n", + "5. `datasets_config.json`: where datasets are registered\n", + "6. How the DNA modality is registered\n", + "7. Pointing at the DNA encoder\n", + "8. Special tokens\n", + "9. Model build: inspect, then modify the projector and LoRA\n", + "10. One batch end to end (tokenization view and forward pass)\n", + "11. Local smoke run on this node\n", + "12. Submitting all four stages with `qsub`\n", + "13. Monitoring and results\n", + "14. Troubleshooting\n", + "\n", + "> **Before you start:** follow section 0 to build the environment and connect this notebook to the **PRISM BioReason (Aurora)** kernel." + ] + }, + { + "cell_type": "markdown", + "id": "cell-env-install", + "metadata": {}, + "source": [ + "## Setup for a new Aurora user (do this once, outside the notebook)\n", + "\n", + "Two independent parts. **Part A** builds the PRISM environment; it has nothing to do with Jupyter, and the training scripts use it directly. **Part B** adds Jupyter on top of it for this notebook. Skip B if you only submit jobs from the shell.\n", + "\n", + "### Part A. PRISM environment (login node, internet access)\n", + "\n", + "It follows the README's Aurora rules: a `uv`-managed venv on `/flare`, the frameworks module loaded **before** the venv, and PRISM installed with `--no-deps` so pip never replaces the module's XPU PyTorch.\n", + "\n", + "```bash\n", + "# Your PBS allocation (the project you pass to qsub -A). Reused by every step below.\n", + "export PROJECT=\n", + "\n", + "# uv, the package manager the venv builders use (skip if `uv --version` already works)\n", + "curl -LsSf https://astral.sh/uv/install.sh | sh\n", + "source \"$HOME/.local/bin/env\" # puts uv on PATH in this shell\n", + "\n", + "# Code\n", + "cd /flare/$PROJECT/$USER # or any directory with room for a clone\n", + "git clone https://github.com/AI-ModCon/PRISM.git\n", + "cd PRISM && git checkout prism-bioreason-demo\n", + "\n", + "# Environment: loads frameworks/2026.1.0, creates a --system-site-packages venv at $PRISM_VENV,\n", + "# installs the extras the module lacks (torchtune from the torchtune-aurora fork, webdataset, wandb) and PRISM itself (--no-deps).\n", + "export PRISM_VENV=/flare/$PROJECT/$USER/prism-envs/bioreason-fw2026.1\n", + "bash applications/bioreason/helpers/setup_prism_env.sh\n", + "```\n", + "\n", + "> **Why not `tools/build_aurora_env.sh`?** The README's canonical Aurora build pins `frameworks/2025.3.1` and its lockfile. The BioReason scripts and this tutorial run on **`frameworks/2026.1.0`**, the stack everything here was tested on, so they use their own venv. Don't mix the two: a venv built against one frameworks module and loaded under the other fails with import or ABI errors. Note that 2026.1.0 ships transformers 5.x, so only Nucleotide Transformer **v1** checkpoints (e.g. the 2.5B multi-species one used here) load. NT-v2's remote code doesn't.\n", + "\n", + "### Part B. Jupyter for this notebook (login node)\n", + "\n", + "Jupyter is **not** part of PRISM's requirements. This step installs `ipykernel`, `ipywidgets` and `jupyter_server` into the Part A venv and registers the \"PRISM BioReason (Aurora)\" kernel, which always starts with the module and venv loaded.\n", + "\n", + "```bash\n", + "bash applications/bioreason/helpers/setup_jupyter_kernel.sh # same shell: PRISM_VENV is still exported\n", + "```\n", + "\n", + "Check it (still on the login node):\n", + "\n", + "```bash\n", + "ls ~/.local/share/jupyter/kernels/prism-bioreason/kernel.json # the registered kernel\n", + "source $PRISM_VENV/bin/activate && jupyter server --version # Jupyter is installed in the venv\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "fd6c415d", + "metadata": {}, + "source": [ + "### Part B (continued). Start a Jupyter server on a compute node\n", + "\n", + "Models and training need an XPU, so run the server on a compute node. Request one **from the login node**:\n", + "\n", + "```bash\n", + "qsub -I -l select=1 -l walltime=1:00:00 -q debug -A $PROJECT -l filesystems=home:flare\n", + "```\n", + "\n", + "Once the job starts you are on the compute node. Load the environment **in this order** (module first, then the venv) and start the server:\n", + "\n", + "```bash\n", + "export PROJECT= # the notebook reads this to build its paths and to qsub with -A\n", + "export PRISM_VENV=/flare/$PROJECT/$USER/prism-envs/bioreason-fw2026.1\n", + "module use /soft/modulefiles && module load frameworks/2026.1.0 # module first ...\n", + "source $PRISM_VENV/bin/activate # ... then the venv\n", + "\n", + "cd /flare/$PROJECT/$USER/PRISM # the clone from Part A\n", + "jupyter server --no-browser --ip=$(hostname -f) --port 8888 # prints the URL + a new random token\n", + "```\n", + "\n", + "The kernel inherits this shell's environment, so `PROJECT` has to be exported **before** the server starts.\n", + "\n", + "Lost the Jupyter URL? Run `jupyter server list` on the node.\n", + "\n", + "To reuse **one fixed token** across nodes, create it once with\n", + "`python3 -c \"import secrets; print(secrets.token_hex(24))\" > ~/.jupyter_token && chmod 600 ~/.jupyter_token`, then start the server with:\n", + "\n", + "```bash\n", + "test -s ~/.jupyter_token && jupyter server --no-browser --ip=$(hostname -f) --port 8888 \\\n", + " --IdentityProvider.token=\"$(cat ~/.jupyter_token)\"\n", + "```\n", + "\n", + "### Part B (continued). Connect VS Code (Remote-SSH to the login node)\n", + "\n", + "1. Open this notebook, then **Select Kernel → Select Another Kernel… → Existing Jupyter Server**, and paste `http://.hostmgmt.cm.aurora.alcf.anl.gov:8888/?token=`. Use the **full** hostname: it matches `no_proxy=*.alcf.anl.gov`, so the request isn't routed through the ALCF proxy (`502 Bad Gateway`).\n", + "2. Pick **PRISM BioReason (Aurora)**, not \"Python 3\".\n", + "3. If it hangs on \"Connecting to kernel\", set `\"http.proxySupport\": \"off\"` in **Remote Settings** and run **Developer: Reload Window**. `403 Forbidden` means a stale token: each server start makes a new one unless you use a fixed token.\n", + "\n", + "### Optional site paths\n", + "\n", + "The README's `PRISM_*` variables (`cp .env.template .env`, Aurora values in `.env.aurora.example`) are **not** needed here: section 1 sets everything this notebook and its jobs use from `PROJECT`.\n", + "\n", + "Section 2 below verifies the result (12 XPU tiles, venv active, no missing packages).\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell-001", + "metadata": {}, + "source": [ + "## 1. Settings and environment variables\n", + "\n", + "Everything site-specific is set here and nowhere else. Later sections pass the same values to batch jobs through `qsub -v`.\n", + "\n", + "| Variable | Why |\n", + "|---|---|\n", + "| `HF_HOME` | Hugging Face cache for **models and datasets**. It must be on `/flare`: home has a small quota, and an earlier projector job died with `Disk quota exceeded` at its first checkpoint. The submit scripts look for `$HF_HOME/hub/models--…`. |\n", + "| `PRISM_DATA_ROOT` | `datasets_config.json` resolves `${PRISM_DATA_ROOT}/dna_local_shards/kegg` from it. |\n", + "| `PRISM_CKPT_ROOT` | Checkpoints (~13 GB each): every run writes to `$PRISM_CKPT_ROOT//` (`hydra.run.dir`). |\n", + "| `PRISM_OUTPUT_ROOT` | Logs, eval results, and generated scripts. |\n", + "| `HTTP(S)_PROXY` | Aurora compute nodes reach the internet only through the ALCF proxy. |\n", + "| `HF_HUB_OFFLINE` | Off while downloading (section 3), on for training, which is what the batch jobs do. |\n", + "| `MAX_SEQ_LENGTH` | Cap on the **merged** sequence (text + 2 × 1024 DNA slots ≈ 2.8–3.6k tokens for KEGG). The 2048 default makes the merged-length guard reject every DNA example (section 10). |" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cell-002", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "repo ${REPO}\n", + "run tag demo-20261011-2324\n", + "HF_HOME ${HF_HOME}\n", + "PRISM_DATA_ROOT ${PRISM_DATA_ROOT}\n", + "PRISM_CKPT_ROOT ${PRISM_CKPT_ROOT}\n", + "PRISM_OUTPUT_ROOT ${PRISM_OUTPUT_ROOT}\n" + ] + } + ], + "source": [ + "import datetime\n", + "import getpass\n", + "import os\n", + "import sys\n", + "from pathlib import Path\n", + "\n", + "USER = getpass.getuser()\n", + "PROJECT = os.environ.get(\"PROJECT\", \"\") # PBS allocation (-A) and project directory name; export PROJECT=\n", + "assert PROJECT != \"\", \"set the PROJECT environment variable to your PBS allocation before starting Jupyter\"\n", + "\n", + "REPO = Path.cwd().resolve().parents[1] if Path.cwd().name == \"bioreason\" else Path.cwd().resolve()\n", + "# Aurora layout, built from parts; each root can be overridden from the environment.\n", + "LUS = Path(\"/\", \"lus\", \"flare\", \"projects\", PROJECT, USER) # Lustre project space\n", + "FLARE = Path(\"/\", \"flare\", PROJECT, USER) # home-style project space\n", + "PRISM_DATA_ROOT = os.environ.get(\"PRISM_DATA_ROOT\", str(LUS / \"cache\"))\n", + "PRISM_OUTPUT_ROOT = os.environ.get(\"PRISM_OUTPUT_ROOT\", str(LUS / \"output\"))\n", + "PRISM_CKPT_ROOT = os.environ.get(\"PRISM_CKPT_ROOT\", str(LUS / \"modcon\" / \"checkpoints\")) # = the submit scripts' default\n", + "HF_HOME = os.environ.get(\"HF_HOME\", str(FLARE / \"cache\" / \"model_weights\")) # = the submit scripts' HF_HOME\n", + "PRISM_VENV = os.environ.get(\"PRISM_VENV\", str(FLARE / \"prism-envs\" / \"bioreason-fw2026.1\"))\n", + "PROXY = \"http://proxy.alcf.anl.gov:3128\"\n", + "\n", + "RUN_TAG = datetime.datetime.now().strftime(\"demo-%Y%m%d-%H%M\") # one tag for all runs from this session\n", + "SUBMIT = False # flip to True in section 12 to actually qsub\n", + "\n", + "ENV = {\n", + " \"PRISM_DATA_ROOT\": PRISM_DATA_ROOT,\n", + " \"PRISM_OUTPUT_ROOT\": PRISM_OUTPUT_ROOT,\n", + " \"PRISM_CKPT_ROOT\": PRISM_CKPT_ROOT,\n", + " \"PRISM_VENV\": PRISM_VENV,\n", + " \"HF_HOME\": HF_HOME,\n", + " \"HF_HUB_CACHE\": f\"{HF_HOME}/hub\",\n", + " \"HTTP_PROXY\": PROXY, \"HTTPS_PROXY\": PROXY, \"http_proxy\": PROXY, \"https_proxy\": PROXY,\n", + " \"no_proxy\": \"localhost,127.0.0.1,.alcf.anl.gov\",\n", + " \"WANDB_MODE\": \"offline\",\n", + " \"MAX_SEQ_LENGTH\": \"4096\",\n", + " \"TOKENIZERS_PARALLELISM\": \"false\",\n", + "}\n", + "os.environ.update(ENV)\n", + "for d in (PRISM_DATA_ROOT, PRISM_OUTPUT_ROOT, PRISM_CKPT_ROOT, HF_HOME):\n", + " Path(d).mkdir(parents=True, exist_ok=True)\n", + "\n", + "os.chdir(REPO) # model YAMLs use repo-relative tokenizer paths\n", + "sys.path.insert(0, str(REPO))\n", + "sys.path.insert(0, str(REPO / \"applications\" / \"bioreason\"))\n", + "sys.path.insert(0, str(REPO / \"applications\" / \"bioreason\" / \"helpers\"))\n", + "import demo_helpers as H # applications/bioreason/helpers/demo_helpers.py\n", + "\n", + "print(\"repo \", REPO)\n", + "print(\"run tag\", RUN_TAG)\n", + "for k in (\"HF_HOME\", \"PRISM_DATA_ROOT\", \"PRISM_CKPT_ROOT\", \"PRISM_OUTPUT_ROOT\"):\n", + " print(f\"{k:18s}\", ENV[k])" + ] + }, + { + "cell_type": "markdown", + "id": "cell-003", + "metadata": {}, + "source": [ + "## 2. Prerequisites\n", + "\n", + "PRISM on Aurora uses the **`frameworks/2026.1.0`** module (XPU PyTorch, oneCCL) plus a thin `--system-site-packages` venv holding only what the module lacks: `torchtune` (the [n-getty/torchtune-aurora](https://github.com/n-getty/torchtune-aurora) fork, for LoRA and the GRPO loss), `webdataset`, `wandb`, and `ipywidgets` for this notebook. Extras are installed with `--no-deps` so pip never swaps the module's XPU torch for a CUDA wheel.\n", + "\n", + "> Do **not** activate the repo's `.venv`. It carries a CUDA torch build, and its transformers 5.x can't run the Nucleotide Transformer **v2** remote code (see section 7)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "cell-004", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "python ${PRISM_VENV_ROOT}/bioreason-fw2026.1/bin/python\n", + "in venv True (sys.prefix=${PRISM_VENV_ROOT}/bioreason-fw2026.1)\n", + "torch 2.13.0a0+gitcf30153 | transformers 5.14.1\n", + "XPU tiles 12 (expect 12 on an Aurora node)\n", + "missing none\n" + ] + } + ], + "source": [ + "import importlib\n", + "\n", + "import torch\n", + "import transformers\n", + "\n", + "print(\"python \", sys.executable)\n", + "print(\"in venv \", Path(sys.prefix).resolve() == Path(PRISM_VENV).resolve(), f\"(sys.prefix={sys.prefix})\")\n", + "print(\"torch \", torch.__version__, \"| transformers\", transformers.__version__)\n", + "print(\"XPU tiles \", torch.xpu.device_count() if hasattr(torch, \"xpu\") else 0, \"(expect 12 on an Aurora node)\")\n", + "missing = [m for m in (\"torchtune\", \"webdataset\", \"ipywidgets\", \"hydra\", \"datasets\", \"wandb\")\n", + " if importlib.util.find_spec(m) is None]\n", + "print(\"missing \", missing or \"none\")" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "cell-005", + "metadata": {}, + "outputs": [], + "source": [ + "# Only needed if something is listed as missing above (same recipe as the submit scripts' header).\n", + "if missing:\n", + " !uv pip install wandb braceexpand==0.1.7 ipywidgets\n", + " !uv pip install --no-deps webdataset==1.0.2 torchao==0.5.0\n", + " # torchtune: the n-getty fork (PyPI's has no torchtune.dev.rl, needed by GRPO)\n", + " TT = Path(PRISM_VENV).parent / \"torchtune-aurora\"\n", + " if not TT.exists():\n", + " !git clone https://github.com/n-getty/torchtune-aurora.git {TT}\n", + " !uv pip install --no-deps -e {TT}\n", + " !uv pip install --no-deps -e {REPO}\n", + " print(\"Installed - restart the kernel and re-run section 1.\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-006", + "metadata": {}, + "source": [ + "## 3. Download models and data from Hugging Face\n", + "\n", + "**Models.** `snapshot_download` is called *without* `cache_dir`, so files land under `$HF_HOME/hub/models----`. That's the layout `transformers` resolves offline, and the layout the submit scripts' pre-flight checks look for.\n", + "\n", + "| Model | Role | Size |\n", + "|---|---|---|\n", + "| `allenai/OLMo-1B-0724-hf` | LLM backbone (d_model 2048) | ~4.7 GB |\n", + "| `InstaDeepAI/nucleotide-transformer-2.5b-multi-species` | DNA encoder **and** DNA tokenizer (6-mers) | ~10 GB |" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "cell-007", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "llm: allenai/OLMo-1B-0724-hf\n", + " -> ${HF_HOME}/hub/models--allenai--OLMo-1B-0724-hf/snapshots/d7cbab742d80589e714b1a2d7f838dcd21cbe143\n", + "dna: InstaDeepAI/nucleotide-transformer-2.5b-multi-species\n", + " -> ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n" + ] + } + ], + "source": [ + "os.environ[\"HF_HUB_OFFLINE\"] = \"0\"\n", + "from huggingface_hub import snapshot_download\n", + "\n", + "MODELS = {\n", + " \"llm\": \"allenai/OLMo-1B-0724-hf\",\n", + " \"dna\": \"InstaDeepAI/nucleotide-transformer-2.5b-multi-species\",\n", + "}\n", + "SNAPSHOTS = {}\n", + "for role, repo_id in MODELS.items():\n", + " local = Path(HF_HOME, \"hub\", \"models--\" + repo_id.replace(\"/\", \"--\"))\n", + " SNAPSHOTS[role] = snapshot_download(repo_id, local_files_only=local.exists())\n", + " print(f\"{role}: {repo_id}\\n -> {SNAPSHOTS[role]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-008", + "metadata": {}, + "source": [ + "**Data.** \n", + "\n", + "`helpers/download_bioreason_dna_datasets.py` saves each Hugging Face dataset with `save_to_disk` (Arrow) under `//`. \n", + "\n", + "PRISM's streaming loader globs `*.arrow` from the `local_path` given in `datasets_config.json`, so training never touches the Hub.\n", + "\n", + "| Dataset | Rows (train) | Fields |\n", + "|---|---|---|\n", + "| `wanglab/kegg` | 1,159 | `question, answer, reasoning, reference_sequence, variant_sequence` |\n", + "| `wanglab/variant_effect_coding` | 48,850 | `question, answer, reference_sequence, variant_sequence` |\n", + "| `wanglab/variant_effect_non_snv` | 35,215 | `question, answer, reference_sequence, mutated_sequence` |\n", + "\n", + "KEGG's `val` and `test` splits go to separate roots, because the script names the output folder after the dataset, not the split." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "cell-009", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "find: warning: you have specified the global option -maxdepth after the argument -name, but global options are not positional, i.e., -maxdepth affects tests specified before it as well as those specified after it. Please specify global options before other arguments.\n", + "${PRISM_DATA_ROOT}/dna_local_shards/kegg/data-00000-of-00001.arrow\n", + "${PRISM_DATA_ROOT}/dna_local_shards/kegg/dataset_info.json\n", + "${PRISM_DATA_ROOT}/dna_local_shards_test/kegg/data-00000-of-00001.arrow\n", + "${PRISM_DATA_ROOT}/dna_local_shards_test/kegg/dataset_info.json\n", + "${PRISM_DATA_ROOT}/dna_local_shards_val/kegg/data-00000-of-00001.arrow\n", + "${PRISM_DATA_ROOT}/dna_local_shards_val/kegg/dataset_info.json\n", + "${PRISM_DATA_ROOT}/dna_local_shards/variant_effect_coding/data-00000-of-00001.arrow\n", + "${PRISM_DATA_ROOT}/dna_local_shards/variant_effect_coding/dataset_info.json\n", + "${PRISM_DATA_ROOT}/dna_local_shards/variant_effect_non_snv/data-00000-of-00001.arrow\n", + "${PRISM_DATA_ROOT}/dna_local_shards/variant_effect_non_snv/dataset_info.json\n" + ] + } + ], + "source": [ + "DL = REPO / \"applications/bioreason/helpers/download_bioreason_dna_datasets.py\"\n", + "SHARDS = Path(PRISM_DATA_ROOT, \"dna_local_shards\")\n", + "\n", + "for key in (\"kegg\", \"variant_effect_coding\", \"variant_effect_non_snv\"):\n", + " if not (SHARDS / key / \"dataset_info.json\").exists():\n", + " !python {DL} --dataset {key} --output-dir {SHARDS}\n", + "for split in (\"val\", \"test\"):\n", + " if not Path(PRISM_DATA_ROOT, f\"dna_local_shards_{split}\", \"kegg\", \"dataset_info.json\").exists():\n", + " !python {DL} --dataset kegg --split {split} --output-dir {PRISM_DATA_ROOT}/dna_local_shards_{split}\n", + "\n", + "os.environ[\"HF_HUB_OFFLINE\"] = \"1\" # from here on, run like the batch jobs: fully offline\n", + "os.environ[\"HF_DATASETS_OFFLINE\"] = \"1\"\n", + "os.environ[\"TRANSFORMERS_OFFLINE\"] = \"1\"\n", + "!find {SHARDS} {PRISM_DATA_ROOT}/dna_local_shards_val {PRISM_DATA_ROOT}/dna_local_shards_test -maxdepth 2 -name \"*.arrow\" -o -maxdepth 2 -name dataset_info.json | sort" + ] + }, + { + "cell_type": "markdown", + "id": "cell-010", + "metadata": {}, + "source": [ + "## 4. Multimodal task the model is being trained for and the respective datasets" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "cell-011", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "kegg 1,159 rows columns=['question', 'answer', 'reasoning', 'reference_sequence', 'variant_sequence']\n", + "variant_effect_coding 48,850 rows columns=['ID', 'question', 'answer', 'reference_sequence', 'variant_sequence']\n", + "variant_effect_non_snv 35,215 rows columns=['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__']\n", + "kegg (test) 146 rows columns=['question', 'answer', 'reasoning', 'reference_sequence', 'variant_sequence']\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAD6CAYAAACBOBLaAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAPYBJREFUeJzt3Xd0FNX7x/H3pndSKAkghEDoSu+9iCBNQAUBFRSkK4hYsKBiBQGpFixYqAqIhZ/SBL9I79IhEAIEAqSQXnd+f0RWAinLGlLw8zqHc9h57tx55qbsk5k7d02GYRiIiIiIyC2xK+wERERERIojFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDh8JOQORW/Pjjj0RFRQHg4OCAv78/9erVw8/Pr5AzK9quXr3KypUr6dGjB76+voWdjojIHcGkj32R4qRZs2bExsbStm1bMjIyOHToEMeOHeODDz7gscceK+z0iqxjx47RoEEDtmzZwj333GP1ftu3b+fKlSt07dr1NmZXdI8vIpIbXYmSYqdu3brMnTvX8nrYsGGMGzeOXr164enpWYiZ3XmWLl3K3r17C62IKezji4jkRkWUFHsdOnRg4cKFnDhxgvr16wNgNpvZunUrZ86cwc/Pj5YtW+Lu7g7AuXPnWLduHf3798fJyQmA33//nbNnz/Loo49iMpkAWLt2LW5ubrRo0QKAU6dOsX//fpycnGjYsCFlypTJNa8tW7YQGxtLq1at2LNnD+fPn6d27drUrl072/YHDx7k0KFDuLq60rp1a7y9vW/qq3Xr1mzevJmIiAi6dOlCyZIls+0rJSWFjRs3EhUVRY0aNXB1dc32eLt27QLA1dWVwMBAGjVqhJ1d5lTJHTt2cPToUa5cucKCBQsAqF27Ng0bNsxz32usGbOczju344uIFAUqoqTYO3z4MHZ2dvj7+wNw6dIlevfuzcWLF2nWrBlHjhzhypUrLF26lCZNmmBnZ8fo0aMJCgqidevWAIwZM4bQ0FDq1KlDnTp1MAyDYcOG8dxzz9GiRQvefPNN5s2bR9u2bbG3t+ell15i7NixPPHEEznmtXDhQjZt2oS9vT2BgYE4OjoyYsQIRo8ezeTJky3t4uPjGTRoEJs3b6Zt27ZERkYyatQoli5dSvPmzbP05ejoSMWKFSlXrhxt2rTJ9rgXL17k/vvvJykpiUaNGvHee+9RoUKFm9qFh4ezc+dOABITE9m+fTu+vr6sWrUKPz8/wsLCuHTpEnFxcZZ2Xl5eNGzYMM99gTzHLK/zzu34IiJFgiFSjDRt2tRo2bKl8eWXXxqfffaZMW7cOMPPz8+YPHmypc3QoUONe+65x4iMjDQMwzDS09ONvn37GnXq1DFSUlIMwzCM+vXrG5MmTTIMwzBCQkIMLy8vo0OHDsb06dMNwzCMAwcOGO7u7sbhw4eNpKQko0SJEsbKlSstx0hOTjY2bNiQa64jR4403N3djW+//daybfXq1Ya7u7uxZcuWLO38/f2NY8eOWba98sorRp06dYy0tLQsff322295jtGQIUOMxo0bG7GxsZZcO3bsaLi7uxv79+/Pcb/U1FSjbdu2xvPPP2/ZNm7cOKNt27Z5HvPGfa0ZM2vO29rji4gUBi1xIMVOVFQUO3fuZMuWLfz0009Uq1aNESNGAJm38VasWMHQoUMtT6HZ29vz3HPPcfLkSfbu3QtA+/bt2bhxIwAbNmygYcOG9OrViw0bNli2BQQEUKNGDRwdHXF3d2fbtm0kJCQA4OzsTLt27fLM1d/fn/79+1ted+nShRo1arB8+XIg87bb0qVLGTRoEFWrVrW0Gz9+PCdPnuR///ufZVuVKlXo1KlTrse7dv5PPfWUZX6Ys7MzY8aMybZ9bGws69evZ9GiRSxcuBA/Pz/Lbbq85LZvXmN2K+ctIlJU6XaeFDvXTyyPjo6mTZs2DB8+nO+++46IiAiSk5OpWLFiln0CAwMBOHPmDE2aNKF9+/Z88sknxMTEsGHDBtq1a0e7du2YNGkSycnJlm2QWYQtXLiQSZMmUbFiRerUqUOHDh0YNmxYnksrVKhQwTLH6pqKFSsSFhYGZN56TE5O5urVq5Z5P9e4ublx6tQpSx55zcECiIiIICUlJcfzv97y5csZPXo0wcHBBAUF4e7uTkREBLGxsXkeJ6998xqzWzlvEZGiSkWUFGs+Pj688cYbPProo2zatIlmzZphb2/PlStXsrS79vpaIdKqVSvs7e3ZsGEDf/zxB2PGjKFmzZr4+PiwYcMGtmzZwuzZsy37t23blk2bNhEdHc2WLVuYOnUqy5cvZ/fu3bnmd2MeAJGRkdSqVQvInONjMpkICwu7qdh66KGHuOuuuyyvb4xnx8/PL9fzv9748eMZM2YMEydOtGx79tlnWbduXZ7HsWbf3MbsVs5bRKSoUhElxV7Pnj0JDg5m+vTprFq1ihYtWrB48WIGDRpkeYNetGgRPj4+1K1bFwAPDw8aNWrE9OnTMZvNNGrUCIB27drx7rvvkpSUZLkSEhcXR2JiImXKlMHHx4euXbuSlpbGwIEDiYuLy3VZhVOnTrF9+3aaNGkCwPHjx9mzZw+jRo0CoESJErRp04YKFSpkWbYBMq8qXXui0FpOTk40a9aMpUuX0q9fP8v2RYsWZWmXnp7O1atXs1yhio2N5aeffsryJJ+XlxeJiYm3vG9eY2bteWd3fBGRokJFlBR7dnZ2PPPMM4wePZoDBw4wdepU7rvvPrp3707nzp05dOgQixYt4uOPP6ZEiRKW/dq3b89bb71Ft27dcHBwsGxbtGgRNWvWtFy1io+Pp1OnTrRo0YLatWuTlJTEggUL6N27d57rUvn6+jJw4EAGDRqEg4MD8+fPp2XLlvTp08fSZs6cOfTo0YMePXrQrl07HBwcOHjwIFu3bmXt2rV4eHjc0ni899573HffffTp04d27dqxbds2y9Nt1zg4ONC7d29efvllzp07h729Pd988w3p6elZ2jVv3pzp06fz9ttvU65cOcsSA3nta82YWXPeOR1fRKQo0IrlUqy8++67lCpViiFDhmTZnpKSwvPPP0+rVq148MEHuXjxIkuWLCE0NBRfX18eeOCBm1bqPnLkCHPmzKFnz56WCduXLl3ijTfeoHnz5gwYMMDSNikpiZUrV7J//35cXFxo3LgxnTt3xt7ePsdcR40aRUhICB9++CG//PIL58+f5+6772bAgAGW9amu7/+HH35g7969ODo6Urt2bXr16oWLiwuQucTB5cuXGTt2rFXjFBISwsKFC4mOjqZ69ep06dKF999/n5deeony5csDmVeUlixZwr59+3B2dqZNmzaYzWa2bdvG66+/bunr119/ZfPmzcTExNC+fXt69+5t1b7WjFle553T8UVEigIVUSK3ybUi6tdffy3sVERE5DbQEgciIiIiNtCcKJHbpHnz5gQHBxd2GiIicpvodp6IiIiIDXQ7T0RERMQGKqJEREREbKAiSkRERMQGKqKkWDAMA7PZjKbwiYhIUaEiSooFwzC4EB5GRkZGYadSpJnNZs6fC8VsNhd2KkWexsp6Givraaysc6eMk4ooERERERuoiBIRERGxgYooERERERuoiBIRERGxgYooERERERuoiBIRERGxgYooERERERs4FHYCIrei2ZsnSEzLPnbo3RoFm4yIiPyn6UqUiIiIiA1URImIiIjYQEWUiIiIiA1URImIiIjYQEWUiIiIiA1URImIiIjYQEWUiIiIiA20TpTc0cxmM4mJiZbXLi4uODjk/G2fV/sb4wD29va4urre1Fd6enquxxIRkeJNV6IEgD17dvPhjGmFnUa+O3z4MJUrV6Zy5coEBASwZMmSf9V+3759+Pv7W9pUrlyZzp07W+KGYTBlyhQqV66Mn58fdevWZd26dbfl3EREpHDpz2QBIC4ujrNnzxZ2Gvmudu3aREREABAUFJQv7Z2dnS1tbvT5558zZ84cfvzxR+6++24WL15Mv3792LVrF4GBgbadhIiIFEm6EiViA7PZjNlsvmn7L7/8wsMPP0zdunWxt7dn4MCBBAUFsXDhwkLIUkREbicVUSK3wN7eHkdHRwIDAylVqhStW7dm48aNlridnR1paVk/3C8tLY2//vqrgDMVEZHbTbfzJFthYWF8Nv8T+jz4EHXq1AVg48bf2bFjO4mJCdSoXpNevfvg5OQEZBYK33+/jKNHjuDn50eHjvfy1YIvmPT6ZNzd3fOMFxd16tTh4sWLAMTHxzNz5kweeughtmzZQnBwMD179uSFF16ga9eu1KlTh8WLF3Pq1CnKly9fyJmLiEh+UxElNzlwYD8zP5zBkKFPWQqob77+ip07d9Dvkf54enqyatUPTJ82lRdfehmAOXNmcSE8nH6P9CcpKYl5c2dz8eJFMjIyrIrfKC0tLcsVHcO4+dZZYfPw8ODll19m+fLl/PbbbwQHB/PYY48RGxvLyy+/zKVLl+jcuTNdunTJ9uk9EREp3lRESRYb1q9j0aJvef6FF6lWrToAcXGx/PjjD3wwbQYVKwYCUKVKMIMHPcqFCxdwcXbmz83/Y9bseZQtWxbIvK31wdT3AYiOiso1np0Vy79n2bJ/noxzcXFmypQpt+OUiY+Px9XVFXt7e5v2T0tLy7Lv6NGjGT16NABJSUncc889vPLKK/mSq4iIFB0qosTi4sULzJ07m4kvv2opoADOnz+P2Wxm7pxZWdobhsGFC+G4uLjg5ORkKZAAgoIqW/5/4eKFXOPZ6d3nQbr36HndsczERF+x6bzi4+Mt+aakpBAfH4+zszOOjo5cuXKFwMBAVq9eTevWrfNs/8Ybb1CpUiXatWtHeno6s2fPJjIykvvvvx+AgwcP8v333/PEE08QFxfH5MmTKVWqFP369bMpdxERKbo0sVws/P0DGPzEEObNncOZM6GW7R4enphMJgYNfpKnho2w/Hv7nfeoVq0aHu4epKamkpKSYtknLi72n/3ziGfH0dERNzc3yz9XVzebz+vaek6JiYlMnDiRypUr8/HHHwNw+vRpfH19qVOnjlXtR44cyd69e+ncuTNdu3YlPDyc1atXU7FiRQBq1aqFn58fXbt2pU+fPpQpU4aff/4ZZ2dnm/MXEZGiSVeiJItu3brj6urK65NeZeLLrxIcXJWyZcsSGFiJ3bt20X/AQOzt7UlNTeXHH3+gYsVAypUvT+nSZVi1aiUPP9yPjIwMfvhhpaXPvOK3W05rOgH8+eefTJgwgRIlSljVvlSpUsyYMSPHuMlkYsyYMYwZM8a2ZEVEpNhQESU36dChIy7Ozrw1+U2ef+FFatWqzXMTXmDevNk8MfhxvL1LEB0dTZcuXXFwcMBkMvH0M2OZOuU91q9bR3p6GnXr1QfA3t4Oe3v7XOOFaezYsYV6fBERKb5URAkA9es3yLJCd4uWrahcpQqpqZlPyPn7+/Pmm28THx9PTEwM/v7+WT4Xrnr1Gnw6/wsiIi7i4+NLaOhptvy52XIbLq+4iIhIcaMiSgDw9PTE09MzyzZ//4Cb2nl4eODh4XHT9r/+OkDp0mUoW7YcKSkpLF/+PXXr1sPOzs6quIiISHGjIkryhY+PD2+99QYYBjExMdx1VwXGj59gdVxERKS4UREl+aJ8+buYNWsuERERuLi44O3tfUtxERGR4kZFlOQbk8mEv7+/zXEREZHiRBNSRERERGygIkpERETEBiqiRERERGygIkpERETEBppYLsXK1teCsyzyKSIiUlh0JUpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGzgUNgJiNyKZm+eIDGtsLMoDo4VdgLFiMbKehor62msDr1bo7BTuO10JUpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBiqiRERERGygIkpERETEBlonKp/FxcVy5MgR7OzsaNiwEYZhcPjwYSKvXKZe/QZ4enoWdoo2SU1NZevWLTRv3gJHR8fCTkdERIq5L75YwIWLl5gyZQoeHh4AGIbB6tWr2bp1K2azmSZNmtCjRw9MJlO2feTV/qOPPuLgwYNZ9hkyZAj16tWzvN6+fTurVq3CbDbTrVs3WrZsafU5FKsrUXv37uHixQuFnUaOrsbEMG7s06xbt5ajR48AMG/eHOZ/+jF79+4hMTGxkDO0XUJCArNmziApKQko+l8LEREputasWcMXXy7g66+/Jjk52bL9xRdfZMyYMZQoUQI/Pz+ee+45nn766Rz7yav9+vXriY6OplGjRpZ/vr6+lviKFSu4//77cXBwwNXVlV69erFw4UKrz6NYXYk6fuwYjo6O+PsH5Et/e/bspmzZcvj7++dLf1u2/klgpSAmTnwFgMTERDb+voHPPvuSEt7e/7r//M7338jvr4WIiPw3REdHM2HCBF54fgLPTXghS+zbb79l+vTp9O3bF4CqVasyaNAgZs6ciZ3dzdd9rGnfoEEDBg0alG0ur776KhMnTmT8+PEAlC5dmldffZV+/fphb2+f57kUqytRffs9Qu3ad+dbf0sWL+Tw4UP51t/ly5fx8/OzvI6MvIKdnV2+FFCQ//n+G/n9tRARkf+G8ePHM3jwYAIDK94UCwgI4MqVK5bXV65cwd/fP9sCytr2GzduZNy4cUydOpUTJ05Ytp8+fZozZ87QtWtXy7bu3btz6dIljhw5YtW53JYrUadOnSIpKZFatWpbthmGweb//cE999ShhLc3Bw/+RWRkJHYmE6VKlaZylSpZ5trs3r2L8uXvwmQyERp6msDASpw/f46AgADL1Q9r+7CzsyP09Cl8fHypEhwMwJEjh4mLj+f4sWPY29vj5eVFvXr18zy3lJQUQkJOkpiYSOWgyvj8fVnw4MG/CD19GrPZzKZNGwG4FBEBwKZNG3FxdqZJ02a59nH9MU6fOkVCQgLVa9TA3d39X+V7Y1/XxMXFERJyEhMmqgRXwd3dI8u+UVGRhJw8iZ9fSXx8fLLE9u7dk+VrkdtYXxMZGcmpkMz+ypUvz7ZtWzXHSkTkP2TVqlWEhITwySefsH7dbzfFFyxYwMiRI/njjz8wmUyEhYXlenstr/YjRozg7NmzpKSksHv3bpo1a8aCBQvo1q0bFy9eBMhyd6dMmTIAREREULt2bfJyW4qomJhoZs2cwfzPvrS8Qf711wE+/fRjPvt8AQCnQkI4ffoUGWYzZ8PCwARvvvm2ZeL1ooXf4u7uztWrMVQMrIR3CW+WLllMx473Wt64renDw8OD2NhYAgICOHToIG3btWfw4CcJDT1NYkIiYWGhpKQk4+8fkGdRcvzYMaZNm4KfX0k8PT2ZffQIQ4YOo1Wr1pwKCeHy5UsYBuzbuweAuPh4zGYz+/buwcPDgyZNm+XaB8DRo0eYPm0qXl4l8PX15auvvmTC8y/YlG9Ofd11VwV2797FrJkzCAyshNls5uzZMMY/9zx3330PADt2bGfmh9OpVq06SUlJODk5Zen7xq9FbmMNsG3bVmbP+pBq1auTnJSEg6Mjhw4epF69+tkWUWlpaaSl/fNJw4ZhzvVcRUSkaLt06RITJkzghx9+yPFW2eLFi4mPj6dp06bY2dkREhLCN998Q926dW1q36FDhyzty5cvzyuvvEK3bt0sOaSnp1vi1/6f05WvG92WIqpOnbo4Ojqxa9dOmjVrDsDvG9bTvEVLnJ2dAejR8wFLe8Mw+GDq+/z884888siAfzoywfQZs3IcbGv6cHZ2Ztr0D7Gzs+Po0SO8+spEBgx4lC5duvL7hvV0vPc+2rfvkE3vWaWnp/Phh9Pp+UBv7r8/89Lf0SNHePvtyTRo0JAePR8g4lIE5owMhg0fCcDJkyc4+NcBnhn7rFV9ODk58eGM6XTs2ImH+/YDIDoqiti4WNvyzaGv1NRU5s2dzUMP9aVb9x4AfP/dMubMnsW8jz4hIyODTz6ex+DBT9Lx3k4AfP75fA4e/CvXY+Y01oZh8OknHzFk6DDatWsPwJdffs6hG56YuN6K5d+zbNkSy2sXF2emTJmS53mLiEjRtGjRIhwcHJg7dy6GYXD+/Dkgc3L4gAEDKFeuHDNnzmTnzp3UqFEDgF69elGzZk0GDx5805Wh48eP31J7gIYNG/Lhhx9iGAYVK2beTgwLC6NkyZIAnDlzBsASy8ttKaLs7e1p27Ydv29YT7NmzUlMTGTbtq288cZbWdqdPRtGeHg4ycnJuLm5cfrUqSzx5s1a5DmxK68+GjduYqkoK1UKwmw2Ex0dbblkZ63w8HAiIi7i5OhouV0HmVdIws6cofrfX8B/04eLqyuXL1+ie48elpiPr+9Nt/usce7cuRz7On36FDExMdzb6T5L7N57O7F48UIuXrxASkoqsbGxtLuuWOvY8V5W//JzrsfMaawTEhKIi4ujdes2lrbt23fg559+zLGv3n0epHuPnpbXhmEmJvpKju1FRKRo69ixI95/zxE2DAN3d1fWr99AvXr1KFOmDK6urkDmFatrRVHE39NiXFxcAFi2bBnh4eGMHTs2z/YxMTEcPXqUpk2bWo65YsUK7rnnHkwmE2XKlKFBgwYsWLCA+vUz7+x89dVXVK9enaCgIKvO6bY9ndeufQfGPjOa6Ohodu3aSenSpalarZrlRKZOeY/jx49RpUowLi4ulhO/nqubW479W9uH43W3oa6tG2E23/qtoaSkREwmE4cOZb160qhRYxydrJvTk1cfqSkp2NnZ4ejolEMP1sutr9TUVOzs7HBw+OfL7+Sc2S4lJZXU1BQcHByyFLBOTs55HjOnsbalP0dHxyy3+cxmMzHReaYgIiJFVO3atS1Xh8xmM2t+W81HH31C3759LVeCRo8eTd++fWnfvj12dnasX7+eIUOGUKVKFQA2bNhAQkICAHfddVeu7RMSEpg8eTLJyclUrFiR/fv3k5iYyLJlyyw5ffjhh/To0YMTJ05gb2/Pnj17WL58udXndNuKqLJlyxIcXJVNm35nx/ZtWa5qhIefZ+fOHXz19ULc/i6UvvziM44dO2Z1//nRh4ODI+aMDKva3nVXBRwdHelyf1eqVq1m2R4dFWX103d59ZGcnIyDgwP79u2lYcNGQGaxGBcXh5eX1y3lW/6uu3Lsq2zZctjZ2XH48CHLHKi//voLJycnAgICSE9Px2w2c/z4MUueNxZ+t6JcuXKYzWZOnjxBlSrB/7o/EREp/sqVL8fs2bOzLEL93nvv8eSTT/LXX5nTR1599VWqVfvn/XLLli188803VrV3d3fnl19+Yc+ePZw8eZJBgwbRtGnTLHN869Wrx/79+9m4cSOGYbBgwQJLQWeN27pOVLv2HViyeCGxsbE8/8JEy3Zvbx/c3NxY8OXnVK1WndOnQtiy5U/KlLF+/aP86KNiYCC//74BO3t7fHx8cp2o7ebmxuODnuD9996l47334udXkrNnw9i3by8zZsyyahJaXn24ubkxYOBjzPxwOp273I+vry/bt22jd58HueeeOrecb259PfBAb2ZM/4AePR/AnGFm1aqVPPRwX8vl0W7dezB1yvv06NmTpKQkfv99g9XjeiNPTy/uv78bU6e8R7fuPbL0l8MitCIicofz9fHh8ccfv+n9Mzg4mOAbnu6GzAeOJk+eTJ06daxqf039+vUtt+uyzcPXl969e99i9pluaxHVokULDh06SLmy5Sz3QSGzOnz7nfdYv24dx44dJSgoiLHjxnP0unUZ6jdocNO8pXr16uMfEGBzH3Z2drRu3cZSKAwc+Cj/93+rOXzoIKVKlc7zabfOnbtQtWo1duzYxulTIdxVoSL9+w+03BYLrhKc5Vahp6eX5ak7a/vo1q07VapUYfeuXYSHh/PQw/2oVauWTfnm1tcj/QdQtWo1DhzYByYTY8eNz9Lfo48+TmDFQI4cOYKfnx+vvfYGy5Yuttxiu/5rYc1YPz5oMJUqVeLo0aP4+vkyevTTvPbqyzg7u+R6DiIiIpA5zaNnz555NyxAJsMwjMJOQu58cXFxWS7Z/vzzT6z57VdmzZ5r1f5ms5kL4WE88GkSiWl5txcRkcJ16N2cH7i69js9oGwFq5cTKIqK1ce+3E7Jycls374t25i/vz/VqlUv4IxyV9zy3b9/L39u3kyduvWIiLjIb7/+H8P/XgpCRESkOFIR9bfU1FTLIpk3qlGzVpErSopbvi1btsbFxZX9+/bi7OLC669PtjytKSIiUhzpdp4UC7qdJyJSvPwXbucV38xFRERECpGKKBEREREbqIgSERERsYGKKBEREREb6Ok8KVa2vhac5TP/JKs7ZbJmQdBYWU9jZT2N1X+LvsIiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDh8JOQORWNHvzBIlphZ1FcXCssBMoRjRW1iuaY3Xo3RqFnYL8R+lKlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNtE6UiIjcsS5evIi9vT2lSpXKsv38+fPExMRk2Va2bFl8fHxy7e/s2bN4eXlRokSJLNvDw8OJjo7GMAyiIiMIKFshX44nRZuKqDyYzWa+/fZrtm75Ez+/krz19rvs37ePb775isjIK0ya9CaBlSoVdpoiInKdX3/9lYkTJ5KYmEhcXBwVK1Zk/vz51KpVC4DXXnuNdevWUbp0acs+r7zyCj179rypL8MwmDNnDjNnzsTR0ZHIyEiaNGnCZ599RpkyZQCYP38+P//8M7GxscTGxnL+/PksfdzK8aT4KPTbeZPffJ1NmzYWdho52rdvL9u2bmXS62/y0sSXycjIYPbsD+n5QC9mzprLXRUq5N2JiIgUqJMnT7JixQqOHj1KaGgoNWvWZOTIkVnaPP744+zcudPyL6eCJi0tjQsXLrBlyxaOHDnCiRMnSExMZMKECZY2kyZNYufOnUyePDnHnKw9nhQfhV5EPTt+As2bt8i3/l5//VU2b/4j3/o7cfw4NWrWxN8/AHd3DyIjrxATE0OLFi3x8vLC3t7+X/Wf3/mKiAiMHj2awMBAABwdHenQoQNnzpzJ0iY1NZWQkBDi4uJy7cvJyYl33nnHchWpRIkSdOrUiePHj99STtYeT4qPQi+i3N3dcXR0zLf+EhMSSE3Nvw9Xi4+Pw+m6/OLj4nFwcMDOLn+GLr/zFRGRTFFRURw6dIg1a9YwY8YMRo0aZYmVL1+e9evX8+CDDxIYGEjPnj1vugWXk/T0dH766Sdat25tdS7/5nhSdJkMwzCsabhmzW9s2bKZ11//51JlZGQkz40fx/tTplK6dBk+nDGNffv2YjLZUapUKTp07Mh993WxtH/ttZdp2LAxx44e4dSpEAYNeoI1a36jdZu2tGnTFsCqPho3asLRY0cJCTmJj7cPjw8aTLVq1fnoo7n8vmE9jo6OODo6UrlyFV597fVczystLY0fVq5gx47tJCYmUL1GTR57bBAlSpTgwxnT2L59GwDOzs4AZGSYSUxMwNPTEx9fX2bMmJVrH9cfY+fOHSQkJtC8eQv69n2E+fM/ueV8czt/gKSkRBYt/JYDB/aDyUS9uvV5pP8AS/557Z+bTZs28svPP3H16lWCgoJ47PHBBAQE5NmvNd87eTGbzVwID+OBT5P0AcQikkVOH0C8bNkypkyZwqVLl6hUqRIfffQRNWvWvKldVFQUjz32GGazmdWrV+d6LMMwGD58OPv27WPdunV4enredMxnnnmG8+fP5/jH9q0c70517Xd6QNkK+XZRojBYPbG8ceMmfDb/E86dO0v58ncBsPH3DZQtW9byJvjUsBGkp6djNps5ExrK3Lmz8Pb2oUmTpgAkxCewYvl3jBo1hqFPDcfd3Z2VK1eQlppqOY41faxcuZyRo8bw+OODWf3Lz0yfNpWPP/mMQYMGc/LECTp0vJeWLVtZdavt44/ncfHCBYYNG4GHpyfLl3/HlPff5e133uOpYSNwcnIiw2zm8ccHAxAaepp33p7MrNnzLF/43PoAmDd3NmfPhjH4iaH4+vqy5c/N7N6106Z8czt/k8nEjOnTSExKZNyzEzCbzXzy8TzmzJ7J+Oeet2r/nERERDB3ziyem/AClYMqczr0ND+uWsmw4SPz7Nea750bpaWlkZb2T7VkGOY8x0ZE5HoPP/wwDz/8MGazmWnTptGlSxcOHz6Mu7t7lna+vr5MmDCB7t27k5SUhKura7b9ZWRkMHz4cA4dOsQvv/xyUwFlLWuPJ0Wf1eWft7c39es34PcNGyzbNm7cQLv2HSyv3dzc8PLywtvbmzp169Kte0+2bPkzSz+du9xPo8ZN8Pb2zvY2njV9dOvWgwYNGlKqVCl69+nDlSuZ85RcXd2wt7fDxcUFLy+vm35QbhQdHc3G3zfw9DPjqBIcjL+/P8OGjSA09DRnz4bh5uaGo5MTTo6OeHl54eXlhZubGwBeXl54eHjk2UdUVCR//LGJp58eR61atQgICKDPgw/RuEnTW843r/O/fPkyu3fvYuTIzLkAQUFBPDVsOFu2/MnVq1fz3D83GRnpODg4UKlSEH4lS9KwYSOeGjbCqrys+d650Yrl3/PowEcs/54a+oRVYyMiYjabycjIsLy2s7OjW7duREZGEhkZiWEYpKenZ9knIiICV1dXy1X7S5cucerUKUs8NTWVRx99lJMnT7J69WpKlixpdT7WHE+Kp1ta4qBd+w7M//Rj+g8YyInjx4mKiqJFi5aW+P59+1i1aiXh4eEkJyeRlpZGpUpBWfrwL+Of6zGs6cPXz8/yf2dnFwCSk5Nv5VQAuHjxAgAvPD8+y/bU1FQuRVzirrvyfvIurz5c3Vyxs7O76Sm+3K765CWn84+KisLe3p6yZctZ4tfO4fKlS5bbi7aMX9my5Rj61HCmT5uKvb09VatV4777ulge782r37y+d27Uu8+DdO/xz5MrhmEmJvpKrjmKiAAkJCTwwAMPMHz4cCpXrsyFCxeYPn06jRs3pkKFCiQmJtKlSxdGjhxJlSpVOHz4MJMmTWLIkCGWOwwfffQRa9as4c8//yQtLY1+/foREhLC/PnzOX/+POfPn8fBwYFq1aoBmVMULl68SHh4OGazmcOHD2MymahZsyZJSUl5Hk+Kp1sqoho0aMgnZjP79u5l+/atNGvW3HIZMi4ulnfemczQocO4+546uLi4sG7tGnbu3JGlD1Mu3zDW9pE764sTHx9fTCYT773/geUK0zXWXl7Nq4/IyEjMZjNXrly5abG3W80371y8ycjIIDoqCh9fXwCuXLn8d+zfL+jWrl172rVrT3JyMuvWreXFFybw6fzPrXowILfvnexcmyd2jdlsJib6X5+CiPwHeHp6Mnv2bGbNmsWcOXPw9vamS5cuDB06FMi84/Hxxx8zc+ZM5s2bR5kyZXjzzTd55JFHLH0cOXLEsgRBbGwsZ8+excnJKcvkdG9vb9auXQvA2rVrmTZtGgBlywYweHDmFJA///zTquNJ8XRLRZSDgwOtW7fl119Xc/ToEV586RVLLD4+AbPZTHDVqpQpU4bw8HA2bvoddzfrblHlVx8lSpQg4uJFq9r6+/tT++67Wbp0MU888SSenl5ERESwYvl3DBk6LF/68Pf3p3btu/n0k48YPfppPDw92bVzBy4urtSpW/eW8s1LQEBZKleuwldffcmIkaMxm818/dUCatSsid8tXHrOzqFDBwkJCaF9+w54eHjg7+9PfHwcqampVhVRuX3viIjkt5o1a/Lxxx/nGK9Ro0aOccMwSEpKshRMfn5+7Ny5M9fj9evXj379+uU4YTq340nxdcsrlrdv34Fx457GPyDAsvIrQEBAAD16PsCLL0zAzs4OHx9fatasSVhYmNV950cfXbt1Z+aHM1i9+heqVq2a59Nuzz47ga+/+pKRI4aTkZGOj48PvXo/eEvLLuTVx7hnn+Pzzz5l5MhhmEwm6tSpy/ARo2zKNzcmk4nxzz3Pxx/NZfCgRwGoWbMW48aOz2PPvAUFVWb37l2MGjmc9PQ03NzcGTFilNXzuCDn7x0RkaLEZDKxatWqwk5DigGrlzi4XlxcLA4ODri6ut0Uy8jIIDU1BVdXN9LS0khNTbW80SYkxOPk5JylQElISMDJySnLtlvtIzY2Fg8PjyxVf0JCAsAtvcmnpKTcNMnv2pweFxcXS26ZSxx4Wd3HNdcmF2ZXoFmbr7Xnf21S5Y1P/Fm7f26yO0dr+83teyc3WuJARHKS0xIHheFOeXT/drtTxsmmz87LqYCAzDfta2+QN85rcXf3uKl9dkXDrfbh5XVzPrdSPF2TXfFzrXi6Prfczj+3Jy1MJlOOV7iszdfa889puQRr989Ndudobb+5jZ2IiEhxckd/APGVK5d5bvy4bGMtW7ZmyNCnCjij3BVmvsVtrERERAqbTbfziguz2Ux8fHy2MUdHxyK3wFlh5lvUx0q380QkJ7qdV/zcKeN0R1+JsrOzu+VbVYWpMPMtbmMlIiJS2Ipv+SciIiJSiFREiYiIiNhARZSIiIiIDVREiYiIiNjgjp5YLneera8F4+Cgb9uc3ClPvBQEjZX1NFYi2dNPg4iIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFlIiIiIgNVESJiIiI2EBFVAEKO3OGdWvXFHYaWSQmJvLzTz8W6DHX/PYrMTEx+dbf+PHj6dy5M507d2bMmDE5tjObzQwZMoTOnTvfFDtx4gTjx4+nd+/evPPOOyQnJ+dbfiIicmdSEVWAToee5v/+b3Vhp5HFypXLiYyMLNBjxsfHs3TJonzr77HHHmPixIkEBQWxb9++HNvNnDmT3bt3s3nz5izbt23bRps2bXBycmLo0KG4uLjw8ssv51t+IiJyZ1IR9R+WnJzMr/+3mi7332/Ztn79OkJDQ/PtGNn1d2+nTmzatJGr+XQ1qk6dOrRu3ZqKFSvm2Obw4cN8+eWXvPLKKzfFxowZw4gRI3j33Xfp0qULzz77LJMnT86X3ERE5M6lIuo/bNvWLZQrV47SpctYtv3262pOnQrJt2Nk15+npxfVa9Tgj/9tyrfj5CY9PZ1hw4YxdepUvLy8ssROnjzJkSNHaNSoEUOHDqVv375MmzYNOzv9aIiISO70TlGI0tLS+PHHVRw6dMiyLSoqko0bf2f16l84derUTfucPHmCX375iW3bthIXF8v33y0jJSXF6vj1/vrrAFWrVre83r5tK9HR0ezatZPvv1vGhg3rrcorKSmJPzZt5KefVnHo0EGr+qtevQZ/HThwC6NluylTplC9enXuu+++m2Lnzp0DYOLEibRu3Zr+/fuzfPly+vXrVyC5iYhI8eVQ2An8V8XFxfL+e+9SpkwZ7r+/KwC7du1k9qyZ1KtXD09PT5Z/v4wHevWme/eeAPz66//x7Tdf0bJla44fP87SJYsJCztDp/s64+zsnGf8RmfPnqVtu/aW1+kZGRiGQUZ6OqlpqaSnpeWZV3JyMuOfHUupUqUICqrM7t272LF9G4OfGJJjfwD+Zfz5/bqi6kZpaWmkXdfeMMw2jfO+ffv46quv2Lp1a7ZxR0dHAMaOHcujjz4KQOXKlWnatCmnTp0iKCjIpuOKiMidT0VUIbhw4QJvv/UmLVu1ol+//gCkpqYyb+5shg0fQfPmLQC4r/P9THhuHG3atMPR0YFvv/mKZ8dPoH79BgCsWPE9C7/9BoCkpMRc49lJSEzAxcXF8rpFi5as+mEFTZo2o337DlbldfnyJWJjrzJ33seYTCYAyxyo7Pq7xs3dncTExBxzW7H8e5YtW2J57eLizJQpU/IY2Zt99913GIZB//6Z4xwdHQ1A586def7556lVqxYAZcuWtezj7++fpa2IiEh2VEQVsKioKCa+9DyP9B9Ip07/3F46f/4cV69e5WxYGN+fP2/ZbmdnR1jYGZydnUlLS6Nu3XqWWOPGTSxF0rlz53KNZ8fdzS3PR/nzyqtq1WqULVuO6dOm0qRpU2rUqEVgYGCe45CUmIibm1uO8d59HqR7j56W14ZhJib6Sp793ujJJ5/Mchtv9+7dvPrqq0ycOJFq1apRpkwZWrVqxdKlS+nQoQMmk4mlS5fi6+tL9erVc+lZRET+61REFTAPTw9KlyrNju3baNOmreU2W0Z6BiaTifSMdDLMGZb2Xbt1x9vbm6SkJADL1Z4b/5/dtuzi1ytXrjwRERdzbZNXXk5OTrz73hSOHz/G0SNH+GHlWwQFBTFyVM7rNQFERERQrlz5HOOOjo6WW22QucZTTA4XhubOnctPP/3E2bNniYyMpHPnzpQsWZJvv/2WoKCgLLfkrs0Pa926tWXbvHnz6NOnD7Vr18bDw4NLly7x+eef4+7unus5iIjIf5uKqALm5OjEiy+9zIzpH/DW5DeY+PIruLq6cVeFCri7u1OjRk3L7TjIvDUWEFCWlJQUHBwcOLB/P3Xq1gVg965dlnblypXPNZ6d2nffw/p1WRf/dHJ2zjJ3Ka+8rly5jJOTMzVq1KRGjZq0aNmSEcOfYsDAxyhRosRN/V1z9NhRat99t9Xjlpv77ruPu2/oK7s5YAANGjRg9eqsa3VVqlSJXbt2cfjwYQzDIDg4OMttThERkeyoiCoEjo6OjH/ueebOmcXrk17jlVcn4enpyYiRo5k960Pq1auPn19Jzp4NIyoqknffm4qbmxt9+/Xngw+m0LZdO5KTkjh27BiQecUpr3h2mjdvzpdffMaVK5cpWbIUAMHBVfll9c9ER0dTqnRp2rfvkGteCQkJvPH6a1StVp2SfiXZt28vwcFVKVGiRI79JSTEc/jQQUaNHJ0v41mlShWqVKliVVtfX98sV6GusbOzo3bt2vmSj4iI/DeYDMMwCjuJ/4rQ06c5cvQwXbpkPo1nGAarV/+MdwlvWrRsBWQuJbBv715irsZQoUJF6tdvkGXNoiNHDnPs6FF8fX3xDwjg5YkvsmjxMsutr7ziN/rm668w2ZkYOPAxIPOpuK1b/uTixYt4e3vT6b7OeeaVkJDAjh3biYqMpGzZsjRs1NhyvOz6W/XDSs6fP5fnLb/rmc1mLoSHUcY/84qbZO/aOAWUraC1rvKgsbKexsp6Givr3CnjpHejAhRYqRKBlSpZXptMJrp27Z6lja+vH+07dMx2/0uXIqhevQY1atQEYMmSRQQGVrIULHnFs9O7Tx/Wrvnnlp6joyOt27S9qV1uebm7u9PuuqUSrpddf46OjvR7pH+OOYmIiBQHKqKKkbNhZ/lg6hTuvvseIiIusnfvHiY8/6LV8ey4u3vwQK/etzv1LO7v2q1AjyciInI7qIgqRho0bIhfyZIcOLCfMmXK8Njjg7J8ZEtecREREck/KqKKmcDAwFzXYcorLiIiIvmj+M7mEhERESlEKqJEREREbKAiSkRERMQGKqJEREREbKAiSkRERMQGKqJEREREbKAiSkRERMQGKqJEREREbKAiSkRERMQGKqJEREREbKAiSkRERMQGKqJEREREbKAPIJZiwTAMAMxmM2azuZCzKbqujY3GKG8aK+tprKynsbLOrY6TyWTCZDLdzpRsYjKuvTuJFGFJSUlERUYUdhoiIlIIAspWwM6u6N08UxElxUJ8fDxPjxnBrNnzcHNzL+x0iqykpESeGvoEn87/AldXt8JOp0jTWFlPY2U9jZV1bnWciuqVKN3Ok2LBzs6Oq1djsbOzL5J/jRQVJpMdyckpmEx2Gqc8aKysp7GynsbKOnfKOBXfzEVEREQKkYooERERERuoiJJiwdHRkYcf7oejo2Nhp1KkaZysp7GynsbKehor69wp46SJ5SIiIiI20JUoERERERuoiBIRERGxgYooERERERtonSgp0qKjovjkk4/Yv38fLi6utO/QgQEDHi3W64pkJ/LKFZ566knL68ceG0TPB3plaXPs2FE+m/8pYWFnKFmyJP369adV6zYFFi8qwsPP8913y9i/by9ms5kaNWry+KDB+PsHWNqsW7eW5d8vIzo6msDASjz11HCCKlcusHhRER4ezpIliziwfx8mkx1333MPgwYNxtfXD4CMjAy++forNm7cQGpqKnXr1WfYsBGUKFGiQOJFUXRUFGPHPo3JBAu++tayPSYmho8/nseB/ftwdnamXfvM30X29vYFEi8qbvxdBeDp6ZllrI4fP8Zn8z/lzJlQ/PxK8nDffrRt267A4gXKECnCJr70gvH225ON6OhoI+TkSWPIk4ONFcu/L+y0bqtnnh5t/LByRZZtV69eNR4d+IixdMliIy4uztj8vz+Mhx/qbRw9eqRA4kXJ3DmzjD82bTRioqONyMgrxnvvvm2MHjXcEt+3d6/Rr++Dxo7t24zY2Fjj668WGIMHPWokJCQUSLwo+eLz+ca2bVuN+Ph4IyIiwnjzjUnGq69OtMSXLF5kDHtqiHH69GkjKjLSeOP114zXJ71aYPGi6O233jTefGOS8fhjA7Jsf/WVicbkyW8YUVFRxqlTIcZTQ58wvvtuaYHFi4orly8bvXv1MOLj47KNx8bGGo89OsBYtOhbIy4uztjy52aj78N9jEOHDhZIvKDdWX/Oyx0l7MwZjh49wqBBT+Dt7U1Q5cr06NmTNWt+LezUCtzmzf/DxcWFhx7ui4eHBy1atqJevfqsXfNbgcSLkpGjxtCqdRtKeHvj6+tHr959CA8P52pMDABr1vxKkyZNadS4CZ6envQfMBCz2cz27dsKJF6UDH5iCE2aNMXd3Z3SpUvToeO9hJw8aYmvWfMbD/TqTWBgID6+vjw+aDAHDuzn4sULBRIvatatWwtA6zZts2w/d+4chw4dZNCgwfj4+FCpUhA9e/ZizW+/Fki8ONmyZTOOjg707fsIHh4eNGvegvoNGrJ27ZoCiRc0FVFSZIWGnsbNzZ2yZctatlWuXIVLly6RmJhYiJkVvDOhpwmqXCXLZ0dVrlyF0NDQAokXZYcPHcLX15cS3t4AhJ4JpUqVYEvc3t6ewMBKhIaeLpB4URUZGcnG3zfQuHETIPP2UUxMNFWqVLG0qVgxEEdHR0JDQ297vKi5fPky3y1bwrDhI2+KnQk9jYuLC+XL32XZVrlKFSIjI4mLi7vt8aJo5IjhDOjfl5defJ79+/ZZtoeGhhIUVDnLlIsqlav88/Nzm+MFTXOipMhKTErCzS3rB1O6u2d++HBSYuJNsTtZYmIi7tmMRVJSYoHEi6rDhw/x3XdLGffsc5Zt2X1vuLm7k/R34X2740XNurVr+OijuQBUqlSJESNGAVj+ELnxA73d3NxITEy87fGixDAM5s6ZxUMP9cXPz++meK6/i5ISb3vc09PzX5xd/vIrWZLlK1YBEBcXy9q1a3jnncm88+77VK5cJZefjyQgt5+f/IkXNF2JkiLLzdX1pl+2CQkJALi4uhZGSoXGzc2NhGzG4tqnn9/ueFF08OBfvPfu2wwfMYqGDRtZtrtm8yadmJCA69+/eG93vKjpeG8nvl/+A/M/+4Lg4KpMfPlFUlNTLW9EiYkJWdon/v0mdbvjRclvv/2KyWSi472dso3n9rvI1dXttseLKk9PL3r3fpDg4Kps3boFyO3nw7VA4gVNRZQUWYGBlUhMTCA8PNyyLSTkJKVLl7b8lfZfUTGwEqdCTmJc9wEDISEnCQwMLJB4UbNnz27ef+9dRo0aQ6tWrbPEAisGcvLkCcvrjIwMQkNPExhYqUDiRZHJZPp7/tiDXIqIICLiIt7e3nh7+3DyujlSZ86EkpaWRmBg4G2PFyUnThznwIH99Ondkz69ezJr5gzi4uLo07sne/fsoWJgJZKTkzl37qxln5CTJ/Hz88PT0/O2x4uTwMBATp0KwWw2W7adDDn5z8/PbY4XNBVRUmRVqFiR6tVrsGDBF8TExHDq1Cl+XLWKezvdV9ipFbgWLVqSnJzMd98tJSEhnj//3MzevXssY3G740XJ1q1bmDF9Gs8++xxNmja7Kd6pU2e2b9/Gzh3biY+PZ9HCb7Gzs6NJk6YFEi8q0tLSmD79A0JDQ0lNTeXSpQiWLV2Mr68fAQGZ8ww7dbqPH1au4MyZUKKjovjqqy+5+557LMtF3O54UTFmzDMsX7HK8u/pZ8bh6enJ8hWrqFe/PuXLl6dWrdosWPAl0dHRhJ4+zapVK+nUqTPAbY8XJb/88hMb1q/jakwMcXFx/PDDCk6cOE7TJpk/i82btyQtLY2lSxeTkBDP1q1b2LN7F/f+fZXvdscLmj47T4q06KiozLVTDuzHxcWF9h063pHrRAEMenzgTZNIe/R4gMcHDQbg6NEjfP7ZfMLCzuDnV5J+/R7J8hTR7Y4XFWOfGcPZs2E3bf9g2gwqVQoCMucBff/9d0RHR1GpUtDN6zzd5nhRsWPHdlYs/44zZ87g7u5OzZq16NvvEcqVKw9kXkX7+usFbPz9d9LSUqlbt17mOk5/T9K/3fGiatOmjXz5xWdZ1j6Kjo7OXLNu317LOk4DBz5mWcfpdseLitjYWJYuWcT27dtISUmlYsWK9HnwIerVq29pk7mO0yecOXMGXz8/+vZ9JJt1nm5fvCCpiBIRERGxwZ3357yIiIhIAVARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImIDFVEiIiIiNlARJSIiImKD/wcOrwEpdYaP/AAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "from datasets import load_from_disk\n", + "from IPython.display import HTML, display\n", + "\n", + "DS = {k: load_from_disk(str(SHARDS / k)) for k in (\"kegg\", \"variant_effect_coding\", \"variant_effect_non_snv\")}\n", + "DS[\"kegg (test)\"] = load_from_disk(f\"{PRISM_DATA_ROOT}/dna_local_shards_test/kegg\")\n", + "for k, d in DS.items():\n", + " print(f\"{k:24s} {len(d):>7,} rows columns={d.column_names}\")\n", + "H.plot_counts({k: len(d) for k, d in DS.items()}, \"Rows per dataset\");" + ] + }, + { + "cell_type": "markdown", + "id": "cell-012", + "metadata": {}, + "source": [ + "### Sample browser\n", + "\n", + "Pick a dataset and a row. Bases are colored by letter, and the positions where the variant differs from the reference are highlighted: single-base substitutions in `variant_effect_coding`, insertions and deletions in `variant_effect_non_snv`. Only a window around the first difference is shown." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "cell-013", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "edd3c2487aa64d4b8c57521daa5fa4e7", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "VBox(children=(HBox(children=(Dropdown(description='dataset', options=('kegg', 'variant_effect_coding', 'varia…" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.sample_browser(DS)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-014", + "metadata": {}, + "source": [ + "### How long are the sequences, in tokens?\n", + "\n", + "The model doesn't see base pairs. The NT tokenizer turns DNA into **6-mers** (≈ bp / 6), and every DNA span is padded or truncated to exactly `max_dna_length = 1024` tokens. That fixed length is what makes the splice in section 10 work. Text goes through the OLMo tokenizer." + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "cell-015", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABpoAAAFKCAYAAAANA4zPAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAA+xtJREFUeJzs3XdUFFcbBvCHXpYmRaqCYAVExC427BULYo0BTUyMGk2iUaNJLFFjjCYxGo0tGgs27CVq7EHF3guoKB2k97Kw8/3Bx8SVBRZEV8PzO4dzmLkzd98Zlml37nvVBEEQQERERERERERERERERFRB6qoOgIiIiIiIiIiIiIiIiN5NbGgiIiIiIiIiIiIiIiKiSmFDExEREREREREREREREVUKG5qIiIiIiIiIiIiIiIioUtjQRERERERERERERERERJXChiYiIiIiIiIiIiIiIiKqFDY0ERERERERERERERERUaWwoYmIiIiIiIiIiIiIiIgqhQ1NREREREREREREREREVCmaqg6AqCIOHDiA5ORkAICWlhaMjY1Rv3591K9fv8zlu3XrBltbW7myLVu2wN3dHa6urnLzpVIpAgICYGlpiZ49e76eDXnB9u3b4ebmBmdn59f+WfT63L17F7GxsejWrZuqQxFdv34dGRkZ6Nixo6pDeecIgoA///wTXl5esLe3f+Of/+DBA9y8eRPDhw9/459dnrt37+L+/fsYMmTIa/uM1NRUHD9+HDKZDMOGDSt1uZiYGJw/fx5aWlpo3749zMzMSiwjlUpx/fp1hIeHw9bWFi1btoSWllaZn3/69GmEh4ejc+fOqF279itvT2UkJSUhODgY6enpsLKyQuvWraGnp6eSWMqSnZ2NnTt3omfPnrCyslJ1OGVKS0vD3r170bdvX5ibm6s6nDKdOnUKZmZmaNKkidx8mUyGa9euISwsDFpaWmjatCnq1KlTYv2LFy8iMjKy1P/TixcvIiQkBPb29vDy8ipRvnv3bmRkZKBVq1Zo1KiR0nEfPXoU9vb2FVqHqLq6cOECQkNDAQDq6uowNzeHi4uLSq47qlJYWBjOnTsHf39/VYeilJCQEFy9ehUjR45UdSjl2rZtGzp37gxLS0u5+dnZ2QgODkZcXBxMTEzQqlUrhddEe/fuhZmZGTp06KCw/r179yItLU3hsT8rKwu7du0CAPTv3x81atSooq0qP74zZ87g2bNn6NWrFywtLcU4FRk+fDh0dHTE6RevA7W1tWFvbw8XFxdoa2srXL+iy78uO3fuhLOzc4lnFaWRyWTYunUr+vbt+1r+NlR1Hj58iODgYLRp0wYNGjRQdThvnejoaPz9998l/pffRgcOHICLiwucnJzk5kulUly6dAmRkZGQSCRo0aIFrK2tS6x/4sQJSKVS9OrVS2H9J06cQFRUFFxcXNCiRQu5MkEQsGXLFhQWFqJr166ws7NTOu5du3ahTZs2FVqHSGkC0TukdevWgrOzszB+/Hhh3Lhxgq+vr2Bvby80adJECAwMVLi8RCIR/Pz8SpSZmpoKS5YsKTF/7969gkQiEczNzYXU1FSl4tqxY4dw586dCm+PIAhCrVq1hBUrVlRqXXo75OfnC02bNhW2b9+u6lDk3L17V7C2thZiYmJUHco7RyqVChKJRNizZ49KPn/VqlWCjY2NSj672L1794Rt27aVmL9kyRKhXr16r+UzExISBH9/f8HR0VFo3LixUKtWrVKX/eOPPwRzc3NhyJAhQr9+/QQrKyvhyJEjcsvs2LFDqF+/vtCkSRPBz89PaNKkieDm5iZcuXKl1Hrv3LkjmJqaChKJRDhw4ECVbZuyCgoKhK+//lowMzMTevbsKXzwwQdC27ZtBTs7O2HTpk1vPJ5iGRkZwoYNG4S4uDi5+VFRUYJEIhHOnj2r0jiU8fDhQ0EikQjXrl17DZFVnWfPngmWlpbC48eP5eYHBwcL7u7uQsOGDYXRo0cLQ4YMEWrWrCkMGTJESEpKklv2008/LfP/9NNPPxUkEolgb28v5Ofny5U9ffpUMDAwECQSibBq1aoKxb5+/XqhTZs2QmFhYYXWI6qOxo8fL1haWgrjx48XPvnkE6FPnz6CsbGxMH36dFWH9kq2bdsmSCSSN/qZjx49Ev78889Krbt27VqhZs2aVRxR1duxY4fg5uYmSKVSufkbNmwQbG1tBU9PT+HDDz8UevbsKZiZmQnffvttiWOxu7u7MGbMmFI/w93dXZBIJMLQoUNLlG3evFmQSCSCRCIR7t69WzUbpeDzX45v/fr1gqGhoTB37ly55Ro3biyMHz++xE9GRoa43I4dOwQnJyfBxcVFGDVqlDBq1CihQYMGQp06dYTly5eX+PyKLv861alTR/jll18qtI6/v78wZcqU1xQRVZWhQ4cKEolEeO+991QdymsTGxsrbNiwQcjKyqrwusePHxckEomQkJDwGiKrOsHBwYKdnZ2QkpIiN//w4cNC3bp1BXd3d+GDDz4Q+vfvL5iZmQnjx48XcnJy5Jbt37+/0LVr11I/o3///oJEIhHatGlTouzs2bPiMfnle+DyzJ49W/D19a3QOkTKYkMTvVNat24tjBgxQm5efn6+MH/+fEEikQhbt24tsXydOnUEAwMD4datW3JlpTU0+fj4CL179xYsLCyEdevWKRVXvXr1FNalDDY0vfvWr18vuLq6vpUP1oYOHSpMmjRJ1WG8c9jQJAgrVqxQ2NDzOhuaoqOjhR07dgg5OTnCV199VWpDU2hoqGBsbCz88ccf4rzZs2cLtra2QnJysjjPz89PWLhwofi/KZVKBW9vb8HFxUWQyWQl6s3LyxNatWolTJo0SWUNTZ999plgZWUlXLhwQW7+H3/8IUgkEiEgIOCNxyQIRY0fEolEOH/+vNz85ORkYfz48UJISIhK41DGu9LQ9OGHHwoffPCB3Lx79+4JFhYWwvjx4+UahqKjowUPDw+hffv2Ql5enjhfmYYmZ2dnwc7OTjh06JBc2YIFCwRPT89KNTRJpVLB0dHxrXvxguhtNH78eKFRo0Zy81atWiVIJBLh4sWLKorq1V26dEkYP378G/3MzZs3C0ZGRpVa911oaCooKBCcnZ2FjRs3ys3ftGmTIJFISswPCgoSatasKUydOlVuvjINTW3bthVMTEyExMREubKePXuK54Y31dD066+/CgYGBiUaXMrbDkEQhC1btggGBgbC77//LnfNV1hYKKxatUro3LnzKy3/ulWmoenevXuCsbGxEB4e/pqiolf1/PlzwcTERBg8eLBgampa4kWh/4riRpCoqKgKr/uuNDR169ZNrgFcEATh1KlTgqGhoTB//ny548j9+/cFR0fHEo34yjQ0tWrVSjAyMirxYvvHH38sHpMr2tCUlJQkmJqalrjfJKoKTJ1H7zwtLS3MmjULN2/exKxZs+Dr6yuXFqlr16548OAB5s6di927d5dZV3x8PP7++29s2LABlpaW2LJlCz744IMy1wkMDEROTg6uXbuGjRs3AoDYrR8A0tPTERQUhLS0NDg6OqJly5ZQU1Mrs84jR44gIyMDgwcPhoaGBoCiNBTXrl2Dmpoa2rdvL5cyoTiV1eDBg3Ht2jU8ffoUtWrVQps2bUrUHRsbi6tXr6KgoABNmzaFg4NDmbEU1z1w4EBcv34dz549Q506ddCyZUuFyysT56BBgxAUFISoqCh4enoqTPtTXqzKxhUeHo7Tp08DALS1tVGrVi20bNlSrht2RfafIqtWrcKIESOgrv7vsHcvbmtxfLVq1ULr1q2hpqaGe/fu4e7duzA3N0eHDh3kvrPlxSyTybBt2zY4ODjA09NTXO/Zs2c4c+aMXAqr9957Dx988AHmzZsHY2NjhfFHRkbi5MmTGDx4MAwMDOTKdu7cCScnJzRr1qzcv4kib8t+KI9MJsM///yD6Oho1K5dG61atSp1W8r7nijznausqjoOFBYW4ty5c4iNjUXt2rXRtm1bbNq0SUwV+ODBA1y+fBn5+fnice3lFFvFKbwq+v9SFhsbG6VS8m3duhVGRkZ4//33xXkTJ07E0qVLsX//fjFdz9SpU+VSjmhqasLX1xfjxo1DZGRkibR48+fPh0Qiwbhx47B+/fpX3p6Kun//PtatW4eZM2eW2J+jR4/GgQMHMGvWLAwaNAg6OjqIj4/HX3/9hSFDhkBfX19c9sSJE9DV1UW7du3k6khNTcWFCxeQlpYGFxcXuLm5yZXn5eXh/PnzSExMhJOTEzw8PKCmpobs7Gzx/Hn06FEx1ZO/vz90dHTQokULGBkZydUVHR2N4OBg8VjxYorbF1PYpaWl4c6dO9DT04Onp2eJY9CLyopDmc9VJD8/H4GBgahZsya6du0qzr927RoePXoEIyMjdOzYERKJRCw7efIkNDU10bx5c1y9ehXPnz9H48aN0bBhwxL1P3jwAPfv34eBgQGaN2+uMJXRi54/f47du3djz549cvO//vprGBgYYMmSJXLHSRsbGyxZsgT9+vXD1q1bMXr06DLrf5G2tjYGDRqEgIAA9OnTR5y/fft2fPzxx7h582aJdaRSKc6fP4+YmBhYWlqiXbt2csc2TU1NDBs2DKtXr8bQoUOVjoWIinTp0gVA0Tm9devW4vzIyEhcvnwZhYWF8PT0lEsJHhsbi2PHjgEo+h+sVasWWrRoIXdeAIqOL1euXEFeXh7c3NxQt27dEp9/584d3Lt3D7q6umjbti1q1qwplhWnmBsxYgRu3LiBJ0+ewNraGp6ennL3Nubm5nIpfpRdDyhKBRQUFITIyEjY2tqiffv22LZtG1q0aFHq8TwsLAznz5+HIAjiNYuVlZVcCvSytkuRuLg4HD16FM2bNxevI/Ly8nDhwgXExsaiVq1a8PT0lLv237p1K5o3bw4TExNcv34d+fn5aN26dYk0d4IgIDg4GJGRkbC2tkaLFi2gq6tbZjxHjhxBQkICBg0aJM7LycnBrFmz0Lt3b/j5+ckt7+npiQkTJuDHH3/E2LFjyz0XvqhZs2bIzc3Frl27MG7cOABF37/z58/j+++/V3huSE1NRVBQEDIyMlC3bt0SKZ4qY+HChVi0aBF++eWXcu/HX5adnY2ZM2dixIgR+Pjjj+XK1NXVMW7cOLlrpIouX5by9oWy35MXHTlyBAYGBiVSCoaGhuLixYsYOXIkNDU14ezsDDc3N6xfvx5z585VKl56s3bs2AF9fX2sWrUKLi4u2Llzp/h/BsinjXv8+DEePnyIGjVqoF27diXSN6anp+PixYtIT09Ho0aN5O55tmzZgpYtW4r/+8UpTTt16iTev4eGhorH5mLx8fEIDg5Gbm4uPDw8UK9ePYWx3blzB48ePUL9+vXF5wQv1lF8Ttq1axdMTExgYGCAwYMHi8s8efIE165dg7q6Olq1aoVatWqVud/S09Oxd+9eNGjQQDw3FhYWIjg4GBEREbCwsECHDh3k9tH+/fthZ2cHR0dHXL9+HWlpaWjWrJnC9LQ3btzAo0ePxPOXoaFhmfHcuXMHFy5cwO+//y43/8svv4SzszNmzpwpd35r1KgRvv32W0yYMAGnTp1C586dy6z/RVZWVrC2tkZAQAAWLlwIoOiYtX//fnz99dcKj8nZ2dkICgpCYmIiatWqhbZt24rPFQHA1NQUvXr1wurVq6vkHp7oRWxoov8MX19f/PXXX7h+/brcQ2I1NTXMmTMH/fv3x4ULF9C2bdtS6wgICICBgQF69+4NIyMjDBgwAA8fPlT48KjY7du3kZeXh+joaFy5cgUAxAfAp0+fxqhRo2Bvb486dergn3/+Qf369REYGKjwob9MJsO0adMQGBiIHTt2QENDA4WFhZg8eTICAgLEsXYmTJiA3377TTxZHzt2DD///DNWr14NbW1tWFhY4OTJk+jYsSMCAgLE+v/88098+eWX8PT0hJGREebNm4cBAwZg9uzZpW5fcd3Lly+Hrq4uzMzMcObMGXTr1g2bNm0ST6AVifO3336Drq4unJyc0KhRI4UNTeXFqmxcycnJ4t8lLy8PN27cQH5+Pnbv3i1eeCm7/xR59OgRHjx4ID4YeHG//fTTT/jxxx9hZWUFIyMj/PXXXxgwYAD09fVx6dIluLi44OLFi7C2tsbff/8tPjwsL2Z1dXUkJyfjs88+w+nTp+Hq6ors7GwMGTIENjY2cjebXl5eyM3NxYkTJ+Dj46NwGwwMDDB16lSoq6vLPbi/f/8+xowZg4MHDyr1N1HkbdkPZcnOzsaAAQMQGhqKdu3a4cmTJwrHb1H2e6LMd66iqvI4kJmZCW9vb4SFhaFdu3YICwuDmZkZTp8+jc2bN8Pe3h7Pnz9HZGQkpFKp3LYUNzRlZ2ejW7duZX5OcSNIeQYOHFhqI2hpbty4gcaNG8tdMJubm8Pe3h7Xr18XGx4U5bV/+vQpNDU1SzxkCg4Oxu+//45z586V+zLA63Lw4EEIglDquBZ+fn547733cPHiRXTq1AlPnjzBxIkT0a1bN7kHiitWrBAbAYrt3LkTkydPhqOjI+rUqYPp06ejb9++WLlyJYCim83evXvD0tISDRs2xLNnz6ChoYHdu3ejoKAAt27dAlDUcJKUlASgqIEnJSUFEydOxJEjR8SG3WXLlmHu3Llo27Yt9PT0MGHCBIwcORLLli0DUPQAb+LEidi7dy+ioqLg5uaGmzdvIicnB6dPn1aYwxwo+g6WFocyn/uylJQUjBgxAhkZGeK4EwkJCRgxYgQePnyI9u3bIyoqCpMnT8a+ffvg4uICoOjlgvDwcBQUFKB+/foQBAFjx47FvHnzMGnSJLH+iRMnYt++fejYsSMKCwvx5ZdfYu7cuRg4cKDCeICiBjQNDQ25xvPMzEycOnUK48aNUzhOl5eXFxwcHHDgwIEKNTQBwMiRI9GzZ08kJyfD1NQUFy5cQHR0NAYPHozp06fLLfvs2TMMGDAAUqkUzZo1w82bN1FQUIDdu3fLjeXRtWtXLFu2DLGxsaX+LYlIsQcPHgCA2JAkCAJmzJiBNWvWoH379tDW1sbEiROxePFi8diXlpYmnqvz8/Nx584dpKSkYOfOneI4b7t378b48ePRsmVLmJubY8GCBejUqROWLl0KoOg64/3338epU6fQsWNHJCUl4cMPP8SyZcvEsYv++ecfTJ8+HVu3boVUKoWNjQ3OnDkDNzc37N+/X2x0uXz5MiZOnCjGp+x6eXl58PHxwa1bt9ChQwdER0dDW1sb165dw9KlS0u9hkpMTERYWBgEQRD3Q7169dCzZ0+ltutld+/exeDBg9GhQwfxAWzxGE7a2tpo2rQprl27BhsbG+zZs0d8IPnFF1+gadOmiI6ORtOmTREVFYVx48Zh165d4vk4IyMDffr0QXJyMlq2bInnz5/j+fPn2LRpU5n3mwcOHEDLli3lHn4GBQUhOTm51Otdf39/LF68GIcOHcIXX3xRat2KjBgxAgEBAeID8ICAAHh4eCj8Gxw7dgyjR4+Gk5MTateujX/++QcuLi7YtWtXmS+PlGXmzJlYuXIl1q1bV6kxQc+fP4+kpKQy133xGrGiy5dGmX2hzPfkZbdv38batWvx8OFDuZdNZs+eXeI70LVrV+zZs4cNTW+pzZs3Y8CAATA3N0e/fv2wZcsWuYam+/fvY+LEidi/fz+eP3+O+vXr4+LFizA2NsapU6fEF58uXLiAwYMHw8XFBbVq1cIvv/wCW1tb7Ny5EwDwxx9/4Pr16/jpp5/E6V9++QVTp07FnDlzAAArV67E/fv3xeNc8XV0s2bNYGZmhsmTJ+PTTz/FrFmz5GLbs2cPEhIS4ObmBn19/RINTZmZmXj48CEA4NatW9DX14eZmZl4z/rll1/ijz/+gJeXF/Lz8zF27Fh89dVXmDZtmsJ9FhkZCR8fH1hbW2Pz5s0Aihqqhg0bhpSUFLRp0wb379+HIAg4dOgQbGxsAPz7AmFSUhLc3NyQlpaGDz/8EGvWrBFjKSgowNChQ3Hjxg106NAB6enpePr0KVasWCF3Lf6yffv2wdHREY6OjuK8kJAQPHz4EIsXL1Z4LzlixAh8/vnn2L9/f4UamorXnTFjBr777jtoaGjgwIEDkEgkCsdZvX79Onx9fWFiYgJnZ2cEBwfDzMwM+/btk3sJt1u3bpg2bRqkUmm54xcTVYjqOlMRVZyi1HnFrl69KkgkEmHHjh1yy3/00UeCIBR19+/WrZtYpih1noeHhzBhwgRBEIpSFDg5OQmzZs0qNy5FqfMyMjKEunXrCp988onYbTYmJkaoU6eOXBqD4tR5mZmZgq+vr+Du7i6EhYWJ5UuWLBEMDQ3lxr/4448/BBsbGzFF1JIlS0qkDizeHy+m3nB2dhaWLl0qTstkMuHYsWNlbltx3S92C75z545gYmIi93kViXPlypVlfqYysSob18tkMpkwatQoYfDgwSXqKm//KbJ582bBwMBAyM3NlZtfXOeLY9ysX79ekEgkwqhRo8RUXtHR0YKxsbHCMcbKilkQBGHEiBFCkyZNhLS0NGHs2LFCgwYNSqS5EARBaNmyZbn5uv38/Ep0254xY4bg7Owsfn9f5fvzNuyH0ixcuFCwt7cXYmNjxc8ZPXp0idR5lf2elBZ3WV5OnVeVx4G5c+cKderUEce4kclkwocfflhie8tKnafM5zx+/Fhh7vyXf0oba6es1HmtWrUS3n///RLzO3TooHBcgWLPnj0TrK2thVGjRsnNz8zMFNzc3IQffvhBEIR/U6y96dR5Y8eOFczNzUstv3XrliCRSIS1a9cKgiAI58+fV5iWon///uK5TxCKxmszNjYWZsyYIc57+vSpYGlpKezbt08QBEGYNm1aibQwwcHBYkqP0lLWvTxG061btwQjIyO5sTLOnTsnGBgYCLt37xYE4d/9O3LkSKGgoEAQhKK0hS4uLuWec0uLoyKfe+3aNSEsLExo2rSpMHjwYCEzM1Ncx8fHR3BychKio6PFeR999JHQpUsXuWXMzMyEe/fuifOWL18umJmZienroqOjBYlEIpeSIiMjQzh37lyZ2zd+/HihY8eOcvPu3LkjSCQS4ffffy91vT59+giNGzcWp5VJnefu7i4IQlEKojVr1giCIAgTJkwQRo4cKeTk5JRIneft7S20b99e3F85OTlCt27dBC8vL7n0IGlpaYJEIinzeE5ERf/vtWrVEjZs2CD88ccfwpw5c4RatWoJQ4cOFa+P1q5dK0gkErkUl7t37xbMzMzkjlMvmzBhgtx1XZs2bYRvv/1WbpmjR4+Kv69cuVIwMzMT7t+/L85buHChYG5uLqbhKo7lt99+E5cJCQkRDAwM5Op6eYwmZddbtmyZYG1tLZf264svvhAkEomwYcOGUrdVEEpPnafsdhWnzjt+/LhgbW0tLFiwQFw+MzNTqFOnjtCvXz/xej89PV1o3ry5MHv2bHG5mjVrCs2aNRPS09PFeUOGDBF69eolTm/fvl2wsbGRG7fk2bNncvEp0rhxY+Hrr78usW0SiUTuXPSiwsJCoUaNGnJpDJVJnffpp58KUVFRgqGhoRiXu7u78Pvvv4sprYpT56WmpgoODg7CZ599JtYRGRkp1KpVq0S8ynB3dxfMzc1L3NMrWs7Ly0vYsGGD3E/xdWzxvnn69KncegcPHpRbvvjvWdHlFVF2XyjzPREE+dR5cXFxQo0aNcTrGUEoeq5gbGws7Ny5U269Q4cOCRKJpFJjWdLrdf36dUEikYjXgidOnBAkEolcSrTi/7EXx+pLTEwUrKyshNWrV4vzhg4dWmIc8hfTp82bN0+8zhOEonNA9+7dhQ4dOojz3NzchIULF8p97ovH6cuXLwtGRkbC9evX5ZYZP368whTkLyotdd7hw4cFiUQinDp1Spy3detWwcDAQBxD98XUedevXxecnJyECRMmiOPTFRYWCi1bthRatWoljo+Un58v9OnTR/D39xfrbd68ueDg4CDe4wuCIEydOlVwcXERpy9cuCBIJBK58+nz58/LHM9XEIquu1++nyzetr/++qvU9Ro3biz06dNHnFYmdV7//v2FnJwcwcbGRnz20q9fP2HmzJnivU3x314qlQrNmjUTfH19xf2VnJwsuLu7l4i3+P6ivG0lqij18puiiN4NxW+2y2QyheVz5szBhQsXcPz4cYXlly9fRkhICIYNGybWN2TIEGzbtg0FBQUVjuf8+fOIjY3FtGnTxDcarK2t8f777yMwMFBu2djYWPTo0QNpaWk4efKkXA+fzZs3o1evXnJd5f38/KCtrS2XWkcikWD48OHidLNmzaCnp4fHjx+L84yMjHDjxg3xLXA1NTV079693G1RU1PD559/Lk67urqiR48ectuhbJxaWloYO3ZsuZ+pTKzKxAUUvSF55swZbN++HX/++SckEon4xmMxZfafIjExMTAxMVGYFk0ikcilDipO6+fv7y++vWljYwM7Ozs8evSowjGvWrUKANCpUyfs2rULf/75p8K0TBYWFoiOji5zO95//31cvHhRjEMqlWL79u0YOXKk+P2t7PfnbdkPpdm1axdGjhwpvuGjpqZW6pufyn5PlIm7IqryOLB3716MGDFCTNGhpqaGzz77rELxKPM5Tk5O+O2338r9KStVSGlkMpnCN8XU1dVLPQekpaVh6NChMDExEd/gLjZz5kxIJJIKv/Fb1QoLC0ukxXhRcZkgCBWqd9u2bdDU1MQ333wjznNwcICPj4+YZsjIyAixsbG4e/euuEyrVq1gampaoc/as2cPLC0tMWrUKHFe+/bt0aZNmxLH5jFjxojnbm1tbTRv3rzcY25VfO6lS5fg5eUFLy8vbN++XXw7ND4+HkePHsXEiRPFtyGBorePg4ODcf/+fXGep6cnnJ2d5aZzc3MRGRkJANDV1YW2tjaCgoKQl5cHoKj3aPv27cvcjpiYmBK97Yq/02Wl39TR0anUtQoADB8+HAEBAcjNzcXevXsVvuWfkpKCkydPYuLEieL+0tXVxeTJk3H58mWEh4eLyxoZGUFXVxcxMTGVioeoOsnPz8eVK1cQHByMQ4cOQUNDA9988414fbR582Z4enrKpbccNGgQbG1tsX37dnGeVCrFP//8g507d2Ljxo3Q1tbG9evXxeOHkZER7ty5g/j4eHGdHj16iL8HBgbC29tbrnfiZ599hsLCQhw+fFicp66ujjFjxojT9evXR82aNcs9diuz3t69ezFo0CC5tLavel5WdrsAYN26dRg5ciSWLl2KmTNnivOPHj2K58+fY/bs2eJx2NDQEB9//DE2bNggV8ewYcPkeh15enqWuBfLycnBxYsXxXO5vb29XHyKVObcoK6uDi0trUqdG2xtbdGxY0ds27YNly5dQnh4uFzaq2JnzpxBQkKCXA9YOzs7jBgxosS5V1lOTk4wNDTEhg0bkJ2dXepyiYmJuHLlitzPnTt3APy7bzQ15RP43Lt3D1euXMHWrVsxceJEZGRkVGp5RSqyL8r7nrzM0tIS/fv3x7p168R5GzZsQI0aNdC/f3+5ZS0sLAAUPV+gt8vmzZtRq1Ytseeal5eXXC+dF3344Yfi72ZmZmjUqJHcd8TY2BhPnjzBs2fPxHm9evUSf+/cuTMePXqEqKgoJCQk4M6dO5g/fz5u3LiB5ORkhIeH48mTJ2LPmk2bNsHe3h6ffPKJWEeLFi3Qvn178V6h2CeffFLp7A+7du1C69at5XrijBgxAnZ2diWGuTh8+DB69eqFTz75BCtWrBD/P69fv4579+5h2rRpMDExAVD0jGny5MkIDAyU+z/t27evXC8eT09PhIeHIz8/HwDE1N9nzpwRj5UWFhZo3rx5mduh6JhcWFgIoOzrdW1t7Uodk3V1dTFw4EBs3boVMTExOHPmjNy9eLG7d+/i4cOHmDJliri/atSogbFjx+LAgQOQSqXissXHivKeExFVFFPn0X9GVFQUAJQ6JkurVq3Qu3dvzJ07F926dStRvnnzZujq6uLRo0fiSVxNTU0ct+nFE7cywsPDoaamVmIMEAcHByQkJCA7O1tMdbRu3Trk5+fj8uXLJR7qhYeHo169eiVO8Pr6+ggLCxOnjY2NS5zwtbW1kZubK06vXbsWM2bMQP369dGgQQN06tQJ48aNKxHjy0xNTUvkqa1duzZOnjxZ4TjNzc1LXMQrokysysR18eJFDB8+HObm5mjUqBGMjIwQHh6OxMREufWU2X+KFBQUyOVnL6vO4i7JxRdEL84vfhBZkZiNjIwwadIkTJo0CSNHjlQ4rlBx/cUXPqXp1KkTateujS1btmDu3Lk4evQokpKS5B7aVvb787bsh9IoGquntLGnlPmeKBt3RVTlcSAqKqpEHmxFuarLosznvM7UeTVq1EBaWlqJ+ampqQrTumRnZ2Pw4MFISEjAsWPHxAtrAEhKSsL69evx8ccfY8uWLWLsAHD69Gnk5eUpfLjyOtjZ2SEtLQ2pqakl/j8AiI0YdnZ2Fao3PDwcpqamYjqNYsnJyeL3Z9KkSYiLi0PPnj1haGgIT09PjBw5UmFKhvI+y97evsT3o06dOrh3757cvJe3UUdHB6mpqRX6vMp87ty5c2FlZYX58+fLpV8s3r+xsbFy/2vFN4RhYWFi49LLsRc3Ahb/D5iammLjxo1YsGABfvzxRzRt2hQ9evTA2LFjy8z7XlBQUOIGtbjR68XGnJdFRETIjdlSEcOHD8f8+fPx888/Q1tbG926dStxE1z82S8fG4unw8PD5co0NTUr3fBFVJ2Ymprit99+A1D0sNvHxwdDhw7FlStXoKenh/DwcLi4uJQ4/2tpaYnH79u3b2PQoEEwNDSEi4sLjIyMEBMTg7y8PGRkZMDY2BjLly/H9OnT4eLiAicnJ3To0AEff/yxOE5TeHh4ifFf9PT0YGVlhYiICLl5L48npKOjU+71sjLrRUZGlrjfsrW1Veq+oTTKbldubi5mzJiBgQMHlnh4V7zcjRs3xIYMAHj8+DGSkpKQlpYmXscoOje8uI29evXCtGnT8MknnyAvLw+tW7fGwIEDMWTIkFLvJ4CiB5gvlxcf8yMiIuDk5FRincTERGRlZVX63DBy5Eh8++23SE5ORo8ePRS+xBUREQEdHZ0S998ODg6IiopCYWGh3HlWGS4uLvj5558xcOBA+Pr6IjAwUGHa2BYtWoj/Oy8rPm9GRUXJXTMVNwJt3LgRFy9erPTyilRkX5T3PVHk448/RteuXRESEoK6deti48aN4ktnLyq+z+I5+O2Sl5eHwMBAODs7488//xTnOzg4YPv27Zg/f75c+rLyviPz5s3DtGnT0KpVK1hbW6Ndu3b44IMP0LRpUwAQU22eOnUKurq6aNCgAVq0aIG6devi7NmzSE1NhZGRkdigEhERAUNDQ7nYgKKXIV681wRQqZcEi0VERCi8z7a3ty9xnTt9+nS4u7vLvVgM/HtN+vjxY7lz4/PnzyEIAp49e4bGjRsDKLpnfJGOjg4EQUB+fj60tbXh4uKCX3/9FT/88AOmTp2KFi1aoE+fPvD39y/z5T9Fx7biY0dZ1+uRkZFwd3cvtbwsI0aMgLe3NxwcHNC4cWO4uroiJCREbpnSrtfr1KmDgoICREdHi2XF51YeK6iqsUcT/WccO3YM+vr6Zb598O233+LOnTslBtnOycnBnj170Lp1a1y9elV8Kyo1NRX16tVT+JZJeSwtLSEIgtj7o1hiYiKMjIzkxtOYNm0aunfvjr59+4qDmxczMTERB+998adTp05yb1Mrw9XVFYcOHUJERAQWLFiAkJAQtG/fvtwHe2lpaSVOQElJSXIXGcrGqezbL8rEqkxcs2bNQteuXXH16lVs3rwZv/32m9xboa+qZs2aSE1NLbUXRWUoG3NMTAzmzZuH5s2bY+fOnbh8+bLC+pKSksod9FhdXV3Mx15YWIjNmzejU6dOcg0Slf3+VFZV74fS1KxZs0Qj0Ks0Cr2O71xVHgcsLCxKHJdenq4KmZmZJeJV9FPejbUiLi4u4jgWxXJycsQHci/Kz8/HiBEjEBoaioMHD5YY/FxLSwt+fn7Izc0t8Ubso0ePcPv27QrHV1nFbxUWj4v2sv3790NfX1/MGV58g/NyQ/LLbwCbmJiI4229+GNqaioOlm5oaIjly5cjPDwcO3bsQK1atdC/f38cPXq0QttgaWmp8P8nMTHxlW5Mq/Jz161bBw0NDQwcOFDurcfim/qnT5/K7acbN27Az89P4dhtZfH29salS5dw7949fPTRRwgICCh3rImaNWuW+H80NzeHu7s7Dh06pLA3W0hICEJCQtC1a9cKxVesVq1aaN++PRYtWgRfX1+FD3WL92Fpx8oX93FeXh4yMzPLPe8QkTx1dXUsWbIEkZGRWL16NYCih2TFYz+++NOqVSvxYdXs2bPh4eGBGzduYMuWLVi5cqX4gkTxMaNevXrYs2cPIiMj8eOPPyImJgbt27dHXFwcAMXH0OL7mNd57H6RouuT1NTUV3oIpux26erqYu/evTh06BC++uorueVNTEygrq6OGzduyP0NMjIy4OfnV+H4ZsyYgdDQUJw6dQpeXl6YMmUKFi1aVOY6ivZNu3btoKOjgwMHDihcp3h+Zc8N3t7eyMjIwObNm8UxXF5maWkpNmi+KDExEebm5hVuZCrWpk0b7N69G1evXsXQoUMrfK3Yvn176OjoKH0NU9HlFXld+6JY69at0aRJE6xfvx6HDx9GXFycwnEZi78nPAe/XQ4dOoS0tDQ4OTnJHUfq16+P1NRUpV7Oe5GlpSX+/PNPREZGYs2aNdDQ0ICXl5d436KlpYV27drh9OnTOH36tDiedOfOnXHq1CmcOnUKHTp0kGv8zMrKKnGuqVevXomxw15lLFtF992A4uv1Xbt24fHjx/D395friVPcePTgwQO5WMPDw+Hn51dmjyJFxowZg1u3buHKlSvw9fXFkiVL5MZcLW07Xj4mN27cGBYWFqXex508eRJZWVmVPia3bdsWNjY2+OWXX8o8JgNFY86+KDExEWpqaiVetizeFqKqxB5N9J9w/PhxbN68GVOmTClz0FFXV1f4+vpi/vz5cg9rDhw4gIyMDKxbt67EW0ibNm3C5MmTxYtERQwNDUs82GvVqhX09fUREBCAyZMnAyh6W2Dnzp3o1KmT3LI6OjrYunUrPvnkE3Tv3h379+8XB+/t168fHj9+jF9//VXuAjU9Pb1CF92CICAsLAxOTk7iwIHm5uZo06YNIiIiFL49X6ygoACBgYFiWsGUlBQcO3ZM7gRcVXFWJFZl4kpKSpJ7I7+goAA7duyoUDxlad68OQoLC3H//n2lBolVhjIxFxQUwM/PD05OTjh69Cg++eQT+Pn54fz583K94goKCvDw4UOlBogfNWoUfvjhB2zduhXHjx+XS8/wKt+fyqrK/VCWTp06Yc+ePZg2bZr4JtnWrVtfa9wVVZX/Xx07dsS+ffswffp0sa6Xe7oARce1yjQCFStOnfc6+Pj4YM2aNThz5ox4PN21axdkMplcCpHCwkL4+/vj2rVrOHz4sMJGOSMjoxJxhoSEYN++ffjoo4/Qr1+/17INinTs2BEdOnTAokWL0KNHD7kL/xs3biAgIACTJk0S0zwUNwQ/fPhQ7JUXFxeH27dvy6Vg9fb2xs6dO/HVV1/JvakrCAKePn0KoOitwLp160JDQwNubm5wc3PDzp07ce3aNbGXE1DUoFcWLy8vrFixApcvXxbTZEZHR+Ps2bPi4MOvorQ4KvK5NjY2OHbsGAYOHIg+ffpg7969MDMzQ926ddGoUSM4OzuXWOfZs2cV6kmWmJgIDQ0N1KhRAxYWFvDx8UFiYiK+/fbbMtdr1qwZjh07BkEQ5G7kv/76awwePBjr1q2TSz9bUFCAmTNnwsLCQi7NSkVNnToVDg4OcqmtXmRtbY2GDRsiICAAvXv3FucHBASgdu3acm/TFzfUvjw4NBGVz8nJCQMHDsSKFSswfvx49OvXD2fOnBF7HBbLyspCeno6gKLrjuLjHlB0bN+2bZtcvY8ePUK9evWgp6eHDh06wMHBAc7OzggNDYWVlRW8vLywZcsWLFy4UDzO7tu3D9nZ2ejYseMb2PKic+DBgwcxb9488UGhousTRQwNDVFYWCi+pV6sItvVvn17HD58GAMGDEB6ejqWL18OdXV1dO/eHVpaWvDx8SkxgPvjx48rlK65+FyiqakJJycnODk5ISgoCNevXy9zvWbNmpXonWtubo5x48bh999/h7+/v3jvCBT1zF68eDE6d+6Mtm3bKh3fi/T19TFnzhzcu3dPfCnlZW3atBHvY8eNGweg6AWfwMBAuWvhq1ev4v79+3j//feV/nxPT0/s3r1b7OW3Y8eOEr3iSmNubo6JEydi5cqV6NevX7nno4our4iy++JVfPTRR5g1axZu3ryJ7t27K8xIcOfOHdSsWbPcjBP0Zm3ZsgUdOnRQeG/09OlTbNmyBd7e3krXV3xM19bWRsuWLeHh4YE///wTt2/fhpubG4Ci49+PP/4IbW1t/PrrrwCKGpq+/PJLZGRkyKUI7devH5YuXYqFCxfKZZqQSqWVSoVcfK/y8jMyLy8vfPXVV4iIiBC/o9evX8fDhw9LXCM3atQIf//9N/r164ehQ4di69at0NPTQ+vWrWFubo4OHTqUeMZRfD+jrOjoaJiamkJPTw+2trZ477338ODBg1KH2yjWrFkznDlzRm6epqYmZsyYgSlTpuDYsWNy6WmzsrIwe/ZsNGrUCIMGDVI6vpd98803OHPmDHx9fRWWu7q6wtzcHAEBAVi4cCGAf68JWrVqJaa/BoqOFRoaGpXuYUVUGjY00TsnIiICGzduhEwmQ3JyMs6fP4+zZ89iwoQJmDVrVrnrz5o1C82aNZN7K2Lz5s1o3bq1wrR7ffv2xaRJk7Bjxw5MmDBBYZ1t27bFjh07YGZmBn19ffTq1QtWVlaYN28epk+fjqdPn8LJyQn79u1DQkKCwofOGhoaWL16Nb788kv07t0bu3btQtu2bTFv3jwMGDAAXl5e6NevHyQSCUJDQ3HixAls3bpV6TcQBEHAkCFD4OzsjGbNmkFdXR3btm1DixYtyu0Roaenh6+//hp37tyBubk5Nm/ejJo1a8rtj6qKsyKxKhPXkCFD8Ouvv6KwsBA1atTA3r17xfRIVcHNzQ3W1tY4d+5clTU0KRPzt99+i5CQEFy4cAFaWlpYtmwZOnbsiLFjxyIwMFB8QFncY0RRusiX2dvbo0OHDpg6dSqMjIzkHrC/yvensqpyP5Rl1qxZaNeuHXr27IkBAwYgNDQUf//992uNu6Kq8v/rxe319vbG48ePxYvpF/dXq1atkJeXhylTpsDV1RUODg5VdqNcli1btqCgoAD37t1Dfn6+mBLBy8tLvKH29PTE6NGj4e/vj4kTJyInJwfLly/H119/LXfTPXPmTBw4cACjR4/G9evX5R7k9OnTR+6trrfFpk2bMHz4cLRp0wajRo2CjY0NHjx4gC1btmDEiBFyN2E2Njbo27cvJk6ciE8++UQcW+3lBzHdunXD2LFj0bFjRwwbNgy1atVCTEwM/v77bwwfPhyTJk3C+vXrERwcDC8vL1haWuLChQtISUnBgAEDABSld2rYsCF+/PFHPH36FJqamvD39y8Rf48ePeDj4wMfHx989NFH0NXVxYYNG+Dq6oqPP/74lfdPaXFU9HPNzc1x+PBh+Pr6okePHjh48CCsra3xxx9/YNCgQXj48CHatm0LmUyGmzdv4tatWwgODlY6hdPz588xZMgQeHl5oVGjRkhNTcW6devKfcjWo0cPTJ8+Hbdu3ZK78evZsyeWLl2KGTNm4PTp02jfvj2ys7Oxa9cupKamYvfu3SUedubk5JRItwVAYQ/L4jGryrJs2TIMGDAAQ4cORYcOHRAcHIzDhw9j+/btculezp07BwcHBzRs2LDM+ohIsS+++AKBgYHYtm0bZsyYgStXrqBDhw4YMGAAjI2NxfP277//DmtrawwdOhTffvstdHV1YWlpiYMHD5YY8/LDDz+EtbU1WrRoAR0dHTGFU3EmiGnTpuHw4cPo3LkzRowYgaSkJKxevRrjx48XUzG9blOnTsW+ffvQrVs3DB48GFFRUdi/fz80NTXLvZ5r1qwZdHR08Pnnn6N58+awtrZGz549K7xdHh4eOHbsGLy9veHn54f169fD1tYWv/32G/z8/DBgwAA4OzsjJSUF58+fh6WlJf744w+lt/HChQtYsmQJevbsCXt7ezx+/BhHjhzB+vXry1yvZ8+e+OabbyCVSuWOt3PmzEFSUhK6deuG9957Dw0bNkRMTAw2bdqEevXqlRhDCih6qP3yuUFTUxPvvfdeiWXLO2/XqlUL3377LWbMmIFHjx7B3t4ee/bsQWZmJubNmycut2fPHvz2228VamgCinpt7dq1C4MHD8bw4cOxfft2sRFS0XYARffu5ubm+Pbbb5GZmYnu3bvD29tb/K4/evQIgYGBaNSokdz1UkWXr+y+eBVDhgzB119/jQsXLpQ6Bta5c+fkHnCT6sXExODkyZP45ZdfFJYPGDAA06ZNkxtDrzxz5sxBeno6PD09YWxsjKNHj6JmzZpyvWU6d+6MadOmQVtbW+yV1L59e0RHR6OgoECu4Xz06NE4f/482rVrB19fX9SsWRPh4eE4evQoZs2aVeE06/Xr1xf/D7t27QpjY2MMHjwYY8aMwa5du9C9e3eMGTMG+fn5WLNmDfr27auwoa1OnTpiY1P//v0RGBgIIyMjbNiwAX5+fggKCoKHhweys7Nx9epVJCcnV+g+PjQ0FJMmTULPnj3h5OSEuLg4rF+/Hl9//XWZ6/Xo0QMrV65EQkKC3P3kxx9/jNjYWAwfPhy+vr5o1qwZkpKSEBAQAD09PQQGBpZIyff8+XOFx7IXx7gu5uvrW2ojE1A0HuySJUvwwQcfIDExEW5ubjh+/Dhu3rxZotfcuXPn4OnpWWZKb6LKYOo8eqd4e3vDzc0NV65cwc2bN5Geno7BgwfjwYMHWLBgQYnc1d7e3iW6+jo6OuL777+Hn58fGjdujNzcXDg4OJTI/VrM1NQUM2fOLHOMmx9++AGTJ09GWFgYrly5gszMTADAuHHjcPLkSZiYmODJkycYOHAgrly5Ivem+fDhw8UH9WpqaliyZAlmzpyJvXv3Ii0tDSYmJjhx4gSmTp2KlJQUhIeHw83NDUFBQeKba40bN1aYjmf48OFo0KABgKJ0HJcvX4aPjw8SEhIQHx+Pzz//HEePHi33wVmNGjVw7tw5WFlZISoqCqNHj8aZM2fk3oh4lThfpmysysQ1a9YsrF27Fjk5OYiJicHEiROxfft2+Pn5icsos//KivWDDz4o8eaoojpNTEwUpl8aOHAgPDw8lI45Li4O6enp+PPPP8U37CUSCbZs2QIrKyvcvHlTrGvbtm3o1auX0m/iT506Fb6+vpg7d65ct/PKfn/elv1QFhsbG1y8eBE9evTAkydP4OjoiAsXLsDPz08uv7Gy3xNlvnPladSokdwYAVV1HACK8kcHBweja9euePLkCRwcHLB3714AkHuDrUGDBjh27BgMDAxw7do1Ma3nq/y/KKM4/YGdnR18fHzE6eTkZLnlli9fjhUrVuD58+fIycnB7t27MXXqVLll7Ozs4OfnB5lMViIVRPFxWpHi72hFb6qqgrm5OY4fP441a9YAKBp7IzAwEC1atMCKFStKnOc2b96Mr776CjExMdDQ0MCOHTvw5Zdfljj3zZ8/H4GBgZBIJAgJCYGpqSnWr18v9gD9/vvvsXTpUmhra+PRo0do0aIFrl+/LpeK8ODBg+jatSvu3LmDK1euACh649nPz0/uRY2NGzdi5cqVyMzMRFxcHL7++mscP35cbnw2RceAdu3aKdUoriiOynyukZER9u3bhx49emDLli0QBAGNGzfGtWvX0KNHD0RERCA5ORn9+vXDlStXxGNily5dSrwJb2pqCj8/P7EnpbOzMy5cuAAPDw88ffoUUqkU69evx+LFi8vcNicnJ3Tu3BkBAQElyj7++GPcunULbdu2xaNHj5CQkIDPP/8cN27cKPE2Ytu2beHt7a0wXWVmZibatm2LgQMHlhqHpqYm/Pz85Aao9/T0xJUrV9CsWTOEhobC2dkZly5dQvfu3eXW3bZt2yv1riKqLtq2batwDEA3NzfMnDkTSUlJ0NfXx+HDhzF79mxkZmYiLCwMDRo0wKlTp8Q0qhMmTBBTH0dERGD06NHYt2+fXAqh06dPY9SoUUhLS0NMTAw+/PBDnDlzRkzlbWRkhKCgIIwbNw4REREQBAHbt2/HDz/8IMbVoEEDjBw5skS8gwcPFsfDAIrutV685lF2PXNzcwQFBaF///4ICwuDhYUF/v77bxQWFopvx5fGzs4OJ06cgIWFBa5du4aHDx9WeruK36I3NjbGoUOHAADDhg1DUFAQnJycEBoaCjU1NcyYMUOukWnkyJElroNevp4bMWIE9uzZAysrK4SGhsLc3Bznz58v83gMFD1clMlkJVK7aWlp4ffffxfHn7x//z6AonFVjx07VuIFhIEDB6JRo0YKzw3F5WX1gLK1tZU71wHA5MmTcfToURgYGCAsLAy+vr7idVyxyMhItG7dusxtLP78l8fU6tChAwIDA2FjY4MjR46Uux3FPSg0NDSwdOlSBAcHw8PDA2FhYYiKikK9evUQGBiIK1euyGVCqejyiiizL5T5ngBFD5lffolRT08PXbp0Qe3atUuce4GinmynTp3iOfgtExYWhlGjRpXaY6l///4YOXIkwsLCxP+xlxs1e/bsKfe/uXXrVkyZMgX5+fl4+vQp+vbti0uXLsldjzds2BDjxo3D9OnTxecjhoaGmDZtGsaNGyc3rq2GhgY2bNiA3377DTKZDKGhobCzs8P+/fvF81RpsSmir6+Pv//+G66urrh165aY0k9TUxNHjhzBnDlz8Pz5c6Snp+PXX3+Vu+59+XOsra1x7NgxuLq6ir1cvby8cPXqVTRr1gxPnjxBTk4ORo8eLdcTqX///iV6J9aqVQt+fn7ivYGXlxdOnDgBR0dHPHr0CFpaWjh48CAmTpxY5vZ16tQJTk5OCnvdzpkzB8HBwahfvz4ePHiAzMxMLFiwABcuXChxb9mtWze0a9dO4bFMKpWiW7duZd4bFd/bvDjcweDBg3H+/HnUrl0bjx8/RqdOnXD16lW5Xq/5+fnYu3cvjxX0WqgJipK9ExH939KlS7F69eoSY0ep2tsUV2pqKpo0aYItW7agffv2qg5H9Pz5czRp0gR//fUXu0STSBAEPH/+XC4P9urVq/HVV1/h8ePHSqccpDfn5MmTGDRoEGbMmFFi7Aj677l06RJ8fHxw+/btd+7/8e+//8a4ceNw8+ZNviFJRBUWFxcn96B0x44dGDt2LO7fv1+h9KX/RUuWLMGxY8deqde9qtSrVw+//vorevXqpepQ3mmZmZlo1KgRpkyZgs8++6xE+bfffov79++X2tuJiKrOzp078d133+H69etyPU3fBevWrcO6detw/vz5Vx4/juhlTJ1HRPSKTExMsHz5coSEhLxVDU23bt3C119/zUYmkiMIAvr16yemort79y62b9+Ob7755p17qF1ddOnSBatWrcKFCxcQExMDGxsbVYdEr1GrVq0wdepU3Lp1642kq6xKjx8/xooVK9jIRESVMnToUDRv3hxOTk4ICQnBli1b8MUXX1T7RiYA+PTTTxEdHY3o6GjY2tqqOhylZWRkYODAgaWO80Tly8zMxK5du3Dw4EFIJBKFvRBkMhlycnKwaNEiFURIVP0MGTIEd+/exYMHD8Rxsd4V8fHxJcZ+Jqoq7NFERGU6fvw4zp07h/nz56s6FDlva1z0dike70cRU1PTCg26WlXehpiSk5Oxa9cuPH78GJaWlujatSsbJImIiEil0tPTsWvXLjx8+BDm5ubo1KkTWrVqpeqwiFQqKSkJs2fPhqmpKUaPHi2Xhp+IiOhtwoYmIiL6z5o8eXKpjTp2dnYqSUP2NsZERERERERERERUWWxoIiIiIiIiIiIiIiIiokpRV3UARERERERERERERERE9G5iQxMRERERERERERERERFVChua6J0gCAJkMhmY6ZGIiP5reI4jIqo+eMwnIqoeeLwnouqGDU30ThAEAbExESgsLFR1KColk8kQHfUMMplM1aGoHPdFEe6Hf3FfFClvP8TFxWHBggWIi4t7w5FRaYrPcbwJJSL67+Mxn4ioeuDxnoiqG01VB0BERERvTlxcHL7//nv06dMHVlZWqg6HiIiIiIjotZDJZNizJxAnT/yN9PR0NGvWHB+O/RhGRkYAgNjYWKxevRIhDx/CwMAAvXv3xcBBPuL65ZUTEdG/2NBERERERERERERE/ynr16/Fvbt38fnnU2Hv4ICbN67j8uVL6Nq1GwoLC7Fw4XeoX78BVq9Zj8jISPywaAFMatSAl1fncsuJiEgeU+cRERFVI1ZWVvjqq6/Ym4mIiIiIiP6z4uPjcfzYUUya9BnqN2gAHR0dtGrdBl27dgMA3L17B8/j4+HvPwZGRkZwcXFBt249cPzYUaXKiYhIHns0ERERVSNWVlaYNWuWqsMgIiIiIiJ6be7duwsDAwPcvHUDc+fOhrq6GtybemDMmA9gaGiE8GfPUNPSEoaGhuI6TnXr4siRQwBQbvnLpFIppFKpOC0IRWPmVvcxhIno3aeurlxfJTY0ERERVSPp6em4fPkyWrZsKeYmJyIiIiIi+i/JSE9HRkYGnj17hhW/rUROTi5++XkJVv62AtNnzER2TjYk+vpy60j0JcjPz0dhYWG55RoaGnJle3YHYufO7eK0rq4OFi9ejPi4qNe3kUREb4CtnYNSy7GhiYiIqBoJCwvDgAEDEBQUBHd3d1WHQ0REREREVOX09PQgCALee28UDA2NYGhohEE+vvhx8SIUFhZCX08fWdnZcutkZWdBW1sbGhoa5Za/bJDPYPTz7i9OC4IMqSmJsLSyU7o3ABHRu4wNTURERERERERERPSf4VCnTol5giBATU0NAGDv4IDn8fHIyMgQ0+M9efwYDg51lCp/mZaWFrS0tMRpmUyG1JSilFNsaCKi6oBHOiIiIiIiIiIiIvrPqFevPpycnLBly2ZkZKTj+fPn2LsnEC1btoKGhgZcXRujpqUlNm78AxkZ6bh//x5OnDiObt17AEC55UREJI89moiIiIiISKH09HQYGBjwTVwiIiJ6p6ipqWH6jFlYu+Z3fPzRh9DV1UWLFi3xvp8/AEBDQwMzZ36D1b+vxNgPx8DAwAD9BwxE585dlConIiJ5bGgiIiKqRrS1teHo6AhtbW1Vh0JE7wA3NzecOXMGDg4Oqg6FiIiIqELMzMww46tZpZZbW1tjztzvKl1ORET/YkMTERFRNeLs7Izbt2+rOgwiUlJihhTJWYVVXq+pRAPmhlrlL0hERERERERUDjY00TulzbxHyJaqOoq3QYiqA3iLcF8UeTP74d73jd7I5xARUZHkrEIEXEyp8npHtKlRqYamtLQ0SCQSaGoW3Ubk5+cr7CFZUFAATU1NFBQUIC8vDxKJRKkyIiIiIiIievcw2ToREVE1cvfuXdjb2+Pu3buqDoVeE1laImRpiaoOg/5jZDIZPvvsM3z00UcoLCzE4sWLYWFhARMTEzRr1gz3798Xl/voo49gaGgIOzs7TJ06FT4+PuWWERG96xIzpAiNy1X4k5jBtyWJiIjov409moiIiKqRgoICJCUloaCgQNWh0GtSmFnU+0Xd2FzFkdB/RW5uLoYMGQJ7e3ts374dhw8fxqlTp3Dt2jUYGRlhx44dmDBhAk6fPo1t27bh2rVriIyMhI6ODvr37y/2eCqrjIjoXVdWD9TK9iIlIiIielewRxMREREREZVq4MCBaNKkCZYuXQo1NTUcPXoU586dg5ubGxwcHDB9+nRER0cDAP755x+MGTMG5ubmMDQ0xIQJE8R6yiojIiIiIiKidxcbmoiIiIiIqFTDhw/H7t27ERcXBwBQV1fHtGnTkJqaKv6EhoYCADQ1NSGV/psiKi8vT/y9rDIiIiIiIiJ6dzF1HhERERERler9999Ho0aN0KVLFxw9ehS+vr4YOHAgWrRogVatWkFTUxNaWlqQSCTo3r07pk6dik6dOkFHRwdLly6FhYUFAJRZRkRERERERO8uNjQRERFVI3Xr1sXJkydRt25dVYdCRO8AIyMjqKurw9fXF3p6ehg0aBD279+PNWvW4LvvvsOTJ08glUrRp08fbN26Fd7e3rh9+zYGDRqEmjVromfPnggLCwOAMsuIiIiIiIjo3cWGJiIiomrEwMAArVq1UnUYRKQkU4kGRrSp8VrqVcbt27fF3/v27Yu+ffsCAAYNGoRBgwYpXOfTTz/FxIkTERUVhU8++QTDhw9XqoyIiIiIiIjeTWxoIiIiqkaio6OxfPlyfPrpp7C1tVV1OERUDnNDLZgbaqk6jApxc3NDeno6zM3NMWDAAIwdO1apMiIiIiIiIno3saGJiIioGklISMCKFSswbNgwNjQR0WsRHh5eqTIiIiIiIiJ6N6mrOgAiIiIiIiIiIiIiIiJ6N7FHExEREREREdH/PX/+HJcvBSMjMwPW1jZo29YT2traYnlmZibOnTuL1NQUODo6oVWr1lBTU1NhxEREREREqsUeTUREREREREQAQkIeYtKn4/Ek7Ak0NDRw5PBBTJ8+FXl5eQCAtLQ0TJ3yOS5fDoYgE7Bxwx9YsXyZiqMmIiIiIlIt9mgiIiKqRszMzDB27FiYmZmpOhQiIqK3zokTf8O9qQcmT/4cANCvX3+M9h+Fe/fuwsOjGfbv3wsDAwN8880caGhowKtzZ0z6dAJ69eqDuvXqqTh6IiIiIiLVYI8mIiKiaqRWrVr4+eefUatWLVWHQkRE9NYxNDSEND9fnC4okEImk8HI0AgAcPvWTbRu3QYaGhoAABsbW9jbO+DmrRsqiZeIiIiI6G3AHk1ERETVSHZ2NkJDQ1G/fn3o6+urOhwiojcuKSkJn376Ke7evYtt27YhIyMDM2fOhEwmw5kzZ1QdXrl69OiBTZs2wdLSUtWh/CcNGjQYK1euwLx5s2FtZY2QkBCMHDlK7K2UlJQMU1NTuXVMTU2RkpyssD6pVAqpVCpOC4IMACCTyV7TFpCqCIIAQRBKLePfnP5r1NX57joREf2LDU1ERETVSGhoKNq1a4egoCC4u7urOhwiKocsLRGFmSlVXq+GQQ2oG5tXeb1l8fb2xrJly1CnTp1K19G7d2+sXr36lXplLl++HBKJBFu3boWTkxPatWuHyZMno2XLliqNS1khISFiw0VV7FOSFx7+DM+ePkXjxo1hYlIDBgYGuHHzOrp26waJxABqasDLTQmCIABqagrr27M7EDt3bhendXV1sHjxYsTHRb3GrSBVyM6VIDsrU3FZlhZiY+IrXGceJEjLVVxmrAvoIKvCdRJVFVs7B1WHQEREbxE2NBERERERvaUKM1OQd2ZXlder08n3jTc0LV26FLa2tq9UR2hoKPLy8l65jsGDB6Nx48bi9NChQ6Gtra3SuCqjKvYpyVu3dg06duyEocOGAwB8ZL6Y8sVnOPrXX/AZ7AtzcwskJiTIrZOQmABX18YK6xvkMxj9vPuL04IgQ2pKIiyt7Ngb4D8mMz4P+hKpwjJ9iQGsLSs+Puaj+Dzsu634ZYMRbWrAoRJ1EhEREb0OvLIlIiIiIqISdu/ejWnTpsnN69atG54+fQoAGDlyJFxdXdGyZUuMHz8eSUlJ4nI9evTA7t274e3tja+++goAMGXKFERHRyu17r59+9CvXz906tQJx48fBwAsXLgQkZGR6N27N1xdXfH48eNSY09OTsakSZPQpk0beHt74/z58wCAr7/+GocPH8bnn38OV1dXuLq6IjMzEx4eHvjyyy/LXBcoSrs3efJktG7dGq6urlizZk2F4lK0PgBERkbC398frVu3xpgxY8T9BABPnz6Fr68vOnXqhI0bN8rV9+I+LW2/AcCTJ0/k6ujRowfi4yveu6I6UFNTQ/4LYzTJZDJIpVKoqRf1WPJo1gwXLgSJvcqePHmM6KgoNPXwUFiflpYW9PX1xR89vaK0terq6vz5j/2oqamV+fO21Mkf/lTVDxER0YvYo4mIiIiIiEro1KkTPv74Y8yePRsSiQRBQUFITEwU07R99913yM7ORl5eHgICAvDll1/ijz/+AFCU3m3VqlWYO3cu7O3tAcj3+lFm3W+++QZhYWEYMWIEYmNjMXr0aPz+++9YtmwZatWqVWqaOkEQMHDgQAwZMgT+/v5iHVeuXMEnn3yCy5cvo2/fvujcuTMAoFmzZti6dSvMzMzKXNfc3Bx9+vSBs7MzfvnlFxgYGMDS0hIFBQVKxSWTyRSuX1hYiN69e6NHjx745ZdfsHXrVvTt2xfXrl2DIAjo27cvBgwYgClTpmDdunWIivo35dqL+7S0/aauro4+ffrA19dXrOPkyZNy4wbRvwb7DsHyX39BdHQUzMzN8fDBAxTKCuHl1QUA0K9ff1y6FIyvZnyJOo5OuHrlMvr16w97ewfVBk5EREREpEJsaCIAQG5uLrKzs2Bqyq73RPRuO3z4MJydneHk5FTushEREXj06BG6dOnyBiJ7PQ4dOgQXFxeF43MoKlNXV4ehoWGl3kI8cOAAPD09YWbGcwVRdWBmZoaOHTtiz549GDVqFDZu3IjRo0eL5WFhYfj1118RERGBnJycEqnnFi5cWOq4R+Wtu3jxYjRp0gTt2rXD/PnzERkZCUdHR2hra6NevXqoW7duqXGHhobi0qVLSEpKwqpVqwAU9VK6fv06evbsCSMjI9SuXRuurq4AinqwuLi4QFNTEyEhIaWu6+DggIiICJw/fx4aGhpyn6lsXIrWv3fvHlJTU/Hjjz9CTU0NLVu2hI2NDR4/foz8/Hzk5eVhwYIFAAAPDw8EBASU+hmK9lt2djYKCwvx3XffKVVHdde2rScaNGiIu3fvICsrC+7uTeHu3hRaWloAAIlEgh9+WIKrV68gNSUFXTp3RcNGjVQcNRERERGRarGhiQAAly4F48D+fVj60y+qDoWI3kFSqRRHjx5FeHg4CgsLMXbsWOjrF6WGuXHjBq5evQqJRILu3bvD3LxoTJDLly/j4sWLAIDmzZvD09Oz1PqPHDmCevXqoV69euXGsmXLFrz33ntKNTTNmjULH330kTj96NEjnDt3DmZmZujZsyd0dXXLLUtJScG+fftQp04ddOrUSVz+woULMDc3R/369cuN41Vs2bIFfn5+ChuaXiz766+/4OTkBDc3N8TGxlbqs/Lz8/H9999jyZIlrxo2Eb0j/P398csvv8DHxwf79u3DokWLABSlgBs2bBh+/fVXNGzYEJGRkZgyZYrcujVr1lRYpzLrGhsbi79ra2vLpTIrT3p6OqysrLB9+3a5+cU9qyq77sOHD2FiYlKikUlZGRkZCtfPysqCsbEx1NSKUrOpq6vD2NgYmZmZkEqlMDExEZfV1tYWz6+KKNpvWVlZFaqDihtZO5Varq2tjbZtS79uISIiIiKqbphUlYiIXklWVhY6d+6MxYsXIzw8HPHx8SgsLAQArF69GoMGDcLdu3dx6NAheHh4iCl/srKyEB8fj927d+PYsWNlfsbWrVtx9+7dKo07IiIC9+/fR/v27QEAGzZswPvvv487d+5g1apVaNOmDbKzs8st69+/Py5evIhp06Zh9+7dAIDU1FTMmTMHtWvXrtKYX0V6ejpyc3NfqY7+/fvj0KFDSEtLq6KoiOht16tXLzx48AA///wzOnbsKL4sEB8fD4lEAh8fHzRt2hRBQUFK1/kq60okEqSnp5e5jLOzM7KzsxEaGiqOw+Tq6gpDQ8Ny6y9rXWdnZyQkJODo0aOVjkvR+o0aNUJ0dLT48sXZs2eRkJCABg0aoFGjRggLC8P169cBFPUsfXE8K2U0bNgQT548we3btwEU9fytaB1EVCQxQ4rQuFyFP7n5MlWHR0RERKQy7NFERESv5LfffoOxsTH2799f4i3tEydOYPbs2fD39wcAdO3aFVevXoWdnR28vLzg5eWFmTNnlln/3bt38fjxY/z111+IiIhAjx490LBhQ9y5cweXL1+GtbU1evToIb4J/qK8vDxs2LABffv2hZ2dnVzZ/v370bVrV3G6WbNmuHDhAtTU1CAIAhwdHREaGgp3d/dSy+rXr4/c3FysWbMGJ06cwOHDh+Hj44NvvvkGM2fOlOsR9bJDhw6hYcOGCAsLQ1xcHHr06AFLS0sAwOnTp3H79m1oaWmhfv366NKli9z2Xbp0Cffv31fYC+zSpUsIvngevfv0k5tvZGQEXV1dPHjwAAMGDMCvv/4KTU1NREREoHv37rC1tRWXDQ4OxoMHD+Dp6YnQ0FAx/Z6WlhZat26NY8eOYciQIWX+3Yjov0FTUxPDhw/H7NmzsW/fPnF+w4YN4erqCltbW+jo6JTZK/Vlr7LuoEGD0LVrV9jY2GDfvn0KU9VJJBJs374d/v7+mDx5stjLJygoSK5njyLlrbt582b4+/tDV1cXBgYGmDRpEj766COl4ypt/dWrV6N3794wMTFBWloa1q5dC4lEAgD49ddf0aFDB5ibm8PZ2RlWVlZK7y+gqJfTkiVL0LZtW1hYWKBRo0YwNTVlryaiSkjOKkTAxRSFZX2aGL3haIiIiIjeHmxoolKlpKRAXU0Nxi/ckEulUuTn50EiMVC4TkZGBiQSCQRBQFJSIszNLeTGASmvnIjePX/99RfGjRuHwMBASKVSdOvWTWwwmThxIr777jvExsbi+fPnkEgkco07ysjNzUV+fj5SU1MRHx+P3NxcbNy4EXPnzkXfvn2xceNGrFy5Evv375dbLyUlBcOHD0efPn1KNDIBRSn9OnbsKE67ubnh6tWr+Oeff3Dv3j20a9cOjRs3LrNMQ0MDhYWF+OabbxAcHIwRI0bgwoULkEqlcmn0FNmyZQsePXoEFxcXyGQyzJ07F0FBQbC2tkZaWhri4+NRUFCAwMBA7Nu3DytWrAAArFu3Dt9//z169+6NTZs24fnz52KdxWWdOnbA/gOHkJCQIJZt374dffv2Rb169RAdHY1p06ahadOmAIDvvvsON2/ehJGREX7//XcsWbJErv4lS5aIqflcXV1x7do1NjQRvSEaBjWg08n3tdSrrNmzZ2P06NFo9MI4NOrq6jh8+DAiIiJgamoKHR0dPHv2TCw/fvy4XAM2ABw8eBD29vYVXnffvn1iD9HZs2dj3LhxSExMRK1atUqNuXPnzggPD0dUVJTYC7O4R9PPP/8s1+B0/fp1aGpqKrVuz549ERMTg2fPniE7O1s83ykbV2nrDxkyBAMHDkRUVBTs7OzE8YAA4L333sOgQYPEtH6hoaFiY1PxPi1vv40ZMwZDhw5FRkYGQkNDMWLECJiampYaJxERERERUUWwoYkUOnToIA4dPICvZs6CsYkJUpKTsXr1Kty5cxuampqoUcMUkyZ/DkdHRwBARHg4li5djKSkJGhoaKJL167Yv28vNmzcDCMjo3LLXyaVSiGVSsVpQWAaAqK3VXR0NJYvX45GjRohJycHX331FU6fPo26detCEASkpaUhOjoacXFxAACZrGL/z82bN4ezszMGDx6MgQMHAih6o33Xrl1o3rw5CgoK0Lx5c5w9exYN6v//mBQRgb59+2LKlCkYNGiQwnpTU1NLHH+ysrIQGxuLZ8+ewcbGBoWFhWIvrdLK9u7di4CAAIwZMwYDBw5E//79sXXrVly4cAEPHz5Enz59xAeJL+vRowcWLlwIAJg6dSrWrFmD2bNnw9vbG4aGhggNDYW1tTV++OEHLF++HGpqali4cCEOHDgAV1dX5ObmioPZA8DChQuxb98+mJkaoYZpTbi5uZW6Xzt16oRly5YBAPr164dLly6hW7du+OGHH3Do0CG4uLiUqB8oejM+JCSk1HqJqGqpG5tD3dhcpTEYGRmVOBYUezFF6Ivj6Ckan+7lcfaUXffl3kGWlpalHldfpKamhlq1apVo+Hl52tnZWel1gaJGtuJr4MrEVdr6WlpaCsfbAwB9fX2xB9KL+0fZ/bZ06VJs2LAB2dnZSE1NxcaNG8uNk6i6SsyQIjmrUGEZ0+MRERERKcaGJpIjk8mwYcN6PLh/Dwu/XwRTUzMAwM8/L4WlpSU2bNwMbW1tHD58EEuX/IBflq2AhoYGfv55Kdzdm8LPfwykUil+/PEHuTrLKldkz+5A7Nz57yDMuro6WLx48evZaCJ6JTVq1ECfPn3w1VdfAQAmTJiAzZs3Y+7cufjyyy8xb9489OrVCwAwfPhwbNiwAZMnT67052VnZyMhIQEeHh4AitI6eXh44OnTp2JD09KlS9GzZ89SG5mAogenWVlZcvM6duyIjh07QhAEtG/fHgcOHMDgwYPLLKtduzZmzJgBAFi0aBH8/Pxw+/ZtfPHFF/D09MTy5ctx9epVhYPHF/coAooa1IrHqvL390dYWBhatGgBPT095OXliWNCpaWliQ98dXV1xd+zsrLEstiYCLkyRZycnMTfa9asidTUVGRlZSE9PR0uLi4l6i+WmZkpN9g8EZGquLu7o6CgQG5emzZtsHbtWhVFVORtjGvkyJHo0aMH9PT0YG9vL9eDi4jkMT0eERERUcXxDoNE+fl5WPzD9ygoKMB38xdCT6/orcm4uDjcu3cX/v5jkJpadMHdokVLbN8WgMiICGjraCMyMgILv18EdXV16OjowNd3CG5cvwYAiImJLrNckUE+g9HPu784LQgypKYkvsatJ6LKcnd3h46Ojjito6ODwsKit0Bzc3PlxinS19dHbm5uhT9DU1NT7Amlr68PExMTPHjwAC4uLhAEAffv34ev77+ppRYtWoSAgADMnTsXs2fPVlinm5sbHj58KE4/ffpUfJNcTU1NLm1RWWXFHj16hCtXrmDGjBn47rvv8NVXX2Ho0KHo1asXYmNjFabve/Hz79+/D1tbWwiCgEOHDiE6Ohp6enp49uwZfv31VwiCAAMDA+jr6+PJkydwcnJCQUEBQkNDARSN/VFcpq+nJVemLIlEIn6mg4MDCgsL8ejRI7llHjx4IJdykN5SMhlkaYkq7wlD9Dpt3boVgiDIzTMwUJze+U16G+OysrKq8NhOREREREREymJDE4kyM7Pw4MEDjBz5ntjIBADJyUlQU1PD4sWL5JaXSAyQkZkBzVwt6OjoyK1jYvJv3v+0tPQyyxXR0tKSe5Ark8mQqvilMiJSsU8//RR9+/ZFcnIycnJysHPnThw/fhxAUQ+m8ePHY+TIkYiLi8ORI0cwffp0AEUNM0eOHMH169ehrq6OZcuWwdvbW2HaoHr16mHNmjWIjIxEz5498cUXX2Do0KEYNmwYbty4AU1NTXTp0gUJz2MAAHp6etixYwf8/PwwY8YMLFq0qESdffv2hb+/vzj9zTffQCKRoE6dOggODkZqaiq6d+9eblmx6dOniz0vmzRpgsWLF+Phw4eIjIwsNZXSzp07kZ6eDplMhl27duHMmTNQU1ODq6srxo4di0aNGuGvv/6Ctra23P728fGBr68vLl26hLy8PLmywYMHo1fP7rh3/6FcWTEHBwc0b94cNWvWVBjTJ598gkGDBsHHxwdXr15FTk6O2BtLJpPhwoULWLBggcJ16e0h5GSgUF2dDU30n1bc+/Jt87bGRURERERE9LqwoYlEpqammDT5M8ybOwf5Uin69u0HALC2tgEAzF+wEObmFuLyMpkM6urqSEhIQG5uLhISEmBhUVQeGRkhLlezZs0yy4no3ebq6oq//vqraGwgMzMEBQXBwcEBADBz5kw0b94cV65cQb169RAcHCwOWp6Xl4f4+HgxBV58fLzChhEA+Pzzz2Fubo6oqCjk5ubi888/R5MmTRAcHIy+ffti2LBhYhqgvn37wsnJCTo6OtiyZQt+/vln3LhxQy5NHVA0loW1tTWuX78ODw8P/Pnnn9i7dy8ePnwIX19fDBo0CHp6egBQZhkAhIeHY8SIEeJ4GN7e3igoKMDDhw+xZ88ehT2gAGDu3LnIzMxEfHw8Tpw4IY7ZERgYiC1btkBTUxN//vkntm7dKjY2TZs2DY0aNcK9e/cwc+ZMPH78WGycmzZtGho2bIjg4AuYOXMmnjx5Ipb17t0bDRo0gImJCT788EO58Zt69OiBBg0aAABmzZoFFxcXPHz4EF9++SUmT54MG5ui88Dx48fRvn178VhORERERERERESkJryc14GqpbNnz+DA/n1Y+tMviImJwdw536Jbt+4Y7DsEALBq1W8If/YUw4aPhJmZGSIjI7F/317MX/A9tLS0sGD+PEilUrz33vvIyc3B2jWrER0dhY1/boahoVG55eWRyWSIjYnAgDU5yJa+7r1BRKW5930jVYdQquLjhLVNbairqyu1TkhICEJCQuDt7f2aoytp2LBhGDlyJPr161el9Za3H+Lj47F582aMGjVKYU+rhw8f4tixY5BKpQgKCkJGRgZOnDgBNTU1BAQEoFOnTmLDE1WNynx3yyKNfgQhKw1qEmNo2darggiJiKiqVPUxn6peaFxumWM0Hb6VXuGyEW1qoL6VrsKyysZS2TqJ6M3g8Z6Iqhse6QhA0YDvNUxNAQA2NjaYv+B7XLlyGUf/OgIA+OijcejYyQv79u7B0iU/IvjiBXw8brz4lv7nX0yFtY0Nflu5HPv37cUgn8EAAE1N5cqJiFShQYMGKmlkAv7tefWmxcbGYs6cOYiNjVVYnpubi/j4eGRkZGDgwIE4fPgw1NTUAAAjRoxgIxMRERG98xIzpAiNy1X4k5svU3V4RERERO8cps4jAECrVq3RqlVrcdrCwgI/LF4iTmtoaKBXrz7o1auPwvV1dXXx8cefiNPnzwfBxKSGmFqqvHIiourmvffeU3UICrm7u8Pd3V3VYahEaGgIoiIjUcPUFI0bu4npGIs9f/4cd+/chraONtzdPWBgYFChciJ6e4SHhyMkJAStW7eGkZERrl+/jrS0NHh5eak6NCJ6A5KzCsvstURE/w0PHz7Ay4mc6tdvII5BCxSldI+KioSBgaHCjA/llRMRURE2NFGVOHPmNDIzMtDE3R3x8fHY9OcGdOnaVelyIiIiVUlOTsLiH76Hmpo6bG1t8ejxIxQUFOC77xbA1NQMAHDpUjCW/fIT3N2bIjMzE+vXrcO87+ajVq3aSpUTERAUFISmTZtCIpFUuo5//vkHzZo1g76+fqXrOHnyJEaOHIkmTZpgxYoV+O6773Dw4EF4eHhUuqGpKuIiIiKiqjVn9jewtbWDru6/aSa//ma2+NLztWtX8euyn2FkbIzUlBQ0cnbB1KnTxPFxyysnIqJ/saGJqkS7du2xY8c2LP91GXR1ddG3nzf69OmndDkREZGqyGQyfDj2Y9StWzSeUWFhIb6c+gX279+H0aM/QGFhIdasXoVhw0bAu/8AAMBPS3/Exg1/4Jtv55RbTvQqEnIzkJiXVeX1mutIYKFrWOX1luXPP/+Eg4PDKzU0jR49GkePHkXdunUrXcfOnTuxaNEi+Pv7AwACAgJw69YtmJubqzQuIiIiqnqfjJ8gXue/KCcnG78u+xm+Q4ahb99+yMzMxPTpU7Fv3x4MGTKs3HIiIpLHhiaqEtra2hg1yg+jRvlVqpyIiN4MY2NjDBgwAMbGxqoO5a1hbm4Bc3MLcVpDQwNmZmbIy80DADx+/Aipqanw6txFXMarcxcsXPAd8vPz8fRpWJnlfOORXkViXhY2Pg2u8nr967Qut6HpyZMnKCgoQIMGDcR5wcHBqF+/PkxNTREcHIzU1FTo6+vDxcUFZmZm4nJnz55Fy5YtERkZiby8PDRu3Bh+fn6oUaOGWE9560ZERCAjIwPNmjWDmpoaHjx4gOzsbPzzzz94/Pgx2rVrV2aKypSUFNy/fx8WFhaoX78+AODOnTu4f/8+rKyscPToUQBAYmIiLl++DFtbWzRp0qTUdYulpqbi7t27yMzMRP369ZGXl6d0XKVtW7G4uDg8fvwY9erVk0vPU956ihQUFOD69etISUmBIAjo2bNnmXWV9/cmIiJ6FyUlJkJdTR2WVpaQSP49P9+8eRMFBQXo0aPo/GhgYICuXbrhzJlTGDJkWLnlREQkjw1NRERE1UidOnWwZcsWVYfxVgt78gR37tzG19/MBlA09pKuri4MDf99KG9hURMymQzJyUnllltZWcvVL5VKIZVKxWlBKBp0XCarosHHhX9/qqxOUhlBEEqMLVBV9Zb3/Xj69ClmzpyJ4OCihq7U1FT06dMHT548gUwmw/bt2/HgwQPk5+fj/v37WLt2Lfr27QsA8PPzQ4MGDZCeno6BAwfCxcUFH374Ifbs2YOGDRuWu66zszOSk5MRFxeH1q1bIyAgAKdOnUJaWho2bNgAPT09NGjQoNRUddu2bcOsWbPg6OiI8PBwtGnTBps2bcKxY8fw6NEjZGVlidtVUFCAX375BV5eXmjcuHGp6wLA1q1bMXnyZDg7O0MikeD9999Hamqq0nGVtm0A8Ntvv2H27NlwdnbGvXv3sGDBAowbN67c9RTJyclB69atoaWlBQuLoob07t27l1lXeX9vZairqyu1HBER0ZvQsGEjHDiwH9k52YiNiYFX5y748MOPoKGhgdiYGNSsaQktLS1xeRtbW8TGxgJAueUve+3X+EREKqLsNT4bmoiIiKqR/Px8JCQkwMLCgj1tFIiKisKCBd9h2PARcHVtXDRTEKCuriG3XPGFlkwmlF/+kj27A7Fz53ZxWldXB4sXL0Z8XFSVbINBVia0CgogzcpEZkxEldRJqpOtLkV2VmbV15udidjMvDKXadjACXFxsTh39hTq1auLzZu3olOnDsjKTEVWZiq+nPoZwsLCEB0dg3v37mPhwgVo5uEGACgsLED79m0x2r+oN3tsTAQKCqRIeB4LYyP9ctft2LEd3hs5Ajk5OWjeojXu37uNAf374Mcff8TCBfNQp46DWO/LMjIy8MUXn2PevDmQ6EsgE2SYNesbHDt6GMOHDcbp0ycxaOBA9OxZ1PDiVLcB1q5ZCU1NTYSG3Ct1XQcHe0ycOAHbAragSRM3uc9UJq6ytg0AZs6cicOH9sHJyQn3HzzAoEFD4NWpHYyMjEpdr0YNE4Wfc+/efRQWFuCvIwfEnk/FMZVWV3l/b2XY2jkotRwREdGbMGfud+Lv0dFR+ObrmbC2toG3d39IpdIS90M6OjooLCxEYWFhueUaGvLX/6/rGl8iMURWVsYr1UFE9CqUvcZnQxMREVE1cv/+fbRr1w5BQUFwd3dXdThvlWdPn2LevNkYMNAH3t79xfkmNWogOzsLubm54kDCKSnJUFNTQ40aJuWWv2yQz2D0e6F+QZAhNSURllZ2VdIboDBGCiGrEFoSAxja1H7l+ki10tLjoS8pPT1cZenrG8DayLLc5fz8/HHkr+NY1LEz9u0/iPnz58PapjZkMhl8fHxw48YN1K1bF4WFhUhKSob1/79zGhqa8PUdJk4DgKamFixqWsPSyq7cdX18hojTdna1oKGpA2ub2tDU1ERNSxu5el8WefkycnPzsGfPfnGes7MLjIxNYW1TG7q6+qhhav5CHWpi3WWtm5Kaidq17dGzV98Sn6lMXGVtW3JyMhwdHdGuvRcAwNqmNuzs7JCRmYsGDV3L3CeKmJlbwdbWDq3btEeHDh3w/vvviz2aSqvLxta+1L83ERHRu87W1g5t2nri7p3b8PbuD0MjI6RnpMstk56WBolEAg0NjXLLX/Y6r/GNjGu80vpERG8CG5qIiIio2gsNDcHCBd9h+IiR6NGjl1xZ/foNoKuri0uXgtGxYycAQHDwRdStWxd6evrllr9MS0tLLgWHTCZDakpRL6gqaWhSA/D/H6axevepqamVOxZPZetV5vsxevRodOjQAX5+foiLi0OXLl2gpqaG0NBQ3LlzB2FhYdDU1MSZM2fw4YcfytWpo6NT4jPU1dXx+PHjctfV1NSUmy6OV11dvdzYLSwsoKGhgX379kFPT6/UbX+xjuLpstZ98uQJ4uLiUFBQUOINZ2XiKmvbLCwsEB8fD5lMBk1NTeTn5+P58+cwNzcXly1tnyiiq6uLv//+G+Hh4Th9+jRGjx6No0ePimNQlVZXaX9vIiKid82LL4EVS3j+HIZGRgCAhg0aYv26NYiLixVTXd+5cxv1GzRUqvxlr/san4jobceGJiIiIqrWoqOjMHfOt3ByqovCQhmOHDkMADA3N0fLlq2go6ODkSNHYe2a3xEVGYnMzAycPn0K33w7BwDKLSd6lzk6OsLR0RGjR4+Gn5+f2OhgZmaG5ORkrFu3DgYGBliyZInSdb7KunZ2dti0aRM8PT3Rrl07GBiU7O3l5OQET09P9OvXD2PHjoWxsTEAoEuXLnIPgBQpa10nJyc0a9YMgwYNgr+/PwwMDFC/fn04OjoqFVdZ6tevDycnJ/j5+WHQoEHYuXMnXF1d4ejoWKF6iiUmJuLq1asAAA0NDairqyt8+/plpf29iYiI3jU3rl/D2XNn4enZDvr6+rh65TJu3bqJhd//AABwdHKCh0cz/LR0CQb7DkFUZCT++eccvp09V6lyIiKSx4YmIiIiqtYEQUAnr84AihqdFOndpy9q1a6NmzduwNjEBD8u+Qm1atVWupzoXTZlyhSsWrUK/v7+4jwLCwts27YNGzZsgLm5OZYsWYKAgACxvGPHjiV6BLVv3x4GBgYVXvfFhpuffvoJS5cuxdWrV9GoUaNSG3R2796NlStXYs+ePUhLSwMAtG3bFlpaWmjatCksLf9NG9i9e3e5N43LWnfv3r1YtmwZAgICkJ2dDT8/Pzg6OiodV1nbduTIEfz4448ICAiAs7Mz1q5dq9R6ioSHh+OXX36BmpoazMzMsHbtWri6uipVl6K/d3Vy/PgxxMbElJjfvn0HODo5AQAyMzNx7txZpKamwNHRCa1atWajHBHRW6ZNW0/o6evj7NkzyEhPh7W1NZb+tAx2dnbiMlOmTsO+fXtw5MghGBgY4utvZsPZ2UXpciIi+peaIAglR6kmesvIZDLExkRgwJocZEtVHQ1R9XXv+0aqDqFUxccJa5va1To1QXn74ebNmxyj6S1T1d9dafQjCFlpUJMYQ8u2XhVESKqUkJuBxLysKq/XXEcCC13DKq+X6F134cJ5JDx/Lk7HxETj5MkT+HX5StjY2CAtLQ3Tp02FlbUV6tWtj3/+OQcXFxd8Oukzpern9crbITQuFwEXUxSW9WlihMO30qu0bESbGqhvpauwrLJxVrZOInozeLwnouqGPZqIiIiqETc3NyQlJZWbPoqI3g4WuoZsECrF8ePHIZPJ5OaZm5ujefPmKoqoyJuM623dB++ytm095aZ/X7USrq6NYWNjAwDYv38vDAwM8M03c6ChoQGvzp0x6dMJ6NWrD+rWY+M+EREREVVPbGgiIiKqRtTV1aGjo6PqMIiIXtmyZctQWFgoN69p06Yqb2R5k3G9rfvgvyInJxv//HMWEyZOEufdvnUTrVu3Fce8srGxhb29A27eusGGJiIiIiKqttjQREREVI08evQIn376KZYvX456fCBGRO+ww4cPqzoEhd5kXG/rPvivOHv2LHR1ddGyZStxXlJSMkxNTeWWMzU1RUpyssI6pFIppNJ/c38LQlEPtJd7otGbJQgCShtF4HWVVeZv/jrqJKoqTAdHREQvYkMTERFRNZKVlYWgoCBkZVX9mC9ERET/JX8fP4bOXbpCU/Pf22Y1NeDlx/6CIBQVKLBndyB27twuTuvq6mDx4sWIj4t6HSGTkrJzJcjOylRYVlCgV+Vl2VlaiI2Jr9I4K1snUVWxtXNQdQhERPQWYUMTERERERER0QtCQh4iIiIc02d8JTff3NwCiQkJcvMSEhPg6tpYYT2DfAajn3d/cVoQZEhNSYSllR17A6hQZnwe9CVShWWamlrQlxhUaZm+xADWlmZVGmdl6yQiIiJ6HdjQRERERERERPSC48eOool7U9SsaSk336NZM1y4EIRBPoOhpaWFJ08eIzoqCk09PBTWo6WlBS0tLXFaJpMhNaUo5RQbmlRHTU0NaqX0QntdZZX5e7+OOomIiIheBzY0EREREREREf1fZmYmLlw4jy++mFqirF+//rh0KRhfzfgSdRydcPXKZfTr1x/29g5vPlAiIiIiorcEG5qIiIiqETs7O6xYsQJ2dnaqDoWIiOitlJ6ehuEj3oNHs+YlyiQSCX74YQmuXr2C1JQUdOncFQ0bNVJBlEREREREbw82NBEREVUj5ubm8Pf3V3UY9JrI0hKBvFxVh0FE9E6zsbGFt7dtqeXa2tpo29bzDUZERERERPR2Y0JfIiKiaiQxMREbN25EYmKiqkOh16AwMwVCQZ6qwyAiIiIiIiKiaoQ9muidcvHbetDUrL5fW5lMhtiYCFjb1K72A79yXxThfqCKioqKwsSJE+Hu7g5zc3NVh0NERERERERERO84PpUkIiIiIiIiIiIiIiKiSmFDExEREREREREREREREVUKG5qIiIiIiIiIiIiIiIioUtjQREREVI1IJBK0a9cOEolE1aEQEREREREREdF/gKaqAyAiIqI3p169ejh69KiqwyAiIiKqNgplAkLjchWWmUo0YG6o9YYjIiIiIqpabGgiIiKqRmQyGaRSKbS0tKCuzo7NRERERK9bZq4Mh2+lKywb0aYGG5qIiIjonccnTERERNXI7du3YWZmhtu3b6s6FCIiIiIiIiIi+g9gQxMRERERERERERERERFVChuaiIiIiIiIiIiIiIiIqFLY0ERERERERERERERERESVoqnqAIiIiIiIiIiISHmFMgGhcbkKy0wlGjA31HrDEREREVF1xoYmIiKiasTZ2RkhISGwsLBQdShEREREVEmZuTIcvpWusGxEmxpsaCIiIqI3ig1NRERE1Yi2tjZsbW1VHQYREREREREREf1HcIwmIiKiauTp06d477338PTpU1WHQkRERERERERE/wFsaCIiIqpG0tLSsG/fPqSlpak6FCIiIiIiIiIi+g9gQxMRERERERERERERERFVChuaiIiIiIiIiIiIiIiIqFI0VR0AERERERER0dskNCQEd+/egb5Egg4dOkJfX18sy8rKRFDQP0hNSYWjoyNatGylwkiJiIiIiFSPPZqIiIiqEWtra8yZMwfW1taqDoWIiOittHbN71i0aAGSk5MRFxuL2d9+DalUCgBIT0/H1Klf4HxQEPLy87Bu3Rr8tuJXFUdMRERERKRa7NFERERUjVhaWmLq1KmqDoOIiOitdOHCeZw7dxbLfl0BU1MzAMDz5/FQU1MDAOzftxcSfX3MnjMPGhoa6Ny5Kz6bPBE9evZC3br1VBk6EREREZHKsEcTERFRNZKamorDhw8jNTVV1aEQERG9dU6fOon27TsiMiISO3dsx6lTJ2FkZAxNzaJ3NG/duonWrdtCQ0MDAGBnZ4fate1x6+ZNFUZNRERERKRa7NFERERUjTx79gxDhw5FUFAQ3N3dVR0OERHRWyUmJhrPn8cjIjIcLs6uOHnibwTu2onFPy6FgYEBkpKSYGpqKreOqZkZkpOTFNYnlUrFtHsAIAgyAIBMJnt9G0HlEgQBgiC8NWWlfR9eR51EVUVdne+uExHRv9jQRERERERERARAEACZIGDu3PnQ0NBAYWEhJk2agGNH/4LPYF+oqQElHu0LAvD/1Hov27M7EDt3bhendXV1sHjxYsTHRb2+jaByZedKkJ2VqbCsoEDvjZZlZ2khNia+SuMsq06iqmJr56DqEJRWWFiIQwcPQCKRoGu37nJlISEPEfLwIQwMDNCqdWtIJAYVKicioiJsaCIiIiIiIiICYG5hDmMjYzE1noaGBuztHRAfH1dUbm6OxIQEuXUSExPh4uKqsL5BPoPRz7u/OC0IMqSmJMLSyo69AVQoMz4P+hKpwjJNTS3ol/Ig+XWU6UsMYG1pVqVxllUnUXW0f/9e7N2zGzVrWso1NAXu2okDB/bB07MdoqOjsX17AL7/fjHMzM2VKicion+xoYmIiIiIiIgIgIdHMxw/dgxSqRRaWlqQSqUIe/IEvfv0BQA09WiGCxfPY5DPYGhpaSEsLAxRUZFo6tFMYX1aWlrQ0tISp2UyGVJTilJOsaFJddTU1KBWSi80VZSV9l14HXUSVTcR4eE4dfIkevbqjevXronzkxITsXPndsya9S2auLtDJpNhzuxvsHPndnwyfmK55UREJI8NTURERNWIjo4OGjZsCB0dHVWHQkRE9Nbp1asPgi9ewIzpX8LZ2Rn37t2DsbExevToCQDw9u6P4OCLmDVzOhwd6+LSpWD07t0XDg4Oqg2ciIhKKCgowPLly/DBh2Px9GmYXNmt27cgkRjArUkTAEUvALRr1x67du1UqpyIiOSxoYmIiKgaadSoEa5evarqMIiIiN5KOjo6mL9gEa5dvYKEhAQ0cW+Kpk09xFR6EokBFi9eisuXLyE1NQVTO0wrNW0eERGpVmDgTjg4OKBpU48SDU2JiQkwMzOT6xloZmaOlJRkCIJQbvnLPQqlUimk0n9TXQqCDEBRT1YioneZsr2k2dBERERERERE9H+amppo1bpNqeU6Ojpo377DG4yIiIgq6smTxzh96hR++vmXUpcpkX7y/5OCIJRb/nLZnt2B2Llzuzitq6uDxYsXIz4uqlLxF5NIDJGVlfFKdRARvQpbOwellmNDExERUTVy+/Zt9OjRA8eOHYObm5uqwyEiIiKiKlYoExAal6uwzFSiAXNDLYVlRP8lO3Zsh62dLf46cgQAcP/BPaSmpiJw1050694DZmbmSE5OklsnKSkJNWrUgLq6ernlLxvkMxj9vPuL04IgQ2pKIiyt7F55zDQj4xqvtD4R0ZvAhiYiIqJqRCaTISMjgykciIiIiP6jMnNlOHwrXWHZiDY12NBE1ULr1m0QFxeLfGk+AKCwoBCAUDQtCHBza4LfV/2GO3duo3FjNwiCgAsXguDu3hQAyi1/mZaWFrS0/v3fkslkSE0pSjn1qg1NRETvAjY0ERERERERERER0X9G585d5Kb37AlEZmYmRox4T5w3cJAPli5ZjI4dOyEyKhKREREYP/5TAICFhUWZ5UREJI8NTURERERERERERPSf1bBhI+jp6cnNGzHiPTRu7IaQhw/RskUrfPbZFBgZGSldTkRE/2JDExEREREREREREf1nOTu7wNnZpcT8xo3d0Lhx6WPXlldORERF2NBERERUjdSvXx9BQUGoX7++qkN5q2z4Yx3inz8HANSuXVsupQYArF+/FgkJCXLz/P1Hw8rKGgAgCAL+OXcW169fg7a2Ntp36MgbUiIiIiIiIiKqFtjQREREVI3o6+vD3d1d1WG8dZq3aIns7GycPXsGDx88KFF+5/ZtNG3qgYaNGonzDA0Nxd+3bwvAiRPH4es7FBkZGVgwfx6+mPIlWrZs9UbiJyIiIlJGoUxAaFyuwjJTiQbMDbXecERERET0X8CGJiIiomokMjISP/30E7744gvUqlVL1eG8NYp7H4WGhuBRRobCZZzqOqFVq9Yl5mdlZWH//r2YMuVLtPh/w1JBQQG2BWxlQxMRERG9VTJzZTh8K11h2Yg2NdjQRERERJWiruoAiIiI6M1JSkrC2rVrkZSUpOpQ3jmnTp7ET0t/xOZNfyI6Okqc//jRIxQUFMC9qYc4r6mHByIiwpGVlamKUImIiIiIiIiI3hj2aCIiIiIqxwcfjkV2djby8/Jw9+4dfPH5ZHz9zWw0buyGtLRUSCQSaGn9+wawiYkJACAtLQ0SiYFcXVKpFFKpVJwWBBkAQCaTvXqgwv9//v97ldRJRKQkdXW+x0hUUWWlssvN53mciIiI3g1saCIiIiIqR3FqPQBo36EjoKaGXbt2oHFjN2hoaMg1HAEQp9XVNUrUtWd3IHbu3C5O6+rqYPHixYiPiyqxbEUZZGVCq6CgKIasTGTGRLxynUREyrK1c1B1CETvnLJS2fVpYvSGoyEiIiKqHDY0EREREVVQ7dq1cf/eXQCAlbU18vLykJKcjBqmpgCAuLg4aGpqwtzcvMS6g3wGo593f3FaEGRITUmEpZXdK/cGKIyRQsgqBABoSQxgaFP7leojIiIiIiIiIioPG5qIiIiqEQsLC0ycOBEWFhaqDuWdERsbi7S0VDRs2AgAkJOTg3/OnUX9+g0AAHXqOKJmzZo48tdhjBw5CjKZDMeO/oXmzVtAU7PkpZaWlpZcmj2ZTIbUlKKUU6/c0KQGQO3/E2pMY0VERERERERErx8bmoiIiKoRW1tbLFq0SNVhvHUOHz6IO3fuICoyEpmZGVi0aCHMzMwwduzH0NfTw+rVK5GakooaNWrg6dMw2NWqhff9RgMoasyZ+OlnWPzDQty+dQvZ2VkQBGDO3Hmq2RgF6fqIiIiIiIiIiF4XNjTRO6XNvEfIlpa/3H9fiKoDeItwXxR5s/vh3veN3ujnUdXJzMzEvXv34OLiAgMDA1WH89Zo1NAZ5ubyvbz09PQAAMYmJpgz5ztERUUhKTERFjUtYGNjK7esi4sLVv2+Fo9CQ6GlrYX69Rso7M30Rqjz8o6IiIiI6F2VkJuBxLwshWXmOhJY6Bq+4YiIiMrHJxFERETVyOPHj9GlSxcEBQXB3d1d1eG8NRydnODo5FTmMnZ2drCzsyu1XF9fH03e4X2amCFF8v/Hd3qRjqYa8goEheuYSjRgbqilsIyIiIiIiCouMS8LG58GKyzzr9OaDU1E9FZiQxMRERERITmrEAEXU0rM79PECIdvpStcZ0SbGmxoIiIiIiIiIqrmOEI0ERERERERERERERERVQobmoiIiIiIiIiIiIiIiKhSmDqPiIioGtHU1ISZmRk0NXkJQERE9LInTx7j0iX5cTHqONRBm7ae4nR2djYunA9Camoq6jg6olmz5m86TCIiIiKitwp7NBEREVUjrq6uCA8Ph6urq6pDISIieus8DQvDmdOnoK2lLf5ovPByRkZGOr6c+jnOnDmNjMwM/L5qJVat+k2FERMRERERqR5fZyYiIiIiIiL6P1NTMwz2HaKwbP/+fdDV1cPcefOhoaGBrl274fPPJqFH955wdHJ6w5FSeRIzpEjOKiwxPzdfpoJoiIiIiP672NBERERUjdy/fx/Dhg3D9u3b4ezsrOpwiIiI3joZGenYsycQujq6aNjIGY6OjmLZzRs30Lp1G2hoaAAAatWqjVq1a+PmzRtsaHoLJWcVIuBiSon5fZoYqSAaIiIiov8uNjQRERFVI/n5+QgLC0N+fr6qQyEiInrrONWtCy+vLigsKERY9BNs2bIJ/fr1x/ARIwEASUlJMDMzk1vHzMwcSUmJCuuTSqWQSqXitCAU9aSRydij5k0QBAGCICg9n2UCv5ukNHV1jsZBRET/YkMTEREREREREYA6dRxRp86/PZg6dvLC3DnfopNXZ1hbWwMASjyiFwRATU1hfXt2B2Lnzu3itK6uDhYvXoz4uKiqDp0UyM6VIDsrs8T8ggI9hfOre1l2lhZiY+IVlhG9zNbOQdUhEBHRW4QNTURERPTOCw0JQR1HR2hpaeHkyRN49uwp+vTpCysra1WHRkRE7zBX18bQ0NBATHQ0rK2tYWFhjqRE+d5LSUlJcHZxUbj+IJ/B6OfdX5wWBBlSUxJhaWXH3gBvQGZ8HvQl0hLzNTW1oC8xULhOdS7TlxjA2tJMYRkRERFRWdjQRERERO+0S8EX8c8/5zD1y+l48OA+9uzeBVtbO/yw6Hv8/Muvqg5PJdR09KDgnXsiIirH40ePULdePXH69u1bKCwshK2dLQDAvakHLl68gIGDfKClpYVnz54hIiIcHk2bKaxPS0sLWlpa4rRMJkNqSlHKKTY0vX5qampQU9DbrLT5LFPj95KIiIgqhQ1NRERE1YijoyP27dsnN7D5u+706VPo1r0HAODypUsY88FYeHg0w4cfjEZCQgIsLCxUHOGbp6ajB8gKVR0GEdE75+zZ0/jjj7WoW7c+0tPTcOlSMIYMGSb2kPX2HoDgixfxzddfwcmpLoKDL6Jnr95wqFNHxZETEREREakOG5qIiIiqESMjI3Tt2lXVYVSpnJwcaGhoAABCQh5iyNChUFNTg56entwA7NWNmrauqkMgInrnfPDhRwh78gQPHt6HvYMDfIcMha2tnVhuYGCAxT8uRXDwRaSmpmDyZ1/Aza2JCiMmIiIiIlI9NjQRERFVI3FxcVi/fj0++OADWFlZqTqcKlGvfn3s3LENDx8+gIaGBvT09JGamor8/Dxx4PbqJlEd0NNQh7GqAyEiegc5OjnB0cmp1HJdXV106uT1BiMiIiIiInq7MfkuERFRNRIXF4fvv/8ecXFxqg6lyvj4DEYdRydERETg43GfAADOnTuL9/38Sx2D4L8uWZqLbKbOIyIiIiIiIqI3gD2aiIiI6J2mp6ePsWM/lpvn7d1fRdEQEREREREREVUv7NFERERE77QTJ/7GvXt3q/V4TEREREREREREqsIeTURERPROi46OwtYtm5CXlwdnZxe4uTWBm1sT2Ds4VNvUeURERNVFYoYUyVmK08Xm5svecDRERERE1RN7NFWxWzdvYvXqVdiyZRMAICUlBVu3bMayZT8jPj5exdFVXlZWFpb98hNycnJUHQoRkVJGjRqFI0eOKLXshg0b8PPPP7/miF6vUaNG4fjx4+WWmZiYYOjQoTAxMan0Z3311VdK79s3wc9vNP7YsAnzFyyCa+PGuHnrBmbOnI4PxvghOjpK1eERERHRa5ScVYiAiykKf/IKBFWHR0RERFQtvFMNTTu2b8Pdu3dUHUapwsOf4aeffkTt2rXRsGEjAMD3C+cjPSMd7u5Noa+vr+IIKy8/Px/nzp0V0xK97X8LIlKNwMBAtGjRAjVr1oSZmZnYwP7s2TMMHDjw/8fHhpgzZw4EoejGv0ePHjAzM5P7OXTokML6R48ejQMHDigVi1QqRWGh4rdbX1RQUICff/4ZY8aMAQDExsZizJgxsLe3h4eHBzZs2CAuW1bZ2rVrUadOHXTp0gWxsbHi/DFjxuDOndd/vCxre18sW7BgAfr06QMHB4dKf9a4cePw3XffVXr910FNTQ2GhoYwNDSCgYEhdHR0IDEwgIaGhqpDIyIiIiIiIiL6T3unUufVb9AA5ubmVVbftm1b0aSJO5ydXaqkvmvXrqJJE3f06tUHQFFvpidPHuO7+Quho6PzyvVXdbyvoqr/FkT07rt48SK++OILrF69Gu3atYOWlhZ0dXUBALNnz4atrS3WrVuHhIQE9O/fH56enujWrRsOHjwImaworUl4eDi6dOmCLl26KPyM/Px8pRqPKuL48eNwcXGBsbExAGDTpk3o378/lixZgvv372PgwIFo37496tatW2qZg4MDfvrpJ/zzzz8IDAzE77//jrlz5+Kvv/6ChYUFGjduXKUxv4qffvoJcXFxyM3NFf8+FWVvbw99fX1cu3YNzZo1q+IIKy4wcCfOnj2D7KwsNG7sBvcm7vB73x9mPE8REREREREREb1271SPpqZNPWBlZV1l9d24fg1xcXFVVl9aaiokEonctJaWVpU0MgFVH++rqOq/BRG9+5YtW4aJEyeiV69eMDQ0lGvEqFmzJvT19WFkZAQTExPo6OiIjdXa2trQ1dWFrq4uduzYAR8fH+jp6ZWof8uWLTh48CDGjBkDMzMz7Ny5EzExMRg69H/t3XdYU2cbBvA7CWEkgDJUQGQ7UUFb9yruat1771GtXbafrdW2Vq2jjlZb62qrVuuoWrVq3a0T996yRZZsSAiE5Hx/oNFIGCqQAPfvurwk5znnzXNOkpPxnPd9B8Dd3R3NmzfHf/8dN5jb33//jTZt2uDhw4e5Yv/99x+aNGmiuz1t2jT06NED9vb2cHR0hFwu1/VIzSsmCIKuR421tTXUajUUCgWWLFmCmTNn5nvchg0bhtmzZ6Nly5bw8fHR6+01depUODg4wNnZGa1bt0ZgYKBuu4cPH+Kdd96Bp6cnxo4dqze0aVRUNLp3724wNnLkSDRo0AB3797FsGHD8O2336JNmzbw9PTEjBkzdOuFh4eja9euujb69eunNzRf06ZNcezYsXz3raRcuXwZaampaPNWAALatkPLVq1ZZCIiIiIiIiIiKiHF0qPp3LmziHz4EH369tMty8zMxKqVKzBw0BBUrlwZe3bvQmhoCEQiESpVqoxWrVvD1bWabv1Nm36Hv39DREc9QnBIMALeaosrVy7Dt25d1K2bc2V4Ydpo0KAhoqOjERwUBDs7O3Tp2hVyuTX++WcfYmPjcOTwQdy4fg1OTs4YMHBQgft2//49XLhwHkqlErVr10bLlq11uVy+fAmCAPzw/RIAQFp6OjQaDX74fgmsra0xZuz4fNt4/j4uXbwIhUKBZs2bw9e37mvl+2JbT126dBHXr1+DCCI0aNAQfv7+etueOnUCt2/dhoODA95s1EgvtnXLZr3HIr9j/dTJE8dx584dODg4oFnzFvhz2xaMn/CuwR+Uiaj0uXv3Lnx9feHr6wu1Wo2+fftizpw5EIvF+Oqrr9C5c2c4ODhAq9Xiq6++QoMGDfS212g02LRpEzZv3myw/UGDBmHfvn3o1asXunfvDnNzc/Tq1Qs+Pj64ePEizp8/jzFjxqBV6wC4ubnptlu5ciV27NiBnTt3wsHBIVe7YWFhePPNN/WW/fDDD7qCz7Jly+Di4lJgbPTo0fDx8YGrqyt27NiB2bNn46OPPoK1tTXyo1arsW/fPmzcuBFarRaDBw9GvXr10KdPHyxYsABz585FdnY2zp8/j0mTJuHKlSsAgMmTJ8PPzw+//fYbTp48iZEjR2LixIkAgBkzvsQbbzYyGMvOzta776NHj+qGAOzSpQv69++P+vXrY9KkSXjjjTewfv16nDhxAiNHjtQNLwgAVatWxa1bt/Ldt5Lyzey5CAp6gOvXrmH7n9sQFhYKbx8f+NX3R8dOnfUuAiEiIiIiovIhMTEBBw78g4jwcNjZ2aFtu/aoXr2GLp6VlYW9e/fg7t27sLG2RsdOnVGzZq1Cx4mI6Jli6dHk5OSEbdu2IC0tVbcs8Mxp3Ll7B5UqVQIAeHl7w79BQ9Sr74fMzEx8/tn/EBEerlv/8qVLWLJ4IUJCQlCjRk1UtLPDlSuXEfPcvBeFa+M7hIWGwMPTE9euXcW8eXMBAB4enpDJZXBz84B/g4aoUbNmgft14MA/mD9vLsyl5qjmWg07dmzH2jWrdblUqlQZjo6O8G/QEP4NGqJmjZoQi8Xwb9BQV5DJrw0A2L9/H2Z/8zW0Wi1cqlbFtq1bcP36tVfKN6+2gJxC0Y/Ll8Guoh1sbW2xZMl32LVrp27bTRt/x/p1v8HJ2RkajQbznxy3p158LPI71gDw+4b12LhxA5xdnKHRajD7m6/05nx6kVqthlKp1P3LyFAWuL9EZFxqtRqnT5/GgQMHsHfvXhw8eBC///47AODjjz9G/fr1ERQUhDNnzmDdunX4999/9bY/fPgw7Ozs0LBhQ4PtSyQSiMVi3ZB8Wq0WJ06cwKxZs1CpUiW8/fbbaNToTZw+fVq3zdKlS3H48GHs2bPHYJHpKbFY/+3wvffew8OHD7Fz507MmDFDr6CSV+yTTz5BXFwcrly5gtTUVERFRaFp06Z4++234eHhge+++y7P+3/33Xfh4+ODGjVqYOLEibqeQidPnkSnTp1Qo0YN9OvXD0FBQcjMzER2djZOnz6NL7/8EpUqVULv3r11vbKys7Nx4eIlzJw5M1fMkAkTJsDLywteXl5o2rQpgoODoVarcfbsWcyYMQOOjo4G2xCLxbqeV8YmkUhQs2Yt9Os/AB9P/RTDho1AdFQ0Nm7cgOjoKGOnR0REREREJSwtLQ2zZ8+CzEqGdu07wNa2AmZ88bnefONLlyzC6VOn0Lp1azhWqoSvvpyB+/fuFTpORETPFEuPJnd3D7i7u+PEiePo2rUbAODYv0cRENAWIpEIAHSFl6ckEgn++WcfJkycpFv25puNMH7CxDzvpzBtNG/eEqPHjAUA1K/vh8mTJkChSEft2nVgY22NGjVrok2btwrcp4wMJX7fsA6fT5+hu98mTZth4oSx6NW7D+rWrYfAwDPQajS69oKCHmDHjj91twtqw8rKCr9vWIepn/wPb76Z04Po7be7IC0tDba2ti+Vr1KpzLOttLQ07Ny5HV/M+BL16/sBAFyrVcPSJYvQuXMXqNVq7NmzC7Nnf6sraNnb2+Pnn3/K9z7zOtZarRZ79+7B3G/nw8en+pP2HPDzih/zbGvnju3Ytm2L7ralpQUWLlxY4H4TkfG4urqif//+qFYtp2dpjx49cO1aTnH733//xZ9//onKlSujcuXK6Ny5M44fP46AgADd9uvWrcPw4cMLfX+CIEAQBL0ikZmZRG8Opzp16uD06dOIjo6Gl5eXwXaqVauG6OcK50DO+4m1tTXatm2LRo0a4cKFC/D19S0wJhKJoNFoMH36dPzyyy/YsGEDGjdujHXr1qFz584YM2YM7O3tc+UgkUj0/n46Z9XYsWPx/fffo2XLljA3N0e1atWgVqt176XP7/vTv59uayhmyPO9SiUSCbKzsw0e2+dzBICoqCi9nmPGdOPGdVw4fx7Xr1/Do0eRcHNzR/PmLVC/vh/c3T2MnZ5JiU9TI1FheJ4zVZa2hLMhIiIiIioeVlZWWLBgEczNzQEAjRo1RnBIMAIDz6Bu3XoIDQ3B+fPnsPzHFXBxqQoAeBwXh+07/sT06TMKjBMRkb5iKTQBQEDbdjh65DC6du2GuLhY3Ll9G1OmfKiLpyQn4/iJ44iOegSVSoWYmBhYWOrPZVSjgO6ohWnj+R8Wn/64l5aWpjekW2FERkZCpVLhyOFDOHrksG65WCxG5MOH+V4pX9g2rKyskJ2dDX//Z8NJiUQi2NravlSuABD58GGebd27dxdarVZvGL169eohKysLUVFRyMrKhFgs1us15Vu34Ins8zrWycnJEIvFuiITAL37NqR3n77o1r2H7rYgaJGcFF9gDkRkPH369MHu3bvRp08fqFQqHDhwAOPGjQMA1KhRA5s3b0bt2rURFxeHo0eP4rPPPtNtGxsbi//++w8rVqzI9z4qVKiAqKicHipSqRRNmzbFwoULMX36dFy6dAmBgeewaNES3fqdOnVCnz590Lt3b2zZsgW1auV+X2nVqhV27nzWo/OLL77A6NGj4eHhgXPnzuH8+fO6eZbyiz31888/o2fPnnB+0iNUJpPBxsYGYrFYrwj2vLVr16Jjx44QBAFr167Fe++9B0EQoFAoUK1aNcjlcixevBhZWVkAcua1euONN7B48WJ8/PHHOHXqFM6ePauL1a9fD0uWLMHUqVP1YoVlbm6Ohg0b4ocffsD777+va+ODDz7QrXPu3Dl8/fXXL9VucTnwz37I5XL07dcP9er5oUKFCsZOyWQlKjT4IzDJYKyr38t/3iAiIiIiMkVmZvo/eWZmZiI2Jlo3hPvdu3dQuXJlXREJAPz8G+DXX9YUKk5ERPqKrdDUqlUbrF/3G8JCQ3Hu3FnUrVtPN2xeVlYWpk37FD7Vq6Nu3XqwtLSE2c0buSZpf/FN4XmFbUNk4CruVxnpR2omhUgkQn0/f72ruv0bNISrq2uRtJGWntP7R63OynffC8PcwiLPtszNzaHVapGdna3LIysz58dLC4ucKz2ys7Oh0WiexbMyC7zPvI61ubnFS7cnlUohlUp1t7VaLZIN/y5GRCZi1KhRuHLlCry9vWFubo5BgwbpeigtWbIEkydPhouLC6ysrDBw4ED0799ft+2mTZvQqVMng719njdo0CCMGDECM2fOxMqVK7FixQq8++67cHJyQpUqVTDr6y/h6empt01AQABWrFiBAQMGYOPGjahXT79w3qVLF8ycORPp6emwtrZGu3btMGjQIDx48ABubm6YN28e/J/MYZdfDAAePnyIgwcPYvfu3QCAoUOHokePHli4cCEmTpyoex98UZMmTdCuXTvExcVh0KBBGDBgAEQiEebMmYO3334bUqkUAwYM0Lvw4KeffsLo0aOxaNEiBAQEoEWLFrrYnNmz8Pn0mVi8eHGuWMWKFfHbb7/p5W3I8uXLMWbMGMybNw8BAQFo2LAhbGxsAAAxMTFISEhAs2bN8m2jpHz6v88KXomIiIiIiMqddb/9ipCQYDx8GIHWrdugc+cuAIDUlFTY2upfoGZra4v0J/OtFxR/ccQHtVqtNz2EIOSMFvB0xImX8XSEibxir9ImEdGrym+UnOcVW6HJ2toajRo1xtGjh3Hx4gUMHjJMF4uNjUFCQjyWfr8MVlZW0Gq1OHPmdD6t5VYUbVhZyaDKyCjUulVdXeHoWAnZ2dlo27adbnlg4BlUqFixSNqwrVABjo6V8PeePeg/YCAAICkpCWmpqXBzd3+5fKtWzbOtqlVdUbFiRRw9chhdur4DADh8+BAcHR3h5JRzBb6NjQ3+++9ftGvXHgBw7OiRQt1vXrlYW9vgxInjCAhom9PesaOv3B4RmSapVIqff/4ZK1as0A3t9lTNmjVx5MgRCIKQKwYAU6ZMKdQbV6tWrRASEoKsrJwiulgsxsGDB3UfxKOjInTr/v7777oP/82bN8fFixcNtmlubo6JEyfil19+wQcffIC2bdvi/PnzBnPNLwYALi4u2LVrl25fqlSpgrNnz+a5/lMtW7bEggULcq03duxYjB07Vnd7zpw5sLDI6blbvXp1nDx5UreNWq3W7a+npweOHz8OkUiUK7Z27Vrd388fIwBYuXKl7nbt2rVx5swZaDQanD17FkOGDEHt2rUBAMuWLdPrkWYKYmNjsX//XkRHRcHGxgaNGjdB06amUQgjIiIiIiLjaNW6Ner7+SE8LAy7dv0Fn+o10KpVa5iZmSFLnaW3rlqthlgshkQiKTD+orymgIiNiXzpnJViNZSKdAA537OfL2AplemITi/4YnAioqJS1dWjUOsVW6EJANq2bY/58+fCwsICTZo01S13dnaBp6cXPv3kY3h4eCA8IhxymQxA3j/Cvago2mjQsCG2b9+Gu3fvoGpVVwwYOCjPdaVSKT7++BMsXboIR48choODAx4+jIC7u4fevuWnoDbEYjE++PAjLF2yCBcunIedvT1iY6LxyafTXinfvNoyNzfHu5Pew7IfluL8+XPQaDSIiIjAx1M/gUQigUQiwfgJ72L5su9x+vRJqDIyIJWaF/q4vsjCwgLjxk/ATz8uw8mTx6HKyIBEkvPUK2xFlIhKj/wKKnnFnu/BWBhPx9l+vt0Xr/h6sc387mPSpEnIzs7O1WZe8ooZ+sJRUFsvs97TIpOhbZ7u3/NXt70YA4DQ0FBMnDgRK1euRI0aNfTaen69bdu24d1334VWq4WLiwuWLFmiGyb266+/zvUYGFNkZCT+9+nHqF69Brx9fJCWmorly75HaEgIBg0eYuz0iIiIiIjISLy9fQAADRu+gaysLOzZvQutWrWGs4sLHsfFQa1W674HRUdFwcnZGQAKjL8orykgqji5vvRvXympsZA9N+WH1PzZ90CZzBrOtlVeqj0iopJQrIUmP39/TJ48BXb29no/SJmZmWHe/IW4efMGFAoFhgwdDpFIhOjoKN06g4cMhatrNb32BgwcBOcnJ/RXacPMzAzvf/ARKj7pgdSzZ2/Uq1cfMdHRkMnlBe5PjZo1sWz5CoSEBCM5ORlubu66fAAg4K22EPDsh84qVZww+b33X6qNOnV8sfzHnxEcHIzMTBVq1aoNmUz2Svnm19abbzbCTytWIigoCCKRCD4+1WFt/exNrGnTZqhevQZCgoPg4OCIqq6uOHs2UDdp/POPRWGOdfPmLVCzZi2EBAfB3sEBglbAF198pjcJPRGRsYhEopcudhWVF3sVFTelUonz589DqVTmu17fvn3RvXt3mJmZGRyC1ZTs3rUTnTq9jREjR+mWde/RE//7dCp69+lrsEBHREREpUd8mhqJCsNzXaqyOIQUEeV2585tyGQyuLt7AMjpjXTv3l1UrlwZAODn5w+RSIyjR4+gc+e3oVQqcfToETRv3qJQ8RflNQWEWCx+6ULT05Ep8orxom0iMkXFWmgSi8Vo3eYtgzGJRAI/P3+9ZU5OTrq/33jjzVzbNGjQ8LXaEIvFaPNCPt7ePrqrGwpDKpWiZs3ck8kDgE/16nq3bWxs0KpV65dqAwAsLS3h6+trMPay+ebXlo2Nba5j+jwHBwfd1esA9I7di9sVdKxDQkLg7OyMRo2bQBAErFjxI2rWqlWiP64SEZkiYxW4CiIWi2FpaWnsNAolJiYGAwYG6C2rVs0Njo6OePz4caHnUiwrEiwtoeaXTyIiKkMSFRr8EWh40t6ufrYGlxNR+VahQkX89OMypKalwsbaBo8eRaJqVVdMnjwFACCXy/HelPfx4/IfcOTwQSQkJMDdwwO9evUpVJyIiPQVa6GpNElNTcVvv641GKtdxxcdO3Yq4YzyV9ryzcrKxCdTP0S1am6IjYuFUqHE559/Yey0iIioDKhcuTKuXbuKunXr6ZbFxMQgPj4elSpVMmJmxpGoVSNb4IUcRERERFR+ubi4YO638xETE43k5GRUcqwEB0dHvXWaNGmKevXqIzw8DNbW1qhWze2l4kRE9AwLTU+Ym5vDP4/ePc/3kjIVpS3fWrVqY/6CRbh//x4sLS3h7e1Taq6UJyIi09ate098Nu0TRISHw8enOlLTUnHq5Al0frsLh80jIiIiIirHnJyc4eRkeF4lAJDJZKhdu84rx4mIKAcLTU9YWlrmGlbPlJW2fIGcoQQNDYlIREQlx83NDWvXroWbW9m5Gq+CrS3mL1iEgwf/wa1bN2FtY4MRI0eXuvfJosRx24mIXl9Q0AOcP38OTZo01Ru+XKlU4szpU0hOToanlxe/4xARERFRucdCExERUTlib2+PgQMHGjuNIvXddwswesxYTJjwrrFTMRliEQtNRESvQ6VSYfmy7xEXF4fKlSrrCk1paan4bNqnsLOzh7ePD1b+vAIN33gD77472cgZExEREREZD3+FICIiKkceP36MVatW4fHjx8ZOpchUdXVFbEyMsdMgIqIyZP3639CiZStYW9voLd+9excsLa0w65s5GDVqDL786mscPXIYIcHBRsqUiIiIiMj4WGgiIiIqRx49eoSpU6fi0aNHxk6lyLRu3QabN/+BwMAzePQoEtHR0bp/arXa2OkREVEpc+3qVTy4fw+9e/fNFbt65QqaNm0GiUQCAKhWzQ3V3Nxw9eqVkk6TiIiIiMhkcOg8IiIiKtV+37AO0dFRWPTdglyxBQsXwcenuhGyIiKi0kihUGDlqhX49NPPYGaW++tyQkICHBwc9JY5ODgiISHeYHtqtVrvogdB0AIAtFptEWZd9gmCAEEQiixW1O2VpRifm1RYnBOUiIiex0ITERERlWpzv12Q5w8mUqm0hLMhIqLS7Ndf1qBly1bw8vLKc51c7ziCAIhEBtfduWM7tm3borttaWmBhQsXIjYmsgiyLT+UKjmUinSDsexsq5eOvco25SGmVEgRHRVrMEb0oqquHsZOgYiITAgLTURERFSqsZhERERFITY2FidOHMc73brjjz82AgAyMpQ4f/4cIBKhffsOqFTJEQnx+r2XEhISUMfX12Cbvfv0RbfuPXS3BUGL5KR4VHFyZW+Al5AemwmZ3PBwuGZmUsjk1i8Ve5VtykNMJreGcxUHgzEiIiKi/LDQREREVI5YW1ujXbt2sLY2/AMDERFReSWTWWHAgEF6y0QiESQSiW4YPf8GDREYeAa9eveBVCpFWFgYIiLC0bDBGwbblEqlehdEaLVaJCflDDnFQlPhiUQiiPLoNfYqsaJuryzF+LwkIiKiV8FCExERUTni4+OD3bt3GzsNIiIik2NjY4u+/frrLTt48ADeeONNvPVWAACge/eeOBsYiJkzPoe3tw/Ong1E57e7wMPT0xgpExUpjVbA/RiVwZi9XAJHG/YiJyIiIsNYaCIiIipHNBoNFAoF5HI5JBKJsdMhIiIyaT179YK3j4/utrW1NRZ+txhnzwYiOTkJH3z4MerX9zNihkRFJ12lxb5rqQZjg5vZsdBEREREeWKhiYiIqBy5ceMGWrZsiVOnTsHf39/Y6RAREZm0rl275VpmaWmp6+FEREREREQsNBERERFh+uf/Q3h4OACgdu06mDHzK724QpGO3379BZcvX4K5uQXatHkLAwYO0s1jUFCciIiIiIiIiKisYqGJiIiIyr2vvp4NrVaLrVs2IyQkOFd8+bIfkJaWhjlz5yEtLQ2LFy2EVCrVzeVRUJyIiIiIiIiIqKziZbZERERU7llYWMDKygoSs9zzVsXHP8aFC+cxatQYuLhURc2atdC7d1/888/+QsWJiIiIiIiIiMoy9mgiIiIiykdoSAjEYrHeZPA1atREcnISkhITC4zb2dvrtadWq6FWq3W3BUELANBqta+frKD/94ttCoIAQRBgSF6xgrYpkryJqEzgcKFE5VN8mhqJCo3BmL1cAkcbaQlnRERERCWNhSYiIqJyxNfXF6GhoahYsaKxUyk1FEolZDI5RCKRbpnc2loXKyj+YqFp547t2LZti+62paUFFi5ciNiYyNfO1VqRDsHKHAAghhjRURF6caVKDqUi3eC22dlWBmN5LQcApUKK6KjY18yaiMqKqq4exk6BiIwgUaHBH4FJBmODm9mx0ERERFQOsNBERERUjkilUlSqVMnYaZQqVpaWUKky9JYplQpdrKD4i3r36Ytu3XvobguCFslJ8aji5PravQE0UWqIkNNbytLcAjYuDnrx9NhMyORqQ5vCzEwKmdy60MsBQCa3hnMVB4MxIiIiIiIiIiofWGiiUiXwy+owMyu/T1utVovoqAg4u7iV+6FJeCxy8DjQywoJCcFnn32G+fPnw8vLy9jplAruHh7Izs5GRHg43NzdAeQMp2dtbQN7Bwe4Z6vzjb9IKpVCKn12Za9Wq0VyUs6QU69daBLh2fB5otzDWIlEIr2eV4WJFbQNzz1ERFQU8ht+TZXFYVqJiIiITFn5/cWeiIioHEpNTcX+/fsxffp0Y6dSajg5OcO3bl1s3LgBH3z4ERQKBf7atRPt2reHSCQqME5EREQFy2/4ta5+tiWcDRFR6fNYlYb4TIXBmKOFHJUsbUo4IyIqT1hoIiIionJv1coVOHHiOLKzs6HVajFk8AC4uFTFd4uWAAA++nAqfvxxGUaNHA6JRIJWrVpj0KAhuu0LihMRERERERWn+EwF1oWeNRgb6dmUhSYiKlYsNBEREVG5N3rMOAwfMUpv2fNDwtnZ22Pml19Do9FALBbn6qlUUJyIiIiIiIiIqKxioYmIiIjKvRfnTcqLRCJ5rXhZo9EKuB+jMhizMBMhM1vItdxeLoGjTcHHmoiIiIiIiIhKBxaaiIiIyhFnZ2fMmzcPzs7Oxk6FyoB0lRb7rqUajHX1szUYG9zMjoUmIiIiIiIiojKEhSYiIqJypEqVKpgyZYqx0yAiIiKiUiS/XsyqLG0JZ0NERESmhoUmIiKiciQpKQn//vsvAgICYGdnZ+x0iIiIiKgUKKgXMxEREZVv4oJXISIiorIiPDwcw4cPR3h4uLFTISIiIiIiIiKiMoA9moiIiIiIiIio2MWnqZGo0BiMcfg1IiIiotKLhSYiIiIiIiIiKnaJCg3+CEwyGOPwa0RERESlF4fOIyIiIiqTRMZOgIiIiIiIiIjKARaaiIiIyhFLS0v4+fnB0tLS2KlQcROx0ERERERERERExY9D5xEREZUjtWrVwunTp42dBpFB+c3dYS+XwNFGWsIZEVF5lJSYiMtXLkOpVMDLywe+vr568YyMDAQGnkFychK8vLzh79/ASJkSEREREZkGFpqIiIiIyCTkN3fH4GZ2LDQRUbELPHMaW7b8gTp1fGFmZoZdf+2En58/3v/gIwBAWloapn/+P9jY2MDb2wc/7tuLRo2bYMKEd42cORERERGR8bDQREREVI5cu3YNAQEB+Pfff+Hn52fsdIiIiExK1aquWLzkB5iZ5XxVbtMmANOmfYKBgwajcuUq2LNnF8ykUnwz+1uYmZmhQ4dO+Oij99GhQyd4eXkZOXsiIiIiIuNgoYmIiKgcEQQBWVlZEATB2KkQERGZHDd3d73bYokEYrEYEknOV+erV66gWbPmukKUm7s73NzdcfXqZRaaiIhMiFqtxuFDB3H06BHExcWiShUn9OjZC61atdatExMTgzWrV+Lu3TuwtrZB167voHuPnoWOExHRMyw0ERERERERET2RkpyMAwf+gVKpwPUb1zFm7Hg4ODgAAOLj4+Hg4Ki3voODIxLi4w22pVaroVardbcFQQsA0Gq1xZS9aRMEIc+LXUoyZip5lJdYeX2+l3VisdjYKeTr4oXziImJwZT3P0SlSpVw7txZLF/2Pezs7FC3bj1oNBrM+3Y2vLx98ONPKxEZ+RALF8yHbYUKeOutgALjxiJkqaBNiYe4gmPBKxMRlSAWmoiIiIiIiIheIAhAdnY2gh7ch6ZDR0gkEgCAyNCKolxLAQA7d2zHtm1bdLctLS2wcOFCxMZEFlPWpk2pkkOpSDcYy862KrFYSd5XeY8pFVJER8UajFHpVtXVw9gp5KtZ8xZo1ryF7nbbtu2wf9/fuH37FurWrYfbt28hOjoac+bOg42NLezs7NChY0ccOvgP3noroMC4sQjZmdBkqVloIiKTw0ITERERERER0RMVKlbEgIGDAABKpRITxo9Bg4ZvoEWLlnB0dER8gn7vpYTEBNTx9TXYVu8+fdGtew/dbUHQIjkpHlWcXE2+N0BxSI/NhEyuNhgzM5NCJrcukVhJ3ld5j8nk1nCu4mAwRlSSUlJS8OjRI3h4eAIAQkNCUMXJCTY2trp1fHyq45/9+woVf1FR9mDNr5cghJx/htpl70IiKg6F/czKQhMREVE5UrNmTZw/fx6enp7GToWoSMSnqZGo0BiM2cslcLSRlnBGRFSahYSE6M21pNFooNVqYfakN5N/gwYIDDyD3r37wszMDBHh4YgID4f/lIYG25NKpZBKn52HtFotkpNyvrCXx0KTSCSCKI/eXyUZM5U8ykusPD7XybRkZWVh8aKFqFffD40aNQYAKDOUkMtkeuvJZXJkZWVBo9EUGH/ay/WpouzBqhSroVSk694/ni9gZavVUKrUSI+KyHM7g20q0xGdnvnSuRARFbYHKwtNRERE5YiVlRXq1Klj7DSoJAhCuRi/PVGhwR+BSQZjg5vZsdBERC/l0MEDiIp+BG8vH2RlZeLixQuoUaMmGr7xJgCge/eeCDxzBl/OnA5vn+oIPHMaHTt11itOERGR6VCpVJg/fy6kUimmTv1UVxS1srSCQqnUW1ehVMDc3BwSiaTA+IuKsgdrSmqsXi9BqbmF7m8zqRQyiQVsXNwK3O55Mpk1nG2rvFQeREQvg4UmIiKiciQiIgILFizAtGnT4OaW+8sJlSGCFpr0pDJfaCIiKkoT352EoAcPcOfubZiZmaFlq9aoXfvZBRo2Njb4btESBJ45jeTkZEx+7300aGC4NxMRERmXQqHA3DnfwLZCBUyd+qleD1MPD0/ExcYiPT0d1tY5xZng4CC4u3sUKv6iouzBml8vQYhy/hlqk70LiciYWGgiIiIqRxITE7F+/XqMGzeOhSYiIiIDfKpXh0/16nnGrays0LZd+xLMiIiIXlZqaipmf/M1XKpWxfvvf5irF1LdevVQqVJlbFj/G0aMHIXIyEgcOXwYw0eMLFS8JAlZKgCAyNzyldvQClrcSYkxGHO0kKOSpc0rt01EBLDQRERERERERERERGXI6dMnERISjLCwUJw5fUq3vGPHThg3fiIkEgmmfzEDP6/4CaNGDodcbo1u3bqj3ZMLCQqKlyRBnTO30usUmlLVKux6dN1gbKRnUxaaiOi1sdBEREREVJYYGDOeiIiIiKg86dy5Czp27Jxr+fNDy7m4VMXsOd/m2UZBcSIieoaFJiIiIqIyRCTmxzsiIiIiKt9EIlGu4fKIiKj4cBY4IiKicqRy5cqYOnUqKleubOxUqNjlMYEwEREREREREVER4iWvRERE5YiLiwtmzZpl7DSIiIiIiIiIiKiMYI8mIiKiciQtLQ0nTpxAWlqasVMhIiIiIiIiIqIygIUmIiKiciQ4OBhdunRBcHCwsVMhIiIiIiIiIqIygIUmIiIiojJKJLUwdgpEREREREREVMax0ERERERURonMLY2dAhERERERERGVcSw0ERERERERERERERER0SthoYmIiKgcMTMzg4uLC8zMzIydChERERERERERlQH8lYmIiKgcqVu3Lu7fv2/sNIiIiIiIiIiIqIxgjyYiIiIiIiIiIiIiIiJ6JSw0ERERlSM3b95EjRo1cPPmTWOnQkREREREREREZQALTUREROVIdnY2oqKikJ2dbexUiIiIiIiIiIioDGChiYiIiKgsEUuMnQERERERERERlSMsNBERERGVJSIRACBVk4V0CT/qEREREREREVHxMjN2AkRERERU9BKzlDAzs0AFYydiouLT1EhUaAzG7OUSONpISzgjIiIiIiIiotKJhSYiIqJyxNvbG/v374e3t7exUyEyqkSFBn8EJhmMDW5mx0ITEdEryq+Qr8rSlnA2RERUEK2gxZ2UGIMxRws5KlnalHBGRFQasdBEpUqzbx5AqTZ2FqbgnrETMCE8FjlK9jjcmle7RO+Pio6NjQ1at25t7DSISoRGK+B+jMpgjD92ElFeHj2KxI3r1yFAQO1adeDh6akXz8jIQGDgGSQnJ8HLyxv+/g2MlKlpyq+Q39XPtoSzISIqO4QsFaDRAKKiHR47Va3CrkfXDcZGejZloYmICoWFJiIionIkKioKq1atwoQJE+Di4mLsdIiKVbpKi33XUg3G+GMnERmyft1vuHLlMnx9fZGdnY2Nv29Ar1590LdffwBAWloapn/+P9jY2MDb2wc/7tuLRo2bYMKEd42cORERlXWCOhPQagAJe94TkelhoYmIiKgciYuLw+LFi9GrVy8WmoiIiF7QpGlTDB8xEiKRCADg61sXP/20HN179IS5uTn27NkFM6kU38z+FmZmZujQoRM++uh9dOjQCV5eXkbOnsj05Ne7mHMiEhERlR0sNBEREREREREBqFVLf3hgaxsbaLVaCIIAALh65QqaNWsOM7Ocr9Ju7u5wc3fH1auXWWgiMiC/3sWcE5Ho1WlT4gEA4gqORs6EiCgHC01EREREREREL1Cr1dj+5za0atUaFhYWAID4+Hg4OOj/qOfg4IiE+Pg821Crn00yKwg588NptWV3njhBEHSFOVONmUoejAll+rVQ1onFRTtPEL0cTXrOXHgsNBGRqWChiYiIiKgAYWFhyMrM1Fvm5u4OS0tL3W2NRoPIyIcwN7eAs7NzSadIRERFKDs7G0uXLAIAjBs/US8menFlQQBEuZYCAHbu2I5t27bobltaWmDhwoWIjYksynRNilIlh1KRbjCWnW1lEjFTyaO8x5QKKaKjYg3GyPRVdfUwdgpERGRCWGgiIiIqR+zt7TFixAjY29sbO5VS5fuli6HOVsPG2ka37L0pH8DV1RUAEBYaivnzv4UgaKFSZcLFxQWfT58BW1tbY6VMRESvKCsrC98tnI/MzEzM/PJrWFlZ6WKOjo6IT9DvvZSQmIA6vr4G2+rdpy+6de+huy0IWiQnxaOKk2uZ7Q2QHpsJmVxtMGZmJoVMbm30mKnkUd5jMrk1nKs4GIwRERFR6cJCExERUTni5uaGn376ydhplEqDBg1Gy5atDcaWfr8YDRu+gfETJkKtVuPrr2Zi/frfMGXKByWcJfK8qp6IiAqWkZGBefPmwFxqji9mfKkbMu8p/wYNEBh4Br1794WZmRkiwsMRER4O/ykNDbYnlUohlT6bg0ar1SI5KWfIqbJaaBKJRBDl8V5kKjFTyYMxUZl9HRAREZU3fEcnIiIqRzIyMnD79m1kZGQYO5VSJy0tDaGhIUhP1x/+JSwsDJEPH6Jnr94Acn5UfOedbgg8cxoajabkExXx4x0R0atavPg7BD14AE9PL+z6aye2btmMrVs2IyU5GQDQvXtPqLPU+HLmdPzyyxp8881X6NipM7y8vIybOBERERGREbFHExERUTly7949tGzZEqdOnYK/v7+x0yk1PDw8cPy//3Do0EFER0XhzTcbYfJ778PKygpRUY9gbm6OypUr69avWtUVmZmZSEiIR+XKVfTaKtaJ4V+ca1vQb9cUJkzPb+LvV5kwnJOTE5mOstAzoWHDN1Ddp3qecRsbG3y3aAkCz5xGcnIyJr/3Pho0MNybiYiIiIiovGChiYiIiKgAH340Vfd3YmICZn39JTZu3IBx4yZArVbD3Nxcb33zJ0MtPV9Qeqo4J4avmJUJyJ7NJaLRaBAdFaG7/SoTtBf15N/5TfydX355bVcck85zcnKiV1MWJobv0qVrgetYWVmhbbv2JZANEREREVHpwEITERER0Uuwt3dAhw6dcOjQQQCArY0tFAoFNBoNJBIJACA1NQUAYGNjm2v74pwYXhMVrHfb0twC9nZOutuvMkF7UU/+bSmzRrrI8DYScyHP/PKaMLw4Jp3n5ORERPmLT1MjUWF4eFhVFnuEEhEREZU3LDQRERER5SMrKytXj6XH8Y9hbZ1TpPD28YFIJMLNmzfg5+cPALh54wacnJ1ha5u70FScE8NrXphrW2om1WvTFCZMT1dpse9aqsFtuvrZvvSE4ZycnIio5CUqNPgjMMlgrKtf7vc+IiIiIirbWGgiIiIqR0QiEczNzfP8gZ1ye/gwAhvWr0Pbdu1hZ2eH27dv4cA/+/Hhhx8DAGxtbdGxU2esWvkzho8YifS0NOzY8SfGjB1v5MyJiIiIiIiIiIofC01ERETliJ+fHxITE42dRqni7e2DIUOH4eDBA0iIj0elSpUwa9Yc1KpdW7fO6NFjsbfSHhz4Zz+kUinenTQZLVu2NmLWREREREREr0craHEnJcZgzNFCjkqWNiWcERGZKhaaiIiIiApQo0ZN1KhRM8+4RCJBj5690KNnrxLMioiIiIiIqPikqlXY9ei6wdhIz6YsNBGRDgefJyIiKkfu3r2LFi1a4O7du8ZOhYiIiIiIqNjcu3cXd+7cxp07txEdHW1wnczMTAQHByE2NvaV4kRElIM9moiIiMoRlUqFa9euQaVSGTsVIiIiIiKiYrN1y2ZkZmbi0aNING7cBJMmT9GLX7p0Ect+WArbChWQnJSE2nV88ckn/4O5uXmh4kRE9Ax7NBERERGVAdqUeCA729hpEBERERGZhC+/moW5386Hr2/dXLGMDCWW/bAU/foPxPLlK/DzyjV49CgSu3btLFTc1GhT4iFkq42dBhGVYyw0EREREZUBmvQkCBoWmoiIiIiICnL16lVkZ2ejU6fOAABra2u0b9cBp0+dLFTcJGi1ORebIee7ALQl+11AK2hxJyXG4L/HqrQSzYWIjI9D5xVSQkICsrOzUaVKFQBARkYGEhMTUKlSZXaZJSIyQTt37kS9evVQvXr1AtcNDQ3F/fv30alTpxLIrHjs3LkTfn5+8PT0zDPm7e1dZPfVsmVLVK5cuUjaIyIiIiIiKknRUVGoXLkKpFKpbplL1aq6uZwKir9IrVZDrX7Wo0gQtAAArVb70rkJggBBEJ5bAP2/n9wWlGnQiMSAjb1uud52L25WxLEUtQq7Iq8ZjI30bAoHc7nBGBGVLmJx4foqGb3QFBcXB7lcDrncNE8+mZmZmPX1TDx+/BheXt74fPoM7NmzG9u2boadvT0+/fQzuLm5GTtNIiKTkJmZiT179iA8PBwajQbvvfee7vweGBiIS5cuQSaToUuXLnBycgIAbNy4EY8ePdJrp2vXrqhbN/fwBrt27ULt2rVRs2bNAnPZtm0bpFJpoQpN06dPx/vvv6+7fevWLfz3339wdHRE165dYW1tXWDs8ePH2L59O7y8vPQKVsePH0eVKlVQq1atAvN4Hdu2bYOVlZXBQtPTmLe3N27fvo158+bB3d39le9LJBLh22+/xffff/8aGRMRERERERmHWq3OdeG4hYUFNBoNNBpNgXGJRKIX27ljO7Zt26K7bWlpgYULFyI2JvKlc1OK1VAq0p8VuZ4bEk8raJGVlYlstRrS7GyoFenQPgqBuTIVGrMKUCrSDbaZna0u0ZhSmY7o9My8dpGISpGqrh6FWs/ohaaNv69Hs2bN0ax5iyJpr6gLV4GBZ6DVCli95leIRCJkZmZi65Y/sGDhIri6Vnvt9k290EZEVFhpaWlo3749rK2t0bx5c5iZmemufPrpp5+wdOlS9OnTB48fP8aXX36JM2fOwM3NDZmZmVCpVACA7OxsfP/99+jWrZvB+/jzzz/Rt2/fQhWaCis0NBRBQUFo1qwZAGD16tXYtGkTmjVrhoMHD2LOnDk4e/Ys5HJ5vrEePXrgjTfewMaNG5GSkoL+/fsjISEBc+fOxd69e4ss39dlYWGBgIAA2NnZvXIb3bp1w/Tp05GUlPRa7VDRE1lYGTsFegnxaWokKjQGY/ZyCRxtpAZjRERERPR6bGxtkZqWqrcsNSUFcrkcEomkwPiLevfpi27de+huC4IWyUnxqOLkWujeAE+lpMZCJn92saOgflawEYvEMDe3gLm5BQSFBlK5NQABAgRIxBK97Z5nZiYt0ZhMZg1n2yoGY0RUNhm90PTx1E+LtL1F381H57e7om3bdkXS3qPISLi5uUEkEgEAEhLikZWVVSRFJqDo8yUiMpbly5fDxcUFO3bsyPVB+vjx45gxYwZGjhwJAOjQoQOuXLkCNzc3jBkzRrferl270KhRI9SpUydX+9evX8f9+/exe/du3L9/H++88w58fX1x8eJFnDt3Dk5OTnijYX2DuSmVSqxZswa9e/dGtWr65++///4b7du3191u0aIFxo8fr7vt5eWFBw8ewN/fP89YjRo1oNFosHz5chw5cgT79u1D//79MWPGDHz11Vf5DrG6c+dO1KlTB/fv30dMTAy6dOkCV1dXAMChQ4dw5coVmJubo3r16ujSpYvesT1x4gRu3bqFVq1a5Wr35MmTuH37dq6YQqHAli1bMGXKFJw+fRq+vr4ICgpCREQEunTpotfT6fjx47o27t+/rxt+z8zMTFdsGzhwYJ77RiUvwcoKggi64TQkIk7HacoSFRr8EZhkMDa4mR0LTUREVG7xYgwqbrVq1sIva1cjJiYaTk7OAIAbN66jRs1ahYq/SCqV6g2zp9VqkZyUM+TUyxaaRCKR7ndIADmf73XBJ//y+Pv57fDiZiUZE4leer+JqHQr9Cs+OTkZiYkJuZZHRT3SXYmeEB+PyMhIREU9QmZm7u6RcXGxUCqVOesmJCAjQ4m4uDgoFArdOi/TRmJiArKysnSxnNtqJCclITIyEnFxcYXdPahUKiQl6X/RT4iPR3z8Y6hUKkRGRiIyMhLh4eEQi8WIjIxETEx0gW08LzMzE8nJya+Vb377/7y0tDSkp+fuvlrY7fOiUqkQH/841/is+bVbmOcOEZV+hw4dwsCBA/H7779j7dq1esPhffDBB9i0aRO+/vprTJ48GXZ2dujQoUOuNtavX4/hw4cbbP/5IQxUKhW0Wi1Wr16NgQMHIiwsDCtWrMDIUWNzjYH9+PFjdO/eHTKZLFeRCQCuXLmC2rVr6277+vri7NmzmDdvHoYPH4527dqhfv36+cZkMhkEQcCnn36KuXPnomHDhjh+/DgkEglatMi/x+62bdswaNAg7N27F+fPn0fLli0RGZkzvMLTfU1KSsJPP/2Ed999V7fdihUrMG7cOISEhOCjjz7C1atXdbENGzZi/PjxBmM7duzA999/j+joaGzbtg1Dhw7F33//jWvXrqFNmza696nly5djwoQJCA0NxUcffYTp06fj/v37esfp8uXL+e4blbxEtQra5wZxl4hzX21JREREZOqeXoxh6F9eBSiiF4WHh+HOndtIS09DckoK7ty5jYcPIwAAXt7eaNjwDSxZvAjnz5/Dzh3bcfLkCfTu3adQcSIi0lfoHk3Xrl3F1i2b8dOKlbpqdVhoKKZN+wSr1/wKS0tLbN/xJ27evAGtVovEhAQ0bdoMkyZP0VX0F8yfBy8vL1y/fg1mUilGjx6LP7dtRfv2HdC+Q0cAKFwb3t64eeM6tFotUlNTMWbseLRv3wF7du9GbGwM9u3bi3//OwZPD88Ce0zFxz/GihU/4sH9+7CwsICFpSU+/PBjVK9eA9t3/IlLly4CAELDQgEA6iw1NBoNFiz4FnYV7fDN7Ln5tgHk/Mj5888/4s7t25DJZHBxqYqPp376Svnmt/8A8PBhBH74fini4mIhCAKqVq2KDz6cCmdn50JtnxdBELBm9Sr8998x2NpWgFqtxvgJE9GkSdMC2y3Mc+dFeU2iSESmKyoqCj/88AMaNmwIlUqFr776CseOHUPNmjWRkZGBxMREpKenIzk5GampqcjMzIRMJtNtHxkZiXPnzmHDhg0G22/QoAFq166Nvn37olevXgByhnD766+/0KBBA6jVarzxRkP8999/uh5KoaGhmDt3LmbOnImuXbsabDc5ORk2NjZ6yzQaDTIyMvD48WPY2tpCrVbDwsIi39iuXbuwbds2NGvWDF26dEH37t2xdetW/Pvvv7hz5w569OiBqlWrGszhnXfewezZswEA06ZNw9q1a/H111+jc+fOAIAHDx6gTZs2WLp0KX7++WeIxWIsXLgQ+/fvR506dZCVlaU3p9XPK1fjn3/+ga+vb67Yi/r164f//e9/AIAePXrg/Pnz6NixI7777jscPnwYNWvWNNhGhQoVcPfu3TzbJSIiIiIiMqaDBw4gPDwMAKBIT8fG3zegVq3aGDZ8BABg6if/w65dO7F//15YW9tgxsyvUKeOr277guJERPRMoQtNTZs2w9o1q3D79i34+ub82HTs2FG88cabqFChAgBgwoRnV1qnp6dj7pxvcPjQQXTp+o5u+d17dzF/wSLdnA5/btuqdz+FaSM0JBgLFi6Gra0tTp48gZU//4SAgLYYOWo0bt++Weih6ARBwNIli+Hm5obPP58BqVSKQ4cOYsni77Bs+QpMmPAuxGIxtBoNJkycBAAICnqAGV98juXLVxSqDTMzMyxetBCVq1TBuvUbYWFhgVu3biEq6tFL51vQ/gPAksWLULNmLSz8bjEEQcCPPy7D0iWLsGDhIl2RJ6/tDY0x+1RERDj+/fcoflqxEvb2DkhLS8XJEycKlVdhnjsvymsSRSIyXfb29ujZs6euaDFlyhRs3LgRs2fPxrRp0zB//nxdL6Zhw4Zh3bp1+Oijj3Tbb9iwAT169MhV9MmLUqlEQkIC/Pz8AAASiQR16tRBeHi4bp1ly5ahffv2eRaZgJyCyYs9QFu0aIEWLVpAEAS89dZb2L17N/r3759vzMXFBR9++CEAYPbs2Rg7diwuXLiAzz//HG3atEHXrl1x8eJFmJnlfut92mMKyCmoHTx4EAAwZMgQPH78GI0bN4alpSWysrKgVCohEomQlpamG2LQ3Nxc97dCoUB6erqul9bzMUOe783l6OiIlJQUKBQKKBQK3VxYhtpITU1FxYoV82yXiIiIiCg/Gq2A+zGGRzlRZfFiU3p94ydMzDduaWmJgQMHv3KciIieKfTQeRYWFmjeoiWOHTsKIGfC9pMnj6Ntu/Z662k0GiTExyM5ORn1/fxw89ZNvXiHDp0KnDi8oDbat+8IW1tbAMCbb74JlUqFxMTEwu6KTmxsDO7evYOAgHaIjY1FZGQk6tTxRXp6OiIiwgtuoBBtREdH48GD+xg1aozuanhfX19dweVV5LX/0dFRiIgIx6DBQyAWiyGRSDBw4GAEBwfh8ePHBW6fH1vbCjAzM8Pdu3eRmZkJGxtbveJffu0W9rnzvN59+uL3jZt1/1av+fXlDxQRlaiGDRvqjcH8fPFarVbr3TYzM9PrtajVarFx48Y8h817fjuNJmeoDJlMBjs7O9y6dQtATuH/3r17esPjzZ8/H4mJiZg+fXqebfr7++v1zHl+eLin7T7NPb/YU3fv3sWNGzfQv39/nDt3DtOmTcOSJUvg7OyMmJgYgzk83QcAuHHjBqpVqwZBEHDo0CHs27cP8+bNw9ChQ3XHTC6XQy6X6/JRq9W6fZDL5ZDJZHjw4EGuWGE9bSM4OBhAznn73r17euvcvn0bDRo0eKl2iYiIiIieSldp8xweLzNbKLgBonJEo835R0RkqgrdowkA2rZth29mfYWxY8fj2rWrkEgkaNCgoS6+Y/uf2LVrJywtLWFpaQmlMgNVqlTWayOvHiwv04bVc0MtSSQ5u5Cdnf0yuwIASExMhEgkwvIff9BbXtHOTjff0Ou2kZWVBbFYXOB+v4y89j8lJRUSiURX7AGgu9o8NSUFlStXznf7/NjZ2WHu3Pk4cuQw9v69G2KxBD169kKjRo0LzAso+LnzorwmUSQi0zVlyhS8/fbbiImJgUqlwu7du3H0aE6Befjw4ZgwYQIGDhyImJgYHDx4EDNmzNBte+zYMVhYWKBZs2b53kfNmjXx888/IygoCN26dcO0adPQr18/9OvXD5cvX4ZMJkNAQIBufQsLC2zatAmjR4/Ghx9+iKVLl+aarLRbt24YMmQI5s6dCwC6/z09PXHu3DmoVCp06tSpwBiQU3iaNm0afvgh5z2hYcOG+Oabb3D16lVERkaiSpUqBvfrr7/+QkJCAgRBwJ49e3Dy5EmIRCL4+/tj2LBhqF27No4cOaK7YAEApk6dil69eqFPnz64cOGCrgAHAOPHjUHv3r0NxqRSKfz9/fXeKwx5//330atXL/Ts2ROXLl1Cdna2rqim0WgQGBiIBQsW5NsGERERERERvT6NlsVXIjJtL1VoqlmzFhwcHBF45jTOnTuLNm8F6H50iouLw+bNm/D9Dz/C1dUVALB9+zZcvHCh0O0XRRtisQSCULiTb1WXnLkyZs78CpUrP/vxT6PR6F2V/zptpKakQBAEhIQEw8enul5cIpG8VL4FcXJygkajwcOICLi5uwMAQkNDIBKJUMXJ6bXazs7Ohpu7O0aPGQsAuHv3Dr76cgZ+/W0D5HJ5gdvn99whorKhdu3aOHr0KPbs2QMLCwt8/vnnujmJPvnkEzRq1AgXL15E1apVMWvWLLi4uOi2zc7OLtTwmB9//DFcXFzw8OFDaLVaTJ48GX5+fjh//jwGDx6M5s0a6c4tvXv3RvXq1SGVSrFu3TqsWLECV69ezdULx9vbG+7u7jh//jwaN26MdevWYd++fbh79y7GjBmD7t27w9zcHADyjQFAREQExowZAw8PDwBA165dIRKJcPfuXezZs0evgP68r7/+Gmq1GrGxsTh+/Djcn5zDt2/fjs2bN8PMzAxbtmzBli1bdPf3wQcfoHbt2rh9+zbmzp2LoKAgeHt7AwDGjBmFxk2a4e7du7ligwYNQs2aNeHl5aU7Rk+98847utuffvop6tWrh7t372LWrFmYMGGC7vH8559/EBAQAAcHhwIfM6KyJD5NnecE5PZyCRxtDL/GiYiIiIhKgpCVMxylyDz3fOhERMXppQpNABAQ0BZ/792DR5GRGD5ipG65VCqFWCzGzZs3kK1WIyQkGH/v2QNnZ+dCt10UbVSuXBnXr19DdZ/qsLSy0vXiMaRCxYro3LkLFi6YjwEDBsLBwRERDyOw9+89mDd/YZ4/CL5MGxUqVkSHDh3x/dIlGDZsOOzs7XHm9GnUrl0bTZo2e6l8C+Lg4IDmzVvgxx9/wLDhI6HVaPHrb2sRENC20HOe5OXmzRs4dPAA2rZrD0cHR1y5fBnm5hZ6P7AWJK/nDhGVHd7e3nrzLj2vTZs2aNOmjcFY586dC9W+paUlRo4cqbesZcuWaNmyJbRaLaKjInTLn86pBOQM4zdlypQ82507d65uGDqRSIR33nkH77zzTq718osBgLu7u65I9FSXLl3QpUuXfPfLzMwMvXr1yrXc3t4ekydP1t1+Ov/VUx07dkTHjh0B5PSeAnJ6gD6NPT2uT2MA0L17dyQnJ0OtVusdIwB6Ody6dQvXrl2DWq3GzJkz4ejoCF/fnIlvlUqlXo80MiFiXsRRnBIVGvwRaLiL9eBmdiw0EZUBYWFhuHjhvO52+w4dc81JqFKpcO7cWSQnJcHL2xv16tUHERGRKRDUmQBYaCKikvfShaa33grAv/8eQ6NGjVG1qqtuuZ2dHaZO/RT79u3FoYMH4OXlhYGDBuHO7du6dSpXrgzZc8OrPV32tEfMq7QhEongUrWqbnL1gYMG4/cN67F06WJUq1YNH0/9NN/9GT1mLI4dO4pDhw4iJSUZ1dzc8cGHH+mKTHYV7fSGHDI3t4DLk15MhW1j7LgJOHjgHxw8eAAqlQrNm7dA4yZNXynfgvb//Q8+wl87d2Drls2ACGjVqjV69uxd6O3z4u/fAOqsLBw6dBCJCQlwcXHBrFmzdftYmHbzeu4QERmbj48PfHx8jHLfL/YqKm63bt1Cy5YtcerUKfj7++e5nlarhUqlglQqxciRI/WKUC8WqMiEvDA0JBERvRytVossdRYyVSrs3fs33njzTb1CU3p6OqZ/Pg0ymRW8vHywZ88uNGvWAmPHjTde0kRERERERvbShSY7e3ssW/6TwViTps3QpKn+/Bpvv91V9/e0z3JPxv5iYeVl25BKpVi+fIXutotLVYP3kxexWIz27TugffsOBuN9++n/mObm5oYlS/XnYyqoDYlEgi5d30GXrrmvgH/ZfAvaf6lUiv4DBqL/gIGvtH1+GjVugkaNm7xyu/k9d4iIyitTLdrUq1cP9erVM3YaREREJcrLywteXl5ISkrC3r1/54rv2bMLEokEs+fMg1QqRceOnfDJJx+hffsO8PD0NELGRERERETG99KFptIkOzsbMTExBmNymQx29vYlnFH+jJlvaTtWREREVLCkrAyos9WwldkaO5XXptEKuB+jyrVclaU1QjZEVF5dvXIZzZo1143s4OHpiWrV3HD5yiUWmoiIiJ7QClrcSTH8O6OjhRyVLF9vig8iMj1lutCUkpKCBQu+NRhr0qQphg4dXsIZ5c+Y+Za2Y0VEREQFS1ArIRJLYYvSX2hKV2mx71pqruVd/Ur/vhlbfJoaiQpNruX2cgnnnSJ6wePH8XBwdNRb5uDggISEBIPrq9VqqNVq3W1ByCmOP51TsbQSBAGCIJTamKnkwdirx0r7a6gsEIvFxk6BTFiqWoVdj64bjI30bMpCE1EZVKYLTQ4ODoUeFs4UGDPf0nasiIiIiKhoJCo0+CMwKdfywc3sWGgiMiDXbHgiEZDHD+I7d2zHtm1bdLctLS2wcOFCxMZEFl+CRSQTcqTk7kgKANDADEpFusFYdraVycdMJQ/GXi2mVEgRHRVrMEYlp6qrh7FTICIiE1KmC01ERESkr169eoiKioJcLjd2KkRERKWOod5LCQnxqFO7jsH1e/fpi27de+huC4IWyUnxqOLkavK9AR7EZmLX9dxFaADo6mcFmdzaYMzMTGryMVPJg7FXi8nk1nCu4mAwRkRERMbBQhMREVE5IpFIYGvLocbKonipGTSiXNfZUwnJaw4pgEPQEZUl/g0a4OzZQPTu0xcSiQQPH0bgYUQE/Pz9Da4vlUp18zkBOUPmJSflDDll6oUmkUgEUR7vK6U9Zip5MPbqMVN//RAREZU3LDQRERGVI0FBQZg6dSoWL14MHx8fY6dDRShRkwWtxAr82cU48ppDCuAQdESlSXJyMo4cPoQMVQYA4MiRw7CraIeWrVrDyckJPXr0ROCZ0/jqyxnw8amO06dPoV279vD25nsqUUnhxR1ERESmh4UmIiKiciQ9PR1Hjx5FerrhMe+JiIjKM0EQkKXOgkQiQZ++/QAAWeosaLVaAICNjS2+W7QUp0+fQnJSEiZOnIQ33nzTmCkTlTu8uIOIiMj0sNBEREREVIZlAkhTq2AjtTR2KkREJs/Ozg6DBw/Ndx2ZTIYOHTqWUEZERETPCOaWuC/kXPwASwtA/OzvTEEwXmJEVO6x0ERERERUhqVkZ8IyO4uFJiIiyiU+TY1EhcZgTJWlLeFsiIioIKkaNf4Ku5BzQ50FSM11f/eq3Sbnb0GAkKWCyJyf/4mo5LDQRERERERERFQOJSo0+CMwyWCsq59tCWdDRERFIlsNQSRioYmIShTniyYiIipHqlatisWLF6Nq1arGToWIiIiIiIiIiMoA9mgiIiIqRypVqoQJEyYYOw0iIiIiIiIiIioj2KOJiIioHElMTMSWLVuQmJho7FSoiIlE/FhHRERERETPCFkqCFkqY6dBROUAf5EgIiIqRyIiIjB27FhEREQYOxUqahJ2VCciIiIiomcEdSYEdaax0yCicoC/SBARERGVZSKRsTMgIiIiKhEarYD7MYZ7b9jLJXC0kZZwRkREROUDC01ERERERM/J60cqVZbWCNkQERFRYaWrtNh3LdVgbHAzOxaaqHQTAMHYORAR5YGFJiIiIiKi5+T1I1VXP1sjZEOUIz5NjUSFxmCMV+kTERGVfVrh5SpNQpYK0GgAiaT4kiIieoKFJiIionJEJpOhcePGkMlkxk6FiIheQqJCgz8CkwzGeJU+EVHBOKwelTeCOhPQml6hSStocSclxmDM0UKOSpY2BmOPVWmIz1S89HZEVDJYaCIiIipHatSogWPHjhk7DSIiIiKiEsVh9YhMQ6pahV2PrhuMjfRsmmfBKD5TgXWhZ196OyIqGWJjJ0BERERERERERERERESlE3s0UakS+GV1mJmV36etVqtFdFQEnF3cIBaX7zoxj0UOHgd6WVevXkXLli1x6tQp+Pv7GzsdIiqD8ppLiMMSERGRqeKwelTWCVoNhCwVROaWxk6lWLzqcHxEVHTK7y/2RERERGWJOO+x181ELERTyclrLiEOS0RERKaKw+pRmafVQFBnltlC06sOx0dERYeFJiIiIqJSTpsSD4jEgKA1GLcUiXPWgXXJJkYA8r5KWpVl+PEiIiIiIipy+VyYRkT0ulhoIiIiIirlNOlJgJVVnnGJSJSzjoSFJmPI6yrprn62RsimaOQ1PB7AAhqRqeHrlej1vOqwevm99jgcH70siQgQ4fXO2SIWmoioGLHQRERERFTKxUvNIIhEgGA4npSVAZHUDBZiUckmRmVWXsPjAaW7gFbecD6t8oGvV6LX86rD6uX32uNwfPTSNGqI8visX5bkN9eSSpNVwtmYjseqNMRnKgzGOAcVmQoWmoiIiMqRWrVq4dq1a6hataqxU6EilKjJglaax48VgoAEtRIyiFDJLLNkEyMik8b5tIiIiMiU5DfXUs+q9Us4G9MRn6nAutCzBmOcg4pMBQtNRERE5YilpSW8vb2NnQYZgczMHOKsDGOnQURERFRm5DesHoemJCKi8oSFJiIionIkLCwMs2fPxsyZM+Hh4WHsdMqUyMhIXL92Febm5nizUWNUrFjR2CnpE3HYPCKiopCZmYkLF84hOSkZnl7e8PX1NXZKRGQk+Q2rx6Epy4agBw9w995dWFtbo3HjJpDJZMZO6ZUJmmwgSwWRueWzZVkqCFrDc4mVFfkNx2cpMYNKk20wxiHpiF4OC01ERETlSHJyMrZu3YopU6YYO5Uy5dSpE/jpx+Vo1qw50tLSsGHDOsz6Zg48Pb1K5P5FkoKHuJJKpBALIgBZcKkoRVSyuvgTIyokTphOpYVCkY4vpn8OqVQKL29v7NixHa1atcboMWONnRoRERWxnTu246+/dqJZs2Z49OgRNv+xCfPmL4C9vYOxU3s12WoIghYic0sIWTk98QR1JlDGC00FDceXVyy/IenymzOJxSsqr1hoIiIiInoNarUaa9esxrDhI9GlS1cAwPLlP2D9ut/w9azZJZOEpOCPdFIzKSSinEJTVTsWmsi0cMJ0Ki327NkNAQK+nbcAUqkUnTq9jf99+jHatmtf7D2F8yvIWpiJkJlteJZ4Dt9FZHryG3KvOC6w4AUdLy8hIQFbtvyBzz+fgQYNG0Kr1eLLL7/Atq1bMfHdScZO77UJ6txztz4tPj3f44kMy2/OpFctXhGVdiw0EREREb2GoKAHSEtLQ5s2bXTLWrdugzmzZyEzMxMWFhZGzO4ZiVgCaLTo7KFGlUoCHAQ1IJHgQLDY2KkREZUaVy5fQvNmLSCV5vwo6+XlBVfXarhy+VKxF5ryK8h29bPl8F1EpUh+Q+4VxwUWvKDj5V27dhVyuTX8GzQAAIjFYrRq1Ro7tv9p5Mxen5ClAjQaQCJ5uiRnCL0nxacXC03ltQCV35B7Kk1WCWeTNw4NSKaChSYqFQQh5+o8rVYLrbb8XpH3dN/L8zF4isciB4/DMzwWOQo6DiIRIJfLIRKV7LESiUQQldE5guLj42FlZQW53Fq3zNHREVqtFomJiXB2dtZbX61WQ61+1ptI+2SoiuzsbIjFL1f0ic9UwEwkAgQBEASInv8f0FsmhgiCMhVelimwhAhyyxSobSrBLEQDZzspIGhgJjLwnMhr+avGTL29kryvfLYRtBpkZxv+UljUBK3hPPLLIa9tcoJF215JHgtTVhzH6FUe+/yIxeIyfb4Hcs75jpUq6S1zdHREfEK8wfWL8pz/Kq87xkzgvYMxxl4yVhzve8XxHlLWz/kJ8fFwcHDQ2z9Hh0pITEyEIAi59rtoz/daSJHTvpCVBWgBjUgCQQSIIMAcTwpEIgnw3N8iLXJiLywHJBAJgPnT5epsQIBuHZFGgFSd/eS2AGSpITI3f5IMnsQAkfkLj7UAXZ65d6L0x1KzVNgXdcNgrKtLvVdqU6PR4HZSlMGYpUQKlcbwqBOZGvUr55lXbKDbm3ickfbSubxKLL9t7M3lcLSQG4yVZ/GZCiRmGR6eMb9j9qrb5aew53uR8PQXfCITlpGRgcSEWGOnQUREr8jZxe2lv2CVFidPHMfq1Svx+8bNumVRUY8w5b1JWP7jz3BxcdFbf+uWzdi2bYvudoUKtpg9u4SG2CMiKmZl+XwPAKNGDsewYcPRtl173bI5c75BlcqVMW78xFzr85xPRGVZWT7nb9u2BRfOn8d3i5boll26dBHzvp2DbX/uzLXfPN8TUVlWmPM9ezRRqaDRaDBz5kwsW74CMln5rXJnZCgxftxorF7zK6ysZMZOx6h4LHLwODzDY5HDVI9DWb3SEQDsHRyhVCqRkaHUHfOEhASIRCLY2dnlWr93n77o1r2H7rZWq4VEIoG5ubnB42Sqj2lRKy/7CZSffeV+li2F3c+yfL4HAAcHByQmJuotS0xIQO1atQ2u/7Ln/OJUmp6rpSlXgPkWN+ZbvF4n37J8zndwcERiYoLessTERFSsWNHgj62ve74vbc+bolSe9x3g/pfn/S9N+16Y8xgLTVQqiMVipKSkQiyWlNmrZQpDJBJDpcqESCQu18cB4LF4isfhGR6LHDwOJa969eqQyeQ4ffo02rfvAAA4c+Y0atasBSsrq1zrS6VS3dwehVFeHtPysp9A+dlX7mfZUl72syB+fv44e/YMevXuA4lEgsjISEREhGPS5PcMrv+y5/ziVJoew9KUK8B8ixvzLV6lLd+SUq9efaSmpuLWrZvw9a0LAAg8cxp+/g0Mrv+65/vy/DiU530HuP/lef/L2r6z0ERERET0GszNzTFixEj8+ssahIeFIj09HefOncXXX3OoDCKisqZHz144c+YUZs36EtV9auDUqRN4660A+PhUN3ZqRERUhCpXrowePXth0XcL8VZAACIfPkRYWCgmTJxk7NSIiEwSC01EREREr6l9h45w9/DA1StX4ODoiAEDB8PJycnYaRERURGztbXFosVLcfLkSSQnJ2HM2PFo1KixsdMiIqJiMHTocNSrWx93796Bf4MGeO+991GhYkVjp0VEZJJYaKJSQSqVon//gSYz7ISx8Dg8w2ORg8fhGR6LHDwOxlO9eg1Ur16jyNstL49pedlPoPzsK/ezbCkv+1kYcrk1Ond+29hpvLTS9BiWplwB5lvcmG/xKm35ljQ/f3/4+fsX+/2U58ehPO87wP0vz/tf1vZdJAiCYOwkiIiIiIiIiIiIiIiIqPQp/bNMERERERERERERERERkVGw0ERERERERERERERERESvhIUmIiIiIiIiIiIiIiIieiVmxk6AKD9JiYlYtepnXLt2FZaWVmjbrh2GDBkGsbhs1UhjY2Oxffs2XL50CdnZ2ahZsyaGjxgJV9dqunVOHP8PW7duRnx8PNzd3TF27ATUqFmz0PHSRqFQ4KMPpyAhIQE7du5+bnk6Vq9ehYsXzsPMzAwtWrTCqNFjdBPnFRQvTVJTU/H7hnW4cOE8RCIxOnd+G3379YdEIgEAXLp0ERvW/4aYmBg4O7tgxMhRaNCgoW77guKlRVxcLH79ZS3u3LkNrVZAjRo1MGr0WLi6ugIAsrKy8Osva3HmzClotVo0atwE48ZNgEwmK1TcVMXFxeLdieN1t0eNGoN3unXXW+funTtYu3Y1Hj6MgGOlShg8aAhatGxVZHEyvrNnA7Fp4wbExcXB1bUaRo0eg7p16xk7rZeSlpaG3bv/wpnTp5GSkgx3dw8MGjwE9erV161T1p6roaEh+Gzap/D09ML8Bd/plkdGRmL1qp9x//492Nraolv3HujWrUeh46bkwIF/sG/f34h//Bi+dethwoR3UalSJQBAcnIyVq5cgevXrsLCwgIBbXM+vz19/yoobiqio6Px269rcffuHQBAzZq1MHrMODg7OwMAMjMzsXbNKpw9exaAgCZNmmLsuAmwtLQsVNxYfv1lLfbt+1t3+/nPWQCg1WqxadPvOHb0KDIzVahf3w8TJk6CnZ1dkcSpaGm1Wvz77zHs37cX0dFRqFSpErp164H2HToCyPk8OWrkML1tLC0tsemPrbrbIcHBWL16JcLCQmFnb48+ffqhffsOxZ77P//sw9o1q9GzV28MGzZCt/zChfP4fcN6xMbGwMWlKkaOGg0/P/9Cx4vLqlU/49DBA3j/g4/Qps1bAID9+/fhl7Wr9dZr1649Jk2eoru9f/8+7N61EykpKfDxqY7xE96Fm5tbseS4Zs0qHPhnv96y4cNHokfPXgBM7/X75cwvcOvWTb1lM2Z8hQYNGxYqnpGhxJrVq3D+/DmIRGI0b94co8eMg4WFRbHkKwgC/t6zGwcP/oPk5GT4+flj3PiJuuPz+PFjrFq5Ardu3YRMJkfHjp3Qf8BAiESiQsWLWmZmJjb/sQmnT59EZmYmWrZsjREjR8HCwgJJSUkYO2ak3voymRy/b/xDdzso6AHWrF6FsLBQ2Ds4oF+/AWjbtl2x5FqepSQnY9Wqn3H16hVYWFigTZsADBs+wuQ+FxWFoKAHmPa/T/SWVavmhu9/WK67fe3qVaxb9yuioh7ByckJw4aPxJtvNirpVIvE+nW/Yc+eXbrb2/7cqfe4arVabP5jE44cOQyVKgP16vth4sR3YW/vUKi4qRs3djQSExMAAE2bNsOn//tMLz5m9AgkJyfrLft55WpUrlwFAJAQH4+VK1fg5s0bkMlkaN+hIwYOHFxs58yilBAfjx07tuPChfPIzFTBx6c6hg4bAS8vL906586dxcaNGxAXGwuXqlUxevRYve/IBcVNkkBkwqZ/Pk2YO3e2kJSUJAQHBQljx4wSdu7Ybuy0itzqVT8Lx44dFRITE4WkpCRhyZJFwvhxo4Xs7GxBEAThzp3bQv9+vYVTp04KaWlpwtYtm4VhQwcJKSkphYqXRst+WCp8M+sroXev7nrLFy6YJ3z+2f+Ex4/jhIcPHwpT3psk/PrLmkLHS4usrCzh448+EL5bOF+Ijo4SlEqF8Oe2rUJISLAgCILw6FGk0L9fb+HggX+E9PQ0Yf/+vcKA/n2E6OioQsVLky+++Ex3HkhNTRGWLFkkvP/+ZF189eqVwvvvTxYePXokxMbGCtP+94mwePF3hY6XBu+9967w957desuSk5KEoUMGCn/+uVVIT08TThz/T+jfr7dw//69IomT8YWEBAv9+/UWjh07KqSnpwl//bVDGDyovxAfH2/s1F7KX3/tEP76a4cQHR0tKBQKYeuWzcLAAX2FuLg4QRDK3nM1KytLmPrxB8Ksr78Upv3vE73l704cL/y84ichNTVFuHbtqjBk8ADh1MkThYqbkh07/hTGjxstXL16RcjIyBCuX78m7Pprpy4+c8Z0YfbsWUJiYqIQEhIsjB83Wvjzz62FjpuK/306VViwYJ6QnJwsJCcnCwsXzBM+mfqRLv7Tj8uEjz/6QIiOjhZiYmKET6Z+JCxf9n2h48Z2+dKlXJ+zBCHn8R07ZpQQHBwsJCYmCnPnzha+mP5ZkcWpaIWEBAtr164WIiLCBZVKJZw7d1YY0L+PcPnyJUEQBCElJUXo3au7kJiYaHB7hUIhjBo5TNiwfp2QlpYmnDt3Vhg4oK9w9eqVYs07OjpKmPTueOGjD98XNmxYp1v+8GGE0L9fb+HwoYNCenqasHfvHmHggL5CbGxMoeLF5eqVK8K0/30iDB82WPjvv391y/ft2yvMnDE9z+3Ong0UBg/qL1y+fElITU0R1qxeJYwfN1pQqVTFkufq1SuFlT//lGfc1F6/M2dMF/bt2/vK8e+XLhb+9+lUITY2Vnj06JHw4QdThJUrVxRHqoIgCMKGDeuEyZMmCrdu3RRUKpVw8eIF4Z/9+wRBEAStVitM/fhDYdF3C4TkpCTh3t27wsgRw4T9+/cWKl4c5s75Rvjii8+EsLBQQaVSCYcPHRQuXDgvCIIgJCYmCr17dc/zN4P09DRh5IihwsaNG4S0tDQhMPCMMKB/H+HGjevFlm959dVXM4RZs74UEhMShNDQUGHC+LHC1i2bjZ1WsXjw4L4woH+fPOOxsTHCwAF9hX17/xbS09OEQ4cOCv379RYePnxYglkWvatXrwi9e3XX/cb31K6/dgpjRo8QgoIeCImJicK8eXOFz6Z9Wuh4afHTj8uEhQvm5Vo+etTwfD9vfPrJxzmfxZOShPv37wmjRw0X/v57TzFmWnTWrftVOHjwgBD/+LGQmpoirFjxozByxDAhIyNDEARBCA8LE/r36y0cPXJYSE9PE/bs2SUMGthPePw4rlBxU1W2uoVQmRIRHo67d+9g5MjRqFixIry8vdG9Rw8cOnTA2KkVuXHjJyIgoC3s7OxQsWJF9OnTD/Hx8Xj8+DEA4PDhQ2jQoCFatGgJa2tr9Os/AJaWljh9+mSh4qXNhfPnEB0dhS5d39FbnpKcjPPnz2HosOFwdKwEV1dX9Os/AEePHoFGoykwXpocP/4fUlNT8P4HH8HJyRlWVjL07dcfnp45Vz8cPXoEnp6e6NipM+Rya7z9dldUq+aGf48dK1S8NAkPC0dAQFtUrFgRNja26NChIx5FRiI7OxtqtRr/HjuKAQMGwcXFBZUrV8aQocMQeOY00tJSC4yXZidPnoBcbo2+fftDLrdGq9ZtUL++H44cPlQkcTK+I0cOo3btOggIaAu53Bo9e/aGvb09Thz/z9ipvZSePXujZ8/ecHJygkwmQ7/+A6DVahEcHASg7D1Xt27djDp16qJmzVp6y69du4qkpESMGDkSNja2qF/fDwFt2+Hgk881BcVNhVKpxPY/t2Hs2PHw8/OHpaUl6tWrr7taPjIyErdu3cTIkaNgZ2cHT08v9OjRC4cOHihU3JSEh4ehXdv2qFChAipUqID27TsiIiIcgiBApVLhxInjGDhoMJycnFClShUMGjwEJ0+eQEaGssC4KTt08CC69+gBLy8v2NnZYeTI0bhz5zYiIiKKJE5Fy9PTC2PGjEO1am6wsLBA48ZNUK2aG4KDggq1/dmzgdBqtRg8ZCisra3RuHETNGnSFIcPHSy2nLVaLZYv+wHDR4yCtbW1XuzokSPw9vZB+w4dIZdbo2vXbnB2dsF///5bqHhxUCgUWL36Z0yaPOWlr6I+dPAAWrVqgwYNGsLGxhbDho9Aeno6Ll++VEzZFpRP2Xn9KhTpOH36FAYPHorKlSvDxcUFAwcOwn//HkNWVlaR319SUhL2/r0H7056D3Xq+MLCwgJvvPEmOr/dBQDw4P59hIWFYuSoMahQsSJq1KyJLl264tDBg4WKF7Vbt27i+vVr+PjjT+Du7gELCwu079Cx0D1DAs+cgUgkwsCBg2FtbY2mTZuhUaPGOFSM54byKDo6GjeuX8eIEaNgZ28PDw8P9OzVGwdN8HNRSfj32DE4O7ugS9d3IJdbo0OHjvD29sHRo4eNnVqxOHToAN7p1h3e3j66c+79+/cQFhpaqHhZFhT0ACEhwRj15JxZvXoNdO3azSS/MxgyYsQodOzYCQ6OjrCxsUX//gORmpqCqEePAOR8169RoybatmsPudwa3br1QKXKlfHff/8WKm6qWGgikxUWFgqZTA4XFxfdMm9vH8TFxUGpNO0v6K/r9q2bsLa20Q1BEx4WCh+f6rq4SCSCl5c3wsLCChUvTVJTU/HLL2sxafL7EIv0T1HhEeHQarV6++rt7YOMjAzExsYUGC9Nbt28AS8vb/z003IMGtgPEyeMw5/btkKr1QIAwsPC4P3cfgI5+xoWFlqoeGnStFkzHP/vX6QkJyMtLQ1Hjx5B48ZNYGZmhujoaGRmZsLHx0e3vo+PD7RaLSIiHhYYL83CwkLh7e2tt8zbx0f3un/dOBlfzuvYR2+Zt7cPwsLDjJNQEbl37y60Wi3c3T0AlK3n6v1793A2MBBDhg7LFQsLC4WLS1VYWT0bttPb2wfhz+1nfnFT8eDBfajVajyOj8eY0SMxfNhgzJ//re7imPCwUFhaWuoN/+vt44OEhASkpaUVGDclTZs2w7//HkVqaipSU1Nx9NgRNG3WHCKRCI8eRUKtVuf6zJGdnY3IyMgC46ZKoVDg8eM4vbxdXFwgk8kQHhb62nEqXmq1GpcvX0JMTAz8GzTQi334wRQMGTwAn037BJcuXdQtDw8LhYenp95QPsV9jt2zZxccHBzQpEnTXLGwF77XPMsntFDx4vDrL2sQENAuz+HuHjy4j8GD+mPM6BFYsvg7xMXF6WJhYWF6n0MtLCxQrVq1Ys335MkTGDigLyaMH4PVq1YiPT0dwOu/vovL5j82YeCAvpg8aQK2bduC7OzsQsUfRjyERqOBT/Vnx9fbxwdZWVmIiooq8jzv3rkNMzMzhIQEYdTI4RgxfCgWL1qIpKQkADnPTTs7ezg4PBvSytvHB5GRD5GdnV1gvKjdvHEDrq7VsHvXXxg6ZCDGjhmJX9auRmZmpt56U96b9OTc8CmuXL6sWx4WFgZPT69c5waey4tWWFgopFKp7nMxAPh4+yA5OQkpLwwpVlZkZ2djxPChGDZ0ML788gvcu3dXF8s5x7/w/cfH9D4PF4WMjAzExMTonXOdnJxgbW2DsLDQAuNlxZLF32HQwH748IMpeoXssLAwVKhQQfe7KJDzWToq6lGxXExQ3G7fuglLS0s4u+QMwR0WnvvzjI/3c9+BC4ibKs7RRCZLmZGRaw4VuVwOAMhQKk1+fpVXFfTgATZu3IB3J72n+1CnVBo+FhlPCm4FxUuT1atXomOnTnB1dcXj576kATmPu5mZGczNzXXLnj4nlEplgfHSJDU1FdeuXcXo0WMxceIkhIeHYd63c2Ftk9M7SWngNSCXyxEVlXN1REHx0mTo0OH4du5sjB6dM3a/a7VqmDnzKwDQXRUuk8l161tZySAWi5GRoYSZmSTfeGmmzMjQPb+fksvlUD7Zr9eNk/EplUrIZbkfo6c/6JdGiYkJ+OH7JejZq7dunpuy8lzNzMzEjz8uw4SJ7xqcGyIjj/fqp+9PBcVNRVpaKgRBwIXz57Dwu8WQSCRYseJHfLdwPhZ+tzj/z28ZygLjNjY2JbMjhTBixCjMnfuNbn4bd3cPzJj5JYBnnyue35fnP3M8nU80r7ipeva+qv8YyWQ5r7nXjVPxmTxpAmJiYiAWizF06HDdjxO2tra6ebjS09Nx7NhRLJj/LWbP+RY1a9Z68pp84Rwrkxfb56SHDyOwf99efLdoqcF4RoYSMvkL5wiZHI8fxxUqXtQunD+H8PAwvTmXntelS1d06dIVWq0WMTExWPfbL5g7ZxYWLf4eUqn0Sb76x1cmlyNDmVEs+Y4bNwHjxk2ARqNBeHg4fl7xI374fgm+mPGlSb5+v5k9F0DOnKr37t3Fsh++R6YqE8OGjygwrjTwPeD595OilpqWiszMTNy6dQvff78M2dnZWLbseyz7YSm++vobKA09N+VyaLVaZGaqCoybmen37iuKfMPCQlG3bj2sXvMr4uPjsXDBPIjFYowaPRZ2dna6c0POxXyHMX/+XMyZOw/Vq9fIee7mer+2hrKYnrvllaHftp6eM5QZGahQsaIRsio+Pj7VsX3HLgA5c3bu3v0Xvpn1FRYv+QFOTk7IyMhAlSpOetuY4ufhoqDKyHkt5XoPlsuhzMgoMF4W/PLregA5RbcrVy7jx+U/wEwiQdt27fM4B8l1Iws8/7ufqYuIiMDatWswctQY3UWFhr77yeRy3ZxVBcVNFXs0kcmSWVnlejNRKBQAAEsrK2OkVOzu37uH2bNnYeSoMWjRoqVuuUxm+FhYPTnpFBQvLc6cOY242Bj06NHLYNxKJkN2drbe1QtPnxMymazAeGliZWWFKlWqoEvXd2BlZYVatWqjTZu3cOH8eQA5+2PwMX/y2igoXlqo1WrM+OIz+FSvjnXrf8eG3zehceMmmP75NGRkKHVv0kqlQrdNRoYSWq0WVlayAuOlmczKCkqFQm+ZQqGA7Ml+vW6cjE8mk0GhzP0YlbZz+1Px8Y/x5cwZaNSoCYYMedbjp6w8Vzf/sQm+vr55TtBqlcd79dP3p4LipsLKUgZBENB/wEA4ODigYsWKGDRoMIKDg5CYmJDv5zcrK1mBcVORmZmJ6V98hrp162P9hk1Yv2ET/Pz88cUXnyMzM1P3uDy/L89/5igobqqevW/qP0ZKZc5r7nXjVHx+WrEKW7Zux9ez5mDv3j3Yu/fvXOtYW1uje/ce8PWtizOnTwF4+p3rhXOsUlFsr8ecIfNGokKFCgbjVlYyKBUvnCOee/4UFC9KCkU61qxZhUmT39fr1WGIWCyGi4sLprz/ASIjIxEaGvJcvvrHV6lQwEpWvJ/JJRIJvLxyhlW8fPkSFIp0k379mpubo169+ujXrz9OnjxeqLjMwOf84nw/sbK0giAIGDxoCCpUrAgHR0f06z8AN25ch0qlgszQc1OhgFgshoWFZYHxIs/XygoSiQRDhw2HTCaDm5sbunR9B+cvnM+1ro2NDXr27I2atWrhzJnTT7Y39H0yHbJifu6WN1YGvrc/PWfIStl395dVsWLFnCED7exx8eIFADnP21zvSSb4ebgoPP1d0+D+WlkVGC9LrKys0Lx5C3To0BEnT554sszwb1oikQiWlkV/ziwuYWFhmPX1TPTp2w8dOnTULTf03U/53Hf9guKmioUmMlkeHp5QKhV63d6Dg4NQuXLlXFc2lwU3blzHnDmzMHbsOLRv30Ev5u7hiaCgB7rbgiAgJCQYHh4ehYqXFg/u30dwcDD69+uNPr17YM6cWQCAPr174Pjx/+Du5g6xWKy3r8HBQU+KMk4FxksTD0/PXMsECLpx4d09PBD83H4COfvq4eFZqHhpER0VhaioKHTv3hM2NraQy63Rq1cfJCQkIDwsHM7OzrCwsEDQc/MQBAUFQSwWw82tWoHx0szDwxPBwcF6y4KDgnSv+9eNk/HlvI7159gIDgmGx3NDa5QW0dHRmPHFdDRt1gyjx4zVi5WV5+qDB/dx6NBB9OndA31698C2bVvw4MF99OndA9HR0fDw8ERU1CO9q6yDg4Pg/tx+5hc3FYaOuyDk/C8SieHu4QmVSoXIyGfDkwYHBcHBwQE2NjYFxk3Fw4cRiIuNRY8ePWFtbQ1ra2v07NkLcbGxiIx8iKpVXSGVSnN95jAzM4Orq2uBcVMll8tRqVJlvbyjoqKgVCrh7uH52nEqXlKpFL6+vnizUWPcuH6tUNu4e3giLDRUbz7T4jrHajQaBAcHYemSxbpz5a1bN7Hrr50YN3Y0gJxzYdCLn2GDnn2GLShelOLi4pCQkIBPP/lIl29aWhqW/bAUs7/5ulBteHh46H0OzczMxMOHD43ymbysvX6ruVWDRCJB0INnxzc4KAjm5uZ6w+8XFY8nc+U+TxByvp+JRCJ4eHgiKSkRCQkJevm4ulaDmZlZgfGi5vnkMROevknn3IC4kPOMeXh4IDQ0JNe5wRSfC6WZh4cn1Go1wp8bGjsoOAgVK9qVud5MhZFzjn/h+0+Q6X0eLgpWVlZwcnLSO+fGxMQgPT0NHh6eBcbLOg8PD6SkpOiN5hEcHAQXl6qlpjfTgwf38fVXM9B/wEB0795DL+bhnvvzTFDwc9+BC4ibKhaayGS5ubujVq3aWLfuVyQnJyMkJAR7du9Gh46djJ1akbt06SIWLpiPyZOnoFXrNrniHTp0xJUrl3H69CkoFOn4c9tWqFQqtGjRqlDx0mLEyFHYsXO37t+MGTnDo+3YuRtt2ryFChUronHjJtj4+wbExz/Go0eR+HPbVrRt2x4SiaTAeGkS8FZbJCcnY/++vVCpVLh79w5OHD+OZs2aAwDatm2H0NBQHDp4AAqFAgf+2Y+HDyMQ0LZtoeKlRRUnJ1hbW2Pv3j1IS0uDQqHA7l1/wdLSElVdq0IqlSKgbTts27oFUVFRePz4MTZt/B3NmjWHjY1tgfHSrGWr1lAo0rF9+zYoFOk4efIErl+/hvZPrpJ53TgZX/v2HXDnzm389+8xKBQK7Nq1E4kJCWjd5i1jp/ZSIiIiMHPG52jfoQOGDh2eK15Wnqtzv52v9x7Wv/9AVK9eAzt27oazszP8/PxhZ2eP9evWIS0tDTduXMd//x5DxyefawqKmwoHR0c0atQY27ZuQWJiApKTk7Flyx+oVas27Ozs4OrqCl/fuli37jckJSUhLDQUu3f/hY4dOwNAgXFT4eJSFTKZDHv+3g2FIh0KRTr27NkNmUwGZ2cXWFpaonXrNti65Q/ExMQgNjYWm//YhJatWsPKSlZg3JR17NQJe3bvRkhICJKTk7Fu3a+oVau2bn6a141T0Trwz34cO3YUycnJUKlUuHr1Cs6fOwvfuvUAAAcP/oPDhw8hKSkJ6enp2Lv3b9y6dRPNmrUAkDMXmVgsxh+bNiI9PR0Xzp/DuXNni+U7l0Qi0TtP7ti5G76+ddGzV2+sWfsrAKBtu3YIDg7CkcOHoFAosG/f34iOjsJbAQGFihclT0+vXPna2Njg/Q8+wswvvwYArPhpOW7duomMjAxER0fjpx+Xw8XFBZ5PihIdO3XGyZPHcfXqFaSlpeH3DethbW2Nhg3fKPJ8AeC7hfMRHByEzMxMhIaG4Ndf18LfvwHkcusn+ZjO6zcqKgprVq9EZGQkMjMzcfPmDWzfvk33HbaguFxujRYtWuKPPzYiLi4O0dHR2LJ1M956K6BYfoR0c3NDnTq++GPzJqQkJyMhIQHb/9yGBg0awsLCAtVr1ICHhwfW/fYLUlJScP/+Pezfv0/3WiooXtTebNQYFSpUwKaNv0OpVOYMW7l/H5o++T75zz/7cOTIYSQ/mQd3z57duHvnDpo1zYk3a94cWq0WW7b8AYUiHefOBuLChfMm97mktHN2dka9+vWxfv1vSEpMRFhYGHb9tbPMHuc/t23F6dOnkJaWhpTkZPz++3okJSXijTfeBAC8FdAW0dFR2L9vLxQKBY4cPoTg4CC0a9feyJkXj44dO2Pv33sQHBykO+fWqFFTd+FxQfHS7MqVy/hz21bExcUiIyMDgYFncOTIYd3oTj4+1eHl5Y3ffvsFKcnJCHrwAPv2/V1qXhu3bt3C7G++xsiRo9Gp09u54u3at8f9+/dw7OgRKBQK/P33bsTFxqJNm4BCxU2VSNC7vIHItCQlJmLlyhW4fv0aLC0t0bZdewwZMkw37n1Z8dm0T/Hgwf1cy+fMnYfatesAAE4c/w9btmxGQkI83NzcMXbceNSsWUu3bkHx0ujK5cuYM2eWbuxoIKe7/qpVK3Hp4gVIJBK0aNkKo0ePhVQqLVS8NLl16xbW/fYLHj6MgJ29PTp3ehs9ej4bVvDSxYtYv/43xMbGwMnZGSNHjEaDhg0LHS8t7t+7h42bNiA0JASCIMDNzR2DBg/RDU+VlZWFX35ZgzOnT0Gr1aJx4yYYN36irnt9QXFTNmzo4Fxd5Xv26o1hw3LGrb9z5zZ+WbsaDx8+hKOjIwYNHoKWLVvr1n3dOBlfYOAZbNr4Ox4/jkPVqq4YPWYs6j758bC0WLVyhd7Erk+NGTseXbp0BVA2n6tbt2zGlSuXMX/Bd7plkZGRWLVyBe7fvwdb2wro1r2H3tVtBcVNhUKRjrVrVuPChfOQSMxQv359jB49Fnb29gCApKQkrFr1M65dvQILCwsEtG2HoUOH6y76KChuKu7cuY1Nm37XTUDt7uGBIUOG6T6bZWZmYu2aVQgMPAMAaNK0GcaOHa8bpraguLEcO3YUP/24LNfytb+sg52dHbRaLTZt/B3Hjh2BSqVC/fp+mDhxku7xfd04Fa20tDRs27oZZ88GQqlUonLlKgho2w7dunWHSCRCeno6tm7djLOBZ6BSZaJaNVf07tMPb77ZSNdGcHAQ1qxeidDQUNjZ2aNvv/65RlgoLl/O/ALVa9TQfbYBcuZF2vD7esTFxsLZ2QWjRo2Bn79/oePFaeSIoRg1eizaPLnoIyjoATb/sQn37t2FlZUV6vjWxeDBQ1GlShXdNvv37cWuXTuRkpKC6tVrYPz4iXBzdy+W/C5fvoTtf25FWFgYbG0r4M03G2HAwEG6HqOm9PoVBAFHjx7Bvr1/IyYmGo6OjmjVug169eoDqVRaYBzIGRJ7zepVOHfuLMRiMZo1a44xY8cbnCuxKKQkJ2P16pW4evUKzM3N0bDhGxgxcjRsbXMuYIuLi8OqlStw+/YtWFlZoVOnt9F/wEDdqBQFxYvaw4cRWLNmFR7cvw9raxu0at0agwYNgVQqRVpaGrZu2YyzZ88gMzML1apVQ99+/fWKoEEPHmDNmlUICwuFvb09+vUfiLZt2xVLruVZSnIyVq36GVeuXIa5uQUCAtpi2PARJve5qCgkxMfjjz824vLlSxAEAZ6eXhgwcBBq1aqtW+fq1StY99uviI6OQpUqThg2fAQaNWpsxKxf3fHj/2HZD7nnJFy9+hc4ODpCq9Vi8x+bcOTIIahUKtSrVx8TJk6Cg4MDABQYN3Xfzp2NS5cu6i2rVas25n47H2q1Gn/v2Y2jR3MK3lWqOKFz57fRsdOzi8/i4x9j5cqfcevmDVhZWaFDh04YOGhwsZ0zi9Lsb77G1atXci3/fPoM3Wewc+fOYuPv6xEXFweXqlUxatQY1K/vp1u3oLgpYqGJiIiIiIiIiIiIiIiIXknZ6hZCREREREREREREREREJYaFJiIiIiIiIiIiIiIiInolLDQRERERERERERERERHRK2GhiYiIiIiIiIiIiIiIiF4JC01ERERERERERERERET0SlhoIiIiIiIiIiIiIiIiolfCQhMRERERERERERERERG9EhaaiIiIiIiIiIiIiIiI6JWw0ERERERERERERERERESvhIUmIiIiIiIiIiIiIiIieiUsNBEREREREREREREREdErYaGJiIiIiIiIiIiIiIiIXsn/Aa9xLVI+2dpQAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
datasetkeggvariant_effect_codingvariant_effect_non_snv
ref_bpmean4006.04096.04096.0
50%4001.04096.04096.0
max4070.04096.04096.0
var_bpmean4008.04096.04096.0
50%4001.04096.04096.0
max4260.04096.04096.0
ref_dna_tokensmean672.0687.0687.0
50%672.0687.0687.0
max681.0687.0687.0
question_tokensmean216.040.042.0
50%165.039.042.0
max980.082.059.0
answer_tokensmean5.011.026.0
50%4.010.020.0
max10.032.0221.0
reasoning_tokensmean460.00.00.0
50%458.00.00.0
max649.00.00.0
\n", + "
" + ], + "text/plain": [ + "dataset kegg variant_effect_coding variant_effect_non_snv\n", + "ref_bp mean 4006.0 4096.0 4096.0\n", + " 50% 4001.0 4096.0 4096.0\n", + " max 4070.0 4096.0 4096.0\n", + "var_bp mean 4008.0 4096.0 4096.0\n", + " 50% 4001.0 4096.0 4096.0\n", + " max 4260.0 4096.0 4096.0\n", + "ref_dna_tokens mean 672.0 687.0 687.0\n", + " 50% 672.0 687.0 687.0\n", + " max 681.0 687.0 687.0\n", + "question_tokens mean 216.0 40.0 42.0\n", + " 50% 165.0 39.0 42.0\n", + " max 980.0 82.0 59.0\n", + "answer_tokens mean 5.0 11.0 26.0\n", + " 50% 4.0 10.0 20.0\n", + " max 10.0 32.0 221.0\n", + "reasoning_tokens mean 460.0 0.0 0.0\n", + " 50% 458.0 0.0 0.0\n", + " max 649.0 0.0 0.0" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "share of reference spans truncated at 1024 tokens: {'kegg': 0.0, 'variant_effect_coding': 0.0, 'variant_effect_non_snv': 0.0}\n" + ] + } + ], + "source": [ + "from transformers import AutoTokenizer\n", + "\n", + "llm_tok = AutoTokenizer.from_pretrained(\"tokenizers/prism-olmo-1b-interleaved\")\n", + "dna_tok = AutoTokenizer.from_pretrained(SNAPSHOTS[\"dna\"], trust_remote_code=True)\n", + "\n", + "LEN = H.length_frame({k: DS[k] for k in (\"kegg\", \"variant_effect_coding\", \"variant_effect_non_snv\")},\n", + " llm_tok, dna_tok, max_rows=1500)\n", + "H.plot_lengths(LEN, max_dna_length=1024)\n", + "plt.show()\n", + "display(LEN.groupby(\"dataset\").describe().T.loc[(slice(None), [\"mean\", \"50%\", \"max\"]), :].round(0))\n", + "print(\"share of reference spans truncated at 1024 tokens:\",\n", + " LEN.assign(t=LEN.ref_dna_tokens > 1024).groupby(\"dataset\").t.mean().round(3).to_dict())" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "cell-016", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "37 distinct KEGG answers; top 15 cover 85%\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAALpCAYAAABPMMW0AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAA0HNJREFUeJzs3Xd0FNX7x/F3ei8kgZAAoffeewkgvYqKFcQOon4VRUHFLooiIBbALgpIE0RFkI703luoCWkkJKTX3d8f+TGypNPCxs/rHM5hZu7c+9y72eyTO3dmbcxmsxkREREREStjW9IBiIiIiIhcCyWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJfuSDkDE2oWEhPDPP/9wzz334OrqauyPjY3l999/p0KFCnTv3t0ol5cGDRrQokULi33nz59nz549JCYmEhAQQK1atQgMDMw3juKWv5kOHz7M/v37uffee0uk/duNyWRiw4YNhIWFUb58ebp3717SIcltaNmyZdSpU4eaNWsW67yLFy+yadMmLl68SNu2balVq9ZNirDosrOzmT17NsHBwVSuXLmkw8klPT2duXPncscdd1ChQoWSDuc/b+3atZQpU4YmTZoU+1zNyIpcp61btzJ69Gji4uKMfVFRUfTu3ZtPPvnE+FC5XG79+vXs2LHD4t+5c+cszh06dChNmjRh5syZrFmzhg8++IB69epx5513EhoaatF+cctf7eDBg8yfP/8GjkjOL6WxY8fe0DpvJ8UZM7PZzMCBA3n++efZtGkTx44du8nRlZw//viDbdu2lXQYVmnnzp2MHDkSX1/fYp135swZGjVqxKxZs9i+fTtRUVF8//33Fr9TSkJmZiajR49m9+7dJRpHfhITExk9ejSHDx8u6VAESEhI4MEHHyQ9Pb3Y52pGVuQGCw0NpV+/fjg5ObFy5Ur8/f0tjr/77rv5zgAkJibSq1cvvL292b9/PwEBAcaxEydO8OijjxISEkKlSpWuqXxeVq9ezbRp07jnnnuup9sW6tWrx3333XfD6rvdFGfMzp07x9q1a9mwYQPNmjW7BdGVnI8//pimTZvSunXrkg7F6kyYMIFHH30UHx+fYp23dOlSvLy8WLZsGQBJSUn07t2buXPnEhQUdDNCFbnhBg4cyNtvv83XX3/N008/XaxzlciK3EAhISH079+fcuXK8euvvxb7Q+mzzz7jzJkzuZJSgJo1a/LHH39w8eLFay5/tcOHD7Nz507S0tL4/vvvAahcuTLBwcHs37+f48ePM2jQIDZu3Mj58+fp2LEjJpOJ9evXA+Do6EhQUBAtW7bEycnJqNff35+mTZsa25frGjx4MLt27eLMmTNUrlw5V8Jz+XJfixYtaNCgQb5xX65v4MCB7N6926ivTZs2QM6M6aFDh/Dz86NTp044ODjkqmPXrl0cP34cd3d3OnToQJkyZSyOh4eHs2vXLrKzs2natKlxebSgMbva3r17+e233wBYs2YN+/fvp1OnTiQlJeU5tpUrV8ZkMrF161bOnj1L2bJl6dSpE46Ojhb1xsfHs27dOjIyMmjZsiVms5l//vmHYcOGGWXmz59PvXr1LMYxJCSErVu38uCDD1rUV1ibv/32G4GBgVSvXp3du3cTHx9P8+bNqVKlilFm+fLlxMTEcPToUWNcOnfuTNWqVfN8DdeuXcvZs2cB8PDwoHbt2rle86K0e+VYnzhxAl9fX1q0aIGnpycA+/btIyQkhCFDhhhlFyxYgLOzM/379zf2zZs3jyZNmlCnTp1ijUmVKlXYvHkzcXFxDB8+vMBY8nPo0CE2bNjAtGnTijVGS5cuZf369WRnZ/P9999jMpnIyMgwzo2NjQXgvvvuM96fSUlJxjKEGjVq0LJlS6O+Ky+3p6SksGfPHry9venRo0e+sScnJ7NlyxYuXrxIgwYNqFevXoF9Lah9gLlz55Keno6NjQ1ly5alcePGFn/4XxljdnY2e/fuxd7enrZt2+Z6DxeXyWRi4cKFuLq60q9fP2P/gQMHOHz4MK6urnTq1AkvLy+L89LS0li3bh1xcXHUq1ePoKAgli5dysCBAy1iKmq5guzfv9+Y5XZxcaFq1aq0aNECW9t/L7D/888/pKSk0KlTJ3bu3El4eDh169alYcOGueo7efIk+/fvx9HRkRYtWhgTL6GhoaxevZoHHnjA+P25Zs0awsLCLH7PrFixAg8PD9q1a1ek8bocW4cOHdi4cSPR0dH07dsXHx8fHnjgAb766itGjRqFjY1NkcYDlMiK3DCHDh3iqaeeolatWixYsAAPD49i1/Hbb7/Rtm3bfGdQPTw8LOotbvmrRUVFERoaSmZmJjt27AAgKyuL4OBgli9fzueff86nn36Kq6sr1apVo379+phMJqNseno6u3fvJjs7m0WLFlGjRg0g50P0ww8/NNbIXq7r888/x8XFBT8/P1atWkX37t358ccfjXguX+577733Ckxkly9fzmeffcbEiROpWLEibm5uLF++nLvuugsHBwd27txJvXr12LRpE0FBQaxYsQJ7+5xfd6mpqQwdOpQ9e/bQqVMnwsLCePzxx/n666+ND69vv/2WV155hQ4dOuDh4cFbb73FkCFDeO211wocs6udP3/euHS5b98+PD09qVevHmvXrs13bIcOHUpsbCxt27bl6NGjZGdns2zZMipWrAjAjh07GDJkCAEBAdSsWZPXXnuNatWqsWXLFosPmNdee42nn37aYhy3bt3Kc889Z5HInj59utA2J06ciKOjIxcvXqRRo0YkJiby2GOPMXPmTGNW+siRIyQmJhIdHW2MS8OGDfNNZE+ePMmePXuAnMR8/fr1dOnShR9//NH4UC5Ku9nZ2dx///1s377d+CPh5MmTTJ8+nY4dOxIaGsojjzxCt27d8Pb2JjY2lkceeQQXFxdCQ0NxcnIiNDSUxx57jFWrVhVrTBwcHIiJiaFx48b4+Pjw4IMPFhhLfn777TeCgoJyrY0tbIwOHDhAeHg4KSkp7NixA5PJRFZWFpBzRebyZdq77roLJycn/vrrL5544gn8/f2pU6cOmzZtok2bNvz444/Y29sb77/OnTtz7tw5WrZsSdOmTfNNZP/44w9GjhyJv78/9erV4+OPP6Zdu3ZMnTo1z/KFtQ85f2CmpqZiNpsJDw9n8+bNvPHGG8Ys3eUYg4ODCQ0NpUmTJhw5coTo6GhWrFhxzeuDk5OTeeSRRzh+/DiLFi0Cci53Dx8+nK1bt9K5c2diY2N5+umnWbBggfFHeGhoKH369MFkMtG8eXM++OADgoKCWLt2LS1btjQS1KKWK0x4eLjx/kpOTmbr1q2UK1eOpUuXGnXMnj2bLVu24OjoSOXKlXF0dOSpp57i2Wef5c033zTqeuONN5g1axadO3fG1taWcePGMWbMGIYPH47ZbGb06NHUqlXLSFKffvppQkNDadasGQ0aNMBsNvPEE0/w6quv0q5duyKN1+XY7OzsqFq1KgEBAQQHB+Pj48Mdd9zBhAkTOHToUIG//3Mxi8h1mT17ttnNzc3s5+dn7tKlizklJaXAclOmTDF/9913Fv/Onj1rNpvN5rJly5pHjx5tcd6ZM2csyu7YscM4VtzyeZk6daq5atWqufZ/8MEHZjc3N/OsWbMKPN9kMpnvv/9+89ChQ419n332mblSpUq56vrll1+MfVu3bjW7ublZxJeYmGgeNWqUedWqVQW2ebm++fPnG/tmzpxpdnNzM48YMcKcnZ1tNpvN5nPnzpk9PT3NS5YsMcq988475goVKhhjbjabzWPGjDFXqlTJHBMTYzabzeZatWqZp02bZhzPzs42r1ixwtjOb8zysmvXLrObm5s5JCQkV/xXjq3JZDK3a9fO3KJFC/PFixfNZrPZnJmZaR44cKD5wQcfNJvNZnNWVpa5ZcuW5oceesjoY2RkpLlatWpmT09Pi3Zr1qxpnjp1qsW+2bNnm318fIrVptlsNrdp08YcFBRkPn/+vLHvlVdeMdeuXdui/i5dupiff/75Io3L1SIjI81BQUHmefPmFavd7du3m93c3CxezwsXLpi3b99uNpvN5kuXLpm9vLzMS5cuNZvNZvOCBQvMTZo0MVetWtW8du1as9lsNn///ffmgIAAc2ZmZrHGxN/f33z69Okix5KfgQMHmu+7775rGqPXX3/d3KZNG2M7MTHR7ObmZv7tt98szj137pzZz8/P/OSTT1r87FSpUsX8zTffGLG6ubkV+HvsstOnT5t9fHzM48ePN5tMJmP/5fdJamqq2c3Nzbx48eIit5+XVatWmb29vc2hoaEWMfbt29ecnp5uNptz3hft2rUzP/XUUwXGfKXL9axcudIcERFhbt++vblHjx7m2NhYo8wTTzxhDggIMJ84ccLY98orr5ibNWtmzsrKMpvNZvMDDzxgbt++vTkpKclsNpvNaWlp5u7du5vd3NzMBw8eNM4rarniSk9PN3fs2NE8fvx4i7g9PT3NW7ZsMfbNmTPH7OHhYb5w4YLZbDabk5OTzZ6enuZly5YZZdLS0sxr1qwxths3bmx+5513zGaz2Xzs2DGzt7e3uWvXruZPP/3UbDabzbt37za7ubmZjx8/XuTxeuKJJ8xubm7m1atX5+pLVlaW2dfX1zxz5sxijYFu9hK5QZo0acL+/fvzfTLBZfv27ct1s1d8fDyQM7t0eWbisosXLxrlxowZw+LFi41jxS1fXM7Ozjz66KO59l++RDZv3jx++OEH3N3d2blzZ4F1eXp6WqwpbdWqFY6OjoSEhBj73N3d+fzzz+nWrVuhsXl6enL33Xdb1AcwfPhwY0avUqVKBAQEcOLECaPcwoULue+++yzWD44dO5aLFy+yevVqo+7du3cbyzJsbW0LvLx6La4e23379rFv3z5eeuklY2bF3t6e5557jiVLlhAfH8/hw4c5fPgwzz//vNFHf39/HnrooWuKoShtXta3b1+Lp2C0b9+esLAwUlNTr6ltgIiICP766y9mz57N8uXL8ff3N2abitru5SsO69evN2Yj/fz8jEvWnp6etGjRgrVr1wI5Vwu6detGcHCwsW/NmjV07NgRe3v7Yo1J7969LZY5FBZLfsLDwylXrtw1j1FRLFy4kNTUVN5+++1cPzuXl4JcNmLECFxcXAqtz9bWltdff93iMnB+75PitH/o0CGWLl3KDz/8QGhoKCaTiX379lmUGTZsmLHUw87OjjZt2lj8LimqgwcP0qVLF2rXrs2yZcuM5WCpqaksWLCARx991LjSBDBmzBiOHTvG5s2bycrK4vfff+eJJ57Azc0NACcnJ0aNGmXRRlHLFdWlS5dYtWoVP//8M3PmzMHPzy/Xz0TDhg2NpVaQ874xmUycOnUKAAcHB1xdXdmyZQvJyclGTFdeWQoODmbdunVAznukZcuWDBgwgDVr1hj7KlWqRM2aNYs0XpfVrl2brl275uqXnZ0dPj4+hIeHF2s8tLRA5Ab56quvePnll7nvvvv45Zdf8k3GCrrZq0KFCpw/f95iX9OmTfn8888BjBs6rrV8cZUtW9Zi7RXkrHF64IEHjMuDHh4enD17lpiYmALr8vb2tti2sbHBwcGBtLS0a4rt6vour+O6+hKdo6OjxZ2w586dy7XGsly5cri5uRl3en/zzTe88sor1KxZkzp16tClSxeeeuqpAm+aK66rx/byWsiTJ09afLDHxMRgNps5ffq08Qv+6pt48lozWhRFafPyWuerx/vymsv09PRCk568vPPOO0yfPp3mzZsTGBiIs7MzSUlJuX6OCmu3Tp06fPbZZ3z88ce8/PLLtGjRgj59+jBixAijbHBwMAsWLAByPnwnT55MXFwcM2bM4M0332TdunW88sorxR6T8uXLW8RWlFjykpWVhZ2d3TWPUVGcPXsWLy8v/vzzT4v9ERERRnJz2dX9yktoaCgVK1bE2dn5hrWflpbG0KFD2bVrF23btsXHxwd7e3tsbGxy9fnq97mTk9M1/S758MMPcXNz45NPPrFYAx0ZGUlGRgZxcXG5Em1nZ2dOnTpFlSpVyMrKyvV74erHjUVERBSpXFHMnz+f5557jlq1alG1alXc3NyIiorK9Qdlfu+by2Pk4ODAnDlzeOONN6hcuTJNmjShW7duPPnkk0Yy37VrV7777jsSEhJYu3YtXbt2pWvXrrz33nukp6ezZs0aI/EtynhdXl5T0M+Xg4OD8UdgUSmRFblBHBwcmD17Ng888ABDhw5l/vz5ef7VWZDu3bszb948kpOTjb/cb2T54sprwf348ePp3bs3M2bMMPZNmzaNLVu23PD2b4Zy5cpx4cIFi30pKSkkJycbNzo0atSIP//8k6SkJLZv38706dPp1KkT+/fvv6a1z3m5emwvf/AcO3aMsLAwi2PDhw/HxcXFmLWLiYmxeExTXomNnZ0d2dnZFvtSUlKK3ebNcO7cOT788EMWLlxIr169jP3BwcGYzeZi1/fwww/z8MMPc/78edatW8c777zDzp07+frrr4GcD+SJEyeyZs0aIiMj6dSpE4mJiYwcOZI1a9YQGxtrvFeLMyZ5vT8KiyUv5cqVM27MuuxGj5G3tzdmsznXzJ2joyMDBw602FeUG218fHyIjo6+oe3/8ssvbN++nf3791O2bFkg5+awH3744Zr6XBTTpk3j888/p2/fvvz6669Gu5dvUDp37lyutocOHUrFihXx8/PDxsYm12t39XZRyxXFmDFjeP755y0eb/jss8+ycePGYtcVHBxMcHCw8Rzijz76iF9//ZXt27cD0KlTJ8xmM2vXrmXjxo28+OKLNGjQADc3N9auXcvWrVt5+OGHgaKN12UF/XzFxsbme3UiP0pkRW4gR0dHfv75Z+6//36GDh3KggUL6NKlS5HP/9///scvv/zCmDFj+OKLL3LNhl5v+bx4eHgUayYjNjbWYiYhKyvrhj2HtqhPLbgewcHBLF68mHHjxhmzFHPmzMHBwYH27duTnZ3NuXPnqFq1Ku7u7nTt2pUyZcrQsWNHzp07R/369Ys9ZkXRunVrypUrR/v27Xn88cctjoWEhFCjRg1SUlIoW7Ys8+bN44033gDyH/+KFSty9OhRi30rV64sdpvF4eHhUaSlBpc/wK/8OTp06BB79uwp9iOjwsPD8fb2xtXVlQoVKvDAAw9w/Phx42kRAC1btsTT05MJEybQsmVL3N3dcXd3p1atWrz55psEBgZSu3Zt4PrGpCix5KVFixb89ddfFvuudYxcXV2xt7fP9Tr079+fKVOm8MwzzxhPZriyX8XVr18/PvroI3755ReGDh1q7A8NDc3zykVR2o+NjcXDw8Pij7Sff/652LEVh4+PD3/88Qf33HMPPXv2ZNmyZVSoUAEfHx86dOhA5cqVmT59usU5EREReHp64uLiQqtWrVi4cCF33XWXcfzq92NRyxUmPT2dhIQEiysw8fHx/PHHH4U+GeNqly5dIj09nXLlyuHj40P//v1JT09nxIgRpKSk4OrqipeXFy1atOCjjz7C1taWZs2aYWNjQ3BwMO+99x4ZGRnG51tRxqswp0+fJikpiebNmxerL0pkRW6wy8nsfffdxz333MPChQvp1KmTcXzBggW5LvtcfnxTUFAQixcvZtiwYbRu3ZoBAwZQvnx5YmJi+Pvvv0lLS7P4EChu+by0adOGpKQkxo4dS926dalSpUqed+Bfds899zBt2jQyMzPx8vJi0aJFxV7TlJ+iPrXgerz55pt06tSJO+64gyFDhhAWFsbXX3/NuHHjqFatGllZWQwZMoQGDRoYv1DnzJlD69atjbEs7pgVhbOzM9999x3Dhg1jy5YtNG/enNTUVHbu3El0dDRr1qzB1dWViRMn8sQTTxAdHU2dOnVYunQply5dylXf448/zhNPPIGnpydVq1Zl1apVHDlypNhtFke7du348ssv+fTTT/H09Mz38Vv169enfv36DBs2jOHDhxMbG8u3336b67FGRRESEsLTTz9Nz549qVGjBpGRkXz11VcWM1b29vZ07NiRP/74g9dff93Y37VrV7744guLpzhcz5gUJZa89OzZk2nTphEVFWVcFbjWMbK1taV169Z8/vnnxMfH4+joyH333Ufz5s2ZMGECPXv2ZOjQoVSrVo2oqCjj8vCVd7MXRfPmzRk/fjxPPvkkGzZsoF69ehw5coSzZ8/muaSpKO3369ePDz74gPvuu48uXbpw6NAhfv/991z3AdxoHh4e/Prrrzz00EPccccdLFu2jOrVq/PFF18wYMAABg4cSHBwMHZ2dhw4cICtW7eyZs0a3NzcmDhxIn369OHee++lY8eObNu2zfhSkCtnHotariBOTk4MGjSIcePGcfbsWWxsbJg9ezYmk6nYfU5KSqJXr1506NCBBg0akJKSwrfffsvdd99t8Q2VwcHBfPDBBwwcONBY/tK1a1fmz59Po0aNjBlsoEjjVZANGzbg4+NT6JryqymRFblONWrUYPjw4RZvficnJ+bOncvbb7/N77//TrNmzYxyV954dNmVj29q3bo1+/fv588//2TPnj0cOnQIf39/nnjiCXr37p3rg6y45a9Wr149/vrrL1asWGE8NzU4OJjGjRtbzB5cNmHCBBo3bmzcJPDcc88RGBjInDlzLOq88gsR8qvrgQcesHhcjrOzM8OHDy80ic2rPh8fH4YPH57r2b133nmnxdce+vv7s337dubMmcPx48fx8PDg999/p3379kBO0rN9+3aWLVvGzp07sbGxYcyYMRa/yPMbs7z4+fkxfPhwixmJ/Majc+fO7Ny5k0WLFnH8+HG8vLwYNmwYvXv3Nsrce++9VK1alSVLlnD27FlGjhxJZmYmTz75pEVd99xzDwEBAaxcuZILFy7w1FNP4eXlxdy5c4vd5oABA4wZy8sqVKjA8OHDLdZ+jhkzhooVK3LgwAGOHTuW7+O3HB0d+fvvv/nxxx85efIkZcqUYcmSJWzfvt3imb9FabdTp06sWbOGRYsWERISgre3N4sXL7a40QXgkUcewc/PjwEDBliMUXJycq6vUr7WMSlqLFfr2LEjtWvXZt68eTz33HPFGqPmzZvnWgLy888/89NPP3HkyBEyMjKMx2+NGTOG3r17s2zZMo4dO0aFChWYNm2a8f64/P4r6le2jh8/nl69evH7779z+vRpWrduzZQpU4Cc99Hw4cMtZg8La79WrVps2bKFuXPncuzYMapXr86WLVv46KOPjN8T+cXYokWLYv0hdHU9zs7OzJ07l/fff5+ff/6Z8ePHU61aNbZv386SJUvYu3cvDg4OdO3alU8//dRYG9yqVSs2b97MnDlzCAkJoWPHjgwbNoxBgwZZxFPUcoX55ptvmDt3Lnv37sXFxYWPP/6YjIwMdu3aZZTp0KGDxU2JkDMrPHz4cGN9aoUKFdi2bRuLFy/mwIEDODs788knn9CzZ0+L8wYPHkxERASDBw829t1xxx0MHz481yPlijJeecV22dy5cxk+fHiu52YXxsZ8sxaeiIjILTFv3jyefPLJPGdmxTosXryY1157jb179xb7g1xKTkJCAnZ2dhazjWPGjOGPP/6wWNpT1HL/Vbt376Z///7s2bNHa2RFRESszZ133sn+/fs5fPiwxRUEub0lJCRw55130qdPH/z8/Ni0aRN///03X3311TWV+686dOgQU6dOLXYSC5qRFRGxetu3b2f27Nm5brIQ+S/atm1brvXgV+rXrx9+fn43rL3Q0FAWL15MWFgYFStWZMCAAXkuqSmoXFJSEgsXLsy3japVq9K5c+cbFnNpokRWRERESo358+ezfv36fI+//PLLxX46xs0WGxvLhAkT8j3evHlzHnnkkVsYkfVQIisiIiIiVklfUSsiIiIiVkmJrIiIiIhYJSWyItfIbDZjMplu2lcnioiISMGUyIpcI7PZTET4uVzfZ29tTCYT58POXNO3w9wuSkMfoHT0ozT0AUpHP0pDH6D09ENuDiWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFWyMZvN5pIOQsQamUwmIsLPMWhWKimZJR2NiEjpd2hi3ZIOQW4zmpEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREatkX9IBiIiIiFyr7OxsLj+AydbWFltbyzm6rKysXOfY2NhgZ2d3S+KTm0szsjfA/v37+HTalCKV3b17F198Pv0mR3T7+WXeXP7+e2VJhyEiIqVMixYt8PX1pUyZMtxzzz0Wx3bs2IGvr6/FvzJlytC2bVuLcp9//jkNGzbE19eXQYMGce7cuVvZBbkOSmRvgISES5w6dapIZePj4zlz5vQ1t1WcpPl2cv58GBcuRAPW2wcREbn97Nmzh0uXLvH444/nOtayZUsuXbpk/IuPj6dq1aoMHTrUKPP+++/z5Zdf8uWXXxIeHs6YMWNYtmzZreyCXActLbAyVapUpV//gSUdxnUpDX0QERHrs2bNGsLCwnjooYcAiIuL45NPPmHu3Ll06NABgI4dO9KxY8eSDFOKQTOyVsbT05Nq1aqVdBjXpTT0QURErM/XX39N//79KVeuHJCz9CArK4tjx44RFBREQEAA99xzD6GhoSUcqRSVZmSLICTkBN99+w0Arq6u1K5dhwEDB+Ho6Jhn+alTJnPhwgWLfXXq1OWhYcON7V27drJhw3pSU1Np17YdXYK7GsdiYi6w7LelnD9/Hh8fH/r1H0hQUBCQc1l+3do1PPvc80DOmtutWzbTuHETtm3bSmpqKt3v6EHDho1YvHghJ0NCCAwM5N777sfDw7NIbVyus2Wr1mz6ZyMuLi48NOxhli79lZMhIXh4eNCt+x00aNAw3zE7evQIfy3/k4yMDJo1b2Fx7Oo+pKSkFFh3QbEW9tpcT90iIlJ6REREsHz5cn777Tdj38WLF8nOzmbNmjXs3LkTW1tbRo4cyUMPPcS6detKLlgpMs3IFkFAQCAPPjSMBx8aRo+evThwYD+ffjo13/IDBgwyyt919z1ERITj4eFhHA8NDeXvv1fSqVNnWrRoycyZX3LgwH4AoqKieOH557C3d6Bvv/4EBATy6viXCQ8PB3Kvx42Pj2f9+nVs2vQPXYK7UrdePT7+6EPefmsCnp5eDBg4iPPnzzPjyy+McwprIz4+no0bN7Dir+V07NiZnr368M03X3H40CF69+5Ly1atWbxoIefPh+XZ/5MnQ3jrzQlUrFiJbt3vYO+e3ezYsd04fnUfCqq7sFgLe22up+6rZWZmkpKSYvxLTU3J92dARERuL99//z1Vq1alU6dOxj53d3fMZjPjxo2jXLly+Pn5MX78eHbu3El0dHQJRitFpRnZInBzc6Nu3XrGdr169Xh4+EMkJyfh5uaeq3y16tWBnEeCTJz4Hi1atGTQ4DuN405OTowZ8xIODg5AzgzlgQP7adiwEYsWzqdly1bG7G3Tps2IvhDN8j9/59HHnsgzPhcXF55/4UUcHBxo1qw5a9euoV79BgwYkLMO1dnZmfffe9coX5Q27OzsGPPiWFxcXACIuXCBNm3b0rxFzuxq27btyMzMzDOe335bSqdOnbnr7py7R5s0acqokXnHXljdhcVa2GtzPXVfbfGihcyfP8/YdnZ2YtKkSfn2S0REbg/Z2dn88MMPjB492mJ/48aNATCZTBZlgVyP8ZLbkxLZIsjKymLlyhUcPHiAxIQE4wf+QvQF3KrmTmQvmzkjZxb0yadGWewvV66ckcQCeLi7k5ycDMDZs2dJSEzg1fGvGMcvxl0kMCAw33aurs/N1c2ivJurGykpycZ2Udrw9y9vJLEADz40jFkzZ7Bzxw7q1q1Hq9ZtqFKlSp7xhJ8/T4+evYxtOzs7qlSpmm/8BdVdWKyFvTbXU/fV7hxyF/0H/HuTmtlsIj4uJt9+iYjIzWcymTCZTJjNZsxmM1lZWbmeE/vXX38RExPD/fffb3FupUqV6N27N++++y5ffvklNjY2vPPOO3Tq1Ak/P79b3RW5Bkpki+Dnn2dzYP9+Bg4ajLe3N3Z2drz15gQyMjLyPWf+L/M4c+Y0b7/zfrEeuuzm7ka1atXo1LmLxX5XV9drDf+a2rCzs/xLtGbNWnz08Sdcio/nwMH9vPfuWzzxxFO0bNU6z/qTk5Is9l1O1PNSUN2FxVrYa3M9dV/NwcHB4g8Gk8lEfFy+3RIRkVvg4YcfZunSpca2r68vzZo1Y+3atca+H374gSFDhuDj45Pr/K+++oqXXnqJtm3b4uDgQNeuXXn//fdvSexy/TRvXgRnz56hefMWdOzYiYYNGxEZEZHnN4Vctmb1KtauW8P4Vyfg7OxcrLbatm3PgQMH8Pf3p27detStWw9HR0diLty4mb9raeP3Zb+RkpKCl7c37dt3JCAwkJCTIXmWbda0OavXrCIxMRHIufHr2LGj11R3YbEW9tpcT90iInL7+/HHHy2eFXvp0iWLJBZg/vz5zJw5M8/zvb29+eqrrwgNDeXUqVN8/fXXxlMN5PanGdki6N27L1OnTGb7jm1kZmRSPiCgwAR1xowvKFu2LB9N+sDYd/VTC/Jzxx09uHAhmueeHU25cv4kJyfj7e3FqFHP3JC+XGsb9vb2jH76KcqU8SEpKRFXVze6d++RZ9neffpy+MhhRo18krJly2IymahevcY11V1YrIW9NtdTt4iIiNzebMyXv6BYCpScnExERDju7h6UL1+e48eOUSmoEi4uriQkJBATE2M8G/XIkcO5znd3d6dSpSAuxccTFxdHlar/rhmNjIwAbChfvryxLz09ncjICNzc3PDzK2vsv7qtvOo7c+YM3t7eeHt7G3WdOnXS4qaogtrIq07IuWv/fFgYTs7OBAQEFDpm0dHRZGZmEBAQSEREOI6OTpQtWzZXH4pSd36xQsGvzfXWXRCTyURE+DkGzUolJe/73kRE5AY6NLFuSYcgtxklsiLXSImsiMitpURWrqY1siIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilfTUApFrdPmpBf7lK2Jvb72PZL7cj4DAIKv9bvHS0AcoHf0oDX2A0tGP0tAHKD39kJtDPxEiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSYmsiIiIiFglJbIiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSYmsiIiIiFglJbIiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSYmsiIiIiFglJbIiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSYmsiIiIiFgl+5IOQMTatX37BCmZJR3FjXCspAO4AUpDH6B09KM09AFKRz9ufR8OTax7y9uU/ybNyIqIiIiIVVIiKyIiIiJWSYmsiIiIiFglJbIiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSY/fKkFxcXFcvBhL9eo1SjqUmy409ByOjk74+/uXdCgiIlICli5dysGDBwFwdXXl+eefz1XGZDKxatUqDh06RJ06dejdu7dxbMmSJRw5csSifP/+/WnUqNHNDVxua5qRLUF79uxm5owv8j1+KT6ekBMnbmFEN8+C+b+wevXfQOnql4iIFM/BgweZMmVKrv2JiYn06tWLcePGERERwZw5cxg9erRxfMmSJaxfv/5WhipWQDOyt7FzoedYv24to2s+V9Kh3FCltV8iIpK/gQMHMnDgQObNm8emTZtyHZ8wYQLZ2dls3rwZJycnAA4fPmxRpn379rz66qu3JF6xDkpkb2MNGzaiYcPSd8mktPZLRESuTXZ2NnPnzuWzzz5j6dKlRERE0KhRI4KDgzGZTEa5gwcPMnnyZPz9/bnjjju0XE20tKCo4uLiOHkyBLPZTHh4OGFhYZjNZuN4UlISBw8e4ODBA5w6dYr09PR8z4+KiuLUqVN5tnP8+DHOnw8Dcl+Cv7KOyMhIzpw+TVZWlsX5qampnDwZQnR0VJ71x8fHExJyguTkpHzju9a6rxYVFUVo6Dmys7Mt9ue1tKCwupOTkzl5MoS4uDiL/YWN+/XULSIit8b58+dJSkpi8uTJLFmyhIiICB5//HEee+wxo8ygQYNo0qQJiYmJLFiwgMaNG7N69eoSjFpuB5qRLaI9e3azcMF8PD09ycjI4NKleMqV82fc+Nfw9PQkKiqS+b/MA3KSq9jYWJ773/M0a9bc4vwyZcqQnp5GrVp1qFGzplG/yWTiu2+/5ujRo4wf/xoABw7uZ+GCBUydNt2oY9HC+fj4+JKamkpCwiVcXF15//0PcXNzY/v2bUz/dBr+/uVIT0+nbLlyjBkzFjc3NzIzM/nii8/YuWM75csHEB5+nn79B3DffQ9cd91Xy8zMZPLHkzhy5DBly5YjMysTZydnygcE5Nmvguo2m83Mnv0Df69cQUBAINHRUbRs2YqRo0Zja2tb6LhfT9159SszM9PYNptNucqIiEjxXZ7waN68OZ999hkAjzzyCM2aNeOpp54ioLwfgwYN4s477zTOeeONN3juueeMG8jkv0mJbDFERUVy333307FTZzIyMnj77TeYO+cnnnxqFNWr1+Dtd94zym7duoUvv/icGTO/ws7Ozjj/0Ucfp3mLFgCsWZPzl2RaWhpTPvkYk8nE2++8h4uLS74xREdHM3LUaBo0aEhmZiZjXxrDunVr6Nu3P78v+427776HAQMHAXDgwH6Sk5Nwc3Nj+Z9/cOTwYT777Eu8vL05d+4cL48dQ+3adYyk71rrvtqff/zO+fDzfP7FTNzd3Tl27CivvTqOxk2a5Nmngupevepvtm7ZwqfTv6BMmTKkpKTw6quvsGbNarp3v6PQcb+euq+2eNFC5s+fZ2w7OzsxadKkfF8rEREpmoCAAOzs7GjevLmxr1atWnh4eHDq1CkCyvvlOqdbt25MnjyZtLQ0nJ2db2W4chvR0oJiKFOmDB07dQbA0dGR3r37smfPHuN4dnY2YWGhHD58CDc3NxISLhEbG2McL+fvbySxlyUnJzPh9fH4+PjwyrhXC0xiAQICAmnQoCEADg4O1KxZk6jInEvmlatUYe/ePRw/foysrCwaNmxEuXI564d279lF167d8PL2BiAoKIjmzVuwe9eu6677anv27KZb1+64u7sDULt2HWrXqZNvnwqqe+vWLdStV4/z58P+f/nASWrUqMn+/fuKNO7XW/eV7hxyF7N/mmv8m/XVt/n2SUREis7Z2ZmOHTuy64rPpGPHjpGYmEjNmjXJyMjg6NGjFuesWbOGypUrK4n9j9OMbDFcPfvo5uZGcnIyAMeOHeXjjybh4OCAt7c39vb2mEwm4uPjjcTJ4/8TuytlZWVx6VICZXx8jJnbgjg7O1ls29nZkW3KuSQzYsSjbNu6hdWrVzFzxpdUqFCBx594Cg8PD1KSk3Fzvyp+d3eSU5Kvu+6rpaQk43r1WLnmnrm9rKC6ExISiIyM5EJ0tMU5VatWBQof9+up+2oODg44ODgY2yaTiXgtqxURKZKtW7eyevVqDh48SGpqKu+9l3M1bdy4cdja2jJx4kT69OlDXFwcQUFBLFiwgEceeYTGjRtz5vQJnnzySSpWrEj16tU5dOgQW7Zs4ccffyzhXklJUyJbDNHR0SQmJuDh4QlAyIkTVKhQAYBFCxfQuUsXHnxwGACXLl3i0UeGgznf6gDw8vLipbHjeOvNCaSmpjJ8+Ihrji89PY227drTtl17AN5883X+/ON3ht57HxUqVOTE8ePQN6es2Wzm+PFjtG/f4brrvlpgYAVOhpyAHj2BnGT99OlTVK5Spdh1V65cGVs7O0aOfNrinNTUFKDwcb+eukVE5MZr0KABDRo0yLW/YcOG7Ny5k99++43U1FS+++47OnbsiMlkwsnJiTVr1rBq1SqOHj3KPffcw6xZsyhbtmwJ9EBuJ0pki2nShx/Qr/8ALkRH8+uvi3j22f8BUMbHh4MH9rNj+zYyMjP54/dl2NjYFKnOsmXL8u57E3nrrQmkpaXxxBNPFfncK02c+B5169SjZq1aJCYmcvbMWYKDuwEwcOBgxo0by88/zaZW7dps2byJ+Lh4evbsdd11X23AgEG8+uor+Pr6UbVaNdatXUNSUlKeZQur+667hzL2pTE4OznRsFFjkpOT2bZ1C40aN6FXr96Fjvv11C0iIjdOmzZtaNOmTYFl/P39efzxx/M8ZmdnR+/evS2+7UtEa2SLoVKlSgwYOIjNmzdx7NhRnn/hRVq3aQvAsGEPU79BQ/76aznbt2/jgQcfomnTZri45qx5LeNdhmrVLL+K9sp93t7evP32eyRcusTqVTnfgOXl5U2NGjXyLH9ZQGAFAgICARg37lUcHR1ZuXIFe3bv4tHHHqNz5y4AVKlalQ8+/IikpET+XrkCD09PJn8yxZhdvp66r1atenXefOsdIiIjWL9uLc1btGTQ4DuNJRZX96uguv39/Zk6bTqOjk6s+nsl+/ftJTi4q5FoFjbu11O3iIiI3N5szFc+DFXytWbNav5a/geTPvqkpEOR24TJZCIi/ByDZqWSkll4eRGR/4pDE+vesLou/64NCAzK89GI8t+mnwgRERERsUpKZIsor0vvIiIiIlJydLNXETVt1oymzZqVdBgiIiIi8v80IysiIiIiVkmJrIiIiIhYJSWyIiIiImKVtEZW5DptmVATe3vrfSuVhkfblIY+QOnoR2noA5SOfpSGPogURj/ZIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJfuSDkDE2rV9+wQpmSUdxY1wrKQDuAFKQx+gdPSjNPQBrK0fhybWLekQRG4pzciKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJbBEkJSXx5puvk5KScsPrTktL4803XychIeGG1309blZct2t/RURKqz///BMvLy/j38aNG/Mt++GHH+Ll5cX06dONfWazmYkTJ9KwYUPKly9P69atmT9//q0IXaRQSmSLIDMzkwP795OVlZXn8etJzrKzsziwfz+ZGRnXG+YNdbPiKun+Hjx4gC+/+KxE2hYRKQm9e/cmNjaW8PBwsrOzMZvNeZY7cOAAv/zyC0FBQZhMJmP/999/zxdffMFXX33FiRMneP7553n88cfZt2/freqCSL6sPpH9atYM9u7dU6IxODg4MHjwEFxcXG56WykpKbz55uskJSXd9LZKo4oVK9Gt2x0lHYaIyC1jY2ODvb09dnZ2+ZbJyMjgiSeeYOrUqTg7O1scO3DgAO3bt6dNmzZ4eHhw77334u/vz8GDB2926CKFsvpE9sSJ41y8eLFEY7Czs6Nx4yY4ODjc9Laysv5/RjOzVHyV1C3n7e1Nrdq1SzoMEZHbyvvvv0+rVq3o1KlTrmN9+vRh+/bt7P//z56lS5eSmppK586dSyBSEUvF+oraOXN+4vjxY9ja2FK2bFmCg7tRp+6/X4c3c8YXNGrchNBz5zh1+hRl/fy4+557iYyMYMWKv0hNTaV9+w60b98BgE2b/uHo0SM8+ujjRh1JSUl8Mvkjnhr5NOXKlcNsNrNmzWr279+HDdCkaTO6dAkGYOHC+YSHh7N06a9s2LCOShUr8ehjTzBzxhc0b9GSM6dPc/JkCH379adater8vuw3Tp8+hbe3N8Fdu1GrVm2jzY8//pCHHhrO2jVruBh3kQb1G9Crdx9sbf/N9S9ciGbhgvlEX4imZo2aDBg4CAcHB9LS0vjgg/d44YWX8PT0BHKWI6xc+RfHjh3DycmJvn36UaVq1SKN89tvv4HJZMLB3oEKFSoycOAgyvj4ADD906kATJ48CXt7e7p0DqZLcFeysrL466/lHDt6BEdHR9q160DzFi0s+jd8+AjWrllDePh5XnzpZSZNmphvO0WxccN6duzcgclkolmz5nTt2s04Vpz+X0vsMTEx/LX8T2JiY6gQWIH+Awbi7e0NQGpqKn/+8TshISfw8PCga7fu1KmT83N68OABNm5Yz8hRowstKyLyX7Bjxw4WLVrE5s2b8zzevXt3Hn74Ydq3b4/ZbMbFxYUvv/ySihUr3uJIRXIr1oxsu7btGTx4CP0HDCQgMJD33nub48f//R7q48ePM+PLL3BwcCA4uCvHjx/nzTde5/vvvqVZs+Y0atSYz6ZP49ixowDUrl2blSv+Iioqyqhj7drVxMXHUa5cOQC+/noWCxfOp2mTpjRq1Jg5P89m9uwfAGjVsjVlypShefMWDB48hM6dg404pn86FZPZRI+evShfPoBx48Zy/PgxOnXuQhkfH96Y8Br79u0F/l0D+/HHk6hcpQqtW7fhjz+W8e03X1v0/7PPPqVKlSp07NiJNWtWMf+XeUDudZ/Z2dm8+cbrbNywgRbNW1C3bj1mzvyS1NTUIo3z4MFDGDx4CD179SY9PY2XX36R9PR0AHr06AlAnz79GDx4CHXq1sNkMvHWmxPYuXM7bdu1p179BsyY8Tnr1q6x6N+UTyYTVLky/QcMNJZD5NdOYbZs3sS3335D48ZN6NChI0ePHOav5X8Wu//XEntERDhjX3oh5+esSzAxsTG8OOZ/xnKL7779ml27dtCpcxfq1W/AnDk/ER5+HoD4+DiOHfv3Z7agslfLzMwkJSXF+JeaeuNv/hMRuZUyMjJ48sknmTp1Kh4eHnmWeeedd1i6dCmbNm0iMjKSuXPn8r///Y/ly5ff4mhFcivWjOyVM2pNmzYjPS2dv/9eacxsAvTo2ZPBdw7Jqdzenvffe4eZs77Gz68sAPv37WXf3r3Url0HP7+yNG3WnL//XsGDDw4D4O+VK+nVqzcAl+LjWbniL9577wPjcrCvrx/vvfc2Q4bcTVDlyri4uFCxYiUaN25iEWunzl245557Adiwfh1JiYl89NEnODo60rZtO1KSU1i4YL7FeQ8+OMyYLQ4MDGT8uJcZeu99xvHHHnuS+vXrA5CaksLKlSt44MGHco3Ttm1bCQ8/zxdfzsTFxRWADh06Frg+6UoNGzYy/t+iRUteefkldu3aSbt27an9/7OFdevWo0yZMgBs3bqFyMhIvvhyprG8wc3NjZ9/nk2X4K5GXQ+PeIRmzZoXqZ3CRERGULNWLbp16w5AmzZtjUS1oP6npVkms9u3byt27NOmTaFFi5YMf3gEAK1at+GF559j5Yq/uHPIXURGRtKxU2fatm0HQMeOnfK9Ua84ZRcvWsj8+fOMbWdnJyZNmlToWImI3K7i4uI4fvw4d955p7EvOzub119/nYULF7Jx40b+/PNP7r77bho1yvnM6N69O+3ateOvv/6id+/eJRW6CFDMRDYyMoKVK1cQHh5OWloqcXFxeHl5WZSpUOHfSw0e7h44OTkZSSyAu7u7xY1KvXr15tNpUxg69D6OHTvKhQvRdOrcBYDwiHBsbGyoUbOmUb5O3bpkZ2cTERFO9eo18o21atVqxv/DzodRtVp1HB0dLepZv36txTlXJuTVq9fAxsaGiIhwypbNmR2+8jKKh6cnycl533B19uwZqlWrbiRxgEXbhTlwYD///LORi7GxZGZlcuFCNFGRkfmWP3PmNBkZGbz33tvGvvS0dCIjIsjOzjb2XTkm19LOlbp1u4ODBw8w7pWx1KtfnyZNmhqJcXH6fy2xnw8LpfP/Ly+BnBsZatWuTWhYKAD33/8gM2d+yZ49e6hbpy6tWre2+Lm8UnHK3jnkLvoPGGhsm80m4uNi8h0jEZHbnb+/P/Hx8Rb72rZty/3338+zzz4LQJMmTViyZAlDhgyhatWqbN++nc2bN/POO++UQMQiloqcyKalpTHulbG0b9+R4C7BODu7sHPXDk4cP25RzsbGpsDtqzVu3ARXVze2b9vKtm1badO2He7u7gC4urqSnZ1NWloarq45SdHl5NHNze1yC3nWe+Xsp6urKynJyRbHk5OTcTXqyJGSkgzkJN3p6WlkZ2fj6vpvmcL6cpmbm3u+SW5hDh48wKQPP+Due4bSokVLHB0d+WXeHDL+f9lCXiG4urrh5+fH4MFDch27MuYrx6Swdgrj5eXFhAlvkZycxLFjx5g75yd21qzFiEceK1b/ryV2V1c3UpItL+snJyfj7eUN5PyRMmXqp1y4cIEDB/bzxoTXGDlqNM2bt8jVRnHKOjg4WNzQZzKZiI8rUjdFRErMuXPnaNy4sbHdr18/bGxs+PbbbxkyZAj29rlTAVtbW+P37sSJExk3bhw9evQgLi6OwMBARo0axcMPP3yruiCSryKvkY2JuUBSUhJD772P1m3aUqduXc6eOXPdAdjY2NCjZy9+/XUx27Zt5Y47ehjHKlSoSNmy5Vj221Jj39IlvxIYGEi5cv4AuLq5Fpo0NW7chJCQExw9egTIScpX/LWcpk2aWZT784/fjefr/b5sGeX8/QkICCh2n5o1a87p06fZsX2bsW/btq2kpaUVeu7p06eoWLEiAwYMpGXLVvj7+3P+/L9rNl1cXLG1tbXoc7NmzYmOjsLJ0YnGjZvQuHETqlWrTkxMjMXNasVppzBbtmwmKioKNzd3mjVrTqvWbThy5Eix+38tsTdp0pR169aQmJjz3N6wsFB27dxB06Y5r+dffy0nPT2dsmXL0rVrNypWrMTxK9bFXqk4ZUVErFFQUBCxsbHExsYSHx/PxYsXiY2NtVhOcKWtW7cas7GQ87SXL7/8ktOnTxMfH8/hw4cZP358kSd3RG6mIs/IBgQE0qhRY559ZhQVKlQkKiqKSpUqFXkGryBdu3Zj7pyfKOfvT7169f8Nzt6eZ5/7H1M++ZjNWzZhMpnISM/gpZdeNpKc9u078uMP37Fj+3YqV67Mo489kav+qlWrMWzYw7zz9ptUrFiJCxeiCQqqzL333W9RLjk5mdGjR+Lo4Ehc3EVeGvtKkde1XqlixYqMevoZpk+fhq+vH9nZ2VSsVIkWLVoWem7r1m35belSRj89Eg8PDxISLhlLGy6PSevWbXj7rTcIDKxAly45Ty14auTTTJo0ES9vbxzs7YmLi89z/W5R2ymMh4cH7777Fvb29jjYOxAVFWk8CaCg/mdnW64/rVixYrFj79uvPyEnQxj99EgCAgIIDQ2lT59+xpMOMjMzGDXyCcqWLUdiUiI22DByVPc86ypOWRERa5XXrGt+ruVzT6Sk2Jjz+4qPfJw7e5ak5CQqVQoiKzOT2NhYYw3r8ePHKFu2nHETUkpKCqdOnaRBg4bG+WFhodja2hIYWMGi3lEjn6RXr94MGDgoV5uZmZmcDwsDGxsqVKiQ63mtMTEXiIrKmdWrUbNmrjguS05O5vz5MDw9vShfvryxPy4ujscefZjvvp9NZkYGF+MuEhRUGScnJ6P9w4cPUb9+A+OXwaVLl4iICKdOnZw1uwcPHqBevfoWsaWnp3M+LAwXV9d8Z3bzOjcjI4OzZ88AUKVKVSLCw3F0crKI+dzZs1z6/+Tz8v7MzEwiwsPJys6mUqVKRn15xV9YO/n16erYw8JCyczMpGLFSrkeop1X//Ort7ixA8TGxBAbG0v5gADjsWdXtn327BmcnZypWKmS8YdPfHw80VFRFs+Sza9sYUwmExHh5xg0K5UUPdZXRG4Dhyb++/jAy7+jAgKDivx77XZUWvohN0exE9mbYd++vXz4wfvMnPU1Hh6ehZ9wg12ZyF6dEInkR4msiNxulMjKf02xnlpwM7z7zlscO3aMO++8q0SSWBERERGxTiWeyA4cNBgvL2+CgoJKLAZ3d3cmvPGW8WQEEREREbn9lXgie+VD+UuKg4NDri9UEBEREZHbmxabiIiIiIhVUiIrIiIiIlZJiayIiIiIWKUSXyMrYu22TKhZrIeN325Kw6NtSkMfoHT0ozT0AUpPP0RKO707RURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSvYlHYCItWv79glSMks6ihvhWEkHcAOUhj5A6ehHaegD3Kp+HJpY95a0I1LaaEZWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySnlpQCl26dIlL8fEEVa58TecnJiYQG3uRKlWq5LktIiI3X3x8PN98842x3a1bN5o0aWJRJj09ndWrV3P27FmqV69O9+7dsbX9d44qIyODZcuWERYWRo0aNejRowc2Nja3qgsiN51mZPOQnZ3N8WPHyMy0zmcq7dq1k88+m3bN5+/bt5epUybnu22tEhMTOHPmTEmHISJSJNnZ2TkTE5cuMWXKFLZu3Wpx/Pjx43To0IGffvqJkJAQXn75ZYKDg0lOTgYgNjaWAQOHMHnyZE6ePMnLL7/M3XffjclkKonuiNwUmpHNQ0ZGBt9++zXjxr+Gl5dXSYfzn5SQkEBc3EUqV65yw+o8deoUa9es5n/Pj7lhdYqI3Cy+vr68/fbbACxZsiTXcWdnZ37//Xf8/f0BSEpKonbt2vz+++8MHTqUKVOm4OLizJo1a7C3t+fSpUs0atSIhQsXcs8999zKrojcNJqRzYOLiwsffPiRktgStGfPbj6dNvWG1tm4cRMlsSJSagQFBRlJLICjoyP29vbY2+fMUR09epQmTRobSw28vLyoX78+f/31V4nEK3IzlOoZWbPZTFRUJCaTiYCAQGNdUFZWFqdOnqR6jRqkpaUSFxdHQEAgdnZ2QM7lnJMhIVStVg0HBweL8unp6cRcuIB/+fI4OTlZtJeZmUlERATe3t54enrmisdkMhEdHYWjoyM+Pr4Wx7Kzs4mKisLR0QE/v7LG/ivbzshIJzw8nCpVqmJnZ0d6ejpRUVH4+Pjg7u6e5xjExFzAZDJRrpx/rmNFOT8vKSkphIWGAuDq5kr58gHGL06wXKMbd/EiSclJVKoUVOQ6UlNTiYqMJD09nePHcr5Vp1JQEGazucB2T58+haeHJ75+fsa+U6dO4eXlha+vb661voX1Q0TEGnz++efEx8ezbt06+vTpw8CBAwGoXLkyO7ZvN8qlpqZyzIqXzYnkpdR+asfGxvL222+QkpyMq6srJrOZF18cS+XKVUhMTGTcuLEEB3dl7949RrL68svjqVGzJmlpqYwbN5ZZs77B18/PKN/9jh7s3bMbBwdHkpISGT/+dWrVrg3AqlV/88P331KmjA9xcRdp0KAhzz73P1xcXAHYvn0bM2d8iaOjA1lZ2TRo2JDRo5/Fzs6OHTu2M+PLL3B1dSEtLZ2yZf146aVXKOPjY7TdrVt39uzZg6+vL2+8+TZLlizm92W/Ua6cPzExMXTs2InHn3jSSNaTU1J47bVxXIqPJyEhgaCgyox9eRweHh4AzJ37c4HnFyT8/Hm+/fbrnHaSk0hKSub5F8bQqFFjIGeN7qJFC/Dz9SMuLo46deow6ulnilzHhQsXWL9+LXFxcUaZUU8/Q0Z6eoHtHjt2jMWLFjD5k6l4eHiy6Z+NfP31LD7+eAqQs9Z34YIFTJ02vUj9uFpmZqbFB4DZrHVmIlLyLl26RHx8POnpOZMdFy9epFy5cjz33HN06tSJgQMH0rJlS1atWpXnJIuINSu1iez69Wsp412GqVOnY2NjQ2RkJOHh5y3WXGZmZTFz1jfY2dkx+8cfmPbpFKZOnZ5vna6ursyY+TU2NjbMmjmDhQvnM/7V1zl37hyzZn7J+Fdfp0mTpiQnJ/H6a68y/5dfGP7wCCIjI5jyyceMevoZOnbs9P/xrSMjI4PExASmTf2EMWPG0rRZM0wmE19++Tnfff8tL7zwotG2nb09M2d9ja2tLWvXrmH9unV8+unn+Pr5kZSUxKvjX2bdurUEB3cFIDIigiFD7qZr126kpaXx9lsTmDdvDo8//mSRzi9IjZo1+eDDj4ztjRs38Pln0/niy5nGrHZkRATDHhpO6zZti11HUFAQ9wy9j9+WLrEoAxTYbq9evTl8+BDTpk1lxIhHmDHjS1544UWLGdri9uNKixctZP78eca2s7MTkyZNKmioRERuuvHjxwM5V/3uuOMOPv74YyZNmkRQUBDL/1zG1m07iYqKYuLEiXz22WeUKVOmhCMWuXFK7RrZShUrERUVyb59e0lPT6d8+fI0a9bcosygQYONhOXOIUMIP3+eyMiIfOvs3buPMWPZsGFDIiMjAdi7ZzfVqlWjSZOmALi5udO7T1927sy5pLNzxw4qVapkJLEAnTt3wcXFhR07tuPr54ebmxvHjx0j5MQJ6tSpy769eyza7tevv7HOadOmf2jUqBGxsbEcP3aM8PPnqVO3HnuvOMfbuwxdu3YDcm4I6NO3H7t37Szy+YXJWbYRRciJE5QrV474+DhiY2ON4+XKlcs3iS1qHddyzsiRTxMVFckrL4+lV+/eNG3W7IbFcOeQu5j901zj36yvvi2wbhGRm+no0aOYzWZj28bGBltbW+OzIiEhAZMpmxEjRjB+/HjKli3LihUruO+++0oqZJEbrtTOyLZs1Ro3d3e2bN7EvLlzcHJyYsSIR6lStapR5vJldgAXF1fs7OxITkrG29s7zzqdnV2M/9va2ZGdnQVAYlIiHh6Wl2s8PDxITEwEICExAU/PvG8cu3TpEvFXXEK/LCAgkKysrCvq+7f+hEvxnD8fxrlz5yzOqV69uvF/dw/LNa/u7h4kJSUX+fyChJw4weTJk8jKysbb2xs7OztMJhOXLsVTrlw5o//XW8e1nOPs7Iy/f3nCz5/Pd4nAtcbg4OCAg4ODsW0ymYiPK7AJEZHrMn36dDIyMoiPj2ft2rUkJyfTpk0b2rdvz6ZNm3jqqado3749Dg4ObNy4MecK4axZQM4TeB597EnjytzChQt5/vnn6dixYwn3SuTGKbWJbHp6OvXq1adevfpAzmXhWbO+5P2J/14KPnnypHFj1ZkzZzCbzfiXL1/stgIDK7Bu7VqysrKMm4VOnDhOYGAFACpVCmL1qr9JT083bhDLzs42jrl7ePDe+x9YXM5OT0/P98ajSpWCcHJy5oknn8rV58uio6JISkoybuI6dfIkgYGBRT6/IAsW/EKbtu0YPnwEAImJiTwyYhhmk7mQM4teh52dLaar1qAWpd0lvy4mMiKCESMeZdrUT5g8eSpe+fxhciP6ISJyM126dImMjAwefvhhYzstLQ2ARx99lI4dO/L333+TkpLCqFGj6Nu3L87OzgD4+fnx9Vcz2LBxC+np6fzxxx80blzwH/gi1qbUJrILF8wnMTGR5s2bY+/gwN69eyyeBgDw44/fk5mZgaOjE3Pn/EyXLsF4eXmRnJxUrLbatWvP/PnzmDplMt3v6EFo6Dn+/ON3xrw4FoC2bduxdMli3n/vHfr26092djZ//LGMV1+dQLt27fl92W9M+nAi3bvfgb29A4cPHyIiMoIX///8q91191DGvjQGVzdXGjZsREpyMlu3baV27Tr06dMXyLlkPvnjSQwYMIjoC1H8+usinhr5dJHPL4inlxfHjh5l7949ZGZk8NtvS4s1XkWpIyAgkKjISHbs2I6XpxeVgoIKPefIkcMsXDifd9+bSNWq1Th16iTTpk3htdffsPimmxvZDxGRm+m1114r8HitWrWoVatWvsd9fX0ZNWpUnr8DRUqDUvuTfe9991OjRg1Wr17F78t+o379BkYid9nIkU+zZ88e/lr+J+07dODJp0YBYGtrR82atbD//8vI9vb21KxZy2LG1N3dg6pVqwHg5OTEhx9+jJ9fWZYu+ZWQkBDGjX+NFi1aGue/+94H1G/QgJUrV7Bl8yaGDXsYFxcX7OzseOfd96lbrx6r16zmzz9/x9XVlWeeeS7ftgMCApgydRqZmZksXformzdvolWr1kYS6uXpRYcOHenRoyd//72CPbt3M+rpZ2jfvkORzvfw8LT4Otqrt4cPH0GNGjVZvGgha9euYchdd9OgQUOcXVyM9oOCCv563MLqqF69Bvff/yDL//yDb7/9mgsXLhR4TlZWFr8v+41HH3vCeF2eeHJkzlMhtm+7pn6IiIjI7c3GfOVK8f+IuLg4Hnv0Yb77frYeRSLXzGQyERF+jkGzUknRYxlF5Docmlj3htd5+XdUQGCQVc/IlpZ+yM2hnwgRERERsUr/yUQ2r8v1IiIiImJdSu3NXgXx8PDI9aB9EREREbEu/8kZWRERERGxfkpkRURERMQqKZEVEREREav0n1wjK3IjbZlQM99vYbMGpeHRNqWhD1A6+lEa+gClpx8ipZ3enSIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCX7kg5AxNq1ffsEKZklHcWNcKykA7gBit6HQxPr3sQ4RETkVtCMrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMjKLfHb0iVs3LC+pMMQKdC7776Lv7+/xb+PPvrIoszBgwe55557qFKlCp06dWL9ev1ci4iUFCWy/0FHjhzmm2++uqVthoScIDQstMTaFymKjIwMevXqxcmTJ41/zz33nHH82LFj9OjRg2bNmrF+/XpmzpzJ7NmzSzBiEZH/Nn0hwn9QuXL+tGrV+j/bvkhB7O3tcXd3z/PYO++8Q48ePXjllVeMfV9//fWtCk1ERK6iGdn/IF9fXxo2bPSfbV+kICtWrKBy5co0bdqUsWPHEh8fbxxbt24dNWvWpEePHlSpUoU77riDjRs3llywIiL/cZqRtXIHDuxn965dtGjZks2b/sFkMtGxU2fq1atvcbxV69Zs3rwJR0dHWrRoyebNm3j00ceNeqKjo1i5cgWxMTEEBlagd5++xqzU5TratG3LPxs3kJqaStu27WneokW+cZ07e5a/V60kIyOD5s0tyx05ctii/YyMDP7+ewUhISF4uHsQ3LUrVatWM8onJiayYsVyzp8/j4+PD7169aFs2bIAnDlzhnnz5gDg5upKrdq16d69B3Z2dtddt/z3vPbaa4wdO5bs7GyOHj3KmDFjGDZsGL/99htZWVnEx8czc+ZMvv32Wxo2bMjs2bO588472b59O1WrVi3p8EVE/nM0I2vlLly4wMqVf/HD999RvXoNvLy9efedt9i7d4/F8Z9/mk21atVp1qwFsbExHNi/36jj/PkwXnj+OZKTkmjcpCknT4bw4pjnSUlJsahj7pyfqV69BgEBgUyaNJFjx47mGVNo6DnGjRuLDTbUrVOX5X/+wZ49e4zjV7f//Xffsm7tWho1akxghQp8NWsGERERAFy8GMv/nnuGqKgomjZpislk4sUxzxMTcwGAMmXKEBzcleDgrtSv34DVq/5m5owvbkjdV8vMzCQlJcX4l5qaUuzXS25vjo6OuLu74+XlRevWrZk+fTpr1qwhMjISe3t7nJycuPvuu+nevTv+/v68+OKLlCtXjhUrVpR06CIi/0makS0FsrKyGDfuVcr4+ABgNplZuGA+TZo0zdk2m3ll3KvGDOs//2ywOP/XxYto1LgJTz41CoBOnTrzv+eeYeWKvxg0+E4AbG3teGXcqzg7OwM5N2/t2bOb2rXr5IpnyZJfadW6DY88+hgA7dp3YORTT+Qbf2joObrfcQfBwV0B6N79DrKysgBYuGABdevV4+mnn8mJrXMXEhMS+H3ZMh4e8cj/JxxtjLqaN2/BY4+NYMQjj+Li4npddV9t8aKFzJ8/z9h2dnZi0qRJ+fZLrJ+TkxOQ80cMQIMGDXBwcLAo4+DggMlkuuWxiYiIEtlSwd+/vJHEAtSpW5fly/8wtsuXD8j35hWAc+fO0alzF2Pb1taWOnXqcO7c2SvqKG8ksQDe3t4kJSblWV/ouXN0736Hse3o6Ej1GjXybf/ue4by1awZHDp0iLp169KiRSvj8v6pUydJT0/ngw/eN8pHRkTg9//HTSYTmzf/w8EDB0lITMBkMmFjY0N0dDSVK1e5rrqvdueQu+g/YKCxbTabiI+LybdfYn1GjRrFM888Q+3atQkNDWXs2LE0a9aMSpUqATBixAjeeustHnzwQerWrcucOXMIDQ2le/fuJRy5iMh/kxLZUiAtLfWq7TRcXFyMbXt7uwLPd3FxybMODw8PY9vGJvd5Zsx51ufs4kxaWlqBMV6pUaPGfDr9C8LCQjlw4ACvvPwSzz73Pxo3boKzszMVKlSg1RWzrgCenp4ALJj/C//8s4HevfviXaYMdnZ27Nm9i/S09Ouu+2oODg4Ws3Emk4n4uHy7JVaoX79+PPbYYxw5cgQ3NzfuuOMOZs2aZRx/+OGHiYyMpHfv3iQlJVGtWjV+/vlnatWqVYJRi4j8d2mNbCkQGxvLrp07AcjOzmb1qr9pUIynAjRs1IgN69cba2IjIyPZtWsnjRo1vqZ4GjZoxPr160hPz0kmz549w9EjR/Itv27tGrKysqhUKYg+ffpSuUplDh06CECLFi05ffo0DRo0pHXrNrRu3YagoMrG8oBjx47SuXMwffr2o1279gDGseutW/57+vTpw6ZNm7hw4QKhoaF8++23BAYGWpR55ZVXCA0N5cKFC+zatYtevXqVULQiIqIZ2VKgbNlyzJnzE0uX/kpsbAwODo6MHDW6yOf37z+Qo0eP8szoUVSqVImTJ0/SpUswrdu0vaZ4+vUfwP79+xj99EgqVKhAbGwMQUFB+ZaPiY1h1MgnqVCxAkmJiSQkJPDYYzlravv07cf582GMGvkkVatVJSU5mYyMTEaPfhaA4OCufPHFZxw4sJ/MzAzg33WN11u3/HddfupFQezt9etTRKSk2ZjN5ryvD4tVWLNmNX8t/4P33v+QM2dOYzKZqFatunEJ/MKFC0RHR1G/fgPjnNjYWMLDz+d6lmtYWBixsTEEBARSrlw5Y39edZw5fRpsbKhSpUqecWVnZ3PmzGkyMzKpUrUqYWGhODs7U7FipTzbT0xM5PTpUzg7OVOtevVcSULcxYuEnQ/Dzc2dypUrWyQasTExnDt3DncPd6pXr8Ge3bupXaeOsS74euouiMlkIiL8HINmpZKSWaRT5DZyaGLdkg4hl8s/UwGBQdjaWucFs9LQBygd/SgNfYDS0w+5OTSlUEo4ODhQs2budXply5bN9VxUX19ffH19c5WtWLEiFStWLFIdVQp5ZqadnR3Vq/97g1eNGjULbN/Dw6PApQxlfHwsbmi7kq+fH75+fsb21c+3vZ66RURE5PalP21ERERExCppRtbKNWzYCH9//5IOQ0REROSWUyJr5fK67C8iIiLyX6ClBSIiIiJilZTIioiIiIhVUiIrIiIiIlZJa2RFrtOWCTWt+uH4peEZjaWhDyIiUnz6jS8iIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyIiIiImKVlMiKiIiIiFWyL+kARKxd27dPkJJZ0lHcCMdKOgAADk2sW9IhiIiIldCMrIiIiIhYJSWyIiIiImKVlMiKiIiIiFVSIisiIiIiVkmJrIiIiIhYJSWyclMkJycTERFR0mFIKRQeHs6MGTPYuXOnxf709HT+/vtvfv75Z/bs2VNC0YmIyK2kRFZuij17dvHhB++XdBhSypjNZp588klef/11VqxYYezfsWMHg++8m6+++or169czePBgHn74YUwmUwlGKyIiN5ueIysiVmPWrFmUKVOGBg0aWOx3cXHhp59+oH79xtja2nL69GkaNmzIqFGjaNWqVQlFKyIiN5tmZEXEKpw6dYpp06YxefLkXMcaNGiAT5kyxnaZMmWwtbXVjKyISCmnGVkhOTmJxMREypcPIDk5idTUVPz8ygKQlZVFTMwFypTxwcnJKde52dnZxMbG4u3tjaOjY75tpKWlER0VBYCrmyu+vn7Y2NjkE0MyaWlp+Pr65qonMzOTuLiL+Pr6YWdnd119KCwmuX2YTCaefPJJ3nzzTcqWLZtnmYyMTGbOnElqairLli3j8ccfp02bNrc4UhERuZWUyArbtm1j8aKF+Pr6EhkZSWJiAvXqN6B3rz7MmjUDW1sbEhISGPPiWJo1a26ct+TXxSxatAB3d3cSEhLo2as3w4Y9nGcb586e5YsvPgNykk4bGxuef+FF6tatZ8Tw66+LqFChIqdPnSIlJZlKlYKY8MZbODs7k52dzY8/fMfKlSvw9PQiKSmR+x94kL59+19zHwqL6WqZmZlkZv77XbRms2b7bpXp06fj4+PDPffck28Zkymb48ePk5KSQnx8PGlpaaSlpeHs7HwLIxURkVtJiawAEBkZwfCHR9CyZStiY2N59plRxMdd5OPJn+Dh4cnSJb/y4w/fG0ngP/9s4I8/ljHpo08ICAggPj6eV199hapVq9GxY6dc9deqXZup06Yb26v+Xsln06cx/bMvsbXNWeESER7Ogw88xCuvjCc1NZWxL73AunVr6dWrN3//vZJNm/5h6rTP8Pf359ixo0x4/VWqVKlK/foNrqkPRYnpSosXLWT+/HnGtrOzE5MmTboBoy8FSU9P55133mH06NHMmDEDgOjoaHbt2sW8efO49957AXB2dmby5MnY2tqSlJRE8+bN+f7773nqqadKMnwREbmJlMgKAAEBAbRsmXNTjK+vL4GBFejUuTMeHp4ANGzUmDlzfjLKb1i/nhYtWpKZkcG5s2cBaNKkKbt37cwzkb3s0qVLJCYmULNWLWJiYoiNjTUuFQcEBNC6TVsg5+adOnXrERERDsD27Vvp2q07/v7+ANSuXYemzZqzbetWI5Etbh+KEtOV7hxyF/0HDDS2zWYT8XExhY6tXB8bGxuGDRtGQkICCQkJAKSmphIbG8upU6eAnPWzLs7//jpzdnbGycmJrKysEolZRERuDSWyAuQkjleys7PDxfnfffZ2dhZJQXx8HKdPn+LIkSMW59WsWTPP+s+cOcPUKZO5eDEWLy8v7OzsMZlMxMfHGUnj1TFc2WZiQiLeXt4Wx729vElIuHTNfShKTFdycHDAwcHB2DaZTMTH5dlduYEcHR355JNPLPbt2bOH7t27M378eAC++eYb9u/fS6dOXTCbzSxfvhyAu++++5bHKyIit44SWbkmFSpUpHadujz66OMW+69cQ3qlefPm0LBhQx559HFsbGxITk7m4eEPYjaZi9Re+fLlOX3mtMW+U6dO0rx5i2vrwA2ISUrO4MGDqV27trH93nvv8dvSxew/cBgbGxuefvppBgwYoPWxIiKlnBJZuSZ3DrmLca+MxcvLi0YNG5OcnMzWbVsICgoybsC6kquLC+fOnePYsaNkZmby6+JFxWqvX/8BvDHhNSpVCqJ27dps3vQP4eHnGTfu1Wvuw/XGJCXn2WefzbWvZcsWDBh4Z57rm0VEpHRSIiu4u7tTvnyAxb7y5cvj7uFhbDs4OlKpUpCxXalSEJM++oSlS3/l+++/xc3NjTZt2xEc3BUAN1d3AgL+rXP48BH8/PNsvv3mK9zdPejZqzdpaak4/v/jsPKKwcfH10hKateuwxtvvs3vy35j86Z/CAwMZNJHkynj43PNfSgsJhEREbm92ZjNZl1HFbkGJpOJiPBzDJqVSkreKyrkGhyaWLfY51x+LQICg6x6RrY09KM09AFKRz9KQx+g9PRDbg79RIiIiIiIVVIiKyIiIiJWSYmsiIiIiFglJbIiIiIiYpWUyIqIiIiIVVIiKyIiIiJWSc+RFblOWybUxN7eet9KerSNiIhYK31qiYiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilexLOgARa9f27ROkZJZ0FDfCsZIOAIBDE+uWdAgiImIlNCMrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsjfBsWNH+eGH7/LdLsm2bmYsIrfCTz/9RLVq1Zg2bZqxLzs7m3nz5tOjRw8aN27Mvffey969e0suSBERuSWUyN4EFy5Es2f37ny3S7KtmxmLyM0WGhrKhx9+iIeHB0lJScb+efPmcfbsWd577z0WLVpE1apV6du3L9HR0SUYrYiI3GxKZEXEKpjNZp566ikmTJiAj4+PxbH777+fl19+iZYtW1KjRg3eeecdEhMT2b9/fwlFKyIit4ISWRGxCjNmzMDDw4O777471zEbGxuL7cWLF+Pu7k7Dhg1vVXgiIlICbouvqD106CB79+6hTZu2bNr0D6mpqbRt245GjRobZWJjY1m96m9iYi4QWKECPXv2wsXFtUjnZ2ZmsnbtGk6GnMDdw4POnYMJCgqyOLdJk2bs3LGd1NRUgrt2pXr1Gvz99wpOhoQQGFiBfv0H4OjoCMC5s2dZuHA+AK6ubtSqXZvg4K65PkyLIzk5mdWrV3H+fBgBAQH07t0XJyenf4+t+pvz58Pw8fHljh498PHxvea2rrRz5w7CQkMZNPhOY9/vy37D19eXtu3aF+m1CQsLZc2a1WRmZNC0WXMiIsLx9i5D+/YditXXwsa1KLFc6zherrtVq9Zs3bIFewd77rvvgRsyxnL9Tp48yZQpU9iwYUO+ZZKSkqhRowYpKSnY2dnx448/4u/vfwujFBGRW+22mJGNiopi+Z9/8P1331K+fACenp689+7bnDhxHIDIyAie/98zREVHUbNWbQ7s38+LY54nNTW1SOf/+MN3/LX8T6pXr4G3tzdffP4pkZGRxrl//vE78+b+TGCFCtjZ2zHh9Vd57923iYyMpE7demzevIlZM7804vX08qJV69a0at2aKlWr8vuy35g1a8Y19z8uLo7/PTeaQwcPUL16DRITEvlses6NLPHx8Tz/v2c4deokNWvVJjExkTEvPM/Fi7HX3N6VzoeFcfjwIYt9R44c5ty5c0DhYxsWFsorL79EeloaQZWrsGTJYub/8gtnz54pdl8LG9fCYrmecbxc97fffI2/vz/16tXPFXtmZiYpKSnGv9TUlOsYeSkqs9nMk08+yRtvvEH58uXzLefm5samTZvYtGkTr7zyCg8++CB79uy5hZGKiMitdlvMyELOpcHxr75mzLKePXOG3bt2UbNmLRYtWki9evV55pnnAOjatRvPPTuav1euYMDAQYWef/rMaXr17kOPHj0B6NWrD1lZmUbbtrZ2jH/1dVxcXADYvXsXgYGBjBjxKAD+/v5M/niSUd7b25sOHToZ261bt+HJJx5l2LDhRvvFsWD+L1SqFMS48a8Z+xISEgBYuGA+NWrU5H/PjzGOpaQk8/vvyxg27OFit3UtChrbpUt+pVWr1jz+xFMAdOrUmVEjn8i3roL6WpRxLSiW6x3H7Oxsxr/6Gh4ennnGvnjRQubPn2dsOzs7MWnSpDzLyo2TnJzM1q1bOXXqFK+//jqQ80fL4cOHWb16NWvWrAFyfjb8/f2xtbXlmWeeYcGCBSxfvpymTZuWZPgiInIT3TaJrL9/eYsksEyZMiQmJgI5l/I7dvw3wbGzs6NevXoWs34FnT/kzrv5+uuZHD92lDp169G8WXPKXHGziL+/v5HEAnh5elG5SlWL7ct1Qc4M0fbt2zh48ACJCQmYTCYAoqOjqVy5SrH7furUSdp3sLwM7+mZk0ydPBlCZmYGn0z+yDgWHn7eIp6brcDX5tw5unXrbhxzcnKievUa+dZVUF+LMq4FxXK941g+ICDfJBbgziF30X/AQGPbbDYRHxeTb3m5Mdzc3Dh58qTFviFDhtCpUydeeOEFACZPnkyTxg0oH1AJs9nMypUrOXLkCK+++mpJhCwiIrfIbZPI2trmXl9qxgzkJEfp6ekWx9LS03F3cy/S+U2bNePT6V9w9swZDhzYz4svvsDzL4yhQYOG+Z5rW8B610WLFrBm9Wp69OxJ9eo1sLe3Y9u2raSnped7TkGcnZ1JS03L85ijoyPly5eneYsWFvu9vLyvqa2r2drZYjKbLfZlZmZalilgbB0dHcnIyLA4dvVrdaWC+lqUcS0olusdRwf7gt8ODg4OODg4GNsmk4n4uAJPkRvg8kzrlRwcHHBzc8PPzw+ALl268NJLLzJs+CNkZWVRvnx53n//fXr27FkSIYuIyC1yW6yRLUyDBg3ZuHEDaWk5SUpMzAV279pJgyLekfzPPzk3iFSrXp2BgwZTtWpV9u/fd83xHD50iG7dujNo0J106RKMq6sbWVlZ11xfk6ZNWbN2tXEZHDCe9dqsWXPCwsJo0aIVHTp0okOHTtSuXRc7O7trbu9Kfr5+hJ47Z8QfGxvL0aNHinx+vfr12bhxg5H8nj8fVuD5BfX1ese1JMdRbq1Fixbx3HPPGdvNmzfn559+ICQkhLCwMA4cOMDjjz9eghGKiMitcNvMyBZkwMBBHD58iOeefZqgoMocP36ctu3a065d+yKdHxoayuwfn6Ry5cokJiYSHR3F8IdHXHM8HTp24qtZMzhy9AiZGRmkpqYaTzS4Fv36DeDUyZM8M3oUtWrVIjo6ivYdOtK0WTP69R/AudBzjHzqCWrWrElycjKJiYk8PfqZa27vSs1btGT+/F94/vlnCQgI5EJ0NL6+fkU+f8CAQezbu4fnnh1NhQoViIiIICAgEFvbvP9GKqiv1zuuJTmOcmtd/RzZy1xcXPL92RMRkdLHxmy+6rpyCYiKiiIiIpwmTf69KSMk5AS2NrZUq14dyFk/eebMaWJjYwkMDCQwsEKxzo+Pj+dkSAhOzk7Url3HuESc17n79u3F37+8cYd0cnIye/bssrgRKTIykrNnz+Du7kGdOnXYsWM79evXx8PDkwsXLhAWGkrTZs0Acm3nJzw8nIjwcAICAwkMDLQ4Fh0dTVhYKG5ublSrVt2Iv7ht5XU8MzPTmEWtUaMmZ8+cwdXVlaDKlYs0tllZWYScOEFGZgbVq9fg/ffeoV379vTt27/YfS1oXIsSy7WOY151F8ZkMhERfo5Bs1JJySy8vBTNoYl1i33O5dciIDDIqhPZ0tCP0tAHKB39KA19gNLTD7k5botEVqzbgQP7adiwEZCTWI57ZSwffTyFKlWqlGxgN5kS2ZtDiax196M09AFKRz9KQx+g9PRDbg6rWFogt7elS35lzs8/4ezszLFjR7n77qGlPokVERGRkqdEVq7ba6+/walTp4iLu8iop5+hbNmyJR2SiIiI/AcokZUbolq1akC1kg5DRERE/kO02ERERERErJISWRERERGxSkpkRURERMQqaY2syHXaMqEm9oV8ve3tTI+2ERERa6VPLRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJISWRERERGxSkpkRURERMQqKZEVEREREaukRFZERERErJJ9SQcgYu3avn2ClMySjuJGOFairR+aWLdE2xcREeujGVkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZG9jJpOJiIgIsrOzSzqUIktNTSUm5kJJhyGl0IULF/j+++/ZtWuXxf6kpCRWrVrFX3+t5PTp0yUUnYiIlAQlsrex1NQURj/9FPFxcSUdSpHt2rWDd995u6TDkFLo2Wef5cUXX2Tp0qXGvvnz59OmTRs+++wzfv/jD9q2bcvbb+vnT0Tkv0LPkZV8paamkpychJ9f2ZIORf7j5s6dS1paGi1atLDYX6ZMGTZu3IiXlxcR4ec4cjSEQYMGMXz4cCpXrlxC0YqIyK2iGVnJ1/bt25j4/nslHYb8x4WHh/POO+8wffr0XMfuuOMOypQpY2xXr14dgISEhFsWn4iIlBwlsrdQcnIyaWlpeR6Lj48nKyuryHXlVT47O5uIiAhMJhMmk4no6GhMJpPFOQWtt710xfHMzEwuxceTlZVJREQEERERZGZmkp6ebmxfunSpyPEW1h5QaN2pqSnExsQYZRMT809Wrq77SnmNw5V1Z2RkEBsbW+z+yM3x9NNP8+KLL1KxYsVCy06fPp26detSr169WxCZiIiUNC0tuAUOHjzAV7NmEBsbi52dHV26BDP84UewtbVl69YtfDVrJiZTNmlpaXTq1JlHH3sCR0fHPOsqqHxCQgKjn36KgYMGs3bNalxdXfl48hTWrFnN/F9+wd7enrS0VPr1H8B99z1g1Pn7st9YsOAXbG3tADOPPf4kgYGBLF36K0lJSbz7zpsAvPjSy6SnpzP906kAJCUl4+7uxnP/e4FatWoXeTzyaq99+w6cPn2qwLq3bNnCkl8XU7lyZQ4fPkxKSjK1atVm/Kuv4+TkVGDdAH/8sSzfcbhcd6VKlTh69Ah169bjxZdetog7MzOTzMx/v4vWbDYhN9c333xDRkYGI0aMKELZ7/j1119ZsWIFdnZ2tyA6EREpaUpkb7LYmBg+mPge99//IH369iM7O5vlf/5Benoaly5dYuqUyYx+5lk6dOjExYuxvDHhNRYtWmCRaF4WGRlRpPKxsbHM+upbHBwc2LJlM4sXLeTd996nUqUgYmNieO318VSuXIV27dqz6Z+N/PLLPCa88SY1a9YiKSmJtWtX0759B4YNH8FvS5cw+ZOpFnF8/sVM4/9//vkH0z+dyrRPP8fWtvAJ/vzaA6hTp26hdZ8/H8bd9wxlzItjSU5OYuxLY1i/fh09evQssO7CxuFy3QMGDOSlsa/kGfviRQuZP3+ese3s7MSkSZMK7bNcG5PJxKuvvspTTz3FDz/8AEBUVBQHDx5k/vz53HPPPUbZSZMm8f0Ps1m+fDm1atUqqZBFROQW09KCm2zb9m2UK+dPn779ALCzs6Nf/wG4uLiyc8cOKlasRIcOnQDw8fGlb9/+bN2yOc+6ilp+yJC7cXBwAGDd2jW0adMWe3sHIiIiyMjMpEWLluzYsR3ASAJr1sz58Hd3d6d//4GF9is1NYWoqCiaNGlCdHQ0Fy8W7VJ8UdorqO7yAQF07JjTfzc3d+rVq8/582GF1l3YOAD4+vrS/Y4e+cZ+55C7mP3TXOPfrK++LVKf5doNGTKECxcusGPHDnbs2EFCQgLh4eHs37/fKPP6668ze/Zs5vz8g5JYEZH/GM3I3mTx8XH4+PrmeexSwiV8fHws9pUpU4b4+LzXnha1vLe3t/H/ixcvcvz4cfbu3WNRplbtOkZ8jZs0KUpXAAgLC+XTaVMICwvDw8MTe3s7TCYTcXFxRXq6QUHtFaVuN1dXi3Ps7e2NtcIF1V3YOIDluOXFwcHB+AMBcmYM463nyWhWx9bWls8//9xiX69evWjVqpXxiK133nmHadOmMXbsWLZu3caRoyHY2NjQtWtXgoKCSiJsERG5hZTI3mSBAYGsXbOGzMxMIwkym80A+PuX55+NGzCZTMal8zNnzlC+fPk86ypueYDAChWoV78+I0Y8arH/8s1OAYEVOBkSkuuYnZ0dtra2mK5aBzrn55+oXqMmEz/4CDs7O1JTUxj20AOYTeYijUdB7d3MugsbB7EOvXr1omrVqsa2n58fDz30EOHh4aSkJOHq6o6NjQ3NmjVTIisi8h+gRPYma9e+AwsXLmDKJx8zYMAgsk3ZLPl1Mc+/8CLt27dn7pyfmTVzBj169ORc6DmWLVvKk0+NyrOu4pYHGDx4CK+OfxlfH18aNmpEcnIy27Zuwb98AP369WfQoMGMe2Us5QMCaNGiFVGREezYsZ1nn3uecuXKcSE6mmPHjuLp6YWfnx+Ojo5ERUVy9swZMjIzWLRwgZGYF0VB7d3MugsbB7EO//vf/yy2R44cCfz/t+CFnyMgMKhIa7VFRKR0UCJ7kzk6OvL+xA9ZMH8e3333DS6uLgwcOBjX/79EPvGDSfwybw5ffDEdT08vRo4abdxlb2trS/ny5bH9/zuwXVxcCyxvZ2eXU/6KD/IqVarwwYcf8eviRWzcuAE3NzfatGnLHT16AlC1ajXenziJXxcvZPu2rQQGVuCBB4cBULt2Hfr07cdXs2aSmprCiy+9zPDhI/jxx+/59NOpuHu407tXH+Li4nBwzJltdnF2xa9s/ksMCmqvsLpdXV3x9fWzqM/Tyws7W7tC6y5sHPKqW0RERG5vNubiTHmJiOHyLOCgWamkZBZeXgp2aGLdaz63tMzIloZ+lIY+QOnoR2noA5SefsjNoZ8IEREREbFKSmRFRERExCopkRURERERq6REVkRERESskhJZEREREbFKSmRFRERExCrpObIi12nLhJrY21vvW0mPthEREWulTy0RERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySfUkHIGLt2r59gpTMko7iRjhW7DMOTax7E+IQEREpGs3IioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhVUiIrIiIiIlZJiawV++67bzh79swNqy86Ooovv/jslrX39VezCAsLu2H1ye1h/fr13HvvvbRo0YLBgwezcuVK45jJZGLu3LkMHDiQli1b8sADD7B79+4SjFZERKyZElkrtnnTJi7GXixy+cIS0YSEBNavX3fD2ivMP/9sID4+7obVV5jjx48xZ85Pt6y9/6JDhw7x2Wef8fDDDzN79mx69OjB0KFD2bRpEwAzZ85k/fr1PPvss3z//fdUqlSJ3r17c+rUqRKOXERErJG+EAH45puv6NGjJ5UqBZV0KDdV7dp1cHf3KOkwSoyXlzc1atQs6TBKtTp16rBgwQJju27duvzyyy+sX7+e9u3b88QTT2BnZ2cc/+CDD5g/fz4bNmygWrVqJRGyiIhYMc3IAps3/cPFizdupvF21a5de3x9fUs6jBLj7+9Pq1atSzqMUu3KJBXg6NGjnDhxgtatW+d5PCIigpiYGKpWrXrLYhQRkdLDamZk5879mZiYGOxsbSnn709wcLdcSdmhQwfZt28vWVlZdOrYmSr//+E4ffo0HnzgIcr4+AAQGRnB0iW/8uRTo1i0cAEpKSksXforGzasp0aNGvTu3ReAgwcPsG/fXjIzMmncpAlNmzbLN76vv5pFly7BnAg5TlhoKBUqVqJnz16EhoayYcM6srOzCe7S1Yhp3do1ZGRm0qNHT6OO2NhY5s75iUcfexwXF9dixwCwdesWjhw5jJOTE61atqZGzX9nIL/77hu6du1G5cpVADCbzaxdu4bTp07iXz6AqlWLPyNWUHwFjXteVv29kugL0dxzz73Y29uTmprKunVrOB92Hh9fH7p17Y6XtzcAR44c5sD+/TRp2pSdO3eQmppKly7BVKlSlbVr13DyZAiBgYH06tUHBwcHIGdpwc6dO7j//gcBCA09x5IlvwLg6upK7dq16dChU7HHQHLr27cvx44dIzY2ljfffJNu3brlKpOens6IESPo1asXnTt3LoEoRUTE2lnNjGyN6jWoX78BtWrVJjoqijEvPEd8fLxxfO6cn/lo0gfY2tri5enFjBmfExp6DshJGpNTUoyyCZcS2LBhPQBVq1XD3t6eKlWqUr9+A2N5wfxf5vHptCk4Oznj4+PDjC+/YOn/Jz15+eefDXz44URiY2MJCAjk18ULmfj+u3z5xWf4+vpiys7m1VdfIe7/Z359/fz4+acfSU9PN+r4++8VREVFGUlscWOYOeMLvvv2G3zK+GAymXjjjdf4558NxvGr17h+/tmnLFq0gHL+/ly4EM3UKR8X+jpcqbD4Chr3K5lMJr777hv+/PMPevbohb29PQkJCbzwwnMcOHCAChUrEBUZyQsv/M94zSMiIli2bCnff/cN3l7epKel8dqr45j4/rucOnmSoKAg1qxezddfzzLaiY6OYvu2bca2u5s79es3oH79BpQrV44F83/hm2++yre/mZmZpKSkGP9SU1PyLftf99VXX/Hnn3/y5ZdfMnnyZObOnWtxPDU1laFDh2Jvb8/3339fMkGKiIjVs5oZ2ZZXXBLufkcP0tLTWbVqJXfddQ+RkZEsXryQDz78iOrVawDQp28/iyQxP82aNcfR0ZHGjZvQuHETICfhWbhwPlOmfkqFChUBaNCgIa+9No4+ffsZM3xX69evPwMHDQbA1s6OH3/4jllffYunpycAhw8fZu++vQQHd6Vhw0Z4e3uzadM/dO3ajezsbFavWsWDDw27phjOnw9j1aq/+WTKNCMZ9ynjww/ff5fnLGNYWBgbNqzn0+lfUL58eQCcnZ0LTJSvdK1jdLWMjAw+/uhD0tLTeO/9iUYSv3DBfIIqBTF27CtG2SlTJrNs2VIeemi4se+119/E1TXnnP3791O2bFmeePIpAAIDKzB1ymRGjnw6z7bL+PjQteu/M4Xt2nVg1MgnuP/+B3FxcclVfvGihcyfP8/YdnZ2YtKkSUXq539NYGAgALVq1WL79u0sWLCA++67D4DExETuuusuPDw8mDdvHs7OziUZqoiIWDGrSWRjY2PZuHE90VHRpGekE34+zEiWTp8+hZeXl5HEAjg4OBQ5mbraqVOnsLOzY/HiRRb709PTiYqKpGLFSnmeV616deP/Zby98fMraySxAN7e3iQmJBjbPXr2YuWKv+jatRs7d2wnIyOdtm3bXVMMZ8+exdu7jMUNa40aN+abb74iISHBIo6c8mcoW7askcRCTiJa1ET2Wsfoap9Nz0m8J7zxlsX6yZCQ42RlZTN9+jRjX0R4OCnJyca2v395I4mFnPGtWu3K16AMCVeMd1727N7NwYMHSEhMwGQyAXAhOpqgypVzlb1zyF30HzDQ2DabTcTHxRSpn/8VCxYsoGzZsnTq1AlbW1tCQkJYvXo1gwfn/IF38eJFBg8eTFBQEN9+++01v0dFRETAShLZS/HxvDjmfzRr3oIaNWrg5ORMUmIiaampADg6OpKRkZHv+TY2NpjNJmM7MyuzwPYc7B2wt7enfv0GFvvr12+Al5dXge1YbNva5CpjNpuN/3fp0pU5P//E6dOnWLHyLzp16oyjo+M1xeDg4EBmpuUYZGRkGseuljNmluOQmVnwuFi0V4T4ijLuffr04/fff2PTPxvp1LmLsd/ewQE/v7IW9dev34Ay3mWMbds8xtfWJve+/Py6eBF//bWc7t3voHxAAHZ2dvyzcQNpaWl5lr/6jyOTycQtfHqYVWjVqhUvv/wyDzzwAPb29mRkZHDfffcxbtw4AL744gt27dpFZGQkjRs3Ns57+umnefrpvGfORURE8mMViezpM6cBG55++hlsbW3JzMxk5Yrl+Pr6AVCzZi3jxqXg4K4AREZGYmtrQ7ly/vj6+nL27BljtnLb1q0W9Tu7uFgkLzVr1QJs8PHxoUmTpkBOArpt21Y8PCxnNq+Hm5sbHTp24uefZrN/3z6GDx9xzTHUqFGT9PR0dmzfZizDWL1qJdWqVc/zMnmNGjVJSkpk//59NGqUk1CsW7e2yLEXJb7Cxh2gTt26NGvWnHfffYu0tDR69OwFQJMmTdm2dSsdOnQ0kvu4ixeJio4qcoyFOXBgPz179uLOIXcBsH//PrKysm5Y/f9FlStXZt68eaSlpZGUlISfn5/F8dGjR/PQQw/lOq+gPxBFRETyYxWJbM2aNXF2dmLsS2MIDAzk5KmTuFyxrs7T05NRTz/Dl198xprVq3BxdSU2Jobxr74OQP8Bg/jyi8/ZsnkzsbGxODk5WdTfpElTvv/+W7Zt3ULNWrXo3bsvzz73Pz6dNoWKlSrh6eHJ6TOnadSoMW3atL2hfevVqzcvjnmeWrVqG08TuNyn4sRQpkwZHnn0caZNm0K9+g1ISkwgOjqaV8a9mme7ZcqUYdjwh/nwg/dp2KgxF2NjcHIq+lrFosRX2LhfVq16dd56+13efusN0tLSGDBwEAMGDOLM6dOMGvkEtWvXITklmdiYGEaOGl3kGAvTtl17vvv2a0JCTpCRmUl83EUjaZbr4+zsnOfaV29vb7z//8kTIiIi18vGfOW17ttYeno6Bw7sJz0tjZq1apGSkkpSUiINGjQ0yiQmJhIScgKz2Uy9evUtPkjPnDlDRPh5KlSshJeXF/v27jEuZZvNZg4fPkxMzAV8fX2NOlNSUjh5MoTk5GSqVq2Gv79/vvH9888G6tdvSJkyOZe+o6OjOX3qJK2vSDr37d1LGR8fgoL+XceanZ3Nw8Mf4uERj9CtW/dc9RYUw+bNm6hdu47FY8hiY2M5eTIEJ0cnataqZbGGNK/yERERnDlzmoDyAfj6+bJr1y66dAnOs495nV/YGBU07rnHLIpDhw7RsmUr3N3dgZyb2MJCQ3F1c6NWrdpGMhwREcH582G0aNHSaGvXzp0EBAYaNxolJyexbds244auqKgozp49Y/Es2bCwUM6cOY27uwf16zdg65bNNGrcpEgzhCaTiYjwcwyalUpK0VdllCqHJtYt6RCAf1+LgMAgbG2t5mEsuZSGfpSGPkDp6Edp6AOUnn7IzWE1iWxptWXLZmZ8+TkzZ32ju7etjBJZJbI3WmnoR2noA5SOfpSGPkDp6YfcHFaxtKA0Sk1N5atZM9izZzd33nmXklgRERGRYlIiW0Ls7e1p2KgxXbt1t1geISIiIiJFo0S2hDg4OBhPWBARERGR4tNiExERERGxSkpkRURERMQqKZEVEREREaukNbIi12nLhJrY21vvW0mPthEREWulTy0RERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySElkRERERsUpKZEVERETEKimRFRERERGrpERWRERERKySfUkHIGLt2r59gpTMko7iRjhW7DMOTax7E+IQEREpGs3IioiIiIhVUiIrIiIiIlZJiayIiIiIWCUlsiIiIiJilZTIioiIiIhV0lMLrpKenk5aaipe3t4lHUqpkpiYiJ2dHa6uriUditwChw8fJiQkhEqVKtG0aVOLYyaTiT179hAeHk6tWrWoXbt2CUUpIiLWTonsVTZt+oe/lv/BpI8+uWVtpqenk5qaincpTp6/mjWD8gEB3H//g/+J/v5XxcbGMnz4cC5evEjlypXZtWsXVapUYfHixbi7u7Njxw5Gjx6Ns7MzAQEBbNy4kZ49e/L1119ja6sLRCIiUjz65LjF0tPTuRQfb7Fv/769TJ06+aa2cTu50f2V20d6ejpvvPEGmzdvZu7cuezevZvTp08ze/ZsADIzM5k9ezbr169n3rx5bNq0iSVLlrB8+fISjlxERKyRZmRvsa1bt/Db0iVM/mSqsa9lq9a0bNX6prZxO7nR/ZXbR2BgIIGBgca2u7s7vr6+pKWlAdCuXTuL8pUrV8bd3Z24uLhbGqeIiJQOSmQLkJWVRWZmJi4uLsa+1NRUsrIy8fDwNPZduf7zyjW2V5+fmZlJclISWdlZxMbEAODp5YXJZLK41F5QHVfHl52djZOTkxGDg4NDnm04ODgY5yUnJ+Hm5m5RV1HbTE5OxsnJCXv7wn90rozv6rbyWlpQWN3Jycm4urpiY2Nj7MvMzCTh0iUAXN1ccXHJew3utdQt1+7PP/8kJiaGDRs24OLiwvDhw/MsN2/ePDIzM+nevfstjlBEREoDJbJ5SM/IYPqnU9m9excpKSk0bNSYl18eh4ODAytX/MWhQwcZ/+rrRvkZX35OUFBlht57H5s2/cPSpb9So3qNXOeHhobyyy9zSUtL45VXxgIwbvxr/9fefUdHUfVhHP9ukk0PHQKhhQRCDb2D9C5dwAYo+tpFsSO9iCCiUgVRFFGpAcRGB6X3jhBKCB0CqaRvsvv+EVlZSCCAsGx4PufknMzMnTu/O2vkyd27E86dO0PIggWMnzAJ4KZ9GI1G0tPT+X7md6xcuRxXV1dKlQoAIKhsWerWrZ/pNUqVKsWC+fP49dclABiNRnr1fpamTZtl65pHQkOZOHE8MTHRODk50bBhI57t8xyurq433D+z2cys72eyfPlSXF1dCQgIxGKxULhIEQC2b99qM95b9b1ixXLmzZ2DyZSKxWKhc+euPNatOwDHjh3l88/GAZCYmEDBgoV4441+BAQG3nXf1zOZTJhM//4tWovFfBv/VT1cNm3aRHh4OAcOHKBKlSqkp6ff0Gbt2rW8/fbbzJgxg8KFC9uhShERcXQKspk4c/o0nTp2pu8b/bhyJY7333uHv/5cS4uWre76/Oeef+GGt/3PnTtzW32sXbOaLVs28cX4SRQuXJhNmzby2bixBJUtS0BAQKbXWLNmNUuX/s6oUWMoUbIku3fvYvTHH1G0aFGCgsre8pqz5/xInbp16dmzN2lpafz551ouXYqgaNFiN9S+du0aNmxYx2efT8DPz48NG9bxxeefEZTFp9Nv1vf2bVuZM/tHBg8ZTkBAABcvXmTI4AEUK1aMOnXrUb58Bb7+5lsALBYLv/6yhAkTPmf8hMkYDIa76vt6ixaGMH/+XOu2u7sbY8eOzdZ/Ew+bjz76CMj4paZDhw4MGTKEqVOnWo8vXbqU//3vf8yYMYN27drZq0wREXFwCrKZKFy4MM2aZ7zV6eOTi0qVgjlz5sawea/Ov1UfmzdvomXL1tZZrPr1G7CkdJmb9rdp4waaNW9BiZIlAahWrTpVq1Zj08aN1iB7s2vmzZuPqKgo4uLiyJ07Ny1vEuo3bdxAq9ZtrGslGzZsxC9LlmTZ/mZ9r169ioYNHyF3rlxEXr6Mi7MzdevVZ+vWLTZh02QyEX/lCvXqN+Cnn34gKjKS/AUK/Cd9X9X1sW506NjJum2xmImJvpz1TX8IXbx4EV9fX+u2k5MTfn5+xMXFWfctXLiQN954gx9++IFmzZrZo0wREckhFGQz4eXlZbPt4uKCKc2URev//vxb9REbG0vevHltjufNl++m/cXExFA9fwGbffny5Scm5t8P2dzsmi+++DJLl/7OF1+MIzEhgarVqtOtW49MlxbExsaSL69tPfny58+ytpv1HRkZyaFDf7Nlyxabc8qVKwfAuXPnmDJ5IseOHcXLywtnZxfS09OJjo4mf4ECd9X39YxGo81aY7PZTIw+o2Rj5cqVzJ07l5YtW+Lt7c22bdtYtGgRixcvBuC3336jT58+PPvss0RERDB3bsYMd6VKlahUqZI9SxcREQekIHub3NzdrZ/Avioy8jIlSpTM1vlOTk5YLJa7qqFQoUI2M7wWi4WzZ05TokSJLK9RyNeX06dP2ew7dSqcSsGVs3VNNzc3unbtRteu3UhOTmbY0MF4enjSuUvXG+vz9eX0mdNZ1nc7fRcpUoRKwcE880wfm3PM5oz1qT/+OItixYoxbPhIjEYjycnJ9Or5pPX43fQtt69nz56UL1+eJUuWcPz4ccqWLcvu3butr73JZKJ79+4kJCSwcuVK63keHh4KsiIictsUZG9TUFBZvvv2GzZv3kSJEiXYuHEDx48fp1q1Gtk6v2CBgly6FEFYWBi5c+UiV+7ct11Dy1atGffpJwSWLk2pUqVYvXoV58+fx4Ahy2u0bduOj0eNpHTpMpQtW5ZNGzdy4sQJ+r31brau+fHHH9GgQUPKlAki/soVYuNiM52NBWjdqg1jx44mMCCQgMBAVq/KqO9O+u7c5TEGD/qQQoUKERxchcSEBLZs3Uy+fPlp374DLs7OREVFceHCeUypJhYsmGcT4u+mb7kzNWrUoEaNzH8eunTpQpcuXe5zRSIiklMpyF7H3d2d3Lnz2Ozz8vbG2ckZgICAAJ5//gUWhizAYDBQu3YdmjZthsc/f3r1VueXK1+eFi1aMWH8ZyQmJvHhgEG4ubqRJ++/59yqj2rVqvPSS6/wx++/YTKlUq16DapWq2592zuzawQHV+bddz/g19+W8PPPi/Ar4seoUWMoVKhQtq75wgsvEbJgHosXLcTd3Y3mzVrQuk3bTO9hlapVeeXV1/j9t4z6qteoSYsWrax/nvb68d6s74CAAEZ+NJpFi0JYtmwpXp5e1K1Xnzb/HH/mmT58++03jBk9Cm9vH9q0bce58+dw+ede3E3fIiIi8mAzWO72fW6xC4vFYn3maXp6Oq+9+jI9e/WiYcNGdq7s4WE2mzl/7hSdpyeReHtLoHOMg6PL27sE4N/XoohfCYf+U7c5YRw5YQyQM8aRE8YAOWcccm9oRtZBfTp2DB07dsbd3Z3f//iN1NQUatSoae+yRERERO4bBVkH1aJlK+bOnU10dBQlS/rz8eixWf5VKxEREZGcSEHWQVWvXoPq1bP3ATMRERGRnEiLTURERETEISnIioiIiIhDUpAVEREREYekNbIid2nzkDK4uDjuj5IebSMiIo5K/2qJiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCEpyIqIiIiIQ1KQFRERERGHpCArIiIiIg5JQVZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEIbnYuwARR1dvxFESTfau4r8Qmu2WB0eXv4d1iIiIZI9mZEVERETEISnIioiIiIhDUpAVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkBdkHVGJiIl9N+5Lk5GR7l3JfhIYeZtHCEHuXIXdh+/bt9OnTh4YNG/LEE0+wdu1a67HU1FSqVatm/Zo0aZIdKxURkZxCQfYBlZKSwooVy0lNTc3y+FfTviQhIf4+V3ZvnD17li1bNtm7DLlDR44cYfjw4XTs2JHJkyfToEEDunbtysaNGwEwGo3MnTuXuXPnkitXLi5dumTnikVEJCfQH0RwUE5OTviXKoWzs15Csb+AgAB+++0363bVqlX5+eefWb16NQ0aNMBgMFC2bFkAPDw87FWmiIjkMJqRdVBGo5HWrdvi7u5u71JEcHGx/YUqPDyco0ePUrNmTTtVJCIiDwNN5/1Hdu7cwZHQUNzc3GjatBl58+UjKiqShSELeOHFl63tDh86xOHDh+jcpSsAR0JD2blzB2azmWrVq1OhQkWbfhMTE9mwYT2XIiIoXaYMDRo0BDKWFsz8bgY9e/XGy8ubxMREfpg1k+49HmfLli03tIeMdYpr1qwmIuIipfxLUbBQIf4+eJCuj3XLdExZ1bbk58WU9PenatVq1rbHjx9j48YN9O79bLZqMZlMrFmziosXLxIQEAAWi82Yf5g1k27derB121bOnT1Lr97PkJqawl9//UXk5cv4FS1K48ZNcHV1BTLW2B48cIBq1auzc8cOkpKSaNS4McWLl2DD+nUcP36MIn5FadmyFc7OztZrpaSksH79Os6ePUP+fPlp3KQpPj4+t/fii1WPHj34+++/OXfuHAMHDqRdu3b2LklERHIwzcj+B7744jOmfzUNZ2dnklOSGTVqBCkpKcRfiWfFiuU2bc+cOc2WLZsB2LdvL8OHD8FsNuPl5cW8uXNYv+4vm/affPIx0dFReHt7883XX/HLkp8BSEszsWLFcpKTMj4MdnVN7ciRwzNtbzabGTZ0MH+uXU2uXLnYtm0rn40bm+W61JvV5uTsxNw5P9m0X7x4EUmJidmuZfjwIaxetZJcPrnYsnkzc67p7+r5w4cPJSoqkuIlihMVFUW/N/ty4MB+cufJzdo1q+n/wXvWNcRnz55l8eJFTJv6JS5GF6Kio/iw//uM/WQ0+/fvI3/+Avzx+298O+Nr63WuXLnCu+/0Y+uWzeTNk5fjYcd55+03iY2NzfSemEwmEhMTrV9JSYmZtnuYjR07ljlz5jB+/HgmTJjAvHnz7F2SiIjkYJqRvUu7d+1i547tTJo8lbx58wLwaLv2ODnd+neEI0dCqVq1Gk/37AVA5y5dibx82abNU0/1pFat2kDG2sING9bRsVPnLPvMqv32bVu5cOE8U6d9jZubGwBjxnxMVOTlTPu5WW3NmjVnzuyfCAsLIyAggJiYGLZv28rHoz/JXi3bt3H2zFmmTptuXRox+uOPiI6Osjm/x+OP07BhIwC+mvYlJf396d9/AACPPtqBN/q+ysqVy3n00Q4AmM3pDBk6DC8vbywWCy8fPEju3Ll55dXXASheogQTJ4y3zpAvDFmAb+EiDBw0xHrNz8aN5bfffuHpp3vdcE8WLQxh/vy51m13dzfGjh2b5WvxMPL39wcgODiYAwcO8NNPP/H444/btygREcmxFGTv0tFjRyhTJsgaYgFy58mTrXMbNnyEdX/9xRdffEZwpWCCK1fB19fXpk2ZMkHW7/MXKEBsXNxN+8yq/YnwEwSVLWcNsQCVgyvz559rbrs2Ly9vGjR8hBXLl/HyK6+yetVKihcvQWBg6WzVEn7iBGXLlrVZ31ulStUbailf/t9lFidOnKBBw3+XJhiNRipVCuZEWJh1X5Eifnh5eQNgMBjImzevTU358+UnLu7f2dYjR0KxWCx8Ne1L677Lly+TksWTIro+1o0OHTtZty0WMzHRmf8i8LBZsmQJBQsWpH79+gCcOXOGNWvW0LJlSztXJiIiOZmWFtwlFxcXTGmmTI8ZnAw37Lu2beHCRZgwcTKdOnYmNTWVj0YOs779ftUNM7vXrCXNTFbtXVxcSEtLszl0/fa1blVb2zbtWL/+LxISEli1agUtWrbKdi3OLs7ZqsVoNFq/dzG6kGa68RwXl3/bOGVyv282M+7s7Ey+fPnwL1XK+tWkaVNatsg8fBmNRjw9Pa1fHh6eWfb9sAkODmbcuHGUKFGCcuXKUb16dWrVqsXgwYOtbbp27Uq1atXYuXMnM2fOpFq1agwdOtSOVYuIiKPTjOxdCg6uzLy5cwg7fpyAwEAAwo4fp2ixYuTLlw+z2cyZM6cpVqw4FouFXTt3Ws89ePAgAQGlCAgMJCAwEIOTgTWrV9906cCdKleuPIsXLSQqKpJ8+fKTlpbGpk0bsmx/q9oCAgMpXrw4kyZNIDY2lkceaZTtWsqXr8CihSFEXr5M/gIFblkLQPlyFdi0aSPtO3TEaDQSGxvLrl27eO75/2X7uterUqUqu3btoHnzltZP3cfGxhIRcfGO+3xYBQQEsGjRIq5cuUJcXBy+vr43PMng888/JyUlxWZf7ty572eZIiKSwyjI3qUyZYLo3v1xBg8eQOXKVUg3m4m/coXhIz7Cy8ubJk2aMnzYUCpXrsypU6ds/nGPi43h3XfeomRJf1yMRnbv2sWTTz19T+qsVCmYuvXq8967bxMcXJmTp05idDFiMGQ+Y5md2lq3acfkSRNo1qw5Xl5e2a6lYsVKNGz4CO+//461Frd/nj6Qlc5durJv317eebsfAQEBHDx4gEqVKt1WgL5ep85dOHb8GK+9+jIVKlQgISGBc+fO8dLLr9xxnw87Hx+fLJ/6cHX9rIiIyH/FYLHc4r1qyZaLFy9y7NhRPD08qRQcbH1b3GKxsH//PmJjYggsXRqz2cLZM6epU7cekPHJ+b8PHsCUZqJ06SAKFy4MZHxy/88/19C0aXPrI6YiIi4SGhrKI480sj6+qnHjpri7u9+y/VWHDx8i4uJF/EsFsGP7Ng4ePMDgIcMyHVNWtV0VEXGRV15+kY9Hf0LZsuWs+7NbS2jo4X9qKYWTkxOnT5+mbt16mZ4PkJ6ezqFDfxMZGYmfn5/NGtyzZ89Yz79q65bNFC1WnGLFigEQHx/Pxo3rad26rc04wsPDOX36FF5eXpQvXz7bSwbMZjPnz52i8/QkEjNfXZJjHRxd3t4l2Lj6WhTxK5GtD1o+qHLCOHLCGCBnjCMnjAFyzjjk3lCQfYgcO3aU0qXLABlhs/8H79GgQUO6de9xR/2FhMxn86ZNfPb5+P+wSsehIPvgyCn/0OWEceSEMUDOGEdOGAPknHHIvaGlBQ+RXbt28tW0qRQrVozDoYfJkzsPbds9etv9nDt3lkULQ9i0aSNvvfXOPahURERE5NYUZB8iPXo8QZ06dQkPD6dFy9aUK1fO5q9cZZe7uztBZcvRqlUbgsqWvQeVioiIiNyaguxDpmRJf0qW9L+rPvLly0+rVq3/m4JERERE7pAWm4iIiIiIQ1KQFRERERGHpCArIiIiIg5Ja2RF7tLmIWVu+CtWjkSPthEREUelf7VERERExCEpyIqIiIiIQ1KQFRERERGHpCArIiIiIg5JQVZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEISnIioiIiIhDUpAVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkBVkRERERcUgKsiIiIiLikFzsXYCIo6s34iiJJntXcXdWvOZh7xJERERum2ZkRURERMQhKciKiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySnlrwH0lNTcVsNuPu7n7LtomJibi5ueHs7Jzp8fT0dJKTk/Hy8vrP6rvVNW/GZDKRlpaGh8edf7I9JSUFg8GAq6vrHfch99+xY8cICwujRIkSlCtXzubYH3/8QUpKis2+Nm3a3NV/JyIiIrdDM7J3KT4+nsGDBvDsMz2ZNHF8ts55843X2bd3b5bHT5wI4/nnnrlpH4mJiaSnp2e7zldfeZFDh/7OdvtrrV+/jqFDBt7RuVdNmTyRkJD5AKSlpZGUlHhX/cm9dfLkSdq2bUv37t2ZOnUqrVq1onPnziQnJ1vbvPzyy0ybNo2QkBDrV1JSkh2rFhGRh41mZK9zuzOXq1atwOhq5Mef5uLkdG9+L8ispg8/fJ/nn3+BypWr3JNr3ks7dmznlyWL+Xj0WHuXIlmIi4tj4MCBNGzYEIDLly9Tu3ZtZsyYwWuvvWZtN3DgQBo1amSvMkVE5CGnGdnr9H39FQ4c2J/t9qdPnaJMmaB7FmKzqmnChMkOGWIB6tatpxD7gAsODraGWIACBQpQokQJLl26ZNPu4MGDLFu2jNDQ0PtdooiIiGMF2bS0NJvtlJQUUlNTgYx1pVe/B7BYLJm+9Z6QEI/ZbLZup6enk5CQAEBSUhIWi4WUlBQSEuJJSUnBZDKRkBB/w5fZbCYpKZErV65gsVhISIjHZLL9805Z1ZDd8WVVE9y4tCApKZGEhPhsv7WbMW7be5FVDde6nTFl1TarpQVpaWlYLJYs+8ustqvjSEiIv2ntd9K3/GvHjh3s3buX9u3bW/c9+uijbNiwgalTp9KsWTM6d+5s/VkSERG5Hx74pQWpqanMnz+XtWtWk5CQQJkyQbze9018fX2ZNnUKFouFmJgYwsKO07FTZ5o2acY330xn//59WCwWyleoyCuvvEb+/PkB6N3raSZMnEKxYsUAOH7sGMOHD+Gn2fP4bNxY4uLimDD+c5ycnKlduzbBlasw45uvrfVYLGaSkpKYOGkKU7+cwpEjoezfv4+lf/xBnz7P0ax5CyIvX75pDdcLP3GCSZMmcObMafLlz0/jxk2sxzKrqe8b/Xjzjdd59dXXqVa9OgCjPhrJyZMnMZvTcXV1pUPHTnTt2i3T60VGRjL645FUqFCJZ/s8B8AvS35myZKfiY+/Qv78BejVqzf16jewnmMypTH1y8ls2bIZk8lEo0ZNeP5/L2A0Gm/o32KxMG/eHH7/7TcsFjPBwZVJSf33Q0FbtmwiZMECxk+YBEBYWBhfTpnE6dOn8PDwoFnzFjz55NPWvtev+4u58+YQefkyXl5edH2sG48+2gGAw4cPMWb0xwCkpZkoWdKfV1/rS4kSJe667+uZTCabX1YsFnOm7XKa0NBQevTowbBhw6hZs6Z1/9SpU63fR0ZG0qJFC8aMGcPIkSPtUaaIiDyEHvgg+9OPs9i7dy/Dho+kaNFiHDp0iAMH9uPr6wvA1q1bGDBwMJUqBWM2m/ng/XcoW7Yc38yYibOzM999O4OJE75g+IiPbnmtQYOH8vxzz/DGm29RpUpV6/4mTZoCGQFt/PjPMZlMFCnix0ejRjNmzMeUDixNt+49gIwZwtGjP8p2DSaTiTFjPqZBw4aM+eRTYmJiGPXRiFvWdL2PRo22fn/q5ElGjhxOQEAgVatWs2kXHh7O6I8/okPHTrRvnxHY1q9fR0jIfAYNGkrpMmXYunULX3w+Dt/CRQgICMjo89RJatWuzYxvvycyMpKPRg5j8aKF9Hj8iRtqWbfuL1YsX8aIkR9RokRJ1q5ZzdSpUyhdukymtf/44/dUqFCRT8aOIzU1hVWrVnLx4gWKFSvO7t27+Oab6XzQfwDly1fg9OnTjBg+hMKFi1CjRk0qVqzEDz/OBjJmVRctCmH8F5/x2efjMRgMd9X39RYtDGH+/LnWbXd3N8aOzdlLJA4cOEDHjh1566236Nu3b5bt8ufPT6dOndi0adN9rE5ERB52D/TSAovFwqpVK+nV6xmKFy+Bk5MTFStWpHnzFtY2devVJzi4MgaDgaNHjnD27Fl6PP4k6elppKam0KXrYxw4sJ+4uLi7rueHH77n8qVL9Ov3dpZrYm+3hsOHD5GQkGCdJSxYsCCPPZb5TGp2mEwm8hfIT9169di9e5fNsT17djNyxFCee+55a4gFWPfXnzRv0ZKgsmVxcnKiXr36VK5chfXr/7K28fb24fHHn8TFxQVfX186d3nM5vi11q9fR8tWrSlVKgBnZ2datGxFqVIBWdbs7eVNSkoyqampeHh40qFDJ4oVKw7AiuXLaNK0GSVLliQxMYH8+fPRqHETNm3amMnYU2nbth1nz54hOjrqP+0boOtj3fjhxznWr+lff5vlmHKCHTt20K5dOz788MMbQmxUVNQNy1L+/vtv/Pz87meJIiLykHugZ2QTEhJITk7Gt7Bvlm3y5ctn/f7S5UukpaXx+muv2LTx9PQiKiqKXLly3XEtS5f+zvbt2/j4409u+izU260hOiqKPHnz4OLy70tRoGDB265v8aKF/PHHb8TFxeHq6kZamomaNWvZtNmwfj2FCxeharXqtjVER1Oteg2bfQULFiQqKsq6nTdvXpunJhQoUIDo6OhMa4mJjqJggbo2+242phdfeoVffvmZkSOGARZq1qpNx46dcXFx4fLly+zZs5s1q1fbnFOxYkUAIiIi+Gralxw8eAAnJyecnV1IS0sjKjKKfPny31Xf1zMajTZLKcxmMzGZ3wKHt2/fPjp06ECTJk0oUKAAixcvBqBEiRLUqFGDAwcOMHToULp06UK+fPlYsWIFf/75J6uvu5ciIiL30gMdZL29vfH29ubkyZPWWbSb8fX1xdXVlW+/+z7TtZsArq6upF7zEPeo6Cib405OTnDdZ4K2bt3CooUhfDRqDD4+Pnddw7UKFipEVGQkKSkpuLm5AXD27Nlb1nSto0ePEBIyn49GjbbOfH474xsiIy/btHvp5VdYsXwZoz4awYcDBuLh4ZlRQ8GCnLvummfOnKFcufLW7cuXL5GammoN8efOnaVAgczDaYGChTh3/pzNvvPnzlrXrV7P29ubp57qCWQ8l3fE8CG4uBjp2LETvr6+VK1ajad79sr03FmzZpI7d26+mzkLDw9PUlNT6fn0E9bZwrvp+2EWExNDs2bNAAgJCbHub9CgATVq1KBRo0ZMnjyZuXPncvDgQSpUqMCYMWM0IysiIvfVAx1kAdq378is77/Dw8ODkiVKsv/Afq5ciaNDh043tC1TJojA0qX54ovP6NH9cXxy5SLs+DGWLVvK4CHDAChdugy//vYLPZ/uRWRkJHPnzLbpI1++/ISGHiYgMBBXV1dOhofz5ZRJvPPu+3h7e5GQEA+Ah4dnpssLslPDtYKCylKwYCGmT5/G448/weXLlwlZMP+mNV0NvFdZzBbMZjPx8fFER0dz6NDfrF698ob1sUajkXff+4DJkyYwfNgQBg0ehre3N61ateHTT8dQvkIFypYty6aNGzl69Aivvvbv28lJSUl8/fVXdO/eg4iICBYtXEjnLl0yfc1atmjF+PGfUa5ceQICAli1aiVnzpyhTqat4ZMxH9OocROCygRx5coVEhP/ffJCx46dGTFiKH5Fi1I5uDIJCQls2boZby9v2j3aHovZTHJyMvHxCURHRzNv3lybJyXcTd8Ps0aNGt3y+bAVK1bUB7tERMSuHvgg+1i37nh5ezN3zmxiY2MIDq7Ms32eB8DNzR03139DncFgoH//gSwMWcDEieNJSIgnICCQp3v2trZ56eVX+ebrr+jf/32KFStG+/YdmDfv3w/wPN2zF7O+n8kff/xGjRo1KVCgIGazhU/HfmJT1ydjx+Hn54e7m5vNzGt2avD08sTZJeNtemdnZwYOGsKMb6YzaOCHFC5ShMe6dWfunJ+yrKnvG/1sagkqW5Zu3XowbeoUkpNTKFeuHC1btubKlX/X5Hp6euHs7IyzszN93+jHN19PZ8yYUfTvP5Bq1avz2utv8OsvS5gZGYmfnx9Dh42gSJEiABhdXKhYsRK+hXwZ9dEIzGYzrdu0oW3bRzN9zWrUrMkzz/RhzuyfSDWlUqN6TRo1amydzXVxMeLp+e+fMX3q6Z7MmzeX7779Bjd3d+rUrUu7do9axzZo0FBCFi5g7pzZeHl5UbdePZq3aAlAr97P8vX0afT/4F28vb1p2/ZRDh86hJOz0133LSIiIg82g+VmD9eUB9bzzz1Dv7feITi4sr1LeWiZzWbOnztF5+lJJJpu3f5BtuI1D4r4lbinf9jjXrr6WjjyGCBnjCMnjAFyxjhywhgg54xD7o0HfkZWbKWlpRF2/DhXrlyxzpiKiIiIPIwUZB3Mjh3bmfndtzzx5FNZfthKRERE5GGgIOtg6tatR9269exdhoiIiIjdabGJiIiIiDgkBVkRERERcUgKsiIiIiLikLRGVuQubR5SxuZPDDuaq4+2ERERcTSakRURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCEpyIqIiIiIQ1KQFRERERGHpCArIiIiIg5JQVZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEISnIioiIiIhDcrF3ASKOrt6IoySa7F3FzR0cXd7eJYiIiPznNCMrIiIiIg5JQVZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEIenxWw+ZY8eOcvjwYdq373Bfr7vurz/x8clFterV7+t1JWsHDhxg+vTphIaGkjdPbl559TUaN25sPf7MM88QFhYGQM2aNfniiy/sVaqIiEimNCP7kDl16hTr/lp736+7Y8d2Dh3+G4ATJ8L4ZcnP970G+dfu3bt5+eWXqVq1KgMHDiSwdCCdOnVizZo11jYffPABEyZMoGzZshw9etSO1YqIiGROM7Jy37m6uuLt42PvMh5q5cuXZ/369RgMBsxmM4EBJTh8+AiLFi2iWbNmAFSoUAGAQoUKceHCBXuWKyIikikFWbnvihYtRtGixexdxkPN3d3dZttisRAeHk6NGjXsVJGIiMjtU5C1k+ioKDZt3ogp1UTlKlU5c/qUdQ3puXPnWLliOQCenp4ElS1LlSpVrecePHiACxcu0Lx5C+u+P9euIU/evFStWg2AtLQ0tm7dwtmzZyhSxI8GDRri5PTvSpKIiIts376NpKQkateqQ4mSJa3H0tLS2LplM2fPniVfvnw0aNgQDw9P4N81tjVr1mLXzh1gMNCu3aM3jC8mJoZNGzeQmppKlapVbY6dOBHG/n376NipM5ARonbt2snxY8fw9vGhXt165M2XL1v13Ope3U3fD5MpX04jLi6OF1980d6liIiIZJvWyNrBpUuXePfdt9m/bx/JKclMmTyRH36YZV1DajQayZMnD3ny5CHVlMqXUyYxd+5s6/nHjh5l65bNNn1u376N0MOHAUhISOC9d99myc+LSU9PZ+eO7UyZPNHaNiIignGffkJcXBwXL1zg/fff4fTpUwAkJiby/nvvsHz5MgB27tzBO2+/RXx8PJCxxjZkwTzGfjKa2NhYfLy9bxhf5OXLvPtOP/bs2U1SchKTJk7g0KFD1uNnz55hzZrV1u25c2cz/atppKenE3HxIiNHDufy5UvZqudW9+pu+r6eyWQiMTHR+pWUlJhpO0czfvx45s6dx5IlSyhQoIC9yxEREck2zcjawc+LF1ImKIj+/QcA0KFDR155+d+ZsIIFC9KpcxfrdvPmLXir3xt06fIYbm5ut+w/JGQ+7u7ujPp4jHUW9sKF89bjqampDBw0lNy5cwMQGRXJls2bKV68BAsXLiBX7lwMGzbS2v6TMR/z669LePLJpwFISkpi4MDB5M8i9CxavJCAgEAGDBz8z/g68eorWc/0HTxwgO49HqdFi5YAxMfHY7FYAG5Zz63u1d30fcO4FoYwf/5c67a7uxtjx47NclyOYOTIkfz444/8MGsm5cqVs3c5IiIit0VB1g7CwsJo3KSpddvLy5ugsrYh4tixoxw4sJ8rcVcwm82YzWYiIi5SvHiJW/Z/JDSUOnXr2iwlKFy4iPV7Pz8/a4gF8C3kS2xsLAChhw9jMBj4fuZ31uPx8fGEnzhh3S5SxC/LEAsQdvw4jRs3sW57e3sTFFQ2y/YtW7Vm7tzZXIqIoFz58lSoUNEa2LNTz83u1d32fa2uj3WjQ8dO1m2LxUxM9OUsx/Wg69+/P3/88QfLli3D1WiwdzkiIiK3TUHWDq5+Uvxa5vR06/fLli1l3tw5NHzkEfLmyYuzszMASYlJ1vOvzipelZaW9m//Tjf2f61rA+5VFv7pz5Cx1jRPnjzWYzVr1rJ5y9nV1XjT8Tk5Od04vpvU07hxE2rWrMnfBw+ye9cupk2dwgcfDCAgMPCW9dzqXt1N39czGo0Yjf+O3Ww2ExN901vxwFq9ejWTJ0/G39+f3r17k5qagqurG3Xq1GHcuHFAxmztihUrOHv2LElJSTzyyCMEBATw/fff27l6ERGRDAqydhAUVJZtW7fQtm07DAYD0VFRhIYepnSZMgBs27qFTp0607lLVwCOHj1iE1Tz5s3L+fPnsVgsGAwGEhMTOXbsKAEBgQBUrFiJ9evX0b59R1xdXQE4fvwYgYGlb1lbpUrB/H3wIO07dLSGwoSEBCIiLmZ/fGXLsmXrZtq0bYeTk9MN47vegQP7qVQpmFq161Crdh0iIi6yecsmAgIDb1nPre7V3fSdk9WsWZN169YBGYH88qULFChY2CbU9+rVi0cftf0g3/VPOxAREbEnBVk76NylKwMHfMCADz+gZEl/Dv59gNy582AwZLy9W7VqNUJC5nP+wnlMqakcDztuDaQANWrWYtas7xkyeCDFihfnSGioTcDo0uUxDh48QL83+1K5chUuXDxPsaLFsxVku3R5jNDDh+nXry+VKlYiISGBsLAw/vdC9j/N3rlzVwYO6M+ADz/A378UB/8+YBOQrrd+/TpmfvctZcqU4cqVK+zfv4/HunXPVj23uld303dOljt3bqr/81fWzGYz58+doohfCZvZen9/f/z9/e1UoYiIyK0ZLNe/Ry33RWJiIrt37STVZKJixYpMnjyRWrVq06FDxhrMQ4f+5sSJE/h4e1OjZi3+/HOtzaOjrly5ws4d28FgIDi4MmFhx/Hx8aFcufJAxmOnDhzYz/lz5/ArWpRKlYKBjKcOnDgRZrOGde/ePRgMBipXrmLdd/jwIU6fPo2XlxfBwZXx+ecPGGR2fmaSkhLZtfPq+Cpx6tRJvL28KVe+PGfPniE0NJRmzZpb24eHh3Mk9DBu7u5UrVKV3NcF36zqyc69upu+b+ZqAOw8PYlEU7ZOsZuDo8tneSyrIOtIcsIYIGeMIyeMAXLGOHLCGCDnjEPuDQVZOzl18qT12a0XLpyn35t9GTFiFEFls/5QlDxYFGQfHDlhDJAzxpETxgA5Yxw5YQyQc8Yh94aWFtjJtGlTyJcvP+7u7mzfvp3GjZsoxIqIiIjcBgVZOxk+YhR79uwmOjqaVq3aKMSKiIiI3CYFWTsxGo3UqlXb3mWIiIiIOCwtNhERERERh6QgKyIiIiIOSUFWRERERByS1siK3KXNQ8rg4qIfJRERkftNM7IiIiIi4pAUZEVERETEISnIioiIiIhDUpAVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkBVkRERERcUgKsiIiIiLikBRkRURERMQhKciKiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSC72LkDE0dUbcZREk72rsHVwdHl7lyAiInLPaUZWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCHpqQX3WWpqKq6urg57vjiurVu3EhcXB0CePHmoVavWDW1iYmI4dOgQ3t7eVKpUCYPBcL/LFBERyTbNyN5H0dHRPPlEd2uYuF0JCfE8+UR3Ii9ftu5LTU29q/PtxWw2YzI9YM+syuEWLFjAlClTePvttxkwYMANx2fMmEH58uUZNGgQPXr0oGnTpkRGRtqhUhERkexRkL2Hbidk3omEhAR693rKIcPGli2bGfDhB/Yu46Eybtw4fv75Z7p163bDsfDwk7z77rvMmjWL1atXs2/fPry8vBg6dKgdKhUREckeBdl7JC4ujief6E50dPQ9u4aXlxdz54WQP3/+e3aNe6V+/QZ8Ou5ze5ch/9i//wA+Pj60bNkSAKPRSJcuXVi0aBEWi8XO1YmIiGROQfYfJpMJs9lss+/aGdVrj5vNZpu2mf1Dn5Zmsp6XmppKenr6DW1uFhCyGx6un/W92fXu1q36vraW69tcP57rlxakp6fbbN/uvbnV+ZmdYzabSU1NJTU19YbX/mGTJ08e4uPjiYmJse47c+YMcXFxXLx40X6FiYiI3IQ+7PWPV195idf7vkGVKlUBiLx8mRdffJ5ZP/yEl5c3/T94jypVq/L3wYOcPn2K115/g7x58/LdtzM4deokPj65aNmyFY91646zszPvvvMWAG/0fRWAbt160LxFxmzXhvXr+P33X7l8+TKBgYG82e8dfH19ATh9+hTfzviGI0dCcXExUr9BA5599jnc3NxuqDkhIZ7evZ5m+vQZ5C9QAIBnn+lJeno6FouFQoUK0bPXM9SpUzfb9+HUyZPMnPktR46E4ubmRqdOXejYqfMt+46OjuZ/zz9Lnz7Ps3Tp70RFRfHNjJns27uH+Qvmcf7cOXwLF+aFF16iUqVgNm3aQMiCBYyfMAmAv/76k99+/YXKlauwbt2fJCcnU7duPV57/Q2cnZ0BWLr0d35Z8jNRUVEULlKEnj17U6tWbev5v/66hKAyQWzbto2kpERatmxFs+YtmPrlFE6cCKNw4cK82e9tSpcuA8DBgwcY9dEIAAwGA2XKBPHSy69QtGixTO+NyWS6LiznnPBbp05tgoKCeOKJJ3jxxRc5efIkc+bMAdBaZhEReWApyN6G9evW8eGHAykVEEBMdDSvv/4Kjz/+JG3ajubM6dOMHj0Kdw8POnbsxPgJk+nzbC+mTvuavHnzAliXGezfv4+PR4/F1dWVL774jDmzf6TfW+8QHx/P0CGDaN+hI/0/HEhSUhLjv/iMObN/4tk+z2Wrxp9mzwMyZih3797FhPGfExAQSMGCBW957pUrcQwdOphmzZvzzrvvk5aWxs+LF5GQkICXl1e2+t69exejPv6EPHnysGfPbiZOHM+b/d6mRo2aXLx4gT//XEulSsGZXv/kyXAaN27CtK++ISoqisGDPmT9+nU0adKUTZs2MvunH/mg/wDKli3H5k0bGffpJ4z99DNKlvQHMkJ482Yt+N8LL3HyZDgDB/Rn3769vPX2exQrVowffvieb76ezphPPgUgOLgyc+eFAJCSkkLIgvmM/+LzLJc8LFoYwvz5c63b7u5ujB079pb31RG4uhpZtmwZU6dOJSQkBD8/P4YMGULfvn0pVKiQvcsTERHJlJYW3Ia2bdsREBiIwWBgy9bNFChYkI6dOuPq6kpAYCAdOnZi7drVt+zn6Z69yZ07Nx4eHjRq1IiTJ08CsGnTRnLnyUP79h0xGAx4enryWLfurF+/7o7qrVy5CkFBZdm/b2+22m/etAlvby969uyNl5cXuXPn5pln++Dl5ZXtvp/u2Zs8efIAsHzZUlq0aEnduvUwGo0UK1acnj17Z3n9ggUL0alzF1xdXSlcuDCVq1Tl5MlwAP78cy3Nm7ekUqVgjEYjjRo3oWLFSvz115/W8wsVKkT7Dh0xGo2ULl2G4sWL06RJM/z9/XFxcaFRo8bW/q7n7OxM18e6cerUSaKjojJt0/Wxbvzw4xzr1/Svv836ZjogFxcXBg4cyOzZsxk3bhzLly+ndevWmb4bICIi8iDQjOxtuPr2PUBUZBS+hXxtjvv6+hKVjScI5MqVy/q90ehKamoKAJcuRXDm9Gl693rKpr3BYCAtLe2W/aanp/PjD7NYv/4vYmNjcXZ2Jj09nfLlK9zyXIBLly9RuHCRTJ8dmt2+C1xzjy5fvkxwcOVsXRsgVy4fm21Xo9G67jYqMpKqVavZHPf19bV5YoOPj+35Li5Gm31GF6PNOt7IyEi+/vorDh7YT3JyMs7OzqSlpREZFUnefPluqM9oNGI0Gq3bZrOZmHv3Wb7/3N9//83Zs2cJDw8nJiaGlStX4unpSb169QB49dVXqVWrFn5+fvzxxx9s2rSJ1atv/YuZiIiIvSjI/sPNzc0m5ETfIqHkL1CALVs32+w7d/as9W12axi8jU98FyrkS6lSpRj7aeZvbaek3Pz8dev+Ytu2LYwYOYoiRfwwGAyMGD402x/88i3ky8YNGzCbzTg52U7W30nfhQoV4vSZ09m69q0UKFCA8+fP2ew7e+4sQUFl77jP77//DldXVyZPmUbu3LkxmUz0fPoJzOk5Z+3rtdauXcvKlSsBKFq0KFOmTKFIkSLWIDtp0iQmTJjA5s2bqVixIp988omWFYiIyANNSwv+ERAQwMqVK4iOjubs2TPMmjXzpu3r1atPbEwM8+bOITYmhv379/HLL0to0aIVAN7e3ri6unLk6BFSUlKyFSbr129ATEwMP/34A5cuXSI6OpqNGzcwdeqUbI3BZDJhsWR8Qj8+/gorli9j//592ToXoF79BqSmpvDN19OJiIjgwoXzTP1yMgkJCXfUd7tH27N2zWpWr15FbGwsR0JD+ebr6dmu51rNW7RkzepVbN+2ldjYWJYu/Z3Qw4dp2rTZHfUHkGYyYTGbwWLh0qVLfDXty2zNfDuq1157jZ9//tnma+rUqdbjuXPnZtiwYcydO5fBgwcrxIqIyANPM7L/6NX7WaZ+OZl+b/bF19eXNm3aciQ0FMiYWTUaXWxmKTP+0R/JrFkzWbbsD3LlykWXro/RqnUbIGNG9tk+z/HdtzOIjY3hsce606JlK4xGI9e+c+/k5ITLP29Xe3l58dGo0cz+6UcGDxpAenoa5StU5Mknn/6ntSHjrW1rB7bbTZs2IzT0MIMGZvzVpho1alCzZi3rp/5vPN+Wl5cXIz8azQ+zZjLgw/fx8PSkc+eueHl53bJvg8Fww9gqVqzE+x98SMiC+fz4w/f4+RXluef/98+4nW3epndycsLF5d9tAGdnF5ydM+55rVq1eeGFl5g7dw6RkZfx8/Nj0OCh1icMZHa+0WjEyTp2MDgZbK7Zs1dvpk39ktdffxVvH2/atmlHnjx5MDjpz7KKiIg4AoNFTzsXuSNms5nz507ReXoSiQ/YE6oOji6f7bZXx1HEr8QNS0ocRU4YA+SMceSEMUDOGEdOGAPknHHIvaH/IkRERETEISnIioiIiIhDUpAVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkPUdW5C5tHlIGFxf9KImIiNxvmpEVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkBVkRERERcUgKsiIiIiLikBRkRURERMQhKciKiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCEpyIqIiIiIQ3KxdwEijq7eiKMkmuxbw8HR5e1bgIiIiB1oRlZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEISnIioiIiIhD0uO3cojU1FRCQubTuXNXPD09M20TFhbGkSOhtGnT9j5XJ/bwzjvvsG/fPgACAwOZNm2a9dilS5d46qmnALBYLKSmpuDq6oabmxu///67XeoVERG5XQ99kM1OAHQEJlMqC0MW0LpVmyzHER5+gjWrV+aYIHvyZDgHDxyg3aPt7V3KA6lPnz7ExcUxb948tm3bZnMsd+7cDB8+HMgIspcvX2TixCnkypXLHqWKiIjckYc+yBoMBlyNrhgM9q7EcaSkpLBw4QK6dHkMDw8Pu9Xh5OSEi9Fot+s/6CpVqgTA5s2bbwiyrq6u1K9fHwCz2cyB/bvZvXs3CxYsuO91ioiI3KmHfo2s0WikW/ceeHg47mzs/ZaSksLCkAUkJyfbtY7ixUvQqlVru9aQUyxcuJiiRYvSrFkze5ciIiKSbTlqRjYpKYnFixfy6KMd2LtnN1HRUVSoUJGgoLI2xzt06MjevXs4d/Yc7R5tzy+//GxdWnC1Tbt27dm/by8RERGUr1CBChUqcvr0KXbu2IG7uzuPNGqMl5eX9dpz587GbDZjdDFStFgx6tSpi7Ozc5bX9fbxIaBUAOXK//unRU+fPsW2rVt5rFv3TMd3s2tkxmw2s3XLZi5evEipgIBM2+zevYvwEyfInScPtWrVwsfn37eWz507y65duzCbzVStUpUSJUsC8OuvSwD4efFC3NzdCQ6uTHBwZQCOhIYSeuQwRqMr1avXoFChQlneg85dumI2m9m2dQuXIy9T1K8oNWvVto7p6preWrVqs3PnDpKTkqhZqxZ+fkWBzJcW7N+/j2NHj+Lj40Ot2nXInTu3TV/VqlVnz55dJCUlUb9+AwoV8mX37l0cP34MP7+i1KtXH8NDNj1vsViYvyCEF1548aEbu4iIOLYcNSObnJzMwpAFDB40gP0H9nPp0iWGDxvC8uVLbY4PGTyI/fv24eTkRFpaGgtDFpCUmHhdm4EcOnSIy5GXGT5sCNO/msakieOJT4hnw4b1DB0yCLPZbL220cWIq9EVU5qJnxcvYtjQwVgsliyvmxAfz08//WBT/+JFCzl3/lyW47vZNTLzyZiPmT37R+KuxLFoYQiLFobYHB/36SdM/2oqcVfi2LxpI/3efIMzZ84AcPjwId55ux+nT58iJiaayZMnsHXL5ow6/nk738WYUY+TU0bw/HbGN4wb9wmRlyM5cSKM9959m0OH/s7yHsTFxfHO22+yYuVyEuITmDNnNkOHDiItLQ3IWNM7b+4cRn/8ERcunOd42HHeebsf586dBTKC/4oVy63jmT9/LhMnfEHclThOnAhj+LAhREZGWvuaO+cnxoweRcTFCA4cOMB7777NpInjWbb0D5KTk/l+5rf8+OOsLO+nyWQiMTHR+pWUlJhlW0eyatUqLly4SM+ePe1dioiIyG3JUTOyV7Vs1YoOHToBUDm4MpMnT6Rx46bW4+07dKR58xYAJCTEZ9rHk08+Rb36DYCMD4Tt2LGNSZOn4ubmRnJyMs8+05NTJ0/iX6oUgM0sao8eT9Dvzb7s2bObatWqZ3rd2NhYFi5cwJkzpylWrDhXrlxh8+ZNDB06IstxZecaV+3atZNDh/5mypdf4ePjg8ViYfjwISQmJACwe9cudu/exeQp08ibNy8An3/2KT/9OIsP+g/gwIH91KhRk1deeQ2A9PR0Ll68CECbNu2YN3cO7dt3tJ77998HWbt2DRMnTbHuK+Vfilnfz2T0mLGZ3oNvZ3xD3rz5GDnyYwwGA917PE7f119hzepVtGrdBoDk5CQGDhpi7XP4sCFs2riRbt173DDmfXv38vgTT9GiRUvrPb52hjE1NZWhw0aQJ08ezGYzL734POnp6Xw4YBAA5cqVZ+qXU+jV65lM7/+ihSHMnz/Xuu3u7sbYsWMzbetIvv32W1q1bEGBAgXsXYqIiMhtyZFBtlat2tbvq9eoSXJyMufOnbOGoSpVqtyyj3LlK1i/L1SwEAGBpXFzcwPA3d0dH59cRMdE409GkI2NiWHHzh1ERUZiSjPh5OTEmTOnbULmtdfNnTs3devWY/nyZTz//AusXbuGQoUK2Sw1uF52rnHVsWNHKVe+Aj4+PkDGh9pq1qjFunV/Wo+XCQqy3hOA2nXq8P3M7zK+r12XVStXMuOb6VQKrkyFChXx8/PLsrZDf/+Nl5cXS5f+++im2NhYwsNP2MwaX3sPjh07Sp06da1h093dncpVqnL02FFrkPXzK2pTY+HChYmJicm0hmbNmhMSsoCYmGjKl69AUFBZ6+wxQJEifuTJkwfI+KBY/vwFKF/B9nWOiYnOcoxdH+tGh46drNsWi5mY6MtZtncE586dY9myZcz87ht7lyIiInLbctTSgquuf7fdYrHYPJXA1dXtln1cv/bU2SmTW/XPdU6fPsXrr7/K7t27SDWlZjwFwclAYoLtW8/XX7dNm3b89edaUlJSWLlyOS1atsqynuxe4yoDhhtuhIWslyEAWMwWIONGlShRgomTplCjZi1OnAjj/fffYdXKFVmfiwVnF2dcja7Wr4IFCvLYY91tlmDc6t5bzGauXaXp7Hzjfc9qHM2at2D06E8oUsSPDevX8UbfVwk/ceKmfTk7Zb3G+HpGoxFPT0/r14P+AcEvv/ySli1bMmPGDMLCwmjZsiWPP/64TZvvvvuOwMBAateuZacqRURE7lyOnJHdtnULnTp3AWD79m14eHji51eUxMR7s6Zx9+7dBAQE8O677wMZn+pftWrlLc8rV748BQoU5KtpX3IpIoImTZpm2fZ2r1EmKIhfflnClStx+PjkwmKxsH37NpvjixcvJDoqirz58gGwectmgoKCAAg7fpziJUpQtWo1qlatRt68eVm2bCktWrayzkybTKnW/ipUqMivv/xC8xYtrTOoFouFo0eOZPmBtDJBQWzbtpUOHTvh5OREYmIie/fu4cknn77lvctMaOhhypYtR4MGDWnQoCFjxnzMxo0brMs/Hjbt2rWjatWqNvtcXV1ttjt16nRDuBUREXEUOTLI/vnnWsLDT+Dq6sqGDet5ts/zuLm53bMgW6FCBWb/9AMTJnxBLh8f9uzdc9MPYV2rdZs2TP9qGg0faWTzxIC7vUbVqtUIDg7mg/ffo0bNmpwIC+NK/BXc/gkyVatWo1btOvTv/x61a9fh9OnTnD17huEjRgEQfjKc8eM/p3yFChhdjGzcuIFH22c8HcDNzY1SpUoxbeqXBJYuTeXKVQgOrkzLVq14951+1KxVGxcXF0IPH6JqteoElS2baY1du3Zj0MD+DBzQn9JlyrBn9y6KFStO02bNs3Xvrrdi+TJmfvctZcoEcSX+Cvv27qHzP7/QPIz8/f3x9/e/aZtKlSphNps5f+7U/SlKRETkP5Qjg+ygQUM4ciSUqOgoRrZpR8A/j57y8PDgySefts4oQsZb3U8++TSeXp5ZtqkUXNn66KmrOnfpQpF/1oyWLl2GcZ+NZ+/e3QC8/35/zpw+jfc/wTSzPq+qXr0mgPUDSlm51TWuHwfAe+/3Z8eO7UREXKRevfp4enpx/Pgx6/G33nqH/fv3cfLkSYKCylK9Rg28vLyBjPWmlSoFs3fvHkwmEwMHDaZ06TLWc4cOG8nWLZuJjY21PrWgZ8/eNGrUhCOhh0lLT6d1qzbW+5bZPcidOzfjPhvPrp07iIyM5Nk+z1OtWnWc/lnGERAQSOs27WzuQ63adaxrakuW9Ld59FbfN/px9OgRQkNDKVGiBD179iZ//vxZ9tWqdRtKXTNbmztPnjueDRYREZH7z2DJ7tShA4iOjuZ/zz/LdzN/cJg/tfnLL0tYuXI5EydO0TM8HczVmczO05NINNm3loOjs/6Q4K1cHUcRvxLWXyIcTU4YA+SMceSEMUDOGEdOGAPknHHIvZEjZ2QdwcWLF1m27A9WrVzJiy++pBArIiIicptyVJC92Vv4DxqDwYC3lzf9+r1NjZo17V2OiIiIiMPJUUHW3d090wflP4gKFSqU5Z+iFREREZFb02ITEREREXFICrIiIiIi4pAUZEVERETEIeWoNbIi9rB5SBlcXPSjJCIicr9pRlZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEISnIioiIiIhDUpAVEREREYekICsiIiIiDklBVkREREQckoKsiIiIiDgkBVkRERERcUgKsiIiIiLikBRkRURERMQhKciKiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCEpyIqIiIiIQ1KQFRERERGHpCArIiIiIg5JQVZEREREHJKCrIiIiIg4JAVZEREREXFICrIiIiIi4pAUZEVERETEIbnYuwARR2WxWAAwm82YzWY7V3PnrtauMdhfThhHThgD5Ixx5IQxwL0Zh8FgwGAw/Gf9if0YLFf/NRaR25KUlERU5EV7lyEiIrepiF8JnJz0pnROoCArcofi4+N5o+8rTJz0JZ6eXvYu544lJSXy4gvPMf3rb/Hw8LR3OXckJ4wBcsY4csIYIGeMIyeMAe7NODQjm3NoaYHIHXJyciI2Ng4nJ2eH/s3eYHAiOTkFg8HJYceRE8YAOWMcOWEMkDPGkRPGADlnHHJv6L8IEREREXFICrIiIiIi4pAUZEXukNFopEePJzAajfYu5a7khHHkhDFAzhhHThgD5Ixx5IQxQM4Zh9wb+rCXiIiIiDgkzciKiIiIiENSkBURERERh6QgKyIiIiIOSc+RFbkDv/32K78sWUxcXBxlygTx0suvUKxYcXuXlSWTycSqVStYuWIFFy6cp2ChQnRo35EWLVtZ2zz7TE+uXLlic9706TPIX6DA/S43S8uXL2X6V9Ns9jVp2oy+fd+0bi9btpTFixYSGxtDYGBpXnzpZUqW9L/PlWbtwIH9DB0y6Ib9gYGlGfvpZwA8/9wzxMTE2ByfOm06hQr53o8SM7VmzWqmTJ5o3R732ReUKhVg02bFiuUsWhhCTEw0AQEBvPjiK/iXKpXt4/fDa6++xIULFwCoUaMmAwYOth5LT09nzZrVLF+2lHPnzpK/QAHatn2Udu0etbZ5+aUXuHQpwqbPCROnUKxYsfszAOCvv/5k4oQvrNufjB1H6dJlrNvXv1YADRo05O133rNur1q1koUh84mOjsbfvxQvvvgyAYGB9774a/Tt+yrnzp4FoGrVagweMsx67Nixo3zw/rs3nFO0aDEmTpoC2L6WV33xxURKlCx574qWB49FRG7Lpo0bLE892cOyZ/duS1xcrOWraVMtL7/0P0tqaqq9S8vSzp07LN98/ZUlPPyEJSkpybJp4wZLj+5dLdu2bbW2eab305b9+/fZscpbW7bsD8uADz/I8vj27dssTz7R3bJz5w5LXFycZcY3X1v+93wfS3Jy8n2s8vakp6dbXnzhecsvv/xs3fdcn96WPXt226+om4iKirJ07dLREhZ23Gb/rp07LU883s2yffs2S1xcnOW7b2dYnuvzjCUxMTFbx++36dOnWUZ9NMJm34ED+y1Tv5xiCQs7bklKSrJs377N8sTj3Szr1v1lbfPSi/+zbN++7X6Xm6nY2FhL1y4dLUePHrHZv3r1Kst7776V5Xl7du+2PPF4N8u2rVsscXFxllnfz7T0ebaXJSEh4V6XnKkZ33xtGTF86C3b9X39Vcv8eXOt26++8qJly5bN97AycQRaWiBym5avWEaTJk2pUrUqPj656P3Ms8TGxrJ79y57l5al6tVr8Pz/XqRkSX/c3d2pV78BZcoEcejQ3/Yu7T+1YvkyGjR8hOrVa+Dj40PPXr1JTk5ix/Zt9i4tSzt37iAuLpYmTZrZu5S7snzFMurXb0DNmrXw8fHh6Z69MJlS2b5ta7aOPwgqVqzEy6+8SqlSAbi7u1OzZi0qVqzEob9z2M/JimXUqVOXWrXr4OPjw1NP98RsNrN16xZ7l5al/fv3ceHCeZq3aGHvUuQBoyArcptOhocTeM3beO7u7hQtWozw8BN2rOr2xMfHc/JkOKX8bd/W/XTsGJ58ojv93uzLqlUr7VTdzYWFHeepJ3vwXJ/ejBs3loiIi9Zj4eEnKF26tHXb1dWV4sVLEB4ebodKs2fF8mXUq1cfHx8fm/2ff/ap9bVYsWK5narLvpPhJ2x+LoxGIyVL+hN+Mjxbxx9ESUlJHD9+/IblD5MmjufJJ7rz5huvsXTp73aq7uZOnTrF0089Tp9nezP2k9GcP3/eeiz8ZLjNUgRnZ2f8/Us90P8PW7F8GTVr1iJfvvw2+7+cMoknn+jOG31f4/fff7VTdWJPWiMrcpsSExPx8vS02efl5UVSYpKdKro9JpOJzz/7lDJlgmjQ8BHr/pnf/whAUlIiu3buZPLkibi4uNCkSVN7lXqD1q3b0rp1WywWCxcuXOD7md8ycsQwPvt8Aq6uriQlJeHp6WVzjpeXF4lJifYp+BYiIiLYs2c3Iz/62Gb/jG+/BzKC1O7du5g8aQIuzs40a/7gzkYlJibhmenPRWK2jj9o0tPTmTDhC/z8/GjWrLl1/7SvvgYgOTmZvXt2M2nSBAwGJ9q0aWuvUm/QrFlzmjVrjsViISIiglnff8eIEUP54ouJuLu7k5SYeMNr4fkAvxaxMTFs27aVDwfYri2f8uVXAKSkpLB37x4mTRyPAQPtHm1vjzLFTjQjK3KbPD09Sbjuf/gJCQl4eHrYqaLsS0lJ4ZNPRpOens77H3yIk9ON/wvw8PCkQcNHaN68JRvWr7NDlbdmMBgoUqQIfd94k/PnzxN2/DgAHh4eJCYm2LRNSEjA08Mzs27sbuXK5RQrXpxy5cpnetzDw4P69RvQsmUr1j+gr8VVnp4eJGb6c+GZreMPEpPJxGeffUpcbAwDBg7GxeXGOR93d3fq1K1Hm7btHuifE19fX/q+0Y/Lly5x5EgoAB6enje8FokP6GsBsHr1KvLnL0CVKlUzPe7m5kbt2nVo1679A/9zIv89BVmR21TS35/jx45at5OTkzl79gz+/vf309e3KykpiVGjRgAwcNAQ3N3d7VzRf8/fvxTHjh2zbptMJk6fPoW/v7/9ispCWloaa1avplWr1vYu5T9R0r+Uzc+FyWTi1KmT+P/zxIhbHX9QpKSk8MmYj4mPv8LgIcPx8vK69UkPOIPBYLPtX9KfY9e8Funp6YSHn3gg/x9mNptZuWoFLVu2umEcIqAgK3LbWrdqw59/rmXv3j1cuXKFWd/PJFeu3FSrVt3epWUpPj6e4cOG4OPtwwcffIirq6vN8Z07drAwZAEREREkJSWxadNGVq9eSf0GDe1UceamTp3C/v37SEpK4sKFC0yeNBHfwoUpFZDxGKhWrduwccN6du3aSXx8PD/M+j7jQzu1atu58htt27aV5OQkGje2Xbqxe/cuFsyfR0TERZKSkti8eROrVq2kwQP2WlyvVavWbNq0kZ07dhAfH8/sn37ExcVIrdp1snX8QZCUlMioj0ZgMBgYNGgoHh6277Ls37+PuXNnc+HCBZKTk9m2bSvLly194H5Opn81jb1795CUlEhExEUmT5pA/vwFKFMmCIBWrdqwdesWtm/ban0tnJycqFOnrp0rv9GePbuJioy8YVnNwYMHmT37Ry5cOE9ycjLbt2/jjz9+f+B/TuS/Z7BYLBZ7FyHiaH779ReWLPmZuLhYygQF8dJLr1C8eAl7l5WlzJ4rCdCoUWPe7Pc2qamp/LLkZ9asWU1sbAy+voVp2+5RWl7znNkHQVhYGLN/+oHDhw/h7u5OhYqVeOqpnhQuXNjaZtnSP1i8eCExMVefI/vKAzkjO2zYYHwL+fLKq6/b7DeZTPz6yxJWr15JTEzGa9GmTVtatW5jp0oznDgRxrvvvHXD/kGDhlKtesYvcSuWL2Oh9Tmxgbz40ss2z5q91fH74dOxY9iyZbPNvsDAQMZ++jmbN21k3LixN5xTq3Yd+vcfgMlk4vfffmXVqpVERUXi61uYlq1a2zxn9n44dfIkb731xg37PxwwiJo1axEeHs7sn37g0KG/cXV1o3z58jz1dE/8/Ipa265auYKQkAVER0dRqlSAXZ4j+/lnn7Jx4wabff7+pfjs8/HW7TFjPsbV1ZW337Z9pmxaWhp//P4bK1cuJzIykkKFfGnZqhXt2rXXzO1DRkFWRERERBySlhaIiIiIiENSkBURERERh6QgKyIiIiIOSUFWRERERBySgqyIiIiIOCQFWRERERFxSAqyIiIiIuKQFGRFRERExCEpyIqIiIiIQ1KQFRERERGHpCArIiIiIg5JQVZEREREHNL/ARK3Yq+8rE8yAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from collections import Counter\n", + "\n", + "from src.data.multimodal import clean_kegg_answer\n", + "\n", + "answers = Counter(clean_kegg_answer(a) for a in DS[\"kegg\"][\"answer\"])\n", + "print(f\"{len(answers)} distinct KEGG answers; top 15 cover {sum(c for _, c in answers.most_common(15)) / len(DS['kegg']):.0%}\")\n", + "H.plot_counts(dict(answers.most_common(15)), \"KEGG train: most frequent answers (after clean_kegg_answer)\");" + ] + }, + { + "cell_type": "markdown", + "id": "cell-017", + "metadata": {}, + "source": [ + "## 5. `datasets_config.json`: where datasets are registered\n", + "\n", + "Every dataset PRISM can stream is an entry in `src/data/datasets_config.json`. Here is the KEGG entry:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "cell-018", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/data/datasets_config.json:394-415`\n", + "```json\n", + " 394 \"dna_bioreason\": {\n", + " 395 \"name\": \"DNA sequences\",\n", + " 396 \"hf_id\": \"wanglab/kegg\",\n", + " 397 \"split\": \"train\",\n", + " 398 \"weight\": 0.50,\n", + " 399 \"handler\": \"dna_bioreason\",\n", + " 400 \"modality\": \"dna\",\n", + " 401 \"local_path\": \"${PRISM_DATA_ROOT}/dna_local_shards/kegg\",\n", + " 402 \"preferred_source\": \"local\",\n", + " 403 \"required_keys\": [\n", + " 404 \"question\",\n", + " 405 \"answer\",\n", + " 406 \"reasoning\",\n", + " 407 \"reference_sequence\",\n", + " 408 \"variant_sequence\"\n", + " 409 ],\n", + " 410 \"fallback_dummy\": false,\n", + " 411 \"skip\": false,\n", + " 412 \"size_formatted\": \"1,159 samples (train split)\"\n", + " 413 },\n", + " 414 \"dna_bioreason_variant_effect_coding\": {\n", + " 415 \"name\": \"Variant Effect (Coding)\",\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/data/datasets_config.json\", pattern=r'\"dna_bioreason\": \\{', context=22)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-019", + "metadata": {}, + "source": [ + "| Field | Meaning |\n", + "|---|---|\n", + "| `handler` | Picks the `_process_` method in `src/data/multimodal.py` that turns a raw row into a chat prompt + DNA (section 6). |\n", + "| `modality` | The modality group it is sampled under (`dna`). Sampling first picks a modality, then a dataset within it by `weight`. |\n", + "| `local_path` / `preferred_source` | `local` loads the Arrow files from section 3. `${PRISM_DATA_ROOT}` is expanded from the environment. |\n", + "| `hf_id` | Hub fallback, and the id that `training.bioreason_dataset` refers to. |\n", + "| `required_keys` | Columns that must exist. A dataset missing one fails loudly at load time. |\n", + "| `skip` | `true` excludes the dataset. The shipped JSON enables only `dna_bioreason` (KEGG); every other dataset is `skip: true`. |\n", + "\n", + "**KEGG only.** Because `dna_bioreason` is the only entry with `skip: false`, all three BioReason presets (`training=bioreason_projector|sft|grpo`) train on KEGG without any data override. The cell below lists the zone-A datasets that are switched on:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-020", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "zone_a = json.load(open(REPO / \"src/data/datasets_config.json\"))[\"zone_a\"][\"datasets\"]\n", + "enabled = {k: v[\"hf_id\"] for k, v in zone_a.items() if not v.get(\"skip\")}\n", + "print(\"enabled zone-A datasets:\", enabled)\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell-021", + "metadata": {}, + "source": [ + "### Adding data: a KEGG + variant-effect mix\n", + "\n", + "The original BioReason recipe mixes KEGG with the variant-effect sets (weights 0.5 / 0.25 / 0.25). The variant-effect entries already exist in the JSON, so a **new data YAML** that turns them on and points them at our downloads is all it takes. Passing `data=bioreason_demo_mix` on the command line applies it on top of the KEGG-only JSON.\n", + "\n", + "`${oc.env:PRISM_DATA_ROOT}` is resolved when the job starts." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "cell-022", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# Written by applications/bioreason/bioreason_demo.ipynb (section 5).\n", + "# KEGG + variant-effect mix (BioReason weights). Use with: data=bioreason_demo_mix\n", + "dataset_overrides:\n", + " dna_bioreason:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.5\n", + " dna_bioreason_variant_effect_coding:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.25\n", + " local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_coding\n", + " dna_bioreason_variant_effect_non_snv:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.25\n", + " local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_non_snv\n", + " ts_qa:\n", + " skip: true\n", + "\n" + ] + } + ], + "source": [ + "MIX_YAML = REPO / \"src/conf/data/bioreason_demo_mix.yaml\"\n", + "MIX_YAML.write_text('''\\\n", + "# Written by applications/bioreason/bioreason_demo.ipynb (section 5).\n", + "# KEGG + variant-effect mix (BioReason weights). Use with: data=bioreason_demo_mix\n", + "dataset_overrides:\n", + " dna_bioreason:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.5\n", + " dna_bioreason_variant_effect_coding:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.25\n", + " local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_coding\n", + " dna_bioreason_variant_effect_non_snv:\n", + " skip: false\n", + " preferred_source: local\n", + " weight: 0.25\n", + " local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_non_snv\n", + "''')\n", + "print(MIX_YAML.read_text())" + ] + }, + { + "cell_type": "markdown", + "id": "cell-023", + "metadata": {}, + "source": [ + "Validate the new mix the way the loader will: resolve the overrides, check each `local_path` exists, and check its Arrow columns against `required_keys`." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "cell-024", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W1011 23:25:11.983000 1232379 site-packages/torch/utils/_pytree.py:630] is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.\n", + "W1011 23:25:12.007000 1232379 site-packages/torch/utils/_pytree.py:630] is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
weightlocal_patharrow foundmissing required_keys
dataset
dna_bioreason0.50${PRISM_DATA_ROOT}/dna_l...True[]
dna_bioreason_variant_effect_coding0.25${PRISM_DATA_ROOT}/dna_l...True[]
dna_bioreason_variant_effect_non_snv0.25${PRISM_DATA_ROOT}/dna_l...True[]
\n", + "
" + ], + "text/plain": [ + " weight \\\n", + "dataset \n", + "dna_bioreason 0.50 \n", + "dna_bioreason_variant_effect_coding 0.25 \n", + "dna_bioreason_variant_effect_non_snv 0.25 \n", + "\n", + " local_path \\\n", + "dataset \n", + "dna_bioreason ${PRISM_DATA_ROOT}/dna_l... \n", + "dna_bioreason_variant_effect_coding ${PRISM_DATA_ROOT}/dna_l... \n", + "dna_bioreason_variant_effect_non_snv ${PRISM_DATA_ROOT}/dna_l... \n", + "\n", + " arrow found missing required_keys \n", + "dataset \n", + "dna_bioreason True [] \n", + "dna_bioreason_variant_effect_coding True [] \n", + "dna_bioreason_variant_effect_non_snv True [] " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import glob\n", + "\n", + "import pandas as pd\n", + "import src.train as T\n", + "from hydra import compose, initialize_config_dir\n", + "from hydra.core.global_hydra import GlobalHydra\n", + "from src.data.dataset_manager import DatasetManager\n", + "\n", + "\n", + "def compose_cfg(*overrides):\n", + " # Same composition as `python src/train.py `.\n", + " GlobalHydra.instance().clear()\n", + " with initialize_config_dir(config_dir=str(REPO / \"src/conf\"), version_base=None):\n", + " return compose(config_name=\"config\", overrides=list(overrides))\n", + "\n", + "cfg_mix = compose_cfg(\"model=prism_olmo_1b\", \"training=bioreason_sft\",\n", + " \"training.resume_weights_only=null\", \"data=bioreason_demo_mix\")\n", + "overrides = T._resolve_dataset_overrides(cfg_mix)\n", + "entries = DatasetManager().get_zone_config(\"zone_a\")[\"datasets\"]\n", + "\n", + "rows = []\n", + "for name, ov in overrides.items():\n", + " e = {**entries[name], **ov}\n", + " if e.get(\"skip\"):\n", + " continue\n", + " path = os.path.expandvars(e[\"local_path\"])\n", + " cols = set(load_from_disk(path).column_names) if glob.glob(f\"{path}/*.arrow\") else set()\n", + " rows.append({\"dataset\": name, \"weight\": e[\"weight\"], \"local_path\": path,\n", + " \"arrow found\": bool(cols), \"missing required_keys\": sorted(set(e[\"required_keys\"]) - cols)})\n", + "display(pd.DataFrame(rows).set_index(\"dataset\"))" + ] + }, + { + "cell_type": "markdown", + "id": "cell-025", + "metadata": {}, + "source": [ + "
Registering a new dataset, rather than enabling an existing one\n", + "\n", + "Add an entry under `zone_a.datasets` in `src/data/datasets_config.json` with the fields above. Reuse an existing `handler` if the row schema matches (e.g. `dna_bioreason_variant_effect_coding` for `question/answer/reference_sequence/variant_sequence`). Otherwise, add a `_process_` method plus a branch in the handler dispatch (section 6). Then run `pytest tests/data/test_dna_bioreason_datasets_config.py`, which pins the DNA entries' schema.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "cell-026", + "metadata": {}, + "source": [ + "## 6. How the DNA modality is registered\n", + "\n", + "PRISM has no plugin registry. A modality is wired in through a handful of explicit touch points, and following DNA through them is the best map for adding your own.\n", + "\n", + "**① The enum.** Every modality is a member of `Modality`. `ModelConfig.modalities` (`[text, dna]` in `prism_olmo_1b.yaml`) selects which ones a model builds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-027", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/modalities.py:23-36`\n", + "```python\n", + " 23 class Modality(str, Enum):\n", + " 24 \"\"\"The modality identifiers PRISM recognizes.\n", + " 25 \n", + " 26 Members are the canonical lowercase strings used as keys throughout the\n", + " 27 framework: ``config.modalities``, the ``encoders``/``projectors``\n", + " 28 ModuleDicts on ``UnifiedTransformer``, batch example dicts, and the\n", + " 29 ``modality`` field in ``datasets_config.json``.\n", + " 30 \n", + " 31 Because the enum subclasses ``str``, a member compares equal to and\n", + " 32 hashes like its value, so ``Modality.IMAGE == \"image\"`` is True and\n", + " 33 ``d[Modality.IMAGE]`` and ``d[\"image\"]`` reach the same entry. The\n", + " 34 explicit ``__str__`` below -- not the ``str`` base -- is what makes\n", + " 35 ``str(Modality.IMAGE)`` render as ``\"image\"`` instead of\n", + " 36 ``\"Modality.IMAGE\"``; removing it would change f-string and log output.\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/modalities.py\", pattern=r\"^class Modality\", context=30)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-028", + "metadata": {}, + "source": [ + "**② The encoder** subclasses `ModalityEncoder`. `DNAEncoder` loads any HF masked-LM checkpoint, probes its hidden size, and adds a `Linear` when that differs from `d_dna`. `tokens_per_instance()` is the fixed number of embeddings one DNA span becomes." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "cell-029", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/encoders/dna.py:14-58`\n", + "```python\n", + " 14 class DNAEncoder(ModalityEncoder):\n", + " 15 \"\"\"\n", + " 16 DNA sequence encoder using Nucleotide Transformer (NT) or Evo2.\n", + " 17 \n", + " 18 Supports:\n", + " 19 - InstaDeepAI/nucleotide-transformer-v2-500m-multi-species (default)\n", + " 20 - togethercomputer/evo2_1b_base (set dna_is_evo2=True)\n", + " 21 \n", + " 22 Input: Tokenized DNA sequences (B, T) as input_ids\n", + " 23 Output: Sequence features (B, T, d_dna)\n", + " 24 \"\"\"\n", + " 25 \n", + " 26 def __init__(\n", + " 27 self,\n", + " 28 d_dna: int = 1024,\n", + " 29 model_name: str = \"InstaDeepAI/nucleotide-transformer-v2-500m-multi-species\",\n", + " 30 dna_is_evo2: bool = False,\n", + " 31 max_dna_length: int | None = None,\n", + " 32 ):\n", + " 33 super().__init__(d_dna)\n", + " 34 self.model_name = model_name\n", + " 35 self._max_dna_length_override = max_dna_length\n", + " 36 \n", + " 37 logger.info(f\"Loading DNA Encoder: {model_name}...\")\n", + " 38 self.model = AutoModelForMaskedLM.from_pretrained(model_name, trust_remote_code=True)\n", + " 39 \n", + " 40 # Probe the actual hidden state dimension by requesting output_hidden_states.\n", + " 41 # AutoModelForMaskedLM.outputs[0] = LM-head logits (vocab_size ~4096), NOT hidden\n", + " 42 # states. Using hidden_states[-1] gives the last encoder layer's representation,\n", + " 43 # which is what downstream tasks want.\n", + " 44 with torch.no_grad():\n", + " 45 dummy_ids = torch.zeros(1, 6, dtype=torch.long)\n", + " 46 try:\n", + " 47 dummy_out = self.model(dummy_ids, output_hidden_states=True)\n", + " 48 if (\n", + " 49 hasattr(dummy_out, \"hidden_states\")\n", + " 50 and dummy_out.hidden_states is not None\n", + " 51 and len(dummy_out.hidden_states) > 0\n", + " 52 ):\n", + " 53 model_hidden_size = dummy_out.hidden_states[-1].shape[-1]\n", + " 54 logger.info(f\"DNA model hidden_states[-1] dim (probed): {model_hidden_size}\")\n", + " 55 elif hasattr(dummy_out, \"last_hidden_state\") and dummy_out.last_hidden_state is not None:\n", + " 56 model_hidden_size = dummy_out.last_hidden_state.shape[-1]\n", + " 57 logger.info(f\"DNA model last_hidden_state dim (probed): {model_hidden_size}\")\n", + " 58 else:\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/encoders/dna.py\", pattern=r\"class DNAEncoder\", context=45)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-030", + "metadata": {}, + "source": [ + "**③ Model wiring.** `UnifiedTransformer.__init__` builds the encoder and its projector when the modality is enabled:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "cell-031", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/model.py:590-623`\n", + "```python\n", + " 590 if Modality.DNA in config.modalities:\n", + " 591 dna_model_name = getattr(\n", + " 592 config,\n", + " 593 \"dna_model_name\",\n", + " 594 \"InstaDeepAI/nucleotide-transformer-2.5b-multi-species\",\n", + " 595 )\n", + " 596 dna_is_evo2 = getattr(config, \"dna_is_evo2\", False)\n", + " 597 \n", + " 598 self.encoders[\"dna\"] = DNAEncoder(\n", + " 599 d_dna=config.d_dna,\n", + " 600 model_name=dna_model_name,\n", + " 601 dna_is_evo2=dna_is_evo2,\n", + " 602 max_dna_length=getattr(config, \"max_dna_length\", None),\n", + " 603 )\n", + " 604 # Output dimension from the encoder (d_dna after its internal projection)\n", + " 605 dna_input_dim = getattr(self.encoders[\"dna\"], \"hidden_dim\", config.d_dna)\n", + " 606 \n", + " 607 self.projectors[\"dna\"] = ModalityProjector(\n", + " 608 dna_input_dim,\n", + " 609 target_dim,\n", + " 610 norm_mode=proj_norm_mode,\n", + " 611 target_norm=proj_target_norm,\n", + " 612 modality_embed_pos=proj_embed_pos,\n", + " 613 modality_embed_scale=proj_embed_scale,\n", + " 614 text_norm_mean=proj_text_norm_mean,\n", + " 615 text_norm_std=proj_text_norm_std,\n", + " 616 text_elem_mean=proj_text_elem_mean,\n", + " 617 text_elem_std=proj_text_elem_std,\n", + " 618 norm_clip_min=proj_norm_clip_min,\n", + " 619 norm_clip_max=proj_norm_clip_max,\n", + " 620 hidden_mult=proj_hidden_mult,\n", + " 621 num_layers=proj_num_layers,\n", + " 622 )\n", + " 623 # === Output decoders (Phase 1) ===\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/model.py\", pattern=r\"if Modality.DNA in config.modalities\", context=34)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-032", + "metadata": {}, + "source": [ + "**④ Data.** The `handler` string from `datasets_config.json` is dispatched to a processor:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "cell-033", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/data/multimodal.py:3530-3537`\n", + "```python\n", + " 3530 # DNA Handlers\n", + " 3531 elif handler_key == \"dna_bioreason\":\n", + " 3532 processor = self._process_dna_bioreason\n", + " 3533 elif handler_key == \"dna_bioreason_variant_effect_coding\":\n", + " 3534 processor = self._process_dna_bioreason_variant_effect_coding\n", + " 3535 elif handler_key == \"dna_bioreason_variant_effect_non_snv\":\n", + " 3536 processor = self._process_dna_bioreason_variant_effect_non_snv\n", + " 3537 \n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/markdown": [ + "`src/data/multimodal.py:2888-2947`\n", + "```python\n", + " 2888 def _process_dna_bioreason(self, item):\n", + " 2889 \"\"\"\n", + " 2890 Format a KEGG BioReason example into the structured chat dict consumed by\n", + " 2891 qwen_dna_collate_fn. Mirrors _format_kegg in BioReason/bioreason/dataset/kegg.py.\n", + " 2892 \n", + " 2893 model_name=\"llm\" — DNA sequences are inlined as plain text; dna_sequences is empty.\n", + " 2894 model_name=\"dna-llm\" — DNA sequences are passed as a separate modality; text is question-only.\n", + " 2895 \n", + " 2896 is_sft=True — assistant turn (reasoning + answer) is appended; collate fn masks prompt tokens.\n", + " 2897 is_sft=False — assistant turn is omitted; answer is kept for reward computation (RL/GRPO).\n", + " 2898 \n", + " 2899 Returns a dict:\n", + " 2900 {\n", + " 2901 \"prompt\": list[dict], # chat-format messages\n", + " 2902 \"dna_sequences\": list[str], # [\"ref\", \"var\"] or [\"\", \"\"] for llm mode\n", + " 2903 \"answer\": str, # ground-truth answer\n", + " 2904 }\n", + " 2905 \"\"\"\n", + " 2906 if self.model_name not in (\"llm\", \"dna-llm\"):\n", + " 2907 raise ValueError(f\"Unsupported model_name for DNA BioReason: {self.model_name!r}\")\n", + " 2908 \n", + " 2909 # --- fallback for None / corrupt items ---\n", + " 2910 if item is None:\n", + " 2911 question = \"What disease does this DNA sequence exhibit?\"\n", + " 2912 ref, var = (\"\", \"\") if self.model_name == \"llm\" else (\"CTGA\", \"CTGA\")\n", + " 2913 if self.model_name == \"llm\":\n", + " 2914 question_text = f\"Reference sequence: CTGA\\nVariant sequence: CTGA\\nQuestion: {question}\"\n", + " 2915 else:\n", + " 2916 question_text = question\n", + " 2917 \n", + " 2918 fallback_prompt = [\n", + " 2919 {\n", + " 2920 \"role\": \"user\",\n", + " 2921 \"content\": [\n", + " 2922 {\"type\": \"dna_reference\", \"text\": None},\n", + " 2923 {\"type\": \"dna_variant\", \"text\": None},\n", + " 2924 {\"type\": \"text\", \"text\": question_text},\n", + " 2925 ],\n", + " 2926 }\n", + " 2927 ]\n", + " 2928 if self.is_sft:\n", + " 2929 fallback_prompt.append({\n", + " 2930 \"role\": \"assistant\",\n", + " 2931 \"reasoning_content\": \"\",\n", + " 2932 \"content\": [{\"type\": \"text\", \"text\": \"No DNA sequence information available.\"}],\n", + " 2933 })\n", + " 2934 return {\n", + " 2935 \"prompt\": fallback_prompt,\n", + " 2936 \"dna_sequences\": [ref, var],\n", + " 2937 \"answer\": \"No DNA sequence information available.\",\n", + " 2938 \"class_weight\": torch.tensor(1.0, dtype=torch.float32),\n", + " 2939 }\n", + " 2940 \n", + " 2941 try:\n", + " 2942 ref = item[\"reference_sequence\"]\n", + " 2943 var = item[\"variant_sequence\"]\n", + " 2944 question = item.get(\"question\", \"What disease does this DNA sequence exhibit?\")\n", + " 2945 answer = clean_kegg_answer(item[\"answer\"])\n", + " 2946 reasoning = item.get(\"reasoning\", \"\")\n", + " 2947 \n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/data/multimodal.py\", pattern=r'# DNA Handlers', context=8)\n", + "H.show_source(\"src/data/multimodal.py\", pattern=r\"def _process_dna_bioreason\\(self, item\\)\", context=60)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-034", + "metadata": {}, + "source": [ + "**Checklist for a new modality**, using DNA as the template:\n", + "\n", + "| # | File | What to add |\n", + "|---|---|---|\n", + "| 1 | `src/modalities.py` | enum member + a `make_dummy_batch` branch |\n", + "| 2 | `src/encoders/.py` | `ModalityEncoder` subclass; `forward → (B, T, D)`; `tokens_per_instance()` |\n", + "| 3 | `src/config.py` | `d_`, `_model_name`, length fields on `ModelConfig` |\n", + "| 4 | `src/train.py` → `build_model_config` | pass the new `cfg.model.*` fields through (fields are explicit keyword arguments, so a new YAML key is ignored until it's wired here) |\n", + "| 5 | `src/model.py` | `if Modality.X in config.modalities:` encoder + `ModalityProjector` |\n", + "| 6 | `src/data/multimodal.py` | key in `modality_groups`; handler dispatch + `_process_`; an `elif modality is Modality.X` branch that renders tags and fills `example[\"\"]` |\n", + "| 7 | `src/data/datasets_config.json` | the dataset entry (section 5) |\n", + "| 8 | `src/conf/model/.yaml` | `modalities`, `d_`, `is_interleaved_qa: true`, `modality_start_end_token_indices` (section 8) |\n", + "\n", + "The full playbook is in `docs/skills/prism-adding-a-modality/SKILL.md`." + ] + }, + { + "cell_type": "markdown", + "id": "cell-035", + "metadata": {}, + "source": [ + "## 7. Pointing at the DNA encoder\n", + "\n", + "The encoder is a model-config field, `model.dna_model_name`, and it accepts a Hub id or a local path. `prism_olmo_1b.yaml` doesn't set it, so it falls back to `InstaDeepAI/nucleotide-transformer-2.5b-multi-species`. Here we point it **explicitly at the local snapshot** we downloaded. The leading `+` adds a key the YAML doesn't have." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "cell-036", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "backbone allenai/OLMo-1B-0724-hf\n", + "dna_model_name ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "d_dna 768 | max_dna_length 1024 | modalities ['text', 'dna']\n" + ] + } + ], + "source": [ + "DNA_ENCODER = SNAPSHOTS[\"dna\"]\n", + "BASE = [\"model=prism_olmo_1b\", f\"+model.dna_model_name={DNA_ENCODER}\"]\n", + "\n", + "cfg = compose_cfg(*BASE, \"training=bioreason_projector\")\n", + "mc = T.build_model_config(cfg) # exactly what src/train.py builds\n", + "print(\"backbone \", mc.llm_backbone_id)\n", + "print(\"dna_model_name \", mc.dna_model_name)\n", + "print(\"d_dna \", mc.d_dna, \"| max_dna_length\", mc.max_dna_length, \"| modalities\", [m.value for m in mc.modalities])" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "cell-037", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2026-10-11 23:25:27,060 - src.encoders.dna - INFO - Loading DNA Encoder: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e...\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "0e9b48f2bd8e424881358d35f908bcac", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/524 [00:00 768\n", + "2026-10-11 23:26:08,415 - src.encoders.dna - INFO - DNA Encoder loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e (model_hidden=2560, output=768, tokens_per_instance=1024)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NT hidden size 2560\n", + "added projection Linear(in_features=2560, out_features=768, bias=True)\n", + "tokens_per_instance 1024\n", + "4,001 bp -> input_ids (1, 1024) -> embeddings (1, 1024, 768)\n" + ] + } + ], + "source": [ + "from datasets import load_from_disk\n", + "from src.encoders.dna import DNAEncoder\n", + "from transformers import AutoTokenizer\n", + "\n", + "dev = torch.device(\"xpu:0\")\n", + "\n", + "# The DNA tokenizer comes from the SAME checkpoint as the encoder (what the data pipeline does).\n", + "dna_tok = AutoTokenizer.from_pretrained(mc.dna_model_name, trust_remote_code=True)\n", + "kegg = load_from_disk(str(SHARDS / \"kegg\"))\n", + "\n", + "enc = DNAEncoder(d_dna=mc.d_dna, model_name=mc.dna_model_name, max_dna_length=mc.max_dna_length).to(dev).eval()\n", + "seq = kegg[0][\"reference_sequence\"]\n", + "ids = dna_tok(seq, return_tensors=\"pt\", padding=\"max_length\", truncation=True, max_length=mc.max_dna_length)\n", + "with torch.no_grad():\n", + " out = enc({k: v.to(dev) for k, v in ids.items()})\n", + "print(\"NT hidden size \", enc.model.config.hidden_size)\n", + "print(\"added projection \", getattr(enc, \"proj\", None))\n", + "print(\"tokens_per_instance \", enc.tokens_per_instance())\n", + "print(f\"{len(seq):,} bp -> input_ids {tuple(ids['input_ids'].shape)} -> embeddings {tuple(out.shape)}\")\n", + "del enc, out\n", + "torch.xpu.empty_cache()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-038", + "metadata": {}, + "source": [ + "### Why the DNA tokenizer must come from the same checkpoint\n", + "\n", + "The data pipeline used to hard-code the **NT-v2-250m** tokenizer while the encoder was **NT-2.5B (v1)**. Both are \"6-mer\" vocabularies of the same size, but they **order the 6-mers differently**, so the encoder was silently fed the wrong k-mers. `StreamingMultimodalDataset` now loads the DNA tokenizer from `model.dna_model_name`. Here is the mismatch:" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "cell-039", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
NT-2.5B (v1) idNT-v2 id
AAAAAT57
ACGTAC726728
GGGCCC40784080
TTTTTT13691371
\n", + "
" + ], + "text/plain": [ + " NT-2.5B (v1) id NT-v2 id\n", + "AAAAAT 5 7\n", + "ACGTAC 726 728\n", + "GGGCCC 4078 4080\n", + "TTTTTT 1369 1371" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "6-mers with a different id: 4,096 of 4,097\n" + ] + } + ], + "source": [ + "import pandas as pd\n", + "from IPython.display import display\n", + "\n", + "\n", + "def nt_vocab(snapshot):\n", + " # Token order of an NT checkpoint, read straight from its vocab.txt (line i = id i).\n", + " return {tok: i for i, tok in enumerate(Path(snapshot, \"vocab.txt\").read_text().split())}\n", + "\n", + "v2_vocab = next(Path(HF_HOME, \"hub\").glob(\"models--InstaDeepAI--nucleotide-transformer-v2-250m-multi-species/snapshots/*/vocab.txt\"), None)\n", + "v2_path = v2_vocab.parent if v2_vocab else None\n", + "if v2_path:\n", + " v1v, v2v = nt_vocab(SNAPSHOTS[\"dna\"]), nt_vocab(v2_path)\n", + " kmers = [\"AAAAAT\", \"ACGTAC\", \"GGGCCC\", \"TTTTTT\"]\n", + " display(pd.DataFrame({\"NT-2.5B (v1) id\": [v1v[k] for k in kmers], \"NT-v2 id\": [v2v[k] for k in kmers]}, index=kmers))\n", + " print(f\"6-mers with a different id: {sum(v1v[k] != v2v.get(k) for k in v1v if len(k) == 6):,} of {sum(len(k) == 6 for k in v1v):,}\")\n", + "else:\n", + " print(\"NT-v2 not cached; skip. (Also: NT-v2's remote code doesn't import under transformers 5, so it can't be the encoder on frameworks/2026.1.0.)\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-040", + "metadata": {}, + "source": [ + "## 8. Special tokens\n", + "\n", + "DNA enters the text stream through **placeholder pairs**, declared in the model YAML:" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "cell-041", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/conf/model/prism_olmo_1b.yaml:65-68`\n", + "```yaml\n", + " 65 is_interleaved_qa: true\n", + " 66 modality_start_end_token_indices:\n", + " 67 dna_reference: [\"\", \"\"]\n", + " 68 dna_variant: [\"\", \"\"]\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tag raw OLMo tokenizer (Hub) tokenizers/prism-olmo-1b-interleaved\n", + " ['<', 'd', 'na', '_', 'ref', '_', 'start', '>'] [''] id=50292\n", + " ['<', 'd', 'na', '_', 'ref', '_', 'end', '>'] [''] id=50293\n", + " ['<', 'd', 'na', '_', 'var', '_', 'start', '>'] [''] id=50294\n", + " ['<', 'd', 'na', '_', 'var', '_', 'end', '>'] [''] id=50295\n" + ] + } + ], + "source": [ + "H.show_source(\"src/conf/model/prism_olmo_1b.yaml\", pattern=r\"^is_interleaved_qa\", context=4)\n", + "TAGS = [t for pair in mc.modality_start_end_token_indices.values() for t in pair]\n", + "llm_tok = AutoTokenizer.from_pretrained(\"tokenizers/prism-olmo-1b-interleaved\") # model YAML tokenizer_id\n", + "raw_tok = AutoTokenizer.from_pretrained(SNAPSHOTS[\"llm\"]) # the Hub OLMo tokenizer\n", + "print(f\"{'tag':16s} {'raw OLMo tokenizer (Hub)':42s} tokenizers/prism-olmo-1b-interleaved\")\n", + "for t in TAGS:\n", + " print(f\"{t:16s} {str(raw_tok.tokenize(t)):42s} {llm_tok.tokenize(t)} id={llm_tok.convert_tokens_to_ids(t)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-042", + "metadata": {}, + "source": [ + "The Hub tokenizer splits each tag into sub-words. PRISM's `tokenizers/prism-olmo-1b-interleaved` (the model YAML's `tokenizer_id`) **ships with the four tags pre-registered** (ids 50292–50295, inside OLMo's padded 50,304-row embedding table).\n", + "\n", + "`UnifiedTransformer.__init__` handles any tag still missing from the vocab: it calls `add_special_tokens`, resizes the embedding table (seeded so every rank initializes the new rows identically), and then **replaces the strings with integer ids in `config.modality_start_end_token_indices`, in place**. Here that path is a no-op, because the tags are already in the vocab." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "cell-043", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "`src/model.py:414-435`\n", + "```python\n", + " 414 {\"additional_special_tokens\": missing}\n", + " 415 )\n", + " 416 if num_added > 0:\n", + " 417 rng_state = torch.get_rng_state()\n", + " 418 try:\n", + " 419 torch.manual_seed(0)\n", + " 420 self.backbone.resize_token_embeddings(\n", + " 421 len(self.backbone_tokenizer)\n", + " 422 )\n", + " 423 finally:\n", + " 424 torch.set_rng_state(rng_state)\n", + " 425 logger.info(\n", + " 426 f\"Resized token embeddings to {len(self.backbone_tokenizer)} \"\n", + " 427 f\"after adding {num_added} modality special token(s)\"\n", + " 428 )\n", + " 429 \n", + " 430 resolved = dict(config.modality_start_end_token_indices)\n", + " 431 for modality, tags in unresolved.items():\n", + " 432 resolved[modality] = tuple(\n", + " 433 self.backbone_tokenizer.convert_tokens_to_ids(tok)\n", + " 434 if isinstance(tok, str)\n", + " 435 else tok\n", + "```" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.show_source(\"src/model.py\", pattern=r\"additional_special_tokens\", context=22)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-044", + "metadata": {}, + "source": [ + "Two consequences, both of which once broke training silently:\n", + "- `train.py` **adopts `model.backbone_tokenizer`** for the data pipeline. A tokenizer without the tags splits `` into sub-words, so the splice never matches.\n", + "- Because the config is rewritten in place, and `train.py` builds the dataset *after* the model, the prompt renderer used to see `(50292, 50293)` and write the literal text `Reference sequence: 5029250293`. No tag token ever reached the model, so DNA was dropped and every stage trained text-only, with no error. `_render_dna_prompt_text` now maps ids back to token strings. Section 10 asserts that the splice happens." + ] + }, + { + "cell_type": "markdown", + "id": "cell-045", + "metadata": {}, + "source": [ + "## 9. Model build: inspect, then modify\n", + "\n", + "### 9a. The stage-1 (projector) model as shipped" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "cell-046", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2026-10-11 23:26:16,416 - src.model - INFO - Initializing HF Backbone: allenai/OLMo-1B-0724-hf...\n", + "[transformers] `torch_dtype` is deprecated! Use `dtype` instead!\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "e74e598b5f0e4ad29d3a3356b80fe75c", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/114 [00:00 768\n", + "2026-10-11 23:26:25,227 - src.encoders.dna - INFO - DNA Encoder loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e (model_hidden=2560, output=768, tokens_per_instance=1024)\n", + "2026-10-11 23:26:25,266 - src.modules.projector - INFO - Modality embedding: scale=0.0055, expected_norm=0.2489\n", + "2026-10-11 23:26:25,266 - src.modules.projector - INFO - ModalityProjector initialized: norm_mode=l2_token, target_norm=0.25, embed_pos=after_norm\n", + "2026-10-11 23:26:25,267 - src.model - INFO - Freezing Modality Encoders...\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + ": UnifiedTransformer (3,835.3M)\n", + " backbone: OlmoForCausalLM (1,279.8M)\n", + " model: OlmoModel (1,176.8M)\n", + " lm_head: Linear (103.0M)\n", + " encoders: ModuleDict (2,549.8M)\n", + " dna: DNAEncoder (2,549.8M)\n", + " projectors: ModuleDict (5.8M)\n", + " dna: ModalityProjector (5.8M)\n", + " decoders: ModuleDict (0.0M)\n", + " text: LMHeadDecoder (0.0M)\n", + " text_decoder: LMHeadDecoder (0.0M)\n" + ] + } + ], + "source": [ + "from src.model import UnifiedTransformer\n", + "\n", + "base_vocab = len(llm_tok)\n", + "model = UnifiedTransformer(mc) # CPU for now\n", + "tok = model.backbone_tokenizer\n", + "\n", + "print(H.module_tree(model, max_depth=2))" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "cell-047", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "vocab 50296 -> 50296; embedding rows 50304\n", + "tag ids: {'dna_reference': (50292, 50293), 'dna_variant': (50294, 50295)}\n", + " id=50292 tokenize -> [''] |row norm| 0.000\n", + " id=50293 tokenize -> [''] |row norm| 0.000\n", + " id=50294 tokenize -> [''] |row norm| 0.000\n", + " id=50295 tokenize -> [''] |row norm| 0.000\n", + "mean |row norm| of original vocab: 0.199\n", + "(tag rows can be zero: the merge replaces each tag pair with DNA embeddings, so they're never looked up)\n" + ] + } + ], + "source": [ + "emb = model.backbone.get_input_embeddings().weight\n", + "print(f\"vocab {base_vocab} -> {len(tok)}; embedding rows {emb.shape[0]}\")\n", + "print(\"tag ids:\", mc.modality_start_end_token_indices)\n", + "for t in TAGS:\n", + " i = tok.convert_tokens_to_ids(t)\n", + " print(f\" {t:16s} id={i} tokenize -> {tok.tokenize(t)} |row norm| {emb[i].norm():.3f}\")\n", + "print(f\"mean |row norm| of original vocab: {emb[:50280].norm(dim=-1).mean():.3f}\")\n", + "print(\"(tag rows can be zero: the merge replaces each tag pair with DNA embeddings, so they're never looked up)\")" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "cell-048", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
paramstrainabletrainable %
component
LLM embeddings / lm_head20604518400.00
LLM backbone107374182400.00
DNA encoder (NT)254780122600.00
DNA encoder output Linear (2560->d_dna)196684800.00
DNA projector (MLP)57733135773313100.00
TOTAL383532839557733130.15
\n", + "
" + ], + "text/plain": [ + " params trainable trainable %\n", + "component \n", + "LLM embeddings / lm_head 206045184 0 0.00\n", + "LLM backbone 1073741824 0 0.00\n", + "DNA encoder (NT) 2547801226 0 0.00\n", + "DNA encoder output Linear (2560->d_dna) 1966848 0 0.00\n", + "DNA projector (MLP) 5773313 5773313 100.00\n", + "TOTAL 3835328395 5773313 0.15" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "P_BASE = H.param_table(model)\n", + "P_BASE" + ] + }, + { + "cell_type": "markdown", + "id": "cell-049", + "metadata": {}, + "source": [ + "### 9b. Modify the projector and add LoRA\n", + "\n", + "There's no `projector_type` switch. The DNA projector is always a `ModalityProjector` MLP (fc → GELU → fc → L2 token-norm → + modality embedding), and its **capacity** is set by two knobs: `projector_hidden_mult` (width) and `projector_num_layers` (depth). They're the `W2X`/`D2X` variants in `src/modules/projector.py:VARIANT_MAP`.\n", + "\n", + "LoRA is applied with torchtune's `apply_lora_torchtune(backbone, rank, alpha)`. That's what the trainer does when `training.lora_enabled=true`.\n", + "> `training.lora_target_modules` is **metadata only**: *every* `nn.Linear` in the backbone except `lm_head` gets an adapter." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "cell-050", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2026-10-11 23:26:25,548 - src.model - INFO - Initializing HF Backbone: allenai/OLMo-1B-0724-hf...\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "cb5554853f1f4a4aad5b3c0cd5756e9c", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/114 [00:00 768\n", + "2026-10-11 23:26:30,288 - src.encoders.dna - INFO - DNA Encoder loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e (model_hidden=2560, output=768, tokens_per_instance=1024)\n", + "2026-10-11 23:26:30,490 - src.modules.projector - INFO - Modality embedding: scale=0.0055, expected_norm=0.2489\n", + "2026-10-11 23:26:30,491 - src.modules.projector - INFO - ModalityProjector initialized: norm_mode=l2_token, target_norm=0.25, embed_pos=after_norm\n", + "2026-10-11 23:26:30,491 - src.model - INFO - Freezing Modality Encoders...\n", + "2026-10-11 23:26:36,815 - src.utils.lora_utils - INFO - torchtune LoRA: replaced 112 linear layers | trainable 12,058,624 / 1,291,845,632 params (0.933%)\n" + ] + }, + { + "data": { + "text/html": [ + "
shipped: hm=1, nl=2, no LoRA\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 paramstrainabletrainable %
component   
LLM embeddings / lm_head206,045,18400.000000
LLM backbone1,073,741,82400.000000
DNA encoder (NT)2,547,801,22600.000000
DNA encoder output Linear (2560->d_dna)1,966,84800.000000
DNA projector (MLP)5,773,3135,773,313100.000000
TOTAL3,835,328,3955,773,3130.150000
\n", + "
modified: hm=2, nl=3, LoRA r=16\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
 paramstrainabletrainable %
component   
LLM embeddings / lm_head206,045,18400.000000
LLM backbone1,073,741,82400.000000
LoRA adapters12,058,62412,058,624100.000000
DNA encoder (NT)2,547,801,22600.000000
DNA encoder output Linear (2560->d_dna)1,966,84800.000000
DNA projector (MLP)28,323,84128,323,841100.000000
TOTAL3,869,937,54740,382,4651.040000
\n", + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "projector before: ModalityProjector(\n", + " (fc1): Linear(in_features=768, out_features=2048, bias=True)\n", + " (act): GELU(approximate='none')\n", + " (fc2): Linear(in_features=2048, out_features=2048, bias=True)\n", + ")\n", + "projector after: ModalityProjector(\n", + " (layers): ModuleList(\n", + " (0): Linear(in_features=768, out_features=4096, bias=True)\n", + " (1): GELU(approximate='none')\n", + " (2): Linear(in_features=4096, out_features=4096, bias=True)\n", + " (3): GELU(approximate='none')\n", + " (4): Linear(in_features=4096, out_features=2048, bias=True)\n", + " )\n", + ")\n" + ] + } + ], + "source": [ + "from src.utils.lora_utils import apply_lora_torchtune\n", + "\n", + "MOD = [\"model.projector_hidden_mult=2\", \"model.projector_num_layers=3\"]\n", + "LORA = dict(rank=16, alpha=32.0)\n", + "\n", + "cfg_mod = compose_cfg(*BASE, \"training=bioreason_projector\", *MOD)\n", + "mc_mod = T.build_model_config(cfg_mod)\n", + "model_mod = UnifiedTransformer(mc_mod)\n", + "apply_lora_torchtune(model_mod.backbone, **LORA)\n", + "\n", + "H.side_by_side(P_BASE, H.param_table(model_mod),\n", + " titles=[\"shipped: hm=1, nl=2, no LoRA\", f\"modified: hm=2, nl=3, LoRA r={LORA['rank']}\"])\n", + "print(\"projector before:\", model.projectors[\"dna\"])\n", + "print(\"projector after: \", model_mod.projectors[\"dna\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "cell-051", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
shippedmodified
key
projector_hidden_mult12
projector_num_layers23
training.lora_enabled / r / alphafalsetrue / 16 / 32
\n", + "
" + ], + "text/plain": [ + " shipped modified\n", + "key \n", + "projector_hidden_mult 1 2\n", + "projector_num_layers 2 3\n", + "training.lora_enabled / r / alpha false true / 16 / 32" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Config diff (what you would put on the train.py command line)\n", + "from omegaconf import OmegaConf\n", + "\n", + "a, b = OmegaConf.to_container(cfg.model), OmegaConf.to_container(cfg_mod.model)\n", + "display(pd.DataFrame([(k, a.get(k), b.get(k)) for k in sorted(set(a) | set(b)) if a.get(k) != b.get(k)]\n", + " + [(\"training.lora_enabled / r / alpha\", \"false\", f\"true / {LORA['rank']} / {LORA['alpha']:g}\")],\n", + " columns=[\"key\", \"shipped\", \"modified\"]).set_index(\"key\"))\n", + "del model_mod" + ] + }, + { + "cell_type": "markdown", + "id": "cell-052", + "metadata": {}, + "source": [ + "To train with these changes, add the same overrides to any stage: `model.projector_hidden_mult=2 model.projector_num_layers=3 training.lora_r=16 training.lora_alpha=32`. Note that a stage-1 checkpoint only loads into an SFT model whose projector has the **same shape**." + ] + }, + { + "cell_type": "markdown", + "id": "cell-053", + "metadata": {}, + "source": [ + "## 10. One batch end to end\n", + "\n", + "Build the **SFT** dataset with the builders `train.py` uses, take one example, and look at exactly what the model sees." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "cell-054", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[INFO] DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "[INFO] DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "2026-10-11 23:26:37,120 - src.data.multimodal - INFO - DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "[INFO] dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "[INFO] dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "2026-10-11 23:26:37,123 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "2026-10-11 23:26:37,124 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "2026-10-11 23:26:37,124 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "[INFO] dataset_overrides applied: ts_qa <- {'skip': True}\n", + "[INFO] dataset_overrides applied: ts_qa <- {'skip': True}\n", + "2026-10-11 23:26:37,125 - src.data.multimodal - INFO - dataset_overrides applied: ts_qa <- {'skip': True}\n", + "2026-10-11 23:26:37,225 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/google/tapas-base/resolve/main/tokenizer_config.json \"HTTP/1.1 307 Temporary Redirect\"\n", + "2026-10-11 23:26:37,236 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/google/tapas-base/00456266840bb0a319cd6748ebf7da3caf98816b/tokenizer_config.json \"HTTP/1.1 200 OK\"\n", + "2026-10-11 23:26:37,334 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/google/tapas-base/tree/main/additional_chat_templates?recursive=false&expand=false \"HTTP/1.1 404 Not Found\"\n", + "2026-10-11 23:26:37,388 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/google/tapas-base/tree/main?recursive=true&expand=false \"HTTP/1.1 200 OK\"\n", + "[INFO] Initializing Streaming Datasets for zone_a...\n", + "[INFO] Initializing Streaming Datasets for zone_a...\n", + "2026-10-11 23:26:37,476 - src.data.multimodal - INFO - Initializing Streaming Datasets for zone_a...\n", + "[WARNING] Skipping PixMo-Cap (Configured skip)\n", + "[WARNING] Skipping PixMo-Cap (Configured skip)\n", + "2026-10-11 23:26:37,477 - src.data.multimodal - WARNING - Skipping PixMo-Cap (Configured skip)\n", + "[WARNING] Skipping Molmo2-Cap (Configured skip)\n", + "[WARNING] Skipping Molmo2-Cap (Configured skip)\n", + "2026-10-11 23:26:37,478 - src.data.multimodal - WARNING - Skipping Molmo2-Cap (Configured skip)\n", + "[WARNING] Skipping PixMo-Points (Configured skip)\n", + "[WARNING] Skipping PixMo-Points (Configured skip)\n", + "2026-10-11 23:26:37,479 - src.data.multimodal - WARNING - Skipping PixMo-Points (Configured skip)\n", + "[WARNING] Skipping PixMo-Count (Configured skip)\n", + "[WARNING] Skipping PixMo-Count (Configured skip)\n", + "2026-10-11 23:26:37,479 - src.data.multimodal - WARNING - Skipping PixMo-Count (Configured skip)\n", + "[WARNING] Skipping Molmo2-VideoPoint (Configured skip)\n", + "[WARNING] Skipping Molmo2-VideoPoint (Configured skip)\n", + "2026-10-11 23:26:37,480 - src.data.multimodal - WARNING - Skipping Molmo2-VideoPoint (Configured skip)\n", + "[WARNING] Skipping The Well (Turbulence) (Configured skip)\n", + "[WARNING] Skipping The Well (Turbulence) (Configured skip)\n", + "2026-10-11 23:26:37,481 - src.data.multimodal - WARNING - Skipping The Well (Turbulence) (Configured skip)\n", + "[WARNING] Skipping The Well (MHD 64^3) (Configured skip)\n", + "[WARNING] Skipping The Well (MHD 64^3) (Configured skip)\n", + "2026-10-11 23:26:37,481 - src.data.multimodal - WARNING - Skipping The Well (MHD 64^3) (Configured skip)\n", + "[WARNING] Skipping The Well (Supernova) (Configured skip)\n", + "[WARNING] Skipping The Well (Supernova) (Configured skip)\n", + "2026-10-11 23:26:37,482 - src.data.multimodal - WARNING - Skipping The Well (Supernova) (Configured skip)\n", + "[WARNING] Skipping Berea Sandstone (Real) (Configured skip)\n", + "[WARNING] Skipping Berea Sandstone (Real) (Configured skip)\n", + "2026-10-11 23:26:37,482 - src.data.multimodal - WARNING - Skipping Berea Sandstone (Real) (Configured skip)\n", + "[WARNING] Skipping PDEBench (Problem Statements) (Configured skip)\n", + "[WARNING] Skipping PDEBench (Problem Statements) (Configured skip)\n", + "2026-10-11 23:26:37,483 - src.data.multimodal - WARNING - Skipping PDEBench (Problem Statements) (Configured skip)\n", + "[WARNING] Skipping Tulu 3 SFT (Configured skip)\n", + "[WARNING] Skipping Tulu 3 SFT (Configured skip)\n", + "2026-10-11 23:26:37,483 - src.data.multimodal - WARNING - Skipping Tulu 3 SFT (Configured skip)\n", + "[WARNING] Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "[WARNING] Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "2026-10-11 23:26:37,484 - src.data.multimodal - WARNING - Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "[WARNING] Skipping TS Reasoning (Proxy) (Configured skip)\n", + "[WARNING] Skipping TS Reasoning (Proxy) (Configured skip)\n", + "2026-10-11 23:26:37,485 - src.data.multimodal - WARNING - Skipping TS Reasoning (Proxy) (Configured skip)\n", + "[WARNING] Skipping ChatTime (Verified Available)\n", + "[WARNING] Skipping ChatTime (Verified Available)\n", + "2026-10-11 23:26:37,485 - src.data.multimodal - WARNING - Skipping ChatTime (Verified Available)\n", + "[WARNING] Skipping Time-MMD (Economy) (Configured skip)\n", + "[WARNING] Skipping Time-MMD (Economy) (Configured skip)\n", + "2026-10-11 23:26:37,486 - src.data.multimodal - WARNING - Skipping Time-MMD (Economy) (Configured skip)\n", + "[WARNING] Skipping TableLLM (Reasoning) (Configured skip)\n", + "[WARNING] Skipping TableLLM (Reasoning) (Configured skip)\n", + "2026-10-11 23:26:37,486 - src.data.multimodal - WARNING - Skipping TableLLM (Reasoning) (Configured skip)\n", + "[WARNING] Skipping TableInstruct (Gen) (Configured skip)\n", + "[WARNING] Skipping TableInstruct (Gen) (Configured skip)\n", + "2026-10-11 23:26:37,487 - src.data.multimodal - WARNING - Skipping TableInstruct (Gen) (Configured skip)\n", + "[WARNING] Skipping UniTabE (Structure) (Configured skip)\n", + "[WARNING] Skipping UniTabE (Structure) (Configured skip)\n", + "2026-10-11 23:26:37,488 - src.data.multimodal - WARNING - Skipping UniTabE (Structure) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20 (Global) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20 (Global) (Configured skip)\n", + "2026-10-11 23:26:37,488 - src.data.multimodal - WARNING - Skipping ChEBI-20 (Global) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "2026-10-11 23:26:37,489 - src.data.multimodal - WARNING - Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "[WARNING] Skipping GraphInstruct (Configured skip)\n", + "[WARNING] Skipping GraphInstruct (Configured skip)\n", + "2026-10-11 23:26:37,489 - src.data.multimodal - WARNING - Skipping GraphInstruct (Configured skip)\n", + "[WARNING] Skipping Matbench (Crystal) (Configured skip)\n", + "[WARNING] Skipping Matbench (Crystal) (Configured skip)\n", + "2026-10-11 23:26:37,490 - src.data.multimodal - WARNING - Skipping Matbench (Crystal) (Configured skip)\n", + "[WARNING] Skipping Verilog (Circuit) (Configured skip)\n", + "[WARNING] Skipping Verilog (Circuit) (Configured skip)\n", + "2026-10-11 23:26:37,490 - src.data.multimodal - WARNING - Skipping Verilog (Circuit) (Configured skip)\n", + "[INFO] [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "[INFO] [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "2026-10-11 23:26:37,491 - src.data.multimodal - INFO - [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "[INFO] [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n", + "[INFO] [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n", + "2026-10-11 23:26:37,497 - src.data.multimodal - INFO - [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "DEBUG: Loading 1 Arrow files directly...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[WARNING] Skipping Variant Effect (Coding) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Coding) (Configured skip)\n", + "2026-10-11 23:26:37,627 - src.data.multimodal - WARNING - Skipping Variant Effect (Coding) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "2026-10-11 23:26:37,627 - src.data.multimodal - WARNING - Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "[INFO] [Init] Pre-Calculated Active Modalities: []\n", + "[INFO] [Init] Pre-Calculated Active Modalities: []\n", + "2026-10-11 23:26:37,628 - src.data.multimodal - INFO - [Init] Pre-Calculated Active Modalities: []\n", + "2026-10-11 23:26:37,629 - src.train - INFO - Using MultimodalCollator with max_seq_length=None (is_interleaved_qa=True, passthrough_time_series=False)\n", + "[INFO] [streaming manifest] dna_bioreason: expected ~1,159 samples (train split) samples/epoch\n", + "[INFO] [streaming manifest] dna_bioreason: expected ~1,159 samples (train split) samples/epoch\n", + "2026-10-11 23:26:37,630 - src.data.multimodal - INFO - [streaming manifest] dna_bioreason: expected ~1,159 samples (train split) samples/epoch\n", + "[INFO] Active Modalities for Sampling: ['dna']\n", + "[INFO] Active Modalities for Sampling: ['dna']\n", + "2026-10-11 23:26:37,630 - src.data.multimodal - INFO - Active Modalities for Sampling: ['dna']\n", + "[INFO] Modality dna: ['dna_bioreason'] (Weights: [1.0])\n", + "[INFO] Modality dna: ['dna_bioreason'] (Weights: [1.0])\n", + "2026-10-11 23:26:37,631 - src.data.multimodal - INFO - Modality dna: ['dna_bioreason'] (Weights: [1.0])\n", + "[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer GPTNeoXTokenizer. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "_metadata 'P T' = '214 524': 214 prompt tokens (no loss) + 524 target tokens (loss)\n", + "DNA: ref (1024,), var (1024,)\n", + "merged length = 738 text - 4 tags + 2 x 1024 DNA slots = 2782\n" + ] + }, + { + "data": { + "text/html": [ + "
grey = prompt (label -100, no loss)   green = target (loss)   blue = DNA splice tag
<|user|>\n", + "Reference sequence: <dna_ref_start> [1024 DNA embeddings]<dna_ref_end>\n", + "Variant sequence: <dna_var_start> [1024 DNA embeddings]<dna_var_end>\n", + "Chromosome Number: 4\n", + "Network Definition of the pathway: KIT* -> PI3K -> PIP3 -> AKT -| BAD\n", + "Genes in the pathway: KIT; KIT proto-oncogene receptor tyrosine kinase | PIK3CA; phosphatidylinositol-4,5-bisphosphate 3-kinase catalytic subunit alpha | PIK3CB; phosphatidylinositol-4,5-bisphosphate 3-kinase catalytic subunit beta | PIK3CD; phosphatidylinositol-4,5-bisphosphate 3-kinase catalytic subunit delta | AKT1; AKT serine/threonine kinase 1 | AKT2; AKT serine/threonine kinase 2 | AKT3; AKT serine/threonine kinase 3 | BAD; BCL2 associated agonist of cell death\n", + "\n", + "Given this context, what is the biological effect of this KIT allele, specifically what disease does this contribute to?\n", + "<|assistant|>\n", + "<think>\n", + "Step 1: The variant is an A>T substitution at position 54733154 on chromosome 4, occurring in the KIT gene, which encodes a receptor tyrosine kinase (RTK).\n", + "Step 2: KIT is a transmembrane receptor that, when bound by its ligand stem cell factor (SCF), activates multiple downstream signaling pathways including PI3K/AKT.\n", + "Step 3: This mutation likely occurs in a critical functional domain of KIT, resulting in constitutive activation of the receptor even in the absence of its ligand.\n", + "Step 4: Constitutively active KIT leads to persistent activation of the PI3K enzyme complex, which consists of catalytic subunits (PIK3CA, PIK3CB, or PIK3CD) and regulatory subunits.\n", + "Step 5: Activated PI3K phosphorylates phosphatidylinositol-4,5-bisphosphate (PIP2) to generate phosphatidylinositol-3,4,5-trisphosphate (PIP3), a critical second messenger.\n", + "Step 6: PIP3 recruits AKT (AKT1, AKT2, or AKT3) to the plasma membrane where it becomes activated through phosphorylation.\n", + "Step 7: Activated AKT phosphorylates numerous downstream targets, including the pro-apoptotic protein BAD.\n", + "Step 8: Phosphorylation of BAD by AKT causes it to dissociate from anti-apoptotic BCL-2 family proteins and become sequestered by 14-3-3 proteins, preventing its pro-apoptotic function.\n", + "Step 9: Inhibition of BAD promotes cell survival by preventing apoptosis, while other AKT targets promote cell proliferation and metabolism.\n", + "Step 10: The hyperactivation of this KIT→PI3K→AKT pathway due to the KIT mutation leads to increased survival and proliferation of hematopoietic cells.\n", + "Step 11: This uncontrolled proliferation and enhanced survival of myeloid progenitor cells contributes to the development of Acute Myeloid Leukemia (AML).\n", + "Step 12: In the context of AML pathogenesis, this KIT mutation represents the 'first event' in leukemic transformation, characterized by disordered cell growth and upregulation of cell survival genes.\n", + "This KIT mutation likely results in constitutive activation of the receptor tyrosine kinase, leading to hyperactivation of the PI3K/AKT signaling pathway and inhibition of the pro-apoptotic BAD protein. This promotes increased cell survival and proliferation, contributing to the development of Acute Myeloid Leukemia (AML) through disordered cell growth and upregulation of survival genes.\n", + "</think>\n", + "\n", + "Answer: <answer>acute myeloid leukemia</answer><|endoftext|>
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "cfg_sft = compose_cfg(*BASE, \"training=bioreason_sft\", \"training.resume_weights_only=null\")\n", + "tc_sft = T.build_training_config(cfg_sft, mc, output_dir=\"/tmp\", seed=0)\n", + "ds_sft = T.build_streaming_dataset(cfg_sft, tok, mc, tc_sft,\n", + " T.build_bio_dataset_kwargs(\"bioreason_sft\", tc_sft, mc),\n", + " max_seq_length=2048)\n", + "collate = T.build_collator(tok, mc, use_bucketed_collator=False, collator_max_seq=2048)\n", + "ex = next(iter(ds_sft))\n", + "P, Tn = map(int, ex[\"_metadata\"].split())\n", + "special = {i: t for t in TAGS for i in [tok.convert_tokens_to_ids(t)]}\n", + "n_spans = sum(1 for t in ex[\"text\"].tolist() if t in special) // 2\n", + "assert n_spans == 2, \"no DNA tag tokens in the prompt - DNA would be silently dropped (see section 8)\"\n", + "\n", + "print(f\"_metadata 'P T' = '{ex['_metadata']}': {P} prompt tokens (no loss) + {Tn} target tokens (loss)\")\n", + "print(f\"DNA: ref {tuple(ex['dna']['dna_reference']['input_ids'].shape)}, var {tuple(ex['dna']['dna_variant']['input_ids'].shape)}\")\n", + "print(f\"merged length = {len(ex['text'])} text - {2 * n_spans} tags + {n_spans} x {mc.max_dna_length} DNA slots \"\n", + " f\"= {len(ex['text']) - 2 * n_spans + n_spans * mc.max_dna_length}\")\n", + "display(HTML(H.token_view(tok, ex[\"text\"].tolist(), P, special, mc.max_dna_length)))" + ] + }, + { + "cell_type": "markdown", + "id": "cell-055", + "metadata": {}, + "source": [ + "The **projector** stage formats the same row differently. Its `_metadata` is the sentinel `[dna_bioreason]`, which means a causal-LM loss over the whole sequence. That gives the freshly initialized projector a dense alignment signal before SFT narrows the loss to the answer.\n", + "\n", + "That merged length (~2.8k) is over the 2048 default of `MAX_SEQ_LENGTH`, so the merged-length guard (`model.py`, issue #123) would raise. That's why section 1 sets `MAX_SEQ_LENGTH=4096` and every job inherits it.\n", + "\n", + "> **Efficiency tip from section 4:** KEGG and variant-effect references are ~4 kb, about **672–687 NT tokens**, yet every span is padded to 1024. `training.max_dna_length=768` would cut ~25% of the sequence with no truncation. It's not applied here, so the demo matches the shipped config.\n", + "\n", + "Now run the forward pass on a tile, the way the trainer calls it (`logits, loss = model(batch, labels=batch[\"text\"])`):" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "cell-056", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2026-10-11 23:26:43,890 - src.model - INFO - [MODEL] Forwarding Backbone: OlmoForCausalLM\n", + "2026-10-11 23:26:44,807 - src.model - INFO - [MODEL] Computed Loss: 8.93231201171875\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "logits (1, 2782, 50304) (= merged length, DNA included) loss 8.932\n", + "The loss is high because the projector is untrained: 2,048 out-of-distribution vectors in the context. Stage 1 fixes that.\n" + ] + } + ], + "source": [ + "def to_dev(x):\n", + " if torch.is_tensor(x):\n", + " return x.to(dev)\n", + " if isinstance(x, dict):\n", + " return {k: to_dev(v) for k, v in x.items()}\n", + " return x\n", + "\n", + "model.to(dev).eval()\n", + "batch = to_dev(collate([ex]))\n", + "with torch.no_grad(), torch.autocast(\"xpu\", dtype=torch.bfloat16):\n", + " logits, loss = model(batch, labels=batch[\"text\"])\n", + "print(f\"logits {tuple(logits.shape)} (= merged length, DNA included) loss {loss.item():.3f}\")\n", + "print(\"The loss is high because the projector is untrained: 2,048 out-of-distribution vectors in the context. Stage 1 fixes that.\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-057", + "metadata": {}, + "source": [ + "## 11. Local smoke run on this node\n", + "\n", + "Before burning queue time, run **20 projector steps on all 12 tiles** of this interactive node, using the same `scripts/training/submit_bioreason_projector.sh` the batch job uses. The scripts already send checkpoints to `$HYDRA_RUN_DIR` on `/flare`. The notebook runs a copy made by `H.patch_stage_script`, which drops the projector script's stray `source .venv/bin/activate` (a CUDA torch venv layered over the XPU one). For older script versions, it also adds the missing `hydra.run.dir=`.\n", + "\n", + "The kernel is holding tile 0, so free it first." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "cell-058", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "68d67\n", + "< source \"${PROJ_DIR}/.venv/bin/activate\"\n" + ] + } + ], + "source": [ + "del model, logits, loss, batch\n", + "import gc\n", + "\n", + "gc.collect()\n", + "torch.xpu.empty_cache()\n", + "\n", + "SCRIPTS = Path(PRISM_OUTPUT_ROOT, \"demo_scripts\", RUN_TAG)\n", + "PROJ_SH = H.patch_stage_script(REPO / \"applications/bioreason/scripts/training/submit_bioreason_projector.sh\", SCRIPTS / \"projector.sh\")\n", + "SFT_SH = H.patch_stage_script(REPO / \"applications/bioreason/scripts/training/submit_bioreason_sft.sh\", SCRIPTS / \"sft.sh\")\n", + "GRPO_SH = REPO / \"applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh\" # native HYDRA_RUN_DIR support\n", + "EVAL_SH = REPO / \"applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh\"\n", + "!diff {REPO}/applications/bioreason/scripts/training/submit_bioreason_projector.sh {PROJ_SH}" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "cell-059", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Step 0: Loss 9.5068\n", + "Training Zone A: 50%|█████ | 10/20 [00:08<00:05, 1.87it/s, loss=9.42]Step 10: Loss 9.4199\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[Step final] Saving Checkpoint to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final...\n", + "[2026-10-11 23:28:20,589][accelerate.accelerator][INFO] - [RANK 0] Saving current state to ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final\n" + ] + } + ], + "source": [ + "SMOKE_DIR = f\"{PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/{RUN_TAG}\"\n", + "SMOKE_LOG = f\"{PRISM_OUTPUT_ROOT}/demo_logs/{RUN_TAG}/smoke-projector.log\"\n", + "Path(SMOKE_LOG).parent.mkdir(parents=True, exist_ok=True)\n", + "smoke_env = \" \".join(f\"{k}={v}\" for k, v in {\n", + " \"RUN_ID\": \"DEMO-SMOKE-PROJECTOR\", \"HYDRA_RUN_DIR\": SMOKE_DIR,\n", + " \"MAX_STEPS\": 20, \"WARMUP_STEPS\": 2, \"SAVE_EVERY\": 20}.items())\n", + "# ~5-8 min: model load on 12 ranks + 20 steps. Progress lines only; full log in SMOKE_LOG.\n", + "!env {smoke_env} bash {PROJ_SH} > {SMOKE_LOG} 2>&1; grep -E \"Step [0-9]+: Loss|Saving|Error|error:\" {SMOKE_LOG} | tail -25" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "cell-060", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEiCAYAAAArqK94AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQaNJREFUeJzt3Xd0FNX7x/F3stn03kMPAQRBilKE0KWXQEITaSJig59fRezSFQQBEQVFaQoIUhVQEClBQMTQixQJIaGk92w2u5vd+f0R2RhDSTBhU57XOZzD3pk7eSbJzidT7l0rRVEUhBBCiFJgbekChBBCVFwSMkIIIUqNhIwQQohSIyEjhBCi1EjICCGEKDUSMkIIIUqNhIwQQohSIyEjhBCi1EjICCGEKDUSMiVg3bp1/Pnnn5Yu477s3LmT3377zdJllAhFUVi5ciVRUVGWLkUI8TcJmRLwxhtvsG/fPkuXcV8WLFjAmjVrSnSbO3bs4PDhwyW6zaIwGo2MHz+eo0ePPvCvLYS4PRtLFyAsq0ePHvj4+JToNufPn0+DBg1o3bp1iW5XCFH+VIiQURSFP/74g+joaPz9/WnRogUODg7m5evWraNx48Z4enpy8uRJMjMzad++PX5+fmi1Wg4cOEBaWhpNmzalXr16hbafkZHBwYMHSU9Pp3bt2rRs2RIrK6u71vTTTz+RmZnJwIEDUalUAFy5coVjx45hZWVFu3bt8PPzu+s2zp49y59//kloaCjHjx/n6tWrBAYG0rJly9uuFxYWxsGDB7l+/TrBwcEEBgaSk5PDwYMHSUhIoGrVqgQHB2Njk/9jr1+/Pq6urgW2ZzKZOHLkCFevXsXb25v27dtjZ2d32+95VFQU1apVo3Xr1qhUKnbu3ElCQgIqlYqVK1cC0L59e2rXrg1wz3q2b9+Or68vQUFBHDp0iNTUVEaOHMmvv/5KYmIiAwcOvOv37HYSEhL47bffyMnJoVGjRjRq1KjA8tzcXA4dOkRcXBw1a9akefPmBWq613IhxJ2V+3eKRqMhJCSEmzdv0rp1axITE4mNjWXlypXmg8kbb7xBrVq1SEpKokWLFly8eJHx48czc+ZMFixYQIMGDTCZTDz//PMsWrSIp556yrz9ffv2MWLECGrWrElgYCAHDhygXr16bNy4ETc3t0L1mEwm3njjDTZu3Mh3332HSqXCaDTyv//9j2+//ZYOHToAMG7cOBYtWnTXg+bPP//Mxx9/zKeffoq9vT1eXl6Eh4fTtWtXvvnmG3PQ3Vpv0aJF2NvbExQURIMGDcjKymLAgAE4OTnxyCOPcOTIEVxcXPj++++pVq0akHe5LCgoiDZt2gAQHR3NkCFDSEhIoE2bNly6dAmdTse2bduoUaMGAFFRUTz11FMkJCTw+OOPk5ycjE6nY/PmzVy4cIHMzExUKhUREREAPPzww9SuXZszZ87cs57Zs2cDkJycTOPGjfHy8mLEiBEsX76ciIiIYofM6tWreeWVV2jWrBleXl688sordOnSha+//hqVSkVcXBzdu3dHrVbTpEkTrl27Rk5ODuvXr8ff3/+ey4UQ96CUc1u2bFF8fHyUjIwMc9u1a9eUM2fOmF9Xr15dadKkiXmd3NxcpVmzZoqLi4ty+PBh83rvvvuu8tBDD5lfZ2ZmKnXq1FFefPFFxWQyKYqiKDdv3lQCAwOViRMnFtj+Z599pmRlZSmDBg1SmjZtqly5csW8fO7cuYqLi4uyf/9+c9vy5cuVKlWqKCkpKXfct7lz5ypOTk7KtGnTzG1nzpxR3N3dlTVr1hRab/HixeY2o9GotGnTRunXr5+i1+sVRVGU9PR0pXnz5srgwYPN63Xv3l156aWXzK/btWunNGvWTElOTjZ/r8LCwpQnn3xSURRFMZlMSuvWrZXOnTsr6enp5n7Hjx9X4uPjFUVRlC5duij/93//V2BfilpP27ZtFR8fH+Xy5csF+i9btkyZNGnSHb9XiqIoBoNBcXJyUtavX68oiqJER0cr3t7eypw5cwp8/zw8PMzfq3nz5imNGzdWDAZDgXViYmKKtFwIcXflPmT27t2ruLq6Kjt27FCMRuNt16levboyY8aMAm3PPvus0qpVqwJt33//veLk5KRkZ2criqIoO3fuVJycnJSoqKgC602ZMkWpVatWge2/++67SnBwsNKtWzfzAfqWJk2aFDiQKkreQbdWrVrK0qVL77hvc+fOVZydnQsEqKIoypAhQ5TQ0NAC67m7uxc4EP7555+Kk5OTEh4eXqDvsmXLFGdnZ0Wj0SiKUjBkzpw5ozg5OSnffvttgT779+9XnJyclMTEROXkyZOKk5OTsnv37jvWfbuQKWo9bdu2VYYPH37Hbd/Nv0Nm0aJFire3t5KTk1NgveHDhytPPPGEoiiK8tVXXynVq1dX/vjjj9tu817LhRB3V+4vl3Xq1InJkyfzyiuvkJ2dTatWrejfvz9PPvmk+V4IgIeHR4F+tra2uLu7F2oD0Ol0ODg4EB0djZWVlfky0S21atUiMTGR7OxsHB0dAVi6dCl6vZ4//vgDT0/PAutHR0dTt25d8z2KWxwdHbly5cpd98/T0xMXF5cCbTVq1GDPnj0F2ry9vQvcJ4iJiTHX+k+BgYEoikJMTAz169cvVCfk3Tv6Z62pqalA3mWy+Ph4AIKCgu5a978Vp56SugwVExNDQEBAoftJgYGB/P777wCMHDmSyMhIBg0ahI2NDW3atGHw4MH06dOnSMuFEHdXIR5hfu2117hw4QL79++nW7duvP3220yfPv0/b9fPzw9FUUhOTi7QnpSUhKurqzlgIO++T7du3ejTpw+XLl0qsL67uzsJCQlEREQU+NexY0cefvjhu9aQnp5Obm5ugbbk5ORCDw38+0GEW8uTkpIKtCcmJhZY/u86AS5evFigzsuXLzNq1CicnJzMAXorbIqqOPXc66GK4nzNf//sbtVw6+vZ2toya9YsoqKi2LZtG02bNmXkyJF8/fXXRVouhLi7ch8y0dHRGAwGIO8v1LFjx9KtWzeOHTv2n7fdqlUrHB0d+fbbb81tubm5rF+/no4dOxZY187OjjVr1tC+fXu6devGqVOnzMv69u2Lk5MTCxcuZNGiReZ/s2bNomvXrnetITc3l40bN5pfp6am8vPPPxf6+v9Wv359AgICCo2BWbt2LU2bNi10ZgfQokULAgICaN26dYE6Fy1axKuvvsrDDz9M8+bN8ff356uvvirQNy0tjczMTABcXFzQarX/uZ5/2r9/f4HvQ1F07NiRjIwMtm3bZm5LT09n+/btdOrUCYDIyEgURcHKyooGDRowYcIEmjZtav79uddyIcTdlfvLZREREXzwwQf06NGDWrVqERUVxZYtW1i8ePF/3ra/vz/Tp0/nzTffJCoqiqCgIL7//nsSExP57rvvCq2vUqlYsmQJr7/+Or169WLDhg20adOG6dOn079/fzp16mQOnEuXLrF7927WrFmDr6/vHWtwcHDgvffe48yZM3h7e7Nq1Sp8fX0ZN27cXWu3t7dn/vz5jBgxgrS0NB577DH27NnD4cOHCxx0/8nW1pYVK1YwbNgwjhw5QvPmzcnJyeHo0aPExsayf/9+bG1t+eqrrxg6dCh9+vShW7duJCcn8+OPP7Jt2zZcXFxo06YNCxcu5JNPPsHNzc38CHNx6/mn+3m6rFmzZrz00kuMGTOG559/Hk9PT1avXo2Xlxevv/46AD/88AMbNmygW7duVKlShdOnT3P69GlmzJhRpOVCiLsr92cyAwcOZOvWrVSpUoVLly7h7u7O/v37GTJkiHmdoUOHFros1aZNG3r06FGgrXr16owaNarANfwXXniBPXv24O7uTmRkJKGhoURERBAYGHjb7VtZWTF37lzeeecdtmzZQnp6Ou7u7uzevZuJEyeSmppKdHQ0jRs35uDBgzRp0uSu++fh4cGvv/6Kv78/169fZ/To0YSHh+Pk5GRe55FHHmHw4MGF+vbt25fDhw8TFBTEpUuXaNOmDREREbRo0aLAev+8PNW2bVuOHTtG69atuXLlChqNhhEjRhSY0aBTp06cOHGCrl27EhUVhaenJz/++CMBAQEAvPLKK3z44YfcvHmTiIgI8yWyotTTp0+fQuOAADp06HDPgLG2tmbUqFHmMTmQ90j02rVrMZlMxMTEMG7cOA4ePGgeGzRhwgSWLVuGi4sLly5dok6dOhw9epTg4OAiLRdC3J2VoiiKpYsQtzdv3jyWLFlS6B5PSWrdujXdu3dn6tSppfY1hBCVV7k/kxH35/z580yfPp1z587RrVs3S5cjhKigJGTKsDtdBisJCQkJ5OTksHXrVvNofyGEKGlyuUwIIUSpkTMZIYQQpUZCRgghRKmRkBFCCFFqynXIKIqCyWRCbisJIUTZVO5DJvZmjISMEEKUUeU6ZIQQQpRtEjJCCCFKjYSMEEKIUiMhI4QQotRIyAghhCg1EjJCCCFKTaUMGX2uifM3cyxdhhBCVHiVMmS+2JvE4M+i+OineLR6k6XLEUKICqvShczleB3L9idjUmDlgRRCP7nC75c1li5LCCEqpEoXMjW9bRnf1Qdbm7yPHL6WYmDMshgmbbpJutZo4eqEEKJiKdefJ2MymYi9GUNAlRpYWxcvL6MSdUzZHMuxq1pzm7eLindD/OnWyLWkSxVCiEqp0oZMXn+FDRFpzNuRgEaXf2+mS0MX3g3xw9dVXZLlCiFEpVOpQ+aWuHQDM36II/x8lrnNxd6aib18GdDcHSsrq5IoVwghKh0Jmb8pisLOM5nM3BpHiib/3kzL2o5MDQ2gprftfy1XCCEqHQmZf0nT5DL7xwS2nkg3t9nZWDG+qw8jgz2xUclZjRBCFJWEzB0cvJTFtC1x3EwzmNsaVrVnWlgADarYl+jXEkKIikpC5i40OhMLdyWw5nAqt75LKmt4pr0XL3b2xk5d6Z4AF0KIYpGQKYJTMVomb47lcrzO3FbL25ZpYQE0D3Qsta8rhBDlnYRMEelzFZaGJ7EkPIncf4zZHNLKnQk9fHG2V5Xq1xdCiPKoWCGTk5PDls2bOH36FLm5ubRo2ZKwsIHY2Njcdv21a9dw8MCvBdoGDBxM585PmF/v+nkn+/btRafT0aRpU5588ins7OyKVM+DDJlbLsfrmLwpllPX8gdx+rnaMKm/P50auDyQGoQQorwoVsh8+OFMUpKTGf3MGEwmE0uXfkmD+g/z3PMv3Hb9JV8sJkenY/DgJ81trq6uODk5AfDLL7tY9c1KXnxpPG5ubixftpSAgABem/hGkeqxRMgAGE0Kaw+nsmBXAlp9/revZ2NX3u7rh5fz7UNXCCEqmyIfmTMzM4j44wjPjHmWBg0epmHDRowePYbdu3eRmZlxx36Ojo4EBASY/90KGIAfvt9C//5htG7dhocfbsgLL7zEb78dIjEx8b/tVSlTWVsxPNiTra8EEVw3f392nM6g7/wr/HA8jXJ8FVIIIUpMkf/k1mrzLg/9MyScnJwwGo1ERkbStGmz2/aL+OMI586exc3NjRYtWtKjZy9sbGzQaDTExt6kwcMPm9etU7cutra2REZexsfHp9C2DAYDBkP+I8WKkjcVjMlkmen6/d1UfD6qKttPZjD7xwTStSbStUbe2RDL9hPpTO7vT1UPmZpGCPHgPMirOkVR5JDx8fHFz8+fH7dvZ+xzzwOwffs2ADRZWbft8+TQYYT0C8VoNBJ9NYpvvvmaa9ev8eKL49Bo8vq4uBS8j+Hs7ExWZuZtt7d500bWr19nfm1vb8ecOXOIj7te1N0oFc394MsnbfnioIF9f+U9FfDb5Wz6L7jC6MfVhDyiQmUtgziFEKWvarVali6hgCKHjJWVFa9NfJ2Fn3zM06NGYG1tRceOnbC2tsbB8faP8bq5ueHm5gZAtWrVsLd3YPbsmYwZMxZb27yb+znagp9QqdVq73jjP2zAQPqG9DO/VhQTaalJ+PlXs3h6BwAL68D+C1nM2BpPfHouObnw+UEDh6JVTAv1o45f0R5oEEKIiqJYd6iDgurwycJFZGRkoFKpuHr1Kjt2/ERQUJ0i9Xd1c8VoNKLT6XB3d8fFxYWYmGjq1K0LQGJiIlqtlmrVq9+2v1qtRq3Ov/xkMplIS807PbR0yNzS6WFXWtR2Yv7OBL47kgbA6Ws5DFp0lbEdvHmukxe2NmWjViGEKG3FOtqFh+8jISEBV1dXNBoNK5Z/Rddu3c1nK2tWr+LThQvM63/99Qo0mrxPnczMzOS7dWupW7ee+RJZ+/Yd2bZ9K1lZWRiNRtavX0eNmjUJDKxdQrtnGc72Kib3D+Cb52pS6++JNXON8PneJAZ+GsXJ6GwLVyiEEA9Gsc5kqlWrxozpU8jOzkar1fJEl66MGjXavDw9PY3klGTzay8vb/5v/ItYW1uTmZlJ48ZNmPDa6+blTw0bzoIF83l2zNOo1Wq8vX14beLrVBSPBTqy+eVAvtibxPJfk8k1QWSCnuFLohnW2oOXu/niZCdnNUKIiuu+RvynpCTj4uJa4NIVQHp6OkZjLp6eXgXaU1NScHZxKbT+LVlZWej1ukL97sVS42Tux4XYHCZviuXcjfx7UFXc1UwJ9adtPWcLViaEEKVHppV5gHKNCqt/S+HTXxLJMeR/20OaufFGb188nGQQpxCiYpGQsYCYZD1Tt8RyJDL/3oynk4q3+vjRq4mrfBKnEKLCkJCxEEVR2Hw0nY9+iiczJ38waYf6zkzq50+AuwziFEKUfxIyFpaYYeCDbfH8cjZ/AKqTnTUTevgyuKU71jKIUwhRjknIlBG7z2Uw44c4kjLzP0fg0VoOTAsNoLavDOIUQpRPEjJlSIbWyNwdCWyKSDO3qVVWvPiEN8+090KtkrMaIUT5IiFTBv0eqWHq5liupeRPBlrP344ZAwJoVM3BgpUJIUTxSMiUUVq9iUV7Evn6QAqmv39C1lYwsq0n47v44GBbsfZXCFExSciUceduaJm0KZaLsTpzW3VPNVNDA3i8jtNdegohhOVJyJQDBqPCygPJLN6ThD43/8cV1tyNib38cHNQWbA6IYS4MwmZciQqUceUzbEcu6o1t3m7qHg3xJ9ujVwtWJkQQtyehEw5YzIpbIhIY96OBDS6/EGcXRq68F6IHz6uMohTCFF2SMiUU3HpBmb8EEf4+fxPJXWxt2ZiL18GNHeXqWmEEGWChEw5pigKO89kMnNrHCma/EGcLWs7MjU0gJp/f5aNEEJYioRMBZCmyWX2jwlsPZFubrOzsWJ8Vx9GBntiI4M4hRAWIiFTgRy8lMW0LXHcTMsfxNmwqj3TwgJoUMXegpUJISorCZkKRqMzsXBXAmsOp3LrJ6uyhmfae/FiZ2/s1PJ9EkI8OBIyFdSpGC2TN8dyOT5/EGctb1umhQXQPNDRgpUJISoTCZkKTJ+rsDQ8iSXhSeTmPxfAkFbuTOjhi7O9DOIUQpQuCZlK4HK8jsmbYjl1LX8Qp5+rDZP6+9OpgYsFKxNCVHQSMpWE0aSw9nAqC3YloNXn/8h7Nnbl7b5+eDnbWLA6IURFJSFTydxMNTB1SyyH/tKY29wcVLzZx5eQZm4yiFMIUaIkZCohRVHYdiKDD7fHk67Nv1kTXNeJKaH+VPWQQZxCiJIhIVOJJWflMmtbPDtOZ5jbHGyt+F83X55q7YHKWs5qhBD/jYSMIPx8JjN+iCMuPdfc1ri6PTMGVKGOn50FKxNClHcSMgKArBwj83cm8N2RNHObjQqe6+jN2I5e2NrI91cIUXwSMqKAY1HZTN4cy9UkvbktyNeW6WEBNK0pgziFEMUjISMK0RlMfLE3ieW/JpP790fWWFnBsNYevNzNFyc7+V4LIYpGQkbc0YXYHCZviuXcjRxzWxV3NVNC/Wlbz9mClQkhygsJGXFXuUaF1b+l8OkvieQY8n9VQpq58WZvX9ydZBCnEOLOJGREkcQk65m6JZYjkdnmNk8nFW/39aNnY1cZxCmEuC0JGVFkiqKw+Wg6H/0UT2aOydzeob4zk/r5E+CutmB1QoiySEJGFFtihoEPtsXzy9lMc5uTnTUTevgyuKU71jKIUwjxNwkZcd92n8tgxg9xJGXmT03zWC0HpoUFEOgjgziFEMUMmYyMDNas/obTp0+Rm5tLixatGDnqaezt7/3RvjM/mMG5c2eZOWs2NWvWAiAzM5PVq7/h9KmT6PUGateuzfARI83L70VCxvIytEbm7khgU0Sauc3WxooXO3szur0XapWc1QhRmRXryDxv7hzi4+OZPGU6U6fNIDr6Kp9/vuie/Xbu3EGOLoecnBxMpvxr+cuWfsnVqCjeeXcSH82dh4eHB+/PmFZgHVG2uTqomB4WwLJna1DdM++ejD5X4ZNdiQxZFMXZ69p7bEEIUZEVOWTS0tI4e/YMI0aOIiAggKpVqzF8xEh+O3SQ1NTUO/aLi4vjh+83M2bM2ELLIq9E0rFTJ6pXr4Gnpxch/UJJSUm56/ZE2fR4kBNb/leb0e09uXVL5mKsjqGLr/LRT/Fo9fKHgxCVUZEHORhz8yZPVKnyu9jY2GAymbgaFYWHh0ehPiaTic8++4Rhw0fg6upWaPljjzbnyJHfadeuA/b29uwP30ft2kG33RaAwWDAYDCYXyuKyfx1hOXZ2cCE7j50b+TClM1xXIzTYVJg5YEUdp/LZGp/P1oFOVm6TCEqtLJ266DIIePl7U2NmjXZuHE948aNx2QysWHDegA02Vm37bN921acnV1o27b9bc9Ohj41jFmz3mfUyGFYW1vj5eXNe5Om3PGbtHnTRtavX2d+bW9vx5w5c4iPu17U3RAPgKcVfNzfig0nbVgdkYvBCNdTDDy7/Do9GqgY20aNi73cqxGiNFStVsvSJRRQrOHar7/+Jku++JxnRo/CxsaG/v3DOHH8GM7OhT8n/ubNG2zd+j0ffTT/jtv74P3pODk58elnn+Pg4MDuX3YxedI7fPzxQtzc3QutHzZgIH1D+plfK4qJtNQk/Pyrlbn0FjChOoQ+rmfq93Ecv5p3b2bneSNHr8E7ff3o2qjw740QomK5r0eYjUYj1tbWnDxxgjlzZrF02QqcnArOZbV/fzifffoJtrZ5n7KoKAo6nQ5bW1v69A2hb99+jH56BLNmzaHeQw+Z+z01dDD/9/IrtG7d5p51yNNl5YPJpLAhIo15OxLQ6PIvbXZp6MJ7IX74uMogTiEqqmKdyezY8SN16tSldu0gLl/+i6XLvqRvSD9zwHz11RISExN55533aNu2HS1btjL3TU9PZ9xLzzN12gxq1w5CrVbj6enFnj2/UL1GDWxtbQkP34fBYKB69Rolu5fCoqytrRjSyoMO9Z2Z8UMc4efzLq/uPpfJkUgNE3v5MqC5u0xNI0QFVKyQadKkKYsXf8alixdxdXWlV68+hIYNMC836PXo9ToAVCoVDg4O5mU5OXkz+dra2qJW5/3l+tbb77Bi+TKeGT0Sk8mEr58fL//vVapVq/afd0yUPf5uaj4bUY2dZzKZuTWOFI2RzBwTUzbH8ePJDKaGBlDT29bSZQohStB9Xy5TqVSF2g0GA4qimC+R/ZOiKOTk5GBnZ1fo0paiKJhMpttu827kcln5labJZfaPCWw9kW5us1dbMa6LDyODPbGRQZxCVAgyrYywqIOXspi2JY6bafmPpjesas+0sAAaVLn3TBJCiLJNQkZYnEZnYuGuBNYcTuXWb6PKGp5p78WLnb2xU8vPVojySkJGlBmnYrRM3hzL5Xidua2Wty3TwwJ4LNDRgpUJIe6XhIwoU/S5CkvDk1gSnkRu/uTODGnlzoQevjjbF+++nRDCsiRkRJl0OV7H5E2xnLqWP8Gmn6sNk/r706mBDOIUoryQkBFlltGksPZwKgt2JaDV5/+a9mzsytt9/fByLtYT+EIIC5CQEWXezVQDU7fEcugvjbnNzUHFm318CWnmJoM4hSjDJGREuaAoCttOZPDh9njStfk3a4LrOjEl1J+qHjKIU4iySEJGlCvJWbnM2hbPjtMZ5jYHWyv+182Xp1p7oLKWsxohyhIJGVEuhZ/PZMYPccSl55rbGle3Z8aAKtTxs7NgZUKIf5KQEeVWVo6R+TsT+O5ImrnNRgXPdfRmbEcvbG3kd0IIS5OQEeXesahsJm+O5WqS3twW5GvLjAFVaFLD4S49hRClTUJGVAg6g4kv9iax/Ndkcv/+yBorKxjW2oOXu/niZCe/H0JYgoSMqFAuxOYweVMs527kmNuquKuZEupP23rOd+kphCgNEjKiwsk1Kqw6lMJnuxPJMeT/eoc0c+PN3r64O8kgTiEeFAkZUWHFJOuZuiWWI5HZ5jZPJxVv9/WjZ2NXGcQpxAMgISMqNEVR2Hw0nY9+iiczx2Ru71Dfmcn9/fF3U1uwOiEqPgkZUSkkZhh4f2s8u89lmtuc7KyZ0MOXwS3dsZZBnEKUCgkZUan8cjaD97fGkZSZPzXNY7UcmBYWQKCPDOIUoqRJyIhKJ0NrZO6OBDZFpJnbbG2seLGzN6Pbe6FWyVmNECVFQkZUWr9Hapi6OZZrKQZz20MBdkwPC6BRNRnEKURJkJARlZpWb2LR7kS+PpiC6e93grUVjGrrybguPjjYyu+VEP+FhIwQwLkbWiZtiuVirM7cVt1TzdSwAB4PcrJgZUKUbxIyQvzNYFRYeSCZxXuS0Ofmvy0GNHfjtV5+uDmoLFidEOWThIwQ/xKVqGPK5liOXdWa27xdVLwX4k/XRq4WrEyI8kdCRojbMJkUNkSkMW9HAhpd/iDOLg1deC/EDx9XGcQpRFFIyAhxF3HpBmb8EEf4+Sxzm4u9NRN7+TKgubtMTSPEPUjICHEPiqKw80wmM7fGkaLJH8TZsrYjU0MDqOlta8HqhCjbJGSEKKI0TS6zf0xg64l0c5u92orxXXwYEeyJjQziFKIQCRkhiungpSymbYnjZlr+IM6GVe2ZPiCA+gH2FqxMiLJHQkaI+6DRmVi4K4E1h1O59Q6ysYZn2nvxQmdv7NTy+ygESMgI8Z+citEyeXMsl+PzB3HW8rZlelgAjwU6WrAyIcoGCRkh/iN9rsLS8CSWhCeRm/9cAENauTOhhy/O9jKIU1Re9xUyRmPeO0mlKvqbx2AwoNfrsLd3uG0/g8GAWl28sQcSMqIsuRyvY/KmWE5dyx/E6e9mw6R+/nRs4GLByoSwnGKFTEJCPF8u+YI//zyH0WikadNmjBv/Mq6udx8FbTQaefedN/nrr7+YO+9jAgNrm5cdO3qUVau+5ubNG3h6evHMmGdp2bJVkeqRkBFljdGksPZwKgt2JaDV57+1ejZ25e2+fng521iwOiEevGIdmefM/hB7BweWr/iGFStXoQALP/n4nv22bN6Ep6dXofbTp08xf/5HDBo0mDXffsfs2R/x16VLxSlJiDJFZW3F8GBPfnilNsF18yfW3HE6g77zr/DD8TTK8RVqIYqtyCGTnJREVNQVBg0ajL29PY6OjgwaNJgTJ46TmJh4x35Xo6LYF76X4SNGFVq2bu23dOveg+C27QBwc3dn2PAR97EbQpQtVT1sWTK6OrMGVTFPrJmuNfLOhlieX3GNG6l6C1coxINR5HN367/voxgM+WMDbv0/OvoqPj4+hfoYDAY+/fQTxo59HgcHh0LL/vrrEg83bMhzY58hPT0dHx9fnh79DM2bt7htDQaDocDXV5S8OaVMJtNt1xfC0vo0daF1HQdm/5jAjtOZABz6S0O/BVd4uasPQx93R2UtgzhFySlrtw6KHDIeHh7Ur9+A1au+4dmxz6MoJr5dsxpra2t0OTm37bP+u3XUDgqiadNmpKamFlim1WoxmUwcOniA6TM+wNfXj507dzBv7hw+WfgZvr5+hba3edNG1q9fZ35tb2/HnDlziI+7XtTdEMIiXm0HravbsjBcT5IGtHolb/aAo4m82smWWl5l68Agyq+q1WpZuoQCinUXcuLrb7Lqm5W8P2MaarUN/UMHcOHCeVzd3AqtGxV1hb179zDrw9loNFlotdlAXrjo9XocHBywsrKic+cu+PsHANCrV282bviOs2fP0rlz4ZAJGzCQviH9zK8VxURaahJ+/tXKXHoL8W+hVaDro0YW/JzEd3+kAXA+XuGlDTrGdvDi2Q6e2NrI77GoWIoVMh4eHrz8v1fNr3/77RD29vbUrVsPAJ1Oh6Io2Nvbc/PmTfR6Pa9NuLV+3s3OD96fTp8+IQx9ahjVa9TAkJt/+ctkMmE0GlHb3L4stVpd4DFnk8lEWmre6aGEjCgPXB2tmRwaQO+mbkzeHMvVJD25Rvh8bzK7zmYyY0AVmtRwuPeGhCgnihUymzZuoGq1atStU5e/Lv/FsqVf8eTQYdjb583XtGzplyQkJjB16gyCg9sSHNzW3Dc1NZVnxzzN+x/MMj/C3Lt3X9asXkWjRo9QtUpVfvxpO9bWKpo0bVaCuyhE2fNYoCObXw7ki71JLP81mVwTRCboGfbFVYa19uDlbr442ckfTqL8K1bIdOnSleUrlrH0qyV4eXkzYuQoOnbsZF5uZ2eHvf3t/wqztrLC0dGpwBlHly5d0ev1LFv6JdnZ2dSqFciUqdPvOe5GiIrATm3N/7r70r2xK5M3xXLuRg6KAqt/S2Xvn1lMCfWnbT1nS5cpxH8i08oIUQbkGhVWHUrhs92J5Bjy35Ihzdx4s7cv7k4yiFOUTxIyQpQhMcl6pm6J5UhktrnN00nF23396NnYVT6JU5Q7EjJClDGKorD5aDof/RRPZk7+GLAO9Z2Z3N8ff7fizfEnhCVJyAhRRiVmGHh/azy7z2Wa25zsrJnQw5fBLd2xlkGcohyQkBGijPvlbAbvb40jKTP/cwQeq+XAtLAAAn3sLFiZEPcmISNEOZChNTJ3RwKbItLMbbY2VrzY2ZvR7b1Qq+SsRpRNEjJClCO/R2qYujmWayn5g5gfCrBjelgAjarJIE5R9kjICFHOaPUmFu1O5OuDKZj+fvdaW8Gotp6M6+KDg628F0TZISEjRDl17oaWSZtiuRirM7dV91QzNSyAx4Oc7tJTiAdHQkaIcsxgVFh5IJnFe5LQ5+a/lQc0d+O1Xn7mz7IRwlIkZISoAKISdUzZHMuxq1pzm7eLivdC/OnaSKZpEpYjISNEBWEyKWyISGPejgQ0uvxBnF0auvBeiB8+rjKIUzx4EjJCVDBx6Qamfx/H/gtZ5jZXe2sm9vIjrLmbTE0jHigJGSEqIEVR2Hkmk5lb40jR5A/ibBXkyNTQAGp42VqwOlGZSMgIUYGlaXLzPub5RLq5zV5txfguPowI9sRGBnGKUiYhI0QlcPBSFtO2xHEzLX8QZ8Oq9kwfEED9AHsLViYqOgkZISoJjc7Ewl0JrDmcyq13vY01PNPeixc6e2OnlveQKHkSMkJUMqditEzeHMvl+PxBnIE+tkwLDeCxQEcLViYqIgkZISohfa7C0vAkloQnkZv/XABDWrkzoYcvzvYyiFOUDAkZISqxy/E6Jm+K5dS1/EGc/m42TOrnT8cGLhasTFQUEjJCVHJGk8Law6ks2JWAVp9/OOjZ2JW3+/rh5WxjwepEeSchI4QA4Eaqnmlb4jj0l8bc5uag4q0+fvRt5iqDOMV9kZARQpgpisK2Exl8uD2edG3+zZrguk5MDQ2giodMTSOKR0JGCFFIclYus7bFs+N0hrnNwdaK/3Xz5anWHqis5axGFI2EjBDijsLPZzL9+zjiM3LNbU2qOzB9QAB1/OwsWJkoLyRkhBB3lZVjZP7OBL47kmZus1HB8x29ebajN7Y2clYj7kxCRghRJMeispm8OZarSXpzWx0/O6aHBdCkhoMFKxNlmYSMEKLIdAYTX+xNYvmvyeT+/ZE1VlYwrLUHL3fzxclO3oeiIAkZIUSxXYjNYfKmWM7dyDG3VXFXMyXUn7b1nC1YmShrJGSEEPcl16iw6lAKn+1OJMeQfxgJaebGm719cXeSQZxCQkYI8R/FJOuZuiWWI5HZ5jZPJxVv9/WjZ2MZxFnZScgIIf4zRVHYfDSdj36KJzPHZG7vUN+Zyf398XeTQZyVlYSMEKLEJGYYeH9rPLvPZZrbnOysmdDDl8Et3bGWQZyVjoSMEKLE/XI2g/e3xpGUmT81zWO1HJgWFkCgjwzirEzu68iclpZGUlJisfqkp6cTEx2NXq+/r+VCiPKjayNXtr4axIAW7ua2Y1e1hC2M4st9SRiM5fZvW1FMxTqTuRoVxSeffExaWhqKYsLb24fX33gLPz+/u/bT6/VMfO1Vbty4ztx5HxMYWLtYy+9EzmSEKPt+j9QwdXMs11IM5raHAvIGcTaqJoM4K7oiH5kVReGjubNp0OBhli1fybLlX1OrVi0+WTD/nn3XrP6GBg0a3PdyIUT59XiQE1v+V5vR7Ty5dUvmYqyOoYuvMveneLR60903IMq1IodMYmICcbGx9OrdG2tra1QqFb169eHixQvExsbesd+5c+c4ffo0AwcNua/lQojyz8HWmom9/Fj7Ui3q+efdkzEpsOJACqGfXOH3SM09tiDKqyKPlnJwyDutzcrMMrdlZuY9QXL9WgwBAQGF+mi1WhYv/pRXXpmAjU3hL3Wv5f9mMBgwGPJPuRUl7y8gk0n+EhKiPHi4ih3rXqrJ1wdT+HxvMvpchWspBsYsjSHsMTde6+mDq4PK0mWWa2Xt1kGRQ8bFxZXHHmvO0qVfMmLkKBRFYfWqb7C2tr7jzfqvV66gZctW1K1bj9TU1GIv/7fNmzayfv0682t7ezvmzJlDfNz1ou6GEKIM6F0XGnvb8vE+A2dj8/5I3HwsnfDz6Yxvb0vbIAma+1W1Wi1Ll1BAseZ9eHXCa2zetIn1361DrVYzeMiTzP5wJh6enoXWvXDhPEeO/M67704iJjqazKy8s564uDjc3T2Ij4+763IPD49C2wwbMJC+If3MrxXFRFpqEn7+1cpcegsh7i6gCqxpoLAxIp35Pyei0ZlIyYbpO/V0aejMO3388HGVqWnKu2I9XWY0GlGp8v/C2LNnN6tXfc2XXy1HrVaTkpJMbm4uvr5+HD0awepV3+T3NRm5eeMG/v7+dO/ekypVq951eUi//vesR54uE6JiiEs3MP37OPZfyL8c72qfdx8nrLmbTE1TjhUrZNauXYOrqyt169Tjr8uX+G7dWsY+9wLt2rUHYPGiT0lITGDq1BmF+qampvLsmKfv+IjyvZbfjoSMEBWHoijsPJPJzK1xpGjyB3G2CnJkamgANbxsLViduF/FOjKHhg4gOTmZZcu+4tzZs7z+xlvmgAHw9PLCz/f2Y2ZsbFRUr14DW9vb/6Lca7kQomKzsrKiZ2NXtr1am5Bmbub2I5HZhH5yhRW/JpMrgzjLHZlWRghRJh28lMW0LXHcTMt/orRhVXumDwigfoC9BSsTxSEhI4QoszQ6Ewt3JbDmcCq3jlQ21vBMey9e6OyNnVre92WdhIwQosw7FaNl8uZYLsfrzG2BPrZMCw3gsUBHC1Ym7kVCRghRLuhzFZaGJ7EkPInc/OcCGNLKnQk9fHG2l7E1ZZGEjBCiXLkcr2PyplhOXdOa2/zdbJjUz5+ODVwsWJm4HQkZIUS5YzQprD2cyoJdCWj1+Yewno1debuvH17OMoizrJCQEUKUWzdS9UzbEsehv/In2HRzUPFWHz/6NnOVQZxlgISMEKJcUxSFbScy+HB7POna/Js1wXWdmBoaQBUPtQWrExIyQogKITkrl1nb4tlxOsPc5mBrxSvdfBna2gOVtZzVWIKEjBCiQgk/n8n07+OIz8g1tzWp7sD0AQHU8bOzYGWVk4SMEKLCycoxMn9nAt8dSTO32ajg+Y7ePNvRG1sbOat5UCRkhBAV1rGobCZvjuVqUv5nXtXxs2N6WABNajhYsLLKQ0JGCFGh6QwmvtibxPJfk8n9+0N0raxgWGsPXu7mi5OdHDtKk4SMEKJSuBCbw+RNsZy7kWNuq+KuZkqoP23rOVuwsopNQkYIUWnkGhVWHUrhs92J5BjyD30hzdx4s7cv7k4yiLOkScgIISqdmGQ9U7fEciQy29zm6aTinRB/ejziIoM4S5CEjBCiUlIUhc1H0/nop3gyc0zm9o4NnJnUzx9/NxnEWRIkZIQQlVpihoH3t8az+1ymuc3JzprXevoyqIU71jKI8z+RkBFCCOCXsxm8vzWOpMz8qWkeq+XAtLAAAn1kEOf9kpARQoi/pWuNzNuRwKaINHObrY0VL3b2ZnR7L9QqOaspLgkZIYT4l98jNUzdHMu1FIO57aEAO2YMCKBhVRnEWRwSMkIIcRtavYlFuxP5+mAKpr+PktZWMKqdJ+Oe8MHBVo45RSEhI4QQd3H2upZJm2K5FKczt1X3VDM1LIDHg5wsWFn5ICEjhBD3YDAqrDyQzOI9Sehz8w+ZA5q7MbGXH64OKgtWV7ZJyAghRBFFJeqYsjmWY1e15jZvFxXvhfjTtZGrBSsruyRkhBCiGEwmhQ0RaczbkYBGlz+Is0tDF94L8cPHVQZx/pOEjBBC3Ie4dAPTv49j/4Usc5urvTUTe/kR1txNpqb5m4SMEELcJ0VR2Hkmk5lb40jR5A/ibBXkyNTQAGp42VqwurJBQkYIIf6jNE0us39MYOuJdHObvdqK8V18GBHsiU0lHsQpISOEECXk4KUspm2J42Za/iDOhlXtmT4ggPoB9haszHIkZIQQogRpdCYW7kpgzeFUbh1dbazhmfZevNDZGzt15TpWScgIIUQpOBmdzeTNsUQm6M1tgT62TAsN4LFARwtW9mBJyAghRCnR55pYGp7MkvAkcvOfC2BIK3cm9PDF2b7iD+KUkBFCiFJ2OV7H5E2xnLqWP4jT382GSf386djApdjb++OPPwgPD0ev19OrVy8effTRIvXbsGEDERERvPLKK1SpUgXIe0Ju9+7dHDlyBGdnZ3r06EH9+vUL9IuMjGTTpk1kZ2fzxBNP0K5duyLXWuwjs06nIzLyMhcvXsBgMNy7w99u3rzJiePH0WqzC7Tn5ORw8eIFoqOvYjQa79BbCCHKrzp+dqx6oSZv9/HDwTbvSbO49FzGfXOdiWtvkJyVW+RtLViwgNDQULKystBqtfTu3Ztvv/32nv3OnTvHu+++y+LFi0lKSjK3d+jQgS+++AKVSsXly5cJDg5m/fr15uURERE8/vjjXLlyBaPRyJAhQ/jss8+KXG+xzmTOnj3Dgo/n4ebmjslkIisri3fefY/AwNp37afRZPHahFdJTExg7ryPzeuvWb2KvXt34+fnR0pKKjY2Nrzx5tvUqFGjSPXImYwQory5kapn2pY4Dv2lMbe5Oah4q48ffZu53nUQp1arpWrVqixdupSwsDAA1qxZw9tvv83ly5extb39uByDwUDnzp0ZP348Y8aM4bfffqNx48YAnD17lkaNGpnXnTx5Mj///DNHjhwBoEePHtSpU8ccLBs3buSll14iMjISF5d7n4UV+chsMplY8PE8unbtzrz5C/h4wULatmvHwk8W3LPvsqVfERzctlC7s7Mziz//kpmz5rBo8RfUqFmTpUuXFLUkIYQod6p62LJkdHVmDaqC298Ta6Zrjby94SbPr7jGzdQ7XyG6ceMGer3eHBAATZo0ISUlhePHj9+x3+zZs2nSpAkdO3YstOyfAQOgVqtRqfLq0ul0HD58mP79+5uX9+3bF51OZw6heylyyCQkJJCamkrbf1yLa9++AzEx0cTExNyx3x9/HCEuLpbeffoUWtavfyh2dnkfa6pSqaj/UH1SklOKWpIQQpRLVlZWhDzqxrYJtenZOH9izUN/aQhZEMnqQykYTYUvMlWvXh1HR0cOHTpkbjtw4ACQd0vidk6cOMG6deuYOXPmPeuKjo7mq6++4vnnnwfyjvtGo5GAgADzOnZ2dnh6ehIbG1ukfbUp0lpgPi1KSkqiatVqef9PTAQg9uaN217iysjIYNnSr5g0eSpWVnfPM4PBwP794TzWvPld1/nnfSBFyZuczmQy3amLEEKUWR6O1swZEkDvJi5M/yGehIxctHqFWdvj+fFUOvOGVsHfLX/CTbVazcyZM5k4cSJ79uxBURSuXbuGSqVCpVJhMpkK3DrQ6XQ899xzzJ8/H1dXV7Ra7e3KAPLOkkJCQnjqqacYNWoUgPnS3b+PsUajschzsxU5ZJycnOjYsRNLvvicgYMGoygK27dtxdramtw73LD/csnndO/Rg2rVqpGamnrHbRuNRhZ8PA+1Ws1TTw2/43qbN21k/fp15tf29nbMmTOH+LjrRd0NIYQoc+q5wpIhNiw7rLD9bN7xNCldhzbtJrGaggfznj26UP+hOpw4cRJbW1vq1B1Ov34D8PJ0I/ZmDFWr1TKvu23bNhISEti9eze7d+8mOzvvwasFCxYwYMAAevfuDcCVK1fo06cPgwYNYtq0aeb+fn5+2Nracu3aNR555BEAsrKySEtLo3r16kXatyKHDMBL4/6PX3b9zMkTx1Gr1Yx5dixTJr+Hj49PoXVPnTzJuXNn6dipMyeOHydLkzdT6cULF7C1tTWfDRkMBubN+4jMjAwmT5lmvnx2O2EDBtI3pJ/5taKYSEtNws+/mtz4F0KUe7NqwcCr2Uz7Pp73QvwIrFl40ObJkyd5vHVb2gS3x2Qy8cILLxAcHEyb4PZA3o351NRUxo4dyyOPPMIbb7xh7puZmQlAQEAA7u7uAJw/f56QkBCeffZZ3nzzzQJfS61W88QTT7BmzRp69eoFwNq1a3Fzc6NVq1ZF2qdiPV2m1WpxcHAwv9627Qe2b9vK4s+/RKVSERMTg16vo06dupw5c5rvt2w2r5ubm8vZs2d46KH6dO78BF26dkOn0zH7w5mYFBNvvfUu9vbFm9tHni4TQlREuUbljpNqrlmzhs8//5wmTZpw/PhxdDod33//vfmWxbhx44iOjmb79u2F+sbHxxMUFGR+ukyn01GvXj0cHR3p27eveT21Ws0HH3wAwKVLl+jWrRsNGjTA19eXHTt28NlnnzF48OAi7UuxQmbdum/R6/XUrVOXvy7/xc87d/D6G2/RtGkzABYv+pSExASmTp1RqG9qairPjnna/Aiz0WhkyuT3SExMZMyzY1Hb5F13tLa2pknTpkWqR0JGCFEZnT9/nsOHD1O1alU6d+6MWp1/32bfvn1kZGTQr1+/Qv2ys7NZsWIFgwcPxsfHB4PBwJdfflloPZVKxQsvvGB+nZqays8//0x2djYdOnQgKCioyLUWK2SMRiM7d/7EubNn8fLyomvX7tSoWdO8fPu2raSlpzF8+MhCfbOysvh4/lzGPvcC/v7+6PV6Zn9Y+GkHGxsb3n7nvSLVIyEjhBBlm0wrI4QQotTIkVkIIUSpkZARQghRaiRkhBBClJpijZMpa27dTpIR/0IIkc/KyqrII/JLW4UIGRnxL4QQ+QKq1CgzIVPuny4zmUz3ldpabTbPjX2GL79ajoNDxf4oVNnXikn2tWIqiX2VM5kSYm1tfd+PLltZWZOTo8PK6v63UV7IvlZMsq8VU0Xb1/K/B0IIIcosCRkhhBClptKGjFqtZvDgJwvM+VNRyb5WTLKvFVNF29dyfeNfCCFE2VZpz2SEEEKUPgkZIYQQpabShoxer7d0CQ+MwWCwdAkPlF6vR6PJojJcCa5MP9uKtq8mkwmNJsv87277p9PpHmBlJatcj5O5H3/+eY7PP19EXGwsjo6ODBg4mJCQwh/uU97pdDp++nE7u3btJCUlBWdnF3r16s2AgYMsXVqp0mq1vDbhFeLj41ixchWurq6WLqlUnDt3lq9XLufq1au4ubkxbPhIOnbsZOmySpzRaGTlimWEh+/DYDDg7OzMgIGD6Nmzt6VL+88SExOZ+NqrAOTkaBn9zLP06lVwv44dO8rSr5aQlJSEi4sLw4aP5Iknulii3PunVCIZGRnKiOFDlY0b1isGg0E5e/aMMvTJQUpExB+WLq3EnT17Vlm16mslPj7+79dnlKeGDlb27dtr4cpK1xefL1JmfzhTCQsNUdLT0y1dTqmIvHxZGfrkIOWXXT8rOp1OycrKVFauWG7pskrFj9u3Kc+MHqncuHFdURRFOX7smDJwQH/l4oULFq6sZL304nPKjz9uL9CWkJCgPDlkoLJz509Kbm6uEhHxhzJ4UJhy4cJ5C1V5fyrV5bIjvx9GrVbTPzQMGxsbGjZsROs2wezZ/YulSytxDRs2ZPjwkfj6+v79uhH16j1E5OW/LFxZ6Tl58gSXL1+mf/8wS5dSqtavX0ebNsF06doNKysrnJycGfX0aEuXVSpu3LxBvXoPUaVKVQCaPfoo7u4e3Lh5w8KVlb4DB/bj4+NL9+49UalUNG/egiZNmrJ3z25Ll1YslSpkomOiqVGjJiqVytwWWCuQmJhoC1b1YGRmZnL58mXq1K1n6VJKhUajYcmSzxk37v+wVlXsX+s//zyHk5MT4156nuHDnuSF58dy4MCvli6rVLRt246LFy9w6tRJUlNT2bXrZxTFRLNmj1q6tFIXEx1NrcDAAm21AgOJji5fx6tKdU9Gl5ODg2PBCeccHR3JycmxUEUPhl6v56OPPqR+/fq0a9fe0uWUimVLv6RDh47UCgzkcgU+W1MUBY1Gw6+//sq0aTOoXqMGBw/8ysJPPqZ6teqFDkrl3UMP1adT5yeYMX0qNjY2WFlZ8fzzL+Lu7m7p0kpdjk6Hu5t7gbbyeLyq2H/y/YuDoyPZGk2BtiyNBkfHijura05ODjNnzsBWbcvrb7xVISbc+7fjx49x6dIlevbohUaTZX4TZmdnV7gnkqysrHBwcKBtu3bUqFkTKysr2rXvQJWqVTl16qSlyytxX69czulTp1iyZCnrvtvI+x/MYuXK5Rw6eMDSpZU6RwcHNJqsAm2arPJ3vKp4R5y7qF07iCtXIgsceC5dvEBg7SALVlV6NBoN06dNwcHegTffegdbW1tLl1Qqbt64QXp6OuPHv8QLzz/HrJnvA/D6xFfZU86uXxdFUFAdcv8VnrkGAzbqindh4uzZs7Rs2Qovb28gb9/r12/A2XNnLVxZ6atdO4hLly4VeBT/4qUL1C5nx6tKFTKtWj2Ovb09y5cvJT4+nvB9ezl6NIKePXtZurQSl5mZwZTJ7+Hi4sKLL41Hr9eh0WSV6+ft76RP3xBWrf7W/G/a9LyQWbR4CT169LRwdSWvV+8+/Prrfo4ejSA5OZlNGzeQlpZGixYtLV1aiavfoAH794dz4cJ50tLSOHLkd86cOU39+g0sXVqJ0Gg0aDRZmBTF/B69NYavfYeO5ORoWbNmFQkJ8fz0049cuniRbt26W7jq4ql0c5ddv36N5cuXcjUqCg8PTwYMHESbNsGWLqvE/f77YRZ99mmh9pYtW/J/L7/y4At6gKKirjB50nss/nwJLi4uli6nVOzbt5dtW38gIyOd6tVrMHToMOo99JClyypxOp2O775by7FjR8nKzMTLy4tOnZ+oEONkAMY++0yheyz9+vdn4MDBAERGXubrlSu4du0aPj4+DB06jGaPlq+HHipdyAghhHhwKtXlMiGEEA+WhIwQQohSIyEjhBCi1EjICCGEKDUSMkIIIUqNhIwQQohSIyEjhBCi1EjICCGEKDUSMkIUw5EjvzPupectXYYQ5YaEjBBCiFIj08oI8S9xcXF8ueRzLl26iKOjI506P8HgwU9y4/p1Xn315QLrTp4yjSZNmnLq1EnWfruGa9di8PDwpG9ICN27503Oefi3Q6xZs5o2wcHsD9+HXm+gbbt2jBz5NGq12hK7KMQDU/HmBhfiP1q9+htcXV35/IuvMBqN7Nn9C1euRFK3bj3eePNtvvl6BYsWLzGvf/nyX8yZPYvx41+m2aOPce1aDHNmz8LT08s8M3Js7E3SUlOZO28BqSkpfPTRhzg6OjJ06DBL7aYQD4RcLhPiX2zVauzs7LC3t8fd3Z0BAwdR9y4fW71jx0906NCJ1m2Csbe3p27devTs1Ztf94fnb9PWljHPPoeLiws1atZk0KAhhO/b+wD2RgjLkjMZIf5l9DNj2LhhPe++8xaOjo60aNmS7t17YmNz+7dLQnw8f/55jp9/3lGgvW7duub/u7m5Y2dnZ37t7x9ASkoKiqJgZWVVOjsiRBkgISPEv7i4uDL6mWcBSE1J4cMPZ6LX6QkNG4D1bQLBx8eHvn378fToZ+64zfT0NHQ6nTlo4uPj8PT0lIARFZ5cLhPiXxYt+pQTJ46j1WZjMplQFJP5g6U8PDxJS0sjJSXZvH6Pnr3Ys+cXDh06iFabTUpKMj//vIPt27eZ19Hr9axYvozMzEyuXYthw4b1dOzY+YHvmxAPmjxdJsS/XLlyhW+/Xc2F83+iVtvSvHlznhkzFgcHBxRF4bPPFvLHkSNkZ2vMT5edOXOa79atJTo6GgcHe1q0bMWTTw7FxcW1wNNl4fv2YjAYCA5ux6inR8vTZaLCk5ARopQd/u0Q365dw6efLrZ0KUI8cHK5TAghRKmRkBFCCFFq5HKZEEKIUiNnMkIIIUqNhIwQQohSIyEjhBCi1EjICCGEKDUSMkIIIUqNhIwQQohSIyEjhBCi1EjICCGEKDUSMkIIIUrN/wPn8ZzRSyL6zgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "model.safetensors random_states_1.pkl random_states_6.pkl\n", + "optimizer.bin\t random_states_2.pkl random_states_7.pkl\n", + "random_states_0.pkl random_states_3.pkl random_states_8.pkl\n", + "random_states_10.pkl random_states_4.pkl random_states_9.pkl\n", + "random_states_11.pkl random_states_5.pkl training_state.json\n" + ] + } + ], + "source": [ + "H.plot_logs({\"smoke projector\": SMOKE_LOG})\n", + "plt.show()\n", + "!ls {SMOKE_DIR}/checkpoints/step_final" + ] + }, + { + "cell_type": "markdown", + "id": "cell-061", + "metadata": {}, + "source": [ + "## 12. Submitting all four stages with `qsub`\n", + "\n", + "Each stage's output path is **decided here**, from `RUN_TAG`, so the next stage can be told where its input will be *before* it exists. PBS dependencies (`-W depend=afterok:`) hold each job until the previous one succeeds.\n", + "\n", + "```\n", + "projector (debug, 1h) ──afterok──► SFT (prod) ──afterok──► GRPO (prod, 1 tile) ──afterok──► eval SFT+GRPO\n", + " └──afterok──► eval SFT\n", + "```\n", + "\n", + "Step counts here are **demo-sized** (a few hundred steps); the YAML defaults are 5k / 10k / 1k. Set `SUBMIT = True` to actually submit. Otherwise the cell prints the `qsub` commands.\n", + "\n", + "> If `qsub` isn't available on this compute node, copy the printed commands and run them from a login node, from `REPO`." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "cell-062", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "qsub -v PRISM_DATA_ROOT=${PRISM_DATA_ROOT},PRISM_OUTPUT_ROOT=${PRISM_OUTPUT_ROOT},PRISM_CKPT_ROOT=${PRISM_CKPT_ROOT},PRISM_VENV=${PRISM_VENV_ROOT}/bioreason-fw2026.1,WANDB_MODE=offline,MAX_SEQ_LENGTH=4096,RUN_TAG=demo-20261011-2324,HYDRA_RUN_DIR=${PRISM_CKPT_ROOT}/BIOREASON-PROJECTOR-KEGG/demo-20261011-2324,MAX_STEPS=100,SAVE_EVERY=50 -o ${PRISM_OUTPUT_ROOT}/demo_logs/demo-20261011-2324/projector.log ${PRISM_OUTPUT_ROOT}/demo_scripts/demo-20261011-2324/projector.sh\n", + "qsub -v PRISM_DATA_ROOT=${PRISM_DATA_ROOT},PRISM_OUTPUT_ROOT=${PRISM_OUTPUT_ROOT},PRISM_CKPT_ROOT=${PRISM_CKPT_ROOT},PRISM_VENV=${PRISM_VENV_ROOT}/bioreason-fw2026.1,WANDB_MODE=offline,MAX_SEQ_LENGTH=4096,RUN_TAG=demo-20261011-2324,HYDRA_RUN_DIR=${PRISM_CKPT_ROOT}/BIOREASON-SFT-KEGG/demo-20261011-2324,STAGE1_CKPT=${PRISM_CKPT_ROOT}/BIOREASON-PROJECTOR-KEGG/demo-20261011-2324/checkpoints/step_final,MAX_STEPS=500 -W depend=afterok:DRYRUN -o ${PRISM_OUTPUT_ROOT}/demo_logs/demo-20261011-2324/sft.log -l walltime=02:00:00 ${PRISM_OUTPUT_ROOT}/demo_scripts/demo-20261011-2324/sft.sh\n", + "qsub -v PRISM_DATA_ROOT=${PRISM_DATA_ROOT},PRISM_OUTPUT_ROOT=${PRISM_OUTPUT_ROOT},PRISM_CKPT_ROOT=${PRISM_CKPT_ROOT},PRISM_VENV=${PRISM_VENV_ROOT}/bioreason-fw2026.1,WANDB_MODE=offline,MAX_SEQ_LENGTH=4096,RUN_TAG=demo-20261011-2324,HYDRA_RUN_DIR=${PRISM_CKPT_ROOT}/BIOREASON-GRPO-KEGG/demo-20261011-2324,STAGE1_CKPT=${PRISM_CKPT_ROOT}/BIOREASON-SFT-KEGG/demo-20261011-2324/checkpoints/step_final,MAX_STEPS=100,SAVE_EVERY=50 -W depend=afterok:DRYRUN -o ${PRISM_OUTPUT_ROOT}/demo_logs/demo-20261011-2324/grpo.log ${REPO}/applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh\n", + "qsub -v PRISM_DATA_ROOT=${PRISM_DATA_ROOT},PRISM_OUTPUT_ROOT=${PRISM_OUTPUT_ROOT},PRISM_CKPT_ROOT=${PRISM_CKPT_ROOT},PRISM_VENV=${PRISM_VENV_ROOT}/bioreason-fw2026.1,WANDB_MODE=offline,MAX_SEQ_LENGTH=4096,RUN_TAG=demo-20261011-2324,CKPT=${PRISM_CKPT_ROOT}/BIOREASON-SFT-KEGG/demo-20261011-2324/checkpoints/step_final,OUTPUT=${PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/demo-20261011-2324/sft.json -W depend=afterok:DRYRUN -o ${PRISM_OUTPUT_ROOT}/demo_logs/demo-20261011-2324/eval-sft.log ${REPO}/applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh\n", + "qsub -v PRISM_DATA_ROOT=${PRISM_DATA_ROOT},PRISM_OUTPUT_ROOT=${PRISM_OUTPUT_ROOT},PRISM_CKPT_ROOT=${PRISM_CKPT_ROOT},PRISM_VENV=${PRISM_VENV_ROOT}/bioreason-fw2026.1,WANDB_MODE=offline,MAX_SEQ_LENGTH=4096,RUN_TAG=demo-20261011-2324,CKPT=${PRISM_CKPT_ROOT}/BIOREASON-SFT-KEGG/demo-20261011-2324/checkpoints/step_final,GRPO_ADAPTER=${PRISM_CKPT_ROOT}/BIOREASON-GRPO-KEGG/demo-20261011-2324/checkpoints/step_final/lora_adapter.pt,OUTPUT=${PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/demo-20261011-2324/sft+grpo.json -W depend=afterok:DRYRUN -o ${PRISM_OUTPUT_ROOT}/demo_logs/demo-20261011-2324/eval-sft+grpo.log ${REPO}/applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh\n" + ] + }, + { + "data": { + "text/plain": [ + "{'projector': 'DRYRUN',\n", + " 'sft': 'DRYRUN',\n", + " 'grpo': 'DRYRUN',\n", + " 'eval-sft': 'DRYRUN',\n", + " 'eval-sft+grpo': 'DRYRUN'}" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "SUBMIT = False # <- flip to submit for real\n", + "\n", + "RUN = {s: f\"{PRISM_CKPT_ROOT}/BIOREASON-{s.upper()}-KEGG/{RUN_TAG}\" for s in (\"projector\", \"sft\", \"grpo\")}\n", + "CKPT_PROJ = f\"{RUN['projector']}/checkpoints/step_final\"\n", + "CKPT_SFT = f\"{RUN['sft']}/checkpoints/step_final\"\n", + "GRPO_ADPT = f\"{RUN['grpo']}/checkpoints/step_final/lora_adapter.pt\"\n", + "EVAL_OUT = {k: f\"{PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/{RUN_TAG}/{k}.json\" for k in (\"sft\", \"sft+grpo\")}\n", + "LOGS = {k: f\"{PRISM_OUTPUT_ROOT}/demo_logs/{RUN_TAG}/{k}.log\"\n", + " for k in (\"projector\", \"sft\", \"grpo\", \"eval-sft\", \"eval-sft+grpo\")}\n", + "common = {k: ENV[k] for k in (\"PRISM_DATA_ROOT\", \"PRISM_OUTPUT_ROOT\", \"PRISM_CKPT_ROOT\", \"PRISM_VENV\", \"WANDB_MODE\", \"MAX_SEQ_LENGTH\")} | {\"RUN_TAG\": RUN_TAG}\n", + "\n", + "JOBS = {}\n", + "def q(*a, **k):\n", + " return H.qsub(*a, cwd=str(REPO), dry_run=not SUBMIT, account=PROJECT, **k)\n", + "\n", + "JOBS[\"projector\"] = q(PROJ_SH, {**common, \"HYDRA_RUN_DIR\": RUN[\"projector\"], \"MAX_STEPS\": 100, \"SAVE_EVERY\": 50},\n", + " log=LOGS[\"projector\"])\n", + "JOBS[\"sft\"] = q(SFT_SH, {**common, \"HYDRA_RUN_DIR\": RUN[\"sft\"], \"STAGE1_CKPT\": CKPT_PROJ, \"MAX_STEPS\": 500},\n", + " depend=JOBS[\"projector\"], log=LOGS[\"sft\"], walltime=\"02:00:00\")\n", + "JOBS[\"grpo\"] = q(GRPO_SH, {**common, \"HYDRA_RUN_DIR\": RUN[\"grpo\"], \"STAGE1_CKPT\": CKPT_SFT,\n", + " \"MAX_STEPS\": 100, \"SAVE_EVERY\": 50},\n", + " depend=JOBS[\"sft\"], log=LOGS[\"grpo\"])\n", + "JOBS[\"eval-sft\"] = q(EVAL_SH, {**common, \"CKPT\": CKPT_SFT, \"OUTPUT\": EVAL_OUT[\"sft\"]},\n", + " depend=JOBS[\"sft\"], log=LOGS[\"eval-sft\"])\n", + "JOBS[\"eval-sft+grpo\"] = q(EVAL_SH, {**common, \"CKPT\": CKPT_SFT, \"GRPO_ADAPTER\": GRPO_ADPT, \"OUTPUT\": EVAL_OUT[\"sft+grpo\"]},\n", + " depend=JOBS[\"grpo\"], log=LOGS[\"eval-sft+grpo\"])\n", + "JOBS" + ] + }, + { + "cell_type": "markdown", + "id": "cell-063", + "metadata": {}, + "source": [ + "## 13. Monitoring and results\n", + "\n", + "Press **Refresh** at any time. It shows `qstat` for the chain (including finished jobs), the loss (and GRPO reward) curves parsed from each log, and the tail of each log. Each run gets its own panel, so there are no shared or dual axes." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "cell-064", + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "a3566be993b64c92824963b746743a46", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "VBox(children=(Button(description='Refresh', icon='refresh', style=ButtonStyle()), Output()))" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "H.monitor(list(JOBS.values()), {k: LOGS[k] for k in (\"projector\", \"sft\", \"grpo\")})" + ] + }, + { + "cell_type": "markdown", + "id": "cell-065", + "metadata": {}, + "source": [ + "### Results\n", + "\n", + "Once the eval jobs finish, compare SFT with SFT + GRPO on the KEGG test split (146 variants). `format_rate` is the share of completions with a well-formed `…`, which GRPO's format rewards push up." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "cell-066", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "no eval results yet\n" + ] + } + ], + "source": [ + "import json\n", + "\n", + "res = {k: json.load(open(p)) for k, p in EVAL_OUT.items() if Path(p).exists()}\n", + "if not res:\n", + " print(\"no eval results yet\")\n", + "else:\n", + " display(pd.DataFrame({k: {\"accuracy\": r[\"accuracy\"], \"format_rate\": r[\"format_rate\"], \"n\": r[\"n\"]}\n", + " for k, r in res.items()}).T)\n", + " last = list(res)[-1]\n", + " per = pd.DataFrame(res[last][\"per_label\"]).T.assign(acc=lambda d: d.correct / d.n).sort_values(\"n\", ascending=False)\n", + " display(per.head(15))" + ] + }, + { + "cell_type": "markdown", + "id": "cell-067", + "metadata": {}, + "source": [ + "### Quick in-process check\n", + "\n", + "`eval_kegg_aurora.py` is importable, so you can score a checkpoint right here on one tile. It uses the same weight loading and prompt rendering as the batch eval." + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "cell-068", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2026-10-11 23:28:34,630 - src.model - INFO - Initializing HF Backbone: allenai/OLMo-1B-0724-hf...\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "checkpoint: ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "80b769076d3940b988e929fe893df6a8", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/114 [00:00 768\n", + "2026-10-11 23:28:38,927 - src.encoders.dna - INFO - DNA Encoder loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e (model_hidden=2560, output=768, tokens_per_instance=1024)\n", + "2026-10-11 23:28:38,969 - src.modules.projector - INFO - Modality embedding: scale=0.0055, expected_norm=0.2489\n", + "2026-10-11 23:28:38,969 - src.modules.projector - INFO - ModalityProjector initialized: norm_mode=l2_token, target_norm=0.25, embed_pos=after_norm\n", + "2026-10-11 23:28:38,970 - src.model - INFO - Freezing Modality Encoders...\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[Resume] Loading model weights from ${PRISM_CKPT_ROOT}/DEMO-SMOKE-PROJECTOR/demo-20261011-2324/checkpoints/step_final/model.safetensors\n", + "[Resume] Loaded 646 weight tensors successfully\n", + "[Resume] Checkpoint was saved at step final\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[INFO] DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "[INFO] DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "2026-10-11 23:28:53,483 - src.data.multimodal - INFO - DNA tokenizer loaded: ${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species/snapshots/b746b125aacd1b0970c05a32fd71ba726754542e\n", + "[INFO] dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "[INFO] dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "2026-10-11 23:28:53,485 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason <- {'skip': False, 'preferred_source': 'local'}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "2026-10-11 23:28:53,486 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason_variant_effect_coding <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "[INFO] dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "2026-10-11 23:28:53,487 - src.data.multimodal - INFO - dataset_overrides applied: dna_bioreason_variant_effect_non_snv <- {'skip': True}\n", + "[INFO] dataset_overrides applied: ts_qa <- {'skip': True}\n", + "[INFO] dataset_overrides applied: ts_qa <- {'skip': True}\n", + "2026-10-11 23:28:53,487 - src.data.multimodal - INFO - dataset_overrides applied: ts_qa <- {'skip': True}\n", + "2026-10-11 23:28:53,545 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/google/tapas-base/resolve/main/tokenizer_config.json \"HTTP/1.1 307 Temporary Redirect\"\n", + "2026-10-11 23:28:53,551 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/google/tapas-base/00456266840bb0a319cd6748ebf7da3caf98816b/tokenizer_config.json \"HTTP/1.1 200 OK\"\n", + "2026-10-11 23:28:53,600 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/google/tapas-base/tree/main/additional_chat_templates?recursive=false&expand=false \"HTTP/1.1 404 Not Found\"\n", + "2026-10-11 23:28:53,658 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/google/tapas-base/tree/main?recursive=true&expand=false \"HTTP/1.1 200 OK\"\n", + "[INFO] Initializing Streaming Datasets for zone_a...\n", + "[INFO] Initializing Streaming Datasets for zone_a...\n", + "2026-10-11 23:28:53,685 - src.data.multimodal - INFO - Initializing Streaming Datasets for zone_a...\n", + "[WARNING] Skipping PixMo-Cap (Configured skip)\n", + "[WARNING] Skipping PixMo-Cap (Configured skip)\n", + "2026-10-11 23:28:53,686 - src.data.multimodal - WARNING - Skipping PixMo-Cap (Configured skip)\n", + "[WARNING] Skipping Molmo2-Cap (Configured skip)\n", + "[WARNING] Skipping Molmo2-Cap (Configured skip)\n", + "2026-10-11 23:28:53,687 - src.data.multimodal - WARNING - Skipping Molmo2-Cap (Configured skip)\n", + "[WARNING] Skipping PixMo-Points (Configured skip)\n", + "[WARNING] Skipping PixMo-Points (Configured skip)\n", + "2026-10-11 23:28:53,688 - src.data.multimodal - WARNING - Skipping PixMo-Points (Configured skip)\n", + "[WARNING] Skipping PixMo-Count (Configured skip)\n", + "[WARNING] Skipping PixMo-Count (Configured skip)\n", + "2026-10-11 23:28:53,688 - src.data.multimodal - WARNING - Skipping PixMo-Count (Configured skip)\n", + "[WARNING] Skipping Molmo2-VideoPoint (Configured skip)\n", + "[WARNING] Skipping Molmo2-VideoPoint (Configured skip)\n", + "2026-10-11 23:28:53,689 - src.data.multimodal - WARNING - Skipping Molmo2-VideoPoint (Configured skip)\n", + "[WARNING] Skipping The Well (Turbulence) (Configured skip)\n", + "[WARNING] Skipping The Well (Turbulence) (Configured skip)\n", + "2026-10-11 23:28:53,690 - src.data.multimodal - WARNING - Skipping The Well (Turbulence) (Configured skip)\n", + "[WARNING] Skipping The Well (MHD 64^3) (Configured skip)\n", + "[WARNING] Skipping The Well (MHD 64^3) (Configured skip)\n", + "2026-10-11 23:28:53,690 - src.data.multimodal - WARNING - Skipping The Well (MHD 64^3) (Configured skip)\n", + "[WARNING] Skipping The Well (Supernova) (Configured skip)\n", + "[WARNING] Skipping The Well (Supernova) (Configured skip)\n", + "2026-10-11 23:28:53,691 - src.data.multimodal - WARNING - Skipping The Well (Supernova) (Configured skip)\n", + "[WARNING] Skipping Berea Sandstone (Real) (Configured skip)\n", + "[WARNING] Skipping Berea Sandstone (Real) (Configured skip)\n", + "2026-10-11 23:28:53,692 - src.data.multimodal - WARNING - Skipping Berea Sandstone (Real) (Configured skip)\n", + "[WARNING] Skipping PDEBench (Problem Statements) (Configured skip)\n", + "[WARNING] Skipping PDEBench (Problem Statements) (Configured skip)\n", + "2026-10-11 23:28:53,692 - src.data.multimodal - WARNING - Skipping PDEBench (Problem Statements) (Configured skip)\n", + "[WARNING] Skipping Tulu 3 SFT (Configured skip)\n", + "[WARNING] Skipping Tulu 3 SFT (Configured skip)\n", + "2026-10-11 23:28:53,693 - src.data.multimodal - WARNING - Skipping Tulu 3 SFT (Configured skip)\n", + "[WARNING] Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "[WARNING] Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "2026-10-11 23:28:53,693 - src.data.multimodal - WARNING - Skipping ChatTS Align (Synthetic) (Configured skip)\n", + "[WARNING] Skipping TS Reasoning (Proxy) (Configured skip)\n", + "[WARNING] Skipping TS Reasoning (Proxy) (Configured skip)\n", + "2026-10-11 23:28:53,694 - src.data.multimodal - WARNING - Skipping TS Reasoning (Proxy) (Configured skip)\n", + "[WARNING] Skipping ChatTime (Verified Available)\n", + "[WARNING] Skipping ChatTime (Verified Available)\n", + "2026-10-11 23:28:53,694 - src.data.multimodal - WARNING - Skipping ChatTime (Verified Available)\n", + "[WARNING] Skipping Time-MMD (Economy) (Configured skip)\n", + "[WARNING] Skipping Time-MMD (Economy) (Configured skip)\n", + "2026-10-11 23:28:53,695 - src.data.multimodal - WARNING - Skipping Time-MMD (Economy) (Configured skip)\n", + "[WARNING] Skipping TableLLM (Reasoning) (Configured skip)\n", + "[WARNING] Skipping TableLLM (Reasoning) (Configured skip)\n", + "2026-10-11 23:28:53,695 - src.data.multimodal - WARNING - Skipping TableLLM (Reasoning) (Configured skip)\n", + "[WARNING] Skipping TableInstruct (Gen) (Configured skip)\n", + "[WARNING] Skipping TableInstruct (Gen) (Configured skip)\n", + "2026-10-11 23:28:53,696 - src.data.multimodal - WARNING - Skipping TableInstruct (Gen) (Configured skip)\n", + "[WARNING] Skipping UniTabE (Structure) (Configured skip)\n", + "[WARNING] Skipping UniTabE (Structure) (Configured skip)\n", + "2026-10-11 23:28:53,697 - src.data.multimodal - WARNING - Skipping UniTabE (Structure) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20 (Global) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20 (Global) (Configured skip)\n", + "2026-10-11 23:28:53,698 - src.data.multimodal - WARNING - Skipping ChEBI-20 (Global) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "[WARNING] Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "2026-10-11 23:28:53,698 - src.data.multimodal - WARNING - Skipping ChEBI-20-MM (Grounding) (Configured skip)\n", + "[WARNING] Skipping GraphInstruct (Configured skip)\n", + "[WARNING] Skipping GraphInstruct (Configured skip)\n", + "2026-10-11 23:28:53,699 - src.data.multimodal - WARNING - Skipping GraphInstruct (Configured skip)\n", + "[WARNING] Skipping Matbench (Crystal) (Configured skip)\n", + "[WARNING] Skipping Matbench (Crystal) (Configured skip)\n", + "2026-10-11 23:28:53,699 - src.data.multimodal - WARNING - Skipping Matbench (Crystal) (Configured skip)\n", + "[WARNING] Skipping Verilog (Circuit) (Configured skip)\n", + "[WARNING] Skipping Verilog (Circuit) (Configured skip)\n", + "2026-10-11 23:28:53,700 - src.data.multimodal - WARNING - Skipping Verilog (Circuit) (Configured skip)\n", + "[INFO] [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "[INFO] [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "2026-10-11 23:28:53,701 - src.data.multimodal - INFO - [DATA CONFIG] Dataset: DNA sequences | Priority: LOCAL\n", + "[INFO] [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n", + "[INFO] [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n", + "2026-10-11 23:28:53,704 - src.data.multimodal - INFO - [DATA LOAD] Source: LOCAL (Raw Arrow) | Path: ${PRISM_DATA_ROOT}/dna_local_shards/kegg (1 shards)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "DEBUG: Loading 1 Arrow files directly...\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[WARNING] Skipping Variant Effect (Coding) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Coding) (Configured skip)\n", + "2026-10-11 23:28:53,808 - src.data.multimodal - WARNING - Skipping Variant Effect (Coding) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "[WARNING] Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "2026-10-11 23:28:53,808 - src.data.multimodal - WARNING - Skipping Variant Effect (Non-SNV) (Configured skip)\n", + "[INFO] [Init] Pre-Calculated Active Modalities: []\n", + "[INFO] [Init] Pre-Calculated Active Modalities: []\n", + "2026-10-11 23:28:53,809 - src.data.multimodal - INFO - [Init] Pre-Calculated Active Modalities: []\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[eval] stop token: '<|endoftext|>' (id 50279), pad id 1, max_new_tokens=128\n", + "[eval] 1/5 acc=0.000 (4s) gt='amyotrophic lateral sclerosis' pred='techtechtechtechtechtechtechtechtechtechtechtechtechtechtech'\n", + "[eval] 2/5 acc=0.000 (7s) gt='clear cell renal cell carcinoma' pred='ttttttttttttttttttttttttttttttt'\n", + "[eval] 3/5 acc=0.000 (9s) gt='melanoma' pred='ttttttttttttttttttttttttttttttt'\n", + "[eval] 4/5 acc=0.000 (11s) gt='alzheimers disease' pred='ttttttttttttttttttttttttttttttt'\n", + "[eval] 5/5 acc=0.000 (14s) gt='spinocerebellar ataxia' pred=''\n", + "{'accuracy': 0.0, 'format_rate': 0.0, 'n': 5}\n", + "TTTTTTt\n", + "Tt\n", + "Tt-Tttttt\n", + "Tttech\n", + "Tttechttechttechttechttechttechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtechtech\n", + "\n", + "\n", + "Answer: techtechtechtechtechtechtechtechtechtechtechtechtechtechte\n" + ] + } + ], + "source": [ + "from eval_kegg_aurora import evaluate, load_eval_model\n", + "\n", + "ckpt = CKPT_SFT if Path(CKPT_SFT).exists() else (f\"{SMOKE_DIR}/checkpoints/step_final\" if Path(SMOKE_DIR).exists() else None)\n", + "print(\"checkpoint:\", ckpt)\n", + "m, ecfg = load_eval_model(ckpt, overrides=[f\"+model.dna_model_name={DNA_ENCODER}\"], device=dev)\n", + "quick = evaluate(m, ecfg, DS[\"kegg (test)\"], limit=5, max_new_tokens=128)\n", + "print({k: quick[k] for k in (\"accuracy\", \"format_rate\", \"n\")})\n", + "print(quick[\"records\"][0][\"completion\"][:500])\n", + "del m\n", + "torch.xpu.empty_cache()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-069", + "metadata": {}, + "source": [ + "## 14. Troubleshooting\n", + "\n", + "| Symptom | Cause / fix |\n", + "|---|---|\n", + "| `SafetensorError … Disk quota exceeded` at the first save | Checkpoints went to `outputs/` under the home checkout. Pass `hydra.run.dir=` on `/flare` (all submit scripts now do, via `HYDRA_RUN_DIR`). |\n", + "| `cannot import name 'find_pruneable_heads_and_indices'` or `EsmConfig has no attribute is_decoder` | An NT-**v2** checkpoint under transformers 5. Use an NT v1 checkpoint (e.g. 2.5B multi-species) on `frameworks/2026.1.0`. |\n", + "| DNA `input_ids` clamped / out-of-range warnings | DNA tokenizer and encoder came from different NT releases. The tokenizer now follows `model.dna_model_name`, so keep them the same. |\n", + "| `torch_geometric` / IPEX import errors in the kernel | Venv activated before `module load frameworks`. Use the `prism-bioreason` kernel (its wrapper loads them in order). |\n", + "| `IndexError` in DDP side streams / \"device index out of range\" | Each rank must see one tile as `xpu:0`: `ZE_AFFINITY_MASK=$LOCAL_RANK` and `LOCAL_RANK=0` (the scripts' `mpiexec` block). |\n", + "| Training hangs at the first batch on compute nodes | Hub access attempted offline. Check `HF_HUB_OFFLINE=1` and that section 3 populated `$HF_HOME/hub` and `$PRISM_DATA_ROOT/dna_local_shards`. |\n", + "| GRPO job uses one tile only | By design: `ZoneDTrainer` is single-process (no DDP/vLLM yet). |\n", + "| `Merged sequence length 2798 … exceeds max_merged_seq_length=2048` | Set `MAX_SEQ_LENGTH=4096` (section 1; the GRPO/eval scripts default to it). Or lower `training.max_dna_length`. |\n", + "| Model echoes `Reference sequence: 5029250293`, or the DNA projector gets no gradient | Tag ids were rendered as text, so DNA was never spliced. Fixed in `_render_dna_prompt_text`. The section 10 assertion guards against it. |\n", + "| `ImportError: eval_kegg` during SFT | `training.eval_enabled=true` calls the not-yet-merged `eval_kegg.py`. Keep it `false` and use `eval_kegg_aurora.py`. |" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "PRISM BioReason (Aurora)", + "language": "python", + "name": "prism-bioreason" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/applications/bioreason/eval_kegg_aurora.py b/applications/bioreason/eval_kegg_aurora.py new file mode 100644 index 0000000..6beaefe --- /dev/null +++ b/applications/bioreason/eval_kegg_aurora.py @@ -0,0 +1,344 @@ +"""KEGG accuracy eval for BioReason checkpoints on Aurora (XPU), single tile. + +Stand-in until eval_kegg.py (PR #133) lands. It deliberately reuses the +training code paths instead of re-implementing them: + + * model: src.train.build_model_config + UnifiedTransformer, so the + architecture/special tokens match what src/train.py trained; + * weights: load_model_weights_only (base + projector from model.safetensors), + then merge_lora_into_base for the SFT adapter and, optionally, + a GRPO adapter stacked on top (GRPO checkpoints hold only + lora_adapter.pt -- see ZoneDTrainer.save_checkpoint); + * prompt: StreamingMultimodalDataset's own _process_dna_bioreason + + _render_dna_prompt_text in GRPO mode (is_sft=False), i.e. the + exact user turn + "<|im_start|>assistant\\n" generation prompt; + * decode: UnifiedTransformer.generate, as ZoneDTrainer._generate_completions. + +Usage (on a compute node, after `module load frameworks` + venv): + python applications/bioreason/eval_kegg_aurora.py \\ + --ckpt $PRISM_OUTPUT_ROOT/BIOREASON-SFT-KEGG//checkpoints/step_final \\ + [--grpo-adapter $PRISM_OUTPUT_ROOT/BIOREASON-GRPO-KEGG//checkpoints/step_final/lora_adapter.pt] \\ + --data $PRISM_DATA_ROOT/dna_local_shards_test/kegg \\ + --output results.json [--limit 50] [model=prism_olmo_1b ...extra Hydra overrides] +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +import time +from collections import Counter + +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) +if REPO not in sys.path: + sys.path.insert(0, REPO) + +import torch # noqa: E402 + +ANSWER_RE = re.compile(r"(.*?)", re.DOTALL) + + +def pick_device() -> torch.device: + if hasattr(torch, "xpu") and torch.xpu.is_available(): + return torch.device("xpu") + if torch.cuda.is_available(): + return torch.device("cuda") + return torch.device("cpu") + + +def compose_cfg(overrides: list[str]): + """Hydra-compose src/conf/config.yaml exactly as src/train.py would.""" + from hydra import compose, initialize_config_dir + from hydra.core.global_hydra import GlobalHydra + + if GlobalHydra.instance().is_initialized(): + GlobalHydra.instance().clear() + base = ["model=prism_olmo_1b", "training=bioreason_grpo"] + with initialize_config_dir(config_dir=os.path.join(REPO, "src", "conf"), version_base=None): + return compose(config_name="config", overrides=base + list(overrides)) + + +def _adapter_alpha(path: str, fallback: float) -> float: + """Alpha the adapter was trained with (sidecar written by save_lora_adapter).""" + from src.utils.lora_utils import _load_adapter_meta + + meta = _load_adapter_meta(path) or {} + return float(meta.get("alpha") or fallback) + + +def load_eval_model( + ckpt: str | None, + grpo_adapter: str | None = None, + overrides: list[str] | None = None, + device: torch.device | None = None, +): + """Build the model the way src/train.py does and load trained weights. + + Returns (model, cfg). ``ckpt=None`` evaluates the untrained model (useful + as a baseline / smoke test). + """ + from src.model import UnifiedTransformer + from src.train import build_model_config + from src.training.distributed import load_model_weights_only + from src.utils.lora_utils import merge_lora_into_base + + # Merged length = text + 2 x max_dna_length DNA slots (~2.8k for KEGG), above + # the 2048 default of the merged-length guard (ModelConfig.max_merged_seq_length). + os.environ.setdefault("MAX_SEQ_LENGTH", "4096") + cfg = compose_cfg(overrides or []) + model = UnifiedTransformer(build_model_config(cfg)) + + if ckpt: + load_model_weights_only(model, ckpt, device="cpu") + sft_adapter = os.path.join(ckpt, "lora_adapter.pt") + if os.path.exists(sft_adapter): + alpha = _adapter_alpha(sft_adapter, cfg.training.get("resume_lora_alpha") or 64) + merge_lora_into_base(model.backbone, sft_adapter, alpha=alpha) + print(f"[eval] merged SFT adapter {sft_adapter} (alpha={alpha})") + if grpo_adapter: + alpha = _adapter_alpha(grpo_adapter, cfg.training.get("lora_alpha", 32)) + merge_lora_into_base(model.backbone, grpo_adapter, alpha=alpha) + print(f"[eval] merged GRPO adapter {grpo_adapter} (alpha={alpha})") + + device = device or pick_device() + model.to(device) + if model.backbone is not None and device.type != "cpu": + model.backbone.to(torch.bfloat16) + model.eval() + return model, cfg + + +DNA_MODES = ("real", "shuffled", "blank") +# Uninformative stand-in for --dna-mode blank. Non-empty on purpose: an empty string +# would give an all-zero DNA attention mask, which risks NaNs in the encoder softmax. +BLANK_DNA = "N" * 600 + + +def make_prompt_builder(model, cfg, dna_mode: str = "real"): + """Return ``build(item, donor=None) -> (prompt_batch, answer)`` using the dataset's renderer. + + dna_mode ablates only the DNA content (the prompt text and the answer stay real): + real -- the item's own reference/variant sequences + shuffled -- another item's sequences (``donor``), so DNA is plausible but unrelated + blank -- a constant uninformative sequence + """ + if dna_mode not in DNA_MODES: + raise ValueError(f"dna_mode must be one of {DNA_MODES}, got {dna_mode!r}") + from src.data.multimodal import StreamingMultimodalDataset + from src.train import ( + _resolve_dataset_overrides, + build_bio_dataset_kwargs, + build_training_config, + ) + + mc = model.config + tc = build_training_config(cfg, mc, output_dir="/tmp", seed=0) + helper = StreamingMultimodalDataset( + tokenizer=model.backbone_tokenizer, + model_config=mc, + dataset_overrides=_resolve_dataset_overrides(cfg), + **build_bio_dataset_kwargs("bioreason_grpo", tc, mc), + ) + tok = model.backbone_tokenizer + + def build(item, donor=None): + d = helper._process_dna_bioreason(item) + _, prompt_text = helper._render_dna_prompt_text(d["prompt"]) + ids = tok(prompt_text, return_tensors="pt", add_special_tokens=False).input_ids + if dna_mode == "shuffled": + if donor is None: + raise ValueError("dna_mode='shuffled' needs a donor item") + ref, var = helper._process_dna_bioreason(donor)["dna_sequences"] + elif dna_mode == "blank": + ref, var = BLANK_DNA, BLANK_DNA + else: + ref, var = d["dna_sequences"] + dna = {} + for key, seq in (("dna_reference", ref), ("dna_variant", var)): + enc = helper.dna_tokenizer( + seq, + return_tensors="pt", + padding="max_length", + truncation=True, + max_length=mc.max_dna_length, + ) + dna[key] = {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]} + batch = {"text": ids, "dna": dna, "_metadata": [f"{ids.shape[1]} 0"]} + return batch, d["answer"] + + return build + + +def extract_answer(completion: str) -> str: + from src.data.multimodal import clean_kegg_answer + + m = ANSWER_RE.search(completion) + return clean_kegg_answer(m.group(1) if m else completion) + + +def _to(batch, device): + if torch.is_tensor(batch): + return batch.to(device) + if isinstance(batch, dict): + return {k: _to(v, device) for k, v in batch.items()} + return batch + + +# Appended to a generation that never produced an tag, to force one (see _force_answer). +FORCE_SUFFIX = "\n\n\nAnswer: " +FORCE_MAX_NEW_TOKENS = 32 + + +def _force_answer(model, tok, device, batch, gen_ids, completion): + """Make a generation that ended without produce one. + + Re-prompts with prompt + the model's own generation (trailing eos dropped) + + FORCE_SUFFIX and decodes a few more tokens, so every example gets a scored + prediction. Callers record this separately (``forced``) so natural completion + stays visible in ``format_rate``. + """ + eos = tok.eos_token_id + gen = gen_ids.tolist() + while gen and gen[-1] == eos: + gen.pop() + suffix = tok(FORCE_SUFFIX, add_special_tokens=False, return_tensors="pt").input_ids.to(batch["text"].device) + ids = torch.cat([batch["text"], torch.tensor([gen], dtype=batch["text"].dtype, device=batch["text"].device), + suffix], dim=1) + forced = dict(batch, text=ids, _metadata=[f"{ids.shape[1]} 0"]) + with torch.no_grad(), torch.autocast(device_type=device.type, dtype=torch.bfloat16, + enabled=device.type != "cpu"): + out = model.generate(_to(forced, device), max_new_tokens=FORCE_MAX_NEW_TOKENS, do_sample=False, + pad_token_id=tok.pad_token_id, eos_token_id=eos) + cont = tok.decode(out[0, 1:], skip_special_tokens=True) + cont = cont.split("")[0].split("\n")[0] + return completion + FORCE_SUFFIX + cont + "" + + +def evaluate(model, cfg, dataset, limit: int | None = None, max_new_tokens: int = 1024, verbose=True, + dna_mode: str = "real"): + """Greedy-decode each KEGG item and score exact match on the span. + + Generation stops on OLMo's eos (<|endoftext|>, the token SFT supervises at the end of + the assistant turn) or at max_new_tokens. KEGG reasoning traces are 331-649 tokens + (+ answer), so 1024 leaves headroom; a smaller cap truncates before the answer. + """ + build = make_prompt_builder(model, cfg, dna_mode=dna_mode) + tok = model.backbone_tokenizer + if "im_end" in (tok.eos_token or ""): + raise ValueError(f"eos must be OLMo's <|endoftext|>, not a Qwen ChatML token: {tok.eos_token!r}") + print(f"[eval] stop token: {tok.eos_token!r} (id {tok.eos_token_id}), pad id {tok.pad_token_id}, " + f"max_new_tokens={max_new_tokens}", flush=True) + device = next(model.parameters()).device + n = len(dataset) if limit is None else min(limit, len(dataset)) + records = [] + t0 = time.time() + for i in range(n): + # shuffled: donor = next item in the full split (wraps), a deterministic derangement + donor = dataset[(i + 1) % len(dataset)] if dna_mode == "shuffled" else None + batch, answer = build(dataset[i], donor) + with torch.no_grad(), torch.autocast(device_type=device.type, dtype=torch.bfloat16, + enabled=device.type != "cpu"): + out = model.generate( + _to(batch, device), + max_new_tokens=max_new_tokens, + do_sample=False, + pad_token_id=tok.pad_token_id, + eos_token_id=tok.eos_token_id, + ) + completion = tok.decode(out[0, 1:], skip_special_tokens=True) # strip BOS seed + natural = bool(ANSWER_RE.search(completion)) + if not natural: + completion = _force_answer(model, tok, device, batch, out[0, 1:], completion) + pred = extract_answer(completion) + records.append({ + "idx": i, + "answer": answer, + "pred": pred, + "correct": pred == answer, + "has_answer_tag": natural, # the model's own generation, before any forcing + "forced": not natural, + "completion": completion, + }) + if verbose: + acc = sum(r["correct"] for r in records) / len(records) + print(f"[eval] {i + 1}/{n} acc={acc:.3f} ({time.time() - t0:.0f}s) " + f"gt={answer!r} pred={pred[:60]!r}", flush=True) + return summarize(records) + + +def _prf(records: list[dict]) -> dict: + """Macro/weighted precision, recall, F1 over the ground-truth labels present.""" + support = Counter(r["answer"] for r in records) + tp, fp = Counter(), Counter() + for r in records: + if r["correct"]: + tp[r["answer"]] += 1 + else: + fp[r["pred"]] += 1 + per, n = {}, max(len(records), 1) + for lab, sup in support.items(): + p = tp[lab] / (tp[lab] + fp[lab]) if (tp[lab] + fp[lab]) else 0.0 + rc = tp[lab] / sup + per[lab] = (p, rc, 2 * p * rc / (p + rc) if (p + rc) else 0.0, sup) + k = max(len(per), 1) + return { + "precision_macro": sum(v[0] for v in per.values()) / k, + "recall_macro": sum(v[1] for v in per.values()) / k, + "f1_macro": sum(v[2] for v in per.values()) / k, + "f1_weighted": sum(v[2] * v[3] for v in per.values()) / n, + } + + +def summarize(records: list[dict]) -> dict: + n = len(records) + per_label = Counter(r["answer"] for r in records) + per_label_correct = Counter(r["answer"] for r in records if r["correct"]) + return { + "n": n, + "accuracy": sum(r["correct"] for r in records) / max(n, 1), + "format_rate": sum(r["has_answer_tag"] for r in records) / max(n, 1), + "forced_answer_rate": sum(r.get("forced", False) for r in records) / max(n, 1), + # accuracy of always predicting the most common label; real accuracy is only + # meaningful relative to this (KEGG is heavily imbalanced) + "majority_baseline": (per_label.most_common(1)[0][1] / n) if n else 0.0, + **_prf(records), + "per_label": { + k: {"n": v, "correct": per_label_correct[k]} for k, v in per_label.most_common() + }, + "records": records, + } + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--ckpt", help="SFT (or projector) checkpoint dir containing model.safetensors") + ap.add_argument("--grpo-adapter", help="GRPO lora_adapter.pt to merge on top of --ckpt") + ap.add_argument("--data", required=True, help="KEGG split saved with save_to_disk (helpers/download_bioreason_dna_datasets.py)") + ap.add_argument("--output", required=True, help="JSON file for metrics + per-example records") + ap.add_argument("--limit", type=int, default=None) + ap.add_argument("--max-new-tokens", type=int, default=1024) + ap.add_argument("--dna-mode", choices=DNA_MODES, default="real", + help="DNA ablation: real, shuffled (another item's DNA), or blank (constant N)") + args, overrides = ap.parse_known_args() + + from datasets import load_from_disk + + ds = load_from_disk(args.data) + model, cfg = load_eval_model(args.ckpt, args.grpo_adapter, overrides) + res = evaluate(model, cfg, ds, limit=args.limit, max_new_tokens=args.max_new_tokens, + dna_mode=args.dna_mode) + res.update({"ckpt": args.ckpt, "grpo_adapter": args.grpo_adapter, "data": args.data, + "dna_mode": args.dna_mode}) + os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True) + with open(args.output, "w") as f: + json.dump(res, f, indent=1) + print(f"[eval] dna={args.dna_mode} accuracy={res['accuracy']:.4f} f1_macro={res['f1_macro']:.4f} " + f"majority={res['majority_baseline']:.4f} n={res['n']} " + f"format_rate={res['format_rate']:.3f} forced={res['forced_answer_rate']:.3f} -> {args.output}") + + +if __name__ == "__main__": + main() diff --git a/applications/bioreason/flow-diag.png b/applications/bioreason/flow-diag.png new file mode 100644 index 0000000..3725c21 Binary files /dev/null and b/applications/bioreason/flow-diag.png differ diff --git a/applications/bioreason/helpers/demo_helpers.py b/applications/bioreason/helpers/demo_helpers.py new file mode 100644 index 0000000..543310f --- /dev/null +++ b/applications/bioreason/helpers/demo_helpers.py @@ -0,0 +1,508 @@ +"""Display + PBS plumbing for bioreason_demo.ipynb. + +Everything PRISM-specific (configs, model, data, tokens) lives in the notebook +itself so it can be read top to bottom; this module only holds the +presentation helpers (HTML renderers, plots, widgets) and the qsub/qstat +wrappers that would otherwise bury the interesting cells. +""" + +from __future__ import annotations + +import difflib +import html +import json +import re +import subprocess +from collections import Counter +from pathlib import Path + +# --------------------------------------------------------------------------- +# Palette (validated reference categorical slots 1-3 + neutral inks) +# --------------------------------------------------------------------------- +SERIES = ["#2a78d6", "#eb6834", "#1baf7a"] # blue, orange, aqua +INK = "#0b0b0b" +INK_2 = "#52514e" +GRID = "#e4e3df" +# Nucleotide colors (A/C/G/T) for sequence rendering -- identity only, always +# shown next to the letter itself so color is never the sole carrier. +BASE_COLORS = {"A": "#1baf7a", "C": "#2a78d6", "G": "#eda100", "T": "#e34948", "N": "#8a8984"} + + +def style_ax(ax, title=None, xlabel=None, ylabel=None): + """Recessive axes: no top/right spines, light y-grid, muted tick ink.""" + for side in ("top", "right"): + ax.spines[side].set_visible(False) + for side in ("left", "bottom"): + ax.spines[side].set_color(GRID) + ax.tick_params(colors=INK_2, labelsize=9) + ax.grid(axis="y", color=GRID, linewidth=0.8) + ax.set_axisbelow(True) + if title: + ax.set_title(title, loc="left", fontsize=11, color=INK) + if xlabel: + ax.set_xlabel(xlabel, color=INK_2, fontsize=9) + if ylabel: + ax.set_ylabel(ylabel, color=INK_2, fontsize=9) + return ax + + +# --------------------------------------------------------------------------- +# Source excerpts +# --------------------------------------------------------------------------- +def show_source(path: str | Path, start: int | None = None, end: int | None = None, + pattern: str | None = None, context: int = 25, repo: str | Path = "."): + """Render a numbered excerpt of a repo file as a Markdown code block. + + Either give explicit ``start``/``end`` lines or a regex ``pattern``: the + excerpt then starts at its first match and spans ``context`` lines, so the + notebook does not break when line numbers drift. + """ + from IPython.display import Markdown, display + + p = Path(repo) / path + lines = p.read_text().splitlines() + if pattern is not None: + rx = re.compile(pattern) + hit = next((i for i, ln in enumerate(lines) if rx.search(ln)), None) + if hit is None: + raise ValueError(f"{pattern!r} not found in {path}") + start, end = hit + 1, min(len(lines), hit + context) + start = start or 1 + end = end or len(lines) + body = "\n".join(f"{i:5d} {lines[i - 1]}" for i in range(start, end + 1)) + lang = {".py": "python", ".json": "json", ".yaml": "yaml", ".sh": "bash"}.get(p.suffix, "") + display(Markdown(f"`{path}:{start}-{end}`\n```{lang}\n{body}\n```")) + + +# --------------------------------------------------------------------------- +# DNA sample rendering +# --------------------------------------------------------------------------- +def diff_spans(ref: str, var: str): + """Opcodes (tag, i1, i2, j1, j2) where ref and var differ (SNV / indel).""" + sm = difflib.SequenceMatcher(None, ref, var, autojunk=False) + return [op for op in sm.get_opcodes() if op[0] != "equal"] + + +def _bases_html(seq: str, offset: int, highlight: set[int]) -> str: + out = [] + for k, ch in enumerate(seq): + c = BASE_COLORS.get(ch.upper(), BASE_COLORS["N"]) + style = f"color:{c};" + if offset + k in highlight: + style += "background:#fde68a;font-weight:700;outline:1px solid #b45309;" + out.append(f'{html.escape(ch)}') + return "".join(out) + + +def dna_pair_html(ref: str, var: str, window: int = 120) -> str: + """Ref vs variant around the first difference, differing bases highlighted.""" + ops = diff_spans(ref, var) + if ops: + _, i1, i2, j1, j2 = ops[0] + else: + i1 = i2 = j1 = j2 = len(ref) // 2 + a0, a1 = max(0, i1 - window), min(len(ref), i2 + window) + b0, b1 = max(0, j1 - window), min(len(var), j2 + window) + ref_hl = {i for op in ops for i in range(op[1], op[2])} + var_hl = {j for op in ops for j in range(op[3], op[4])} + kinds = Counter(op[0] for op in ops) + summary = ", ".join(f"{v} {k}" for k, v in kinds.items()) or "identical" + mono = "font-family:ui-monospace,Menlo,monospace;font-size:12px;word-break:break-all;line-height:1.5" + return ( + f'
' + f'
ref[{a0}:{a1}] of {len(ref):,} bp | var[{b0}:{b1}] of {len(var):,} bp' + f' | diff: {summary}
' + f'
REF {_bases_html(ref[a0:a1], a0, ref_hl)}
' + f'
VAR {_bases_html(var[b0:b1], b0, var_hl)}
' + ) + + +def sample_html(item: dict) -> str: + ref = item.get("reference_sequence", "") + var = item.get("variant_sequence") or item.get("mutated_sequence", "") + parts = [dna_pair_html(ref, var)] + for key in ("question", "reasoning", "answer"): + if item.get(key): + text = html.escape(str(item[key])) + if key == "reasoning" and len(text) > 1500: + text = text[:1500] + " …" + parts.append(f'

{key}
{text}

') + return "".join(parts) + + +def sample_browser(datasets: dict): + """Dropdown (dataset) + slider (row) -> rendered sample.""" + import ipywidgets as w + from IPython.display import HTML, display + + names = list(datasets) + dd = w.Dropdown(options=names, description="dataset") + sl = w.IntSlider(min=0, max=len(datasets[names[0]]) - 1, description="row", + continuous_update=False, layout=w.Layout(width="60%")) + out = w.Output() + + def render(*_): + sl.max = len(datasets[dd.value]) - 1 + with out: + out.clear_output(wait=True) + display(HTML(sample_html(datasets[dd.value][sl.value]))) + + dd.observe(render, "value") + sl.observe(render, "value") + render() + display(w.VBox([w.HBox([dd, sl]), out])) + + +# --------------------------------------------------------------------------- +# Dataset statistics +# --------------------------------------------------------------------------- +def length_frame(datasets: dict, llm_tok, dna_tok, max_rows: int = 2000): + """Per-row lengths: bp, DNA tokens (6-mer NT), question/answer/reasoning LLM tokens.""" + import pandas as pd + + rows = [] + for name, ds in datasets.items(): + n = min(len(ds), max_rows) + sub = ds.select(range(n)) + for item in sub: + var = item.get("variant_sequence") or item.get("mutated_sequence", "") + rows.append({ + "dataset": name, + "ref_bp": len(item["reference_sequence"]), + "var_bp": len(var), + "ref_dna_tokens": len(dna_tok(item["reference_sequence"]).input_ids), + "question_tokens": len(llm_tok(item.get("question", "")).input_ids), + "answer_tokens": len(llm_tok(str(item.get("answer", ""))).input_ids), + "reasoning_tokens": len(llm_tok(item.get("reasoning", "") or "").input_ids), + }) + return pd.DataFrame(rows) + + +def plot_lengths(df, max_dna_length: int = 1024): + """DNA span length vs the fixed padding, plus text-length histograms (one measure per panel).""" + import matplotlib.pyplot as plt + + fig, axes = plt.subplots(1, 4, figsize=(17, 3.4), gridspec_kw={"width_ratios": [1.2, 1, 1, 1]}) + names = list(df["dataset"].unique()) + + # Panel 1: DNA tokens are ~constant per dataset (fixed-length windows), so a + # bar per dataset against the padded span length says more than a histogram. + ax = axes[0] + med = df.groupby("dataset", sort=False)["ref_dna_tokens"].max().reindex(names) + ax.barh(names[::-1], med.values[::-1], color=SERIES[0], height=0.55) + ax.axvline(max_dna_length, color=INK, linestyle="--", linewidth=1) + for y, v in enumerate(med.values[::-1]): + ax.text(v, y, f" {v:.0f} tok ({1 - v / max_dna_length:.0%} padding)", va="center", + fontsize=8, color=INK) + ax.set_xlim(0, max_dna_length * 1.08) + style_ax(ax, title=f"DNA tokens per span (max) vs max_dna_length={max_dna_length}") + ax.grid(axis="y", visible=False) + ax.grid(axis="x", color=GRID, linewidth=0.8) + + panels = [ + ("question_tokens", "Question tokens (OLMo)"), + ("reasoning_tokens", "Reasoning tokens (OLMo, KEGG only)"), + ("answer_tokens", "Answer tokens (OLMo)"), + ] + for ax, (col, title) in zip(axes[1:], panels, strict=False): + for i, name in enumerate(names): + vals = df.loc[df.dataset == name, col] + if vals.max() == 0: + continue + ax.hist(vals, bins=40, alpha=0.6, color=SERIES[i % 3], label=name, + edgecolor="white", linewidth=0.5) + style_ax(ax, title=title, ylabel="rows" if col == "question_tokens" else None) + axes[1].legend(frameon=False, fontsize=8) + fig.tight_layout() + return fig + + +def plot_counts(counts: dict, title: str): + import matplotlib.pyplot as plt + + fig, ax = plt.subplots(figsize=(6, 0.45 * len(counts) + 0.8)) + names = list(counts)[::-1] + vals = [counts[n] for n in names] + ax.barh(names, vals, color=SERIES[0], height=0.6) + for y, v in enumerate(vals): + ax.text(v, y, f" {v:,}", va="center", fontsize=9, color=INK) + style_ax(ax, title=title) + ax.grid(axis="x", color=GRID, linewidth=0.8) + ax.grid(axis="y", visible=False) + fig.tight_layout() + return fig + + +# --------------------------------------------------------------------------- +# Model introspection +# --------------------------------------------------------------------------- +def module_tree(model, max_depth: int = 2) -> str: + """Compact module tree: name, class, #params, down to ``max_depth``.""" + lines = [] + + def walk(mod, name, depth): + n = sum(p.numel() for p in mod.parameters()) + lines.append(f"{' ' * depth}{name or ''}: {type(mod).__name__} ({n / 1e6:,.1f}M)") + if depth < max_depth: + for child_name, child in mod.named_children(): + walk(child, child_name, depth + 1) + + walk(model, "", 0) + return "\n".join(lines) + + +def _group(name: str) -> str: + if "lora_" in name: + return "LoRA adapters" + if name.startswith("encoders.dna.proj"): + return "DNA encoder output Linear (2560->d_dna)" + if name.startswith("encoders.dna"): + return "DNA encoder (NT)" + if name.startswith("projectors.dna"): + return "DNA projector (MLP)" + if name.startswith("projectors"): + return "other projectors" + if "embed_tokens" in name or "lm_head" in name: + return "LLM embeddings / lm_head" + if name.startswith("backbone"): + return "LLM backbone" + return "other" + + +def param_table(model): + """Total / trainable parameters per component.""" + import pandas as pd + + agg: dict[str, list[int]] = {} + for n, p in model.named_parameters(): + g = agg.setdefault(_group(n), [0, 0]) + g[0] += p.numel() + g[1] += p.numel() if p.requires_grad else 0 + df = pd.DataFrame( + [(k, v[0], v[1]) for k, v in agg.items()], columns=["component", "params", "trainable"] + ).set_index("component") + df.loc["TOTAL"] = df.sum() + df["trainable %"] = (100 * df.trainable / df.params.clip(lower=1)).round(2) + return df + + +def side_by_side(*frames, titles=()): + from IPython.display import HTML, display + + cells = [] + for i, f in enumerate(frames): + t = titles[i] if i < len(titles) else "" + tbl = f.style.format({"params": "{:,.0f}", "trainable": "{:,.0f}"}).to_html() + cells.append(f'
{html.escape(t)}{tbl}
') + display(HTML(f'
{"".join(cells)}
')) + + +# --------------------------------------------------------------------------- +# Tokenization view +# --------------------------------------------------------------------------- +def token_view(tokenizer, ids, prompt_len: int | None, special: dict[int, str], + dna_tokens_per_span: int) -> str: + """Render token ids: prompt (masked) vs target (loss) and DNA splice slots.""" + out = [] + for i, t in enumerate(ids): + t = int(t) + if t in special: + label = html.escape(special[t]) + extra = (f' [{dna_tokens_per_span} DNA embeddings]' + if special[t].endswith("_start>") else "") + out.append(f'{label}{extra}') + continue + piece = html.escape(tokenizer.decode([t])) + in_target = prompt_len is not None and i >= prompt_len + style = ("background:#dcfce7;" if in_target else f"color:{INK_2};") + out.append(f'{piece}') + legend = (f'
' + f'grey = prompt (label -100, no loss)   ' + f'green = target (loss)   ' + f'blue = DNA splice tag
') + return (legend + '
' + "".join(out) + "
") + + +# --------------------------------------------------------------------------- +# PBS: patch, submit, monitor +# --------------------------------------------------------------------------- +def patch_stage_script(src: str | Path, dst: str | Path) -> Path: + """Copy a submit_bioreason_{projector,sft}.sh with two notebook-side fixes. + + * Checkpoints go to ``$HYDRA_RUN_DIR`` (on /flare) instead of Hydra's + default ``outputs/`` under the home-directory checkout, which hit + "Disk quota exceeded" at the first save. ``$EXTRA_HYDRA`` is appended + verbatim for any further overrides. + * Drops the repo ``.venv`` activation (a CUDA torch build) that the + projector script layers on top of the XPU venv. + + Idempotent: a script that already passes ``hydra.run.dir`` (and defines + ``HYDRA_RUN_DIR``) is only stripped of the ``.venv`` line. The originals + are left untouched. + """ + text = Path(src).read_text() + text = text.replace('source "${PROJ_DIR}/.venv/bin/activate"\n', "") + if "hydra.run.dir=" not in text: + last_arg = " wandb.mode='\"${WANDB_MODE}\"'\n'" + if last_arg not in text: + raise ValueError(f"{src}: could not find the train.py argument list to extend") + text = text.replace( + last_arg, + " wandb.mode='\"${WANDB_MODE}\"' \\\n" + " hydra.run.dir='\"${HYDRA_RUN_DIR}\"' '\"${EXTRA_HYDRA:-}\"'\n'", + ) + if not re.search(r"^HYDRA_RUN_DIR=", text, re.M): + # Must exist in the outer shell before the mpiexec string is built. + text = text.replace( + 'cd "${PROJ_DIR}"\n', + 'HYDRA_RUN_DIR="${HYDRA_RUN_DIR:-${PRISM_OUTPUT_ROOT}/${RUN_ID}/${RUN_TAG:-$(date +%Y%m%d-%H%M%S)}}"\n' + 'cd "${PROJ_DIR}"\n', + 1, + ) + dst = Path(dst) + dst.parent.mkdir(parents=True, exist_ok=True) + dst.write_text(text) + dst.chmod(0o755) + return dst + + +def qsub(script, env: dict | None = None, depend: str | None = None, log: str | None = None, + queue: str | None = None, walltime: str | None = None, name: str | None = None, + cwd: str | None = None, dry_run: bool = False, account: str | None = None) -> str: + """Submit ``script`` and return the job id. ``env`` is passed with -v; ``account`` with -A.""" + cmd = ["qsub"] + if account: + cmd += ["-A", account] + if env: + bad = [k for k, v in env.items() if "," in str(v)] + if bad: + raise ValueError(f"qsub -v values cannot contain commas: {bad}") + cmd += ["-v", ",".join(f"{k}={v}" for k, v in env.items())] + if depend: + cmd += ["-W", f"depend=afterok:{depend}"] + if log: + Path(log).parent.mkdir(parents=True, exist_ok=True) + cmd += ["-o", str(log)] + if queue: + cmd += ["-q", queue] + if walltime: + cmd += ["-l", f"walltime={walltime}"] + if name: + cmd += ["-N", name] + cmd.append(str(script)) + print(" ".join(cmd)) + if dry_run: + return "DRYRUN" + res = subprocess.run(cmd, capture_output=True, text=True, cwd=cwd) + if res.returncode != 0: + raise RuntimeError(res.stderr.strip() or res.stdout.strip()) + return res.stdout.strip() + + +def qstat_table(jobids): + """State of each job (works for finished jobs too via -x).""" + import pandas as pd + + ids = [j for j in jobids if j and j != "DRYRUN"] + if not ids: + return pd.DataFrame() + res = subprocess.run(["qstat", "-x", "-f", "-F", "json", *ids], capture_output=True, text=True) + try: + jobs = json.loads(res.stdout).get("Jobs", {}) + except json.JSONDecodeError: + print(res.stderr or res.stdout) + return pd.DataFrame() + state = {"Q": "queued", "R": "running", "H": "held", "F": "finished", "E": "exiting", "B": "array"} + rows = [] + for jid, j in jobs.items(): + rows.append({ + "job": jid.split(".")[0], + "name": j.get("Job_Name"), + "queue": j.get("queue"), + "state": state.get(j.get("job_state"), j.get("job_state")), + "exit": j.get("Exit_status", ""), + "walltime used": j.get("resources_used", {}).get("walltime", ""), + "depends on": (j.get("depend") or "").replace("afterok:", "").split(".")[0], + }) + return pd.DataFrame(rows).set_index("job") + + +def tail(path, n: int = 30) -> str: + p = Path(path) + if not p.exists(): + return f"(no log yet: {p})" + return "\n".join(p.read_text(errors="replace").splitlines()[-n:]) + + +_ZONE_A = re.compile(r"Step (\d+): Loss ([-\d.]+)") +_GRPO = re.compile(r"step=(\d+) loss=([-\d.]+) avg_reward=([-\d.]+)") + + +def parse_log(path) -> dict: + """{'step': [...], 'loss': [...], 'reward': [...]} from a train.py log.""" + out = {"step": [], "loss": [], "reward": []} + p = Path(path) + if not p.exists(): + return out + for line in p.read_text(errors="replace").splitlines(): + m = _GRPO.search(line) + if m: + out["step"].append(int(m[1])) + out["loss"].append(float(m[2])) + out["reward"].append(float(m[3])) + continue + m = _ZONE_A.search(line) + if m: + out["step"].append(int(m[1])) + out["loss"].append(float(m[2])) + return out + + +def plot_logs(logs: dict): + """One small multiple per run (each its own y-scale; never a dual axis).""" + import matplotlib.pyplot as plt + + runs = {k: parse_log(v) for k, v in logs.items()} + panels = [(k, "loss", r) for k, r in runs.items()] + panels += [(k, "reward", r) for k, r in runs.items() if r["reward"]] + fig, axes = plt.subplots(1, len(panels), figsize=(4.2 * len(panels), 3), squeeze=False) + for ax, (name, key, r) in zip(axes[0], panels, strict=False): + if r[key]: + ax.plot(r["step"], r[key], color=SERIES[0] if key == "loss" else SERIES[2], linewidth=2) + ax.annotate(f"{r[key][-1]:.3f}", (r["step"][-1], r[key][-1]), fontsize=9, color=INK, + xytext=(4, 0), textcoords="offset points", va="center") + style_ax(ax, title=f"{name}: {key}", xlabel="step") + else: + ax.set_axis_off() + ax.set_title(f"{name}: {key}", loc="left", fontsize=11, color=INK) + ax.text(0.5, 0.5, "no steps logged yet", ha="center", va="center", + transform=ax.transAxes, color=INK_2) + fig.tight_layout() + return fig + + +def monitor(jobids, logs: dict, n_tail: int = 15): + """Refresh button: qstat table + log tails + loss curves.""" + import ipywidgets as w + import matplotlib.pyplot as plt + from IPython.display import display + + btn = w.Button(description="Refresh", icon="refresh") + out = w.Output() + + def refresh(*_): + with out: + out.clear_output(wait=True) + display(qstat_table(jobids)) + fig = plot_logs(logs) + display(fig) + plt.close(fig) + for name, path in logs.items(): + print(f"--- {name}: {path}") + print(tail(path, n_tail)) + + btn.on_click(refresh) + refresh() + display(w.VBox([btn, out])) diff --git a/applications/bioreason/download_bioreason_dna_datasets.py b/applications/bioreason/helpers/download_bioreason_dna_datasets.py similarity index 78% rename from applications/bioreason/download_bioreason_dna_datasets.py rename to applications/bioreason/helpers/download_bioreason_dna_datasets.py index aad459f..2fce5fe 100644 --- a/applications/bioreason/download_bioreason_dna_datasets.py +++ b/applications/bioreason/helpers/download_bioreason_dna_datasets.py @@ -4,10 +4,10 @@ branch) can pick them up without hitting the HF Hub at train time. Usage: - python applications/bioreason/download_bioreason_dna_datasets.py --dataset kegg --output-dir /path/to/output - python applications/bioreason/download_bioreason_dna_datasets.py --dataset variant_effect_coding --output-dir /path/to/output - python applications/bioreason/download_bioreason_dna_datasets.py --dataset variant_effect_non_snv --output-dir /path/to/output - python applications/bioreason/download_bioreason_dna_datasets.py --dataset all --output-dir /path/to/output_root + python applications/bioreason/helpers/download_bioreason_dna_datasets.py --dataset kegg --output-dir /path/to/output + python applications/bioreason/helpers/download_bioreason_dna_datasets.py --dataset variant_effect_coding --output-dir /path/to/output + python applications/bioreason/helpers/download_bioreason_dna_datasets.py --dataset variant_effect_non_snv --output-dir /path/to/output + python applications/bioreason/helpers/download_bioreason_dna_datasets.py --dataset all --output-dir /path/to/output_root """ import argparse diff --git a/applications/bioreason/helpers/kernel_wrapper.sh b/applications/bioreason/helpers/kernel_wrapper.sh new file mode 100755 index 0000000..ec3a6f5 --- /dev/null +++ b/applications/bioreason/helpers/kernel_wrapper.sh @@ -0,0 +1,28 @@ +#!/bin/bash -l +# Jupyter kernel launcher for the BioReason demo notebook (Aurora). +# +# Registered by setup_jupyter_kernel.sh as the argv of the `prism-bioreason` +# kernel. Jupyter/VS Code start kernels from a bare environment, so the +# frameworks module and the PRISM venv have to be loaded here, in the same +# order the scripts/training/submit_bioreason_*.sh scripts use (module first, then venv -- +# reversing it breaks torch/IPEX imports). +# +# Do not call directly; Jupyter passes `-f ` as "$@". + +FRAMEWORKS_MODULE="${FRAMEWORKS_MODULE:-frameworks/2026.1.0}" +PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" + +module use /soft/modulefiles +module load "${FRAMEWORKS_MODULE}" +module load hdf5 +source "${PRISM_VENV}/bin/activate" + +# One PVC tile per device (12 per node), matching the submit scripts. +export ZE_FLAT_DEVICE_HIERARCHY=FLAT +export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 +# PBS's default TMPDIR overflows the AF_UNIX socket path limit used by +# DataLoader worker IPC (see submit_bioreason_sft.sh). +export TMPDIR=/tmp +export PYTHONWARNINGS=ignore + +exec python -m ipykernel_launcher "$@" diff --git a/applications/bioreason/helpers/setup_jupyter_kernel.sh b/applications/bioreason/helpers/setup_jupyter_kernel.sh new file mode 100755 index 0000000..3ff3f2c --- /dev/null +++ b/applications/bioreason/helpers/setup_jupyter_kernel.sh @@ -0,0 +1,55 @@ +#!/bin/bash -l +# One-time setup for applications/bioreason/bioreason_demo.ipynb on Aurora. +# +# Installs Jupyter into the PRISM venv and registers a kernel `prism-bioreason` whose argv is +# kernel_wrapper.sh, so the kernel always starts with the module + venv loaded (VS Code +# Remote and jupyter lab both pick it up). Jupyter is not part of PRISM's requirements. +# +# Requires the venv from helpers/setup_prism_env.sh (run that first). +# +# Run once from a login node (UAN; it has direct internet access): +# export PRISM_VENV=/flare//$USER/prism-envs/bioreason-fw2026.1 +# bash applications/bioreason/helpers/setup_jupyter_kernel.sh +# Then, from VS Code connected to the interactive compute node, open the +# notebook and pick the "PRISM BioReason (Aurora)" kernel. + +set -eo pipefail + +HERE="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +FRAMEWORKS_MODULE="${FRAMEWORKS_MODULE:-frameworks/2026.1.0}" +PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +KERNEL_NAME="prism-bioreason" + +case "${PRISM_VENV}" in *""*) echo "ERROR: set PRISM_VENV to your venv path, e.g. /flare/\$PROJECT/\$USER/prism-envs/bioreason-fw2026.1"; exit 1;; esac + +module use /soft/modulefiles +module load "${FRAMEWORKS_MODULE}" + +if [ ! -f "${PRISM_VENV}/bin/activate" ]; then + echo "ERROR: ${PRISM_VENV} not found -- run helpers/setup_prism_env.sh first." + exit 1 +fi +source "${PRISM_VENV}/bin/activate" +command -v uv >/dev/null 2>&1 || { echo "ERROR: uv is not on PATH (see setup_prism_env.sh)"; exit 1; } + +# jupyter_server: fallback when VS Code doesn't list the kernel -- run +# `jupyter server --no-browser --ip=$(hostname) --port 8888` on the compute node +# and connect via "Select Kernel -> Existing Jupyter Server". +uv pip install ipykernel ipywidgets jupyter_server + +chmod +x "${HERE}/kernel_wrapper.sh" +KERNEL_DIR="${HOME}/.local/share/jupyter/kernels/${KERNEL_NAME}" +mkdir -p "${KERNEL_DIR}" +cat > "${KERNEL_DIR}/kernel.json" < ${KERNEL_DIR}/kernel.json" +echo " venv: ${PRISM_VENV}" +echo " module: ${FRAMEWORKS_MODULE}" diff --git a/applications/bioreason/helpers/setup_prism_env.sh b/applications/bioreason/helpers/setup_prism_env.sh new file mode 100755 index 0000000..312f564 --- /dev/null +++ b/applications/bioreason/helpers/setup_prism_env.sh @@ -0,0 +1,50 @@ +#!/bin/bash -l +# One-time PRISM environment for the BioReason application on Aurora (no Jupyter here). +# +# Builds (or reuses) the BioReason venv on top of frameworks/2026.1.0 -- the same venv the +# scripts/training/submit_bioreason_*.sh scripts activate -- and installs the training extras +# the frameworks module lacks (torchtune from the torchtune-aurora fork) plus PRISM itself (--no-deps, so the module's XPU torch stays). +# +# Run once from a login node (UAN; it has direct internet access): +# export PRISM_VENV=/flare//$USER/prism-envs/bioreason-fw2026.1 +# bash applications/bioreason/helpers/setup_prism_env.sh +# For the notebook, then run helpers/setup_jupyter_kernel.sh. + +set -eo pipefail + +HERE="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +PRISM_DIR="$( cd "${HERE}/../../.." && pwd )" +FRAMEWORKS_MODULE="${FRAMEWORKS_MODULE:-frameworks/2026.1.0}" +PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +case "${PRISM_VENV}" in *""*) echo "ERROR: set PRISM_VENV to your venv path, e.g. /flare/\$PROJECT/\$USER/prism-envs/bioreason-fw2026.1"; exit 1;; esac + +module use /soft/modulefiles +module load "${FRAMEWORKS_MODULE}" + +if ! command -v uv >/dev/null 2>&1; then + echo "ERROR: uv is not on PATH. Install: curl -LsSf https://astral.sh/uv/install.sh | sh" + exit 1 +fi + +if [ ! -f "${PRISM_VENV}/bin/activate" ]; then + echo "Creating ${PRISM_VENV} (--system-site-packages over ${FRAMEWORKS_MODULE})" + uv venv --system-site-packages -p "$(which python)" "${PRISM_VENV}" +fi +source "${PRISM_VENV}/bin/activate" + +# Same extras as the recipe in submit_bioreason_sft.sh; --no-deps keeps the +# module's XPU torch from being replaced by a PyPI/CUDA wheel. +uv pip install wandb braceexpand==0.1.7 +uv pip install --no-deps webdataset==1.0.2 torchao==0.5.0 # torchao pin: the fork's torchtune imports torchao.dtypes.nf4tensor + +# torchtune: use the n-getty/torchtune-aurora fork, NOT the PyPI wheel. The PyPI torchtune has no +# torchtune.dev.rl, which src/training/trainer_grpo.py needs (GRPOSimpleLoss); the fork also provides +# the LoRA modules SFT uses. Cloned next to the venv; override the location with TORCHTUNE_DIR. +TORCHTUNE_DIR="${TORCHTUNE_DIR:-$(dirname "${PRISM_VENV}")/torchtune-aurora}" +[ -d "${TORCHTUNE_DIR}/.git" ] || git clone https://github.com/n-getty/torchtune-aurora.git "${TORCHTUNE_DIR}" +uv pip install --no-deps -e "${TORCHTUNE_DIR}" +uv pip install --no-deps -e "${PRISM_DIR}" + +echo "PRISM environment ready:" +echo " venv: ${PRISM_VENV}" +echo " module: ${FRAMEWORKS_MODULE}" diff --git a/applications/bioreason/helpers/summarize_eval_sweep.py b/applications/bioreason/helpers/summarize_eval_sweep.py new file mode 100644 index 0000000..6a8cfe8 --- /dev/null +++ b/applications/bioreason/helpers/summarize_eval_sweep.py @@ -0,0 +1,37 @@ +"""Print a table from a submit_eval_sft_sweep.sh output directory of eval_kegg_aurora.py JSONs. + +Usage: python applications/bioreason/helpers/summarize_eval_sweep.py +Stdlib only. Compare the real row against the shuffled/blank rows and the majority baseline. +""" + +import glob +import json +import os +import re +import sys + + +def step_key(name: str): + m = re.match(r"step_(\d+|final)_", name) + return (1, 0) if m and m.group(1) == "final" else (0, int(m.group(1)) if m else -1) + + +def main(d: str): + rows = [] + for p in glob.glob(os.path.join(d, "*.json")): + r = json.load(open(p)) + rows.append((os.path.basename(p)[:-5], r)) + rows.sort(key=lambda x: (step_key(x[0]), x[0])) + cols = ["acc", "f1_macro", "f1_wtd", "majority", "format", "n"] + print(f"{'run':<26}{'dna':<10}" + "".join(f"{c:>10}" for c in cols)) + for name, r in rows: + vals = [r["accuracy"], r.get("f1_macro", float("nan")), r.get("f1_weighted", float("nan")), + r.get("majority_baseline", float("nan")), r["format_rate"], r["n"]] + print(f"{name:<26}{r.get('dna_mode', 'real'):<10}" + + "".join(f"{v:>10.4f}" if isinstance(v, float) else f"{v:>10}" for v in vals)) + + +if __name__ == "__main__": + if len(sys.argv) != 2: + sys.exit(__doc__) + main(sys.argv[1]) diff --git a/applications/bioreason/submit_eval_kegg.sh b/applications/bioreason/scripts/eval/submit_eval_kegg.sh similarity index 91% rename from applications/bioreason/submit_eval_kegg.sh rename to applications/bioreason/scripts/eval/submit_eval_kegg.sh index 49fc462..249d6b9 100755 --- a/applications/bioreason/submit_eval_kegg.sh +++ b/applications/bioreason/scripts/eval/submit_eval_kegg.sh @@ -9,7 +9,11 @@ set -euo pipefail -PROJ_DIR="/home/abalaji/projects/modcon/genome/BaseMM_PRISM" +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi CHECKPOINT_DIR="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_8000" # SFT with sample weigting and answer structure : /lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_3000 # Set to "true" to run interleaved mode, "false" for prefix mode. diff --git a/applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh b/applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh new file mode 100755 index 0000000..e25dc8e --- /dev/null +++ b/applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh @@ -0,0 +1,94 @@ +#!/bin/bash -l +#PBS -N bioreason-eval-kegg-1b +#PBS -l select=1 +#PBS -l walltime=01:00:00 +#PBS -l filesystems=home:flare +#PBS -q debug +# Allocation is not hardcoded: submit with `qsub -A ...`. +#PBS -k doe +#PBS -j oe +# +# Aurora XPU launcher: KEGG accuracy eval for BioReason checkpoints (1 tile). +# Aurora port of submit_eval_kegg.sh (Polaris); runs eval_kegg_aurora.py +# because eval_kegg.py (PR #133) is not in this repo yet. +# +# CKPT SFT checkpoint dir (model.safetensors [+ lora_adapter.pt]). Required. +# GRPO_ADAPTER optional GRPO lora_adapter.pt merged on top of CKPT. +# SPLIT KEGG split to score (default test; saved by the notebook / +# helpers/download_bioreason_dna_datasets.py under dna_local_shards_/kegg). +# LIMIT optional cap on examples. +# DNA_MODE real (default) | shuffled (another item's DNA) | blank (constant N): DNA ablation. +# OUTPUT metrics JSON (default ${PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/${RUN_TAG}/results.json, +# or results_.json when DNA_MODE != real). +# +# qsub -v CKPT=/checkpoints/step_final applications/bioreason/scripts/eval/submit_eval_kegg_aurora.sh +# qsub -v CKPT=...,GRPO_ADAPTER=/checkpoints/step_final/lora_adapter.pt,LIMIT=50 ... + +set -eo pipefail + +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi +RUN_TAG="${RUN_TAG:-$(date +%Y%m%d-%H%M%S)}" +SPLIT="${SPLIT:-test}" +MAX_NEW_TOKENS="${MAX_NEW_TOKENS:-1024}" +DNA_MODE="${DNA_MODE:-real}" +case "${DNA_MODE}" in real|shuffled|blank) ;; *) echo "ERROR: DNA_MODE must be real|shuffled|blank"; exit 1;; esac + +export PRISM_DATA_ROOT="${PRISM_DATA_ROOT:-/lus/flare/projects//${USER}/cache}" +export PRISM_OUTPUT_ROOT="${PRISM_OUTPUT_ROOT:-/lus/flare/projects//${USER}/output}" +RESULT_NAME="results.json"; [ "${DNA_MODE}" != "real" ] && RESULT_NAME="results_${DNA_MODE}.json" +OUTPUT="${OUTPUT:-${PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/${RUN_TAG}/${RESULT_NAME}}" +DATA="${DATA:-${PRISM_DATA_ROOT}/dna_local_shards_${SPLIT}/kegg}" + +if [ -z "${CKPT:-}" ]; then + echo "ERROR: set CKPT=/checkpoints/step_final" + exit 1 +fi + +cd "${PROJ_DIR}" + +export PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +module use /soft/modulefiles +module load frameworks/2026.1.0 +module load hdf5 +source "${PRISM_VENV}/bin/activate" + +export HF_HOME="${HF_HOME:-/flare//${USER}/cache/model_weights}" +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export HF_DATASETS_OFFLINE=1 +export TMPDIR=/tmp +export PYTHONUNBUFFERED=1 +export PYTHONWARNINGS=ignore +export ZE_FLAT_DEVICE_HIERARCHY=FLAT +export ZE_AFFINITY_MASK=0 +# Text + 2 x max_dna_length (1024) DNA slots is ~2.8-3.6k tokens for KEGG; the +# default 2048 merged-length guard (model.py, issue #123) would reject it. +export MAX_SEQ_LENGTH="${MAX_SEQ_LENGTH:-4096}" + +for p in "${CKPT}/model.safetensors" "${DATA}/dataset_info.json" ${GRPO_ADAPTER:+"${GRPO_ADAPTER}"}; do + if [ ! -e "$p" ]; then + echo "ERROR: missing required path: $p" + exit 1 + fi +done + +echo "================================================================" +echo " BIOREASON KEGG EVAL (Aurora, 1 tile)" +echo " Checkpoint: ${CKPT}" +echo " GRPO adapter: ${GRPO_ADAPTER:-}" +echo " Data: ${DATA} (limit ${LIMIT:-all})" +echo " DNA mode: ${DNA_MODE}" +echo " Output: ${OUTPUT}" +echo "================================================================" + +python -u applications/bioreason/eval_kegg_aurora.py \ + --ckpt "${CKPT}" \ + ${GRPO_ADAPTER:+--grpo-adapter "${GRPO_ADAPTER}"} \ + --data "${DATA}" \ + --output "${OUTPUT}" \ + --max-new-tokens "${MAX_NEW_TOKENS}" \ + --dna-mode "${DNA_MODE}" \ + ${LIMIT:+--limit "${LIMIT}"} diff --git a/applications/bioreason/scripts/eval/submit_eval_sft_sweep.sh b/applications/bioreason/scripts/eval/submit_eval_sft_sweep.sh new file mode 100755 index 0000000..9907a0e --- /dev/null +++ b/applications/bioreason/scripts/eval/submit_eval_sft_sweep.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# Submit the BioReason SFT validation sweep: one scripts/eval/submit_eval_kegg_aurora.sh job per config, +# run in parallel (login node; this script only calls qsub). +# +# real DNA on every checkpoint -> accuracy / F1 vs. training step (overfit check) +# shuffled DNA on step_final -> does the model actually use the DNA? +# blank DNA on step_final +# +# Usage: +# bash applications/bioreason/scripts/eval/submit_eval_sft_sweep.sh /checkpoints +# SPLIT=test LIMIT=20 WALLTIME=00:30:00 bash applications/bioreason/scripts/eval/submit_eval_sft_sweep.sh +# Select checkpoints on SPLIT=val (default); run SPLIT=test once on the chosen one. +# Then: python applications/bioreason/helpers/summarize_eval_sweep.py /BIOREASON-EVAL-KEGG/ + +set -eo pipefail +CKPT_DIR="${1:?usage: $0 /checkpoints}" +SPLIT="${SPLIT:-val}" +WALLTIME="${WALLTIME:-01:00:00}" +TAG="${TAG:-sweep-$(date +%Y%m%d-%H%M%S)}" +PRISM_OUTPUT_ROOT="${PRISM_OUTPUT_ROOT:-/lus/flare/projects//${USER}/output}" +OUT_DIR="${PRISM_OUTPUT_ROOT}/BIOREASON-EVAL-KEGG/${TAG}" +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +mkdir -p "${OUT_DIR}" +echo "Sweep tag: ${TAG} split: ${SPLIT} outputs: ${OUT_DIR}" + +submit() { # + local vars="CKPT=$1,SPLIT=${SPLIT},DNA_MODE=$2,OUTPUT=${OUT_DIR}/$3.json,RUN_TAG=${TAG}" + [ -n "${LIMIT:-}" ] && vars+=",LIMIT=${LIMIT}" + qsub -l walltime="${WALLTIME}" -v "${vars}" "${HERE}/submit_eval_kegg_aurora.sh" +} + +shopt -s nullglob +found=0 +for c in "${CKPT_DIR}"/step_[0-9]* ; do + [ -f "${c}/model.safetensors" ] || continue + submit "${c}" real "$(basename "${c}")_real"; found=1 +done +FINAL="${CKPT_DIR}/step_final" +if [ -f "${FINAL}/model.safetensors" ]; then + submit "${FINAL}" real step_final_real + submit "${FINAL}" shuffled step_final_shuffled + submit "${FINAL}" blank step_final_blank + found=1 +fi +[ "${found}" = 1 ] || { echo "ERROR: no checkpoints with model.safetensors under ${CKPT_DIR}"; exit 1; } +echo "Submitted. Summarize with: python applications/bioreason/helpers/summarize_eval_sweep.py ${OUT_DIR}" diff --git a/applications/bioreason/verify_dna_dataset_mix.sh b/applications/bioreason/scripts/test/verify_dna_dataset_mix.sh similarity index 90% rename from applications/bioreason/verify_dna_dataset_mix.sh rename to applications/bioreason/scripts/test/verify_dna_dataset_mix.sh index 92df0c9..f96db59 100644 --- a/applications/bioreason/verify_dna_dataset_mix.sh +++ b/applications/bioreason/scripts/test/verify_dna_dataset_mix.sh @@ -18,12 +18,16 @@ # 3. Training doesn't crash on the new answer formats/handlers # # Usage: -# qsub applications/bioreason/verify_dna_dataset_mix.sh -# bash applications/bioreason/verify_dna_dataset_mix.sh # on an already-allocated compute node +# qsub applications/bioreason/scripts/test/verify_dna_dataset_mix.sh +# bash applications/bioreason/scripts/test/verify_dna_dataset_mix.sh # on an already-allocated compute node set -euo pipefail -PROJ_DIR="/home/abalaji/projects/modcon/genome/BaseMM_PRISM" +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi STAGE1_CKPT="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_8000" cd "${PROJ_DIR}" diff --git a/applications/bioreason/submit_bioreason_grpo.sh b/applications/bioreason/scripts/training/submit_bioreason_grpo.sh similarity index 91% rename from applications/bioreason/submit_bioreason_grpo.sh rename to applications/bioreason/scripts/training/submit_bioreason_grpo.sh index 47690fe..884f0b1 100755 --- a/applications/bioreason/submit_bioreason_grpo.sh +++ b/applications/bioreason/scripts/training/submit_bioreason_grpo.sh @@ -11,7 +11,11 @@ set -euo pipefail -PROJ_DIR="/home/abalaji/projects/modcon/genome/BaseMM_PRISM" +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi OUTPUT_BASE="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output" RUN_ID="GENOME-GRPO" RUN_VARIANT="1B" diff --git a/applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh b/applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh new file mode 100755 index 0000000..d080496 --- /dev/null +++ b/applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh @@ -0,0 +1,141 @@ +#!/bin/bash -l +#PBS -N bioreason-grpo-kegg-1b +#PBS -l select=1 +#PBS -l walltime=04:00:00 +#PBS -l filesystems=home:flare +#PBS -q prod +# Allocation is not hardcoded: submit with `qsub -A ...`. +#PBS -k doe +#PBS -j oe +# +# Aurora XPU launcher: BioReason Stage 3 GRPO on KEGG (wanglab/kegg) only. +# Aurora port of submit_bioreason_grpo.sh (Polaris/CUDA). +# base model : OLMo-1B (model=prism_olmo_1b) -- training=bioreason_grpo alone +# would fall back to config.yaml's default prism_olmo3_7b +# start from : Stage 2 SFT checkpoint (model.safetensors + lora_adapter.pt); +# the SFT adapter (r=32, alpha=64) is merged into the base, then +# a fresh r=16 GRPO adapter trains on top +# frozen : NT-2.5B DNA encoder, DNA projector, base LLM +# saves : checkpoints/step_*/lora_adapter.pt only (no model.safetensors), +# so eval needs the SFT checkpoint + this adapter +# +# ZoneDTrainer is single-process by design (no DDP/vLLM), so this runs ONE rank +# on ONE tile; the other 11 tiles of the node sit idle. +# +# Batch job: +# qsub -v STAGE1_CKPT=/checkpoints/step_final applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh +# qsub -v STAGE1_CKPT=...,MAX_STEPS=50,RUN_TAG=demo applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh +# Interactive node: +# STAGE1_CKPT=... MAX_STEPS=5 bash applications/bioreason/scripts/training/submit_bioreason_grpo_aurora.sh +# +# Outputs land in ${PRISM_CKPT_ROOT}/${RUN_ID}/${RUN_TAG} (override with +# HYDRA_RUN_DIR); EXTRA_HYDRA is appended verbatim to the train.py command. + +set -eo pipefail + +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi +RUN_ID="${RUN_ID:-BIOREASON-GRPO-KEGG}" +RUN_VARIANT="1B" +RUN_TAG="${RUN_TAG:-$(date +%Y%m%d-%H%M%S)}" +# Hyperparameters come from src/conf/training/bioreason_grpo.yaml. Set any of +# MAX_STEPS / NUM_GENERATIONS / MAX_COMPLETION / SAVE_EVERY to override that value for one run. +TRAIN_OVERRIDES="" +[ -n "${MAX_STEPS:-}" ] && TRAIN_OVERRIDES+=" training.max_steps=${MAX_STEPS}" +[ -n "${NUM_GENERATIONS:-}" ] && TRAIN_OVERRIDES+=" training.grpo_num_generations=${NUM_GENERATIONS}" +[ -n "${MAX_COMPLETION:-}" ] && TRAIN_OVERRIDES+=" training.grpo_max_completion_length=${MAX_COMPLETION}" +[ -n "${SAVE_EVERY:-}" ] && TRAIN_OVERRIDES+=" training.save_every_n_steps=${SAVE_EVERY}" +EXTRA_HYDRA="${EXTRA_HYDRA:-}" + +export PRISM_DATA_ROOT="${PRISM_DATA_ROOT:-/lus/flare/projects//${USER}/cache}" +export PRISM_OUTPUT_ROOT="${PRISM_OUTPUT_ROOT:-/lus/flare/projects//${USER}/output}" +export PRISM_CKPT_ROOT="${PRISM_CKPT_ROOT:-/lus/flare/projects//${USER}/modcon/checkpoints}" +HYDRA_RUN_DIR="${HYDRA_RUN_DIR:-${PRISM_CKPT_ROOT}/${RUN_ID}/${RUN_TAG}}" + +# Stage 2 SFT checkpoint. Default: newest SFT step_final under PRISM_CKPT_ROOT. +STAGE1_CKPT="${STAGE1_CKPT:-$(ls -td "${PRISM_CKPT_ROOT}/BIOREASON-SFT-KEGG"/*/checkpoints/step_final 2>/dev/null | head -n 1)}" +if [ -z "${STAGE1_CKPT}" ]; then + echo "ERROR: no SFT checkpoint found -- run scripts/training/submit_bioreason_sft.sh or set STAGE1_CKPT." + exit 1 +fi + +cd "${PROJ_DIR}" + +export PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +module use /soft/modulefiles +module load frameworks/2026.1.0 +module load hdf5 +if [ ! -f "${PRISM_VENV}/bin/activate" ]; then + echo "ERROR: PRISM_VENV not found: ${PRISM_VENV} (bash applications/bioreason/helpers/setup_prism_env.sh builds it)" + exit 1 +fi +source "${PRISM_VENV}/bin/activate" + +if [[ ! "${HOSTNAME}" =~ aurora-uan ]]; then + export HTTP_PROXY="http://proxy.alcf.anl.gov:3128" + export HTTPS_PROXY="http://proxy.alcf.anl.gov:3128" + export http_proxy="http://proxy.alcf.anl.gov:3128" + export https_proxy="http://proxy.alcf.anl.gov:3128" +fi + +export HF_HOME="${HF_HOME:-/flare//${USER}/cache/model_weights}" +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export HF_DATASETS_OFFLINE=1 +export WANDB_MODE="${WANDB_MODE:-offline}" +export WANDB_SILENT=true +export TMPDIR=/tmp +export PYTHONUNBUFFERED=1 +export PYTHONFAULTHANDLER=1 +export PYTHONWARNINGS=ignore +export NUMEXPR_MAX_THREADS=512 +export NUMEXPR_NUM_THREADS=128 +export ZE_FLAT_DEVICE_HIERARCHY=FLAT +export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 +# Text + 2 x max_dna_length (1024) DNA slots is ~2.8-3.6k tokens for KEGG; the +# default 2048 merged-length guard (model.py, issue #123) would reject it. +export MAX_SEQ_LENGTH="${MAX_SEQ_LENGTH:-4096}" + +for p in "${STAGE1_CKPT}/model.safetensors" "${STAGE1_CKPT}/lora_adapter.pt" \ + "${PRISM_DATA_ROOT}/dna_local_shards/kegg/dataset_info.json" \ + "${HF_HOME}/hub/models--allenai--OLMo-1B-0724-hf" \ + "${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species"; do + if [ ! -e "$p" ]; then + echo "ERROR: missing required path: $p" + exit 1 + fi +done + +echo "================================================================" +echo " BIOREASON GRPO (KEGG only, OLMo-1B + LoRA r16, Aurora, 1 tile)" +echo " Node: $(hostname)" +echo " SFT ckpt: ${STAGE1_CKPT}" +echo " Run dir: ${HYDRA_RUN_DIR}" +echo " Overrides: ${TRAIN_OVERRIDES:- (none; using bioreason_grpo.yaml)}" +echo "================================================================" + +mpiexec -n 1 -ppn 1 --cpu-bind depth --depth 8 bash -lc ' + module use /soft/modulefiles + module load frameworks/2026.1.0 + source '"${PRISM_VENV}"'/bin/activate + export ZE_AFFINITY_MASK=0 + export WORLD_SIZE=1 RANK=0 LOCAL_RANK=0 LOCAL_WORLD_SIZE=1 + + cd '"${PROJ_DIR}"' + python -u src/train.py \ + training=bioreason_grpo \ + model=prism_olmo_1b \ + exp.id='"${RUN_ID}"' \ + exp.variant='"${RUN_VARIANT}"' \ + training.resume_weights_only='"${STAGE1_CKPT}"' \ + training.bioreason_dataset=wanglab/kegg \ + '"${TRAIN_OVERRIDES}"' \ + wandb.mode='"${WANDB_MODE}"' \ + hydra.run.dir='"${HYDRA_RUN_DIR}"' '"${EXTRA_HYDRA}"' +' + +echo "" +echo " GRPO adapter: ${HYDRA_RUN_DIR}/checkpoints/step_final/lora_adapter.pt" diff --git a/applications/bioreason/scripts/training/submit_bioreason_projector.sh b/applications/bioreason/scripts/training/submit_bioreason_projector.sh new file mode 100755 index 0000000..3622435 --- /dev/null +++ b/applications/bioreason/scripts/training/submit_bioreason_projector.sh @@ -0,0 +1,182 @@ +#!/bin/bash -l +#PBS -N bioreason-projector-kegg-1b +#PBS -l select=1 +#PBS -l walltime=01:00:00 +#PBS -l filesystems=home:flare +#PBS -q debug +# Allocation is not hardcoded: submit with `qsub -A ...`. +#PBS -k doe +#PBS -j oe +# +# Aurora XPU launcher: BioReason Stage 1 projector warmup on KEGG (wanglab/kegg) only. +# base model : OLMo-1B (model=prism_olmo_1b), backbone frozen +# trainable : DNA projector only (no LoRA) +# frozen : NT-2.5B DNA encoder +# Its checkpoints/step_final is the STAGE1_CKPT for scripts/training/submit_bioreason_sft.sh. +# Outputs land in ${PRISM_CKPT_ROOT}/${RUN_ID}/${RUN_TAG} (override with HYDRA_RUN_DIR). +# +# Hyperparameters (steps, warmup, batch, grad-accum, save cadence) come from +# src/conf/training/bioreason_projector.yaml; the env vars below override them per run. +# +# Batch job: +# qsub applications/bioreason/scripts/training/submit_bioreason_projector.sh +# qsub -v MAX_STEPS=200,GRAD_ACCUM=2 applications/bioreason/scripts/training/submit_bioreason_projector.sh +# +# Interactive node (after `qsub -I -l select=1,walltime=01:00:00,filesystems=home:flare -q debug -A `): +# bash applications/bioreason/scripts/training/submit_bioreason_projector.sh + +set -eo pipefail + +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi +RUN_ID="${RUN_ID:-BIOREASON-PROJECTOR-KEGG}" +RUN_VARIANT="1B" +RUN_TAG="${RUN_TAG:-$(date +%Y%m%d-%H%M%S)}" +# Training hyperparameters come from src/conf/training/bioreason_projector.yaml. Set any of +# MAX_STEPS / WARMUP_STEPS / GRAD_ACCUM / BATCH_SIZE / SAVE_EVERY to override that value for one run. +TRAIN_OVERRIDES="" +[ -n "${MAX_STEPS:-}" ] && TRAIN_OVERRIDES+=" training.max_steps=${MAX_STEPS}" +[ -n "${WARMUP_STEPS:-}" ] && TRAIN_OVERRIDES+=" training.warmup_steps=${WARMUP_STEPS}" +[ -n "${GRAD_ACCUM:-}" ] && TRAIN_OVERRIDES+=" training.gradient_accumulation_steps=${GRAD_ACCUM}" +[ -n "${BATCH_SIZE:-}" ] && TRAIN_OVERRIDES+=" training.batch_size=${BATCH_SIZE}" +[ -n "${SAVE_EVERY:-}" ] && TRAIN_OVERRIDES+=" training.save_every_n_steps=${SAVE_EVERY}" +NPERNODE="${NPERNODE:-12}" # 6 PVC GPUs x 2 tiles per Aurora node (FLAT mode) + +# Site roots: datasets_config.json resolves dna_bioreason's local_path as +# ${PRISM_DATA_ROOT}/dna_local_shards/kegg. +export PRISM_DATA_ROOT="${PRISM_DATA_ROOT:-/lus/flare/projects//${USER}/cache}" +export PRISM_OUTPUT_ROOT="${PRISM_OUTPUT_ROOT:-/lus/flare/projects//${USER}/output}" +# Checkpoints (~13 GB each: frozen backbone + encoder are saved too) go to flare, +# not Hydra's default outputs/ under the repo in /home, which overflows its quota. +export PRISM_CKPT_ROOT="${PRISM_CKPT_ROOT:-/lus/flare/projects//${USER}/modcon/checkpoints}" +HYDRA_RUN_DIR="${HYDRA_RUN_DIR:-${PRISM_CKPT_ROOT}/${RUN_ID}/${RUN_TAG}}" + +cd "${PROJ_DIR}" + +# frameworks module + a --system-site-packages venv adding only what the module +# lacks (wandb, torchtune, webdataset, braceexpand). Not the repo's .venv, +# which carries a CUDA torch build. Build it once (from a UAN) with: +# module load frameworks/2026.1.0 +# uv venv --system-site-packages -p "$(which python)" $PRISM_VENV && source $PRISM_VENV/bin/activate +# uv pip install wandb braceexpand==0.1.7 && uv pip install --no-deps webdataset==1.0.2 torchao==0.5.0 +# # torchtune comes from the n-getty fork (the PyPI wheel lacks torchtune.dev.rl, which GRPO needs): +# git clone https://github.com/n-getty/torchtune-aurora.git $(dirname $PRISM_VENV)/torchtune-aurora +# uv pip install --no-deps -e $(dirname $PRISM_VENV)/torchtune-aurora +export PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +module use /soft/modulefiles +module load frameworks/2026.1.0 +module load hdf5 +if [ ! -f "${PRISM_VENV}/bin/activate" ]; then + echo "ERROR: PRISM_VENV not found: ${PRISM_VENV} (see build steps above)" + exit 1 +fi +source "${PRISM_VENV}/bin/activate" +source "${PROJ_DIR}/.venv/bin/activate" + +if [[ ! "${HOSTNAME}" =~ aurora-uan ]]; then + export HTTP_PROXY="http://proxy.alcf.anl.gov:3128" + export HTTPS_PROXY="http://proxy.alcf.anl.gov:3128" + export http_proxy="http://proxy.alcf.anl.gov:3128" + export https_proxy="http://proxy.alcf.anl.gov:3128" +fi + +# OLMo-1B-0724-hf, nucleotide-transformer-2.5b-multi-species and +# datasets--wanglab--kegg are all pre-cached here; run fully offline. +export HF_HOME="/flare//${USER}/cache/model_weights" +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export HF_DATASETS_OFFLINE=1 +# Text + 2 x 1024 DNA slots is ~2.8-3.6k merged tokens; default guard is 2048. +export MAX_SEQ_LENGTH="${MAX_SEQ_LENGTH:-4096}" +export WANDB_MODE="${WANDB_MODE:-offline}" +export WANDB_SILENT=true +# PBS's default TMPDIR overflows the 108-byte AF_UNIX socket path limit that +# torch's DataLoader worker IPC relies on -- force a short one. +export TMPDIR=/tmp +# DDP side-copy streams index torch.accelerator.device_count(), which is 0 +# under ZE_AFFINITY_MASK on Aurora -> IndexError on first forward. +export PYTORCH_DDP_USE_SIDE_STREAM=0 +export PYTHONUNBUFFERED=1 +export PYTHONFAULTHANDLER=1 +export PYTHONWARNINGS=ignore +export NUMEXPR_MAX_THREADS=512 +export NUMEXPR_NUM_THREADS=128 +export ZE_FLAT_DEVICE_HIERARCHY=FLAT +export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 + +# Falls back to the current host on an interactive node without PBS_NODEFILE. +if [ -n "${PBS_NODEFILE:-}" ]; then + MASTER_ADDR=$(head -n 1 "$PBS_NODEFILE") +else + MASTER_ADDR=$(hostname) +fi +export MASTER_ADDR +export MASTER_PORT=$((20000 + RANDOM % 20000)) + +for p in "${PRISM_DATA_ROOT}/dna_local_shards/kegg/dataset_info.json" \ + "${HF_HOME}/hub/models--allenai--OLMo-1B-0724-hf" \ + "${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species"; do + if [ ! -e "$p" ]; then + echo "ERROR: missing required path: $p" + exit 1 + fi +done + +echo "================================================================" +echo " BIOREASON STAGE 1 PROJECTOR WARMUP (KEGG only, OLMo-1B, Aurora)" +echo " Node: $(hostname)" +echo " Master: ${MASTER_ADDR}:${MASTER_PORT}" +echo " Ranks: ${NPERNODE}" +echo " KEGG shard: ${PRISM_DATA_ROOT}/dna_local_shards/kegg" +echo " Overrides: ${TRAIN_OVERRIDES:- (none; using bioreason_projector.yaml)}" +echo " Run dir: ${HYDRA_RUN_DIR}" +echo "================================================================" + +mpiexec -n "${NPERNODE}" -ppn "${NPERNODE}" --cpu-bind depth --depth 8 bash -lc ' + module use /soft/modulefiles + module load frameworks/2026.1.0 + source '"${PRISM_VENV}"'/bin/activate + + export CCL_PROCESS_LAUNCHER=none + export CCL_ATL_TRANSPORT=ofi + export CCL_OP_SYNC=1 + export CCL_WORKER_COUNT=1 + export CCL_ALLREDUCE=ring + export CCL_OFI_ENABLE_HOSTNAME_SHARING=0 + export FI_CXI_RX_MATCH_MODE=hybrid + + export WORLD_SIZE=${PMI_SIZE:-${PALS_SIZE:-'"${NPERNODE}"'}} + export RANK=${PMI_RANK:-${PALS_RANKID:-0}} + export LOCAL_RANK=${PMI_LOCAL_RANK:-${PALS_LOCAL_RANKID:-0}} + export LOCAL_WORLD_SIZE=${PMI_LOCAL_SIZE:-${PALS_LOCAL_SIZE:-'"${NPERNODE}"'}} + export ZE_AFFINITY_MASK=$LOCAL_RANK + # Each rank now sees exactly one tile, as xpu:0. Accelerate/DDP build the + # device from LOCAL_RANK, so leaving it at 0..11 targets xpu: -> + # "device index is out of range". Same normalization as tools/launch_aurora_web.py. + export ACCELERATE_ORIG_LOCAL_RANK=$LOCAL_RANK + export LOCAL_RANK=0 PMI_LOCAL_RANK=0 PMIX_LOCAL_RANK=0 PALS_LOCAL_RANKID=0 OMPI_COMM_WORLD_LOCAL_RANK=0 + export MASTER_ADDR='"${MASTER_ADDR}"' + export MASTER_PORT='"${MASTER_PORT}"' + + cd '"${PROJ_DIR}"' + python -u src/train.py \ + training=bioreason_projector \ + model=prism_olmo_1b \ + exp.id='"${RUN_ID}"' \ + exp.variant='"${RUN_VARIANT}"' \ + training.bioreason_dataset=wanglab/kegg \ + '"${TRAIN_OVERRIDES}"' \ + wandb.mode='"${WANDB_MODE}"' \ + hydra.run.dir='"${HYDRA_RUN_DIR}"' +' + +CKPT="${HYDRA_RUN_DIR}/checkpoints/step_final" +[ -f "${CKPT}/model.safetensors" ] || CKPT="" +echo "" +echo "================================================================" +echo " STAGE 1 DONE. Projector checkpoint: ${CKPT:-}" +echo " Stage 2: qsub -v STAGE1_CKPT=${CKPT} applications/bioreason/scripts/training/submit_bioreason_sft.sh" +echo "================================================================" diff --git a/applications/bioreason/scripts/training/submit_bioreason_sft.sh b/applications/bioreason/scripts/training/submit_bioreason_sft.sh new file mode 100644 index 0000000..f203635 --- /dev/null +++ b/applications/bioreason/scripts/training/submit_bioreason_sft.sh @@ -0,0 +1,183 @@ +#!/bin/bash -l +#PBS -N bioreason-sft-kegg-1b +#PBS -l select=1 +#PBS -l walltime=06:00:00 +#PBS -l filesystems=home:flare +#PBS -q capacity +# Allocation is not hardcoded: submit with `qsub -A ...`. +#PBS -k doe +#PBS -j oe +# +# Aurora XPU launcher: BioReason Stage 2 SFT on KEGG (wanglab/kegg) only. +# base model : OLMo-1B (model=prism_olmo_1b), backbone frozen +# trainable : DNA projector (random init, no Stage 1) + LoRA adapters +# frozen : NT-2.5B DNA encoder +# loss : assistant span only +# (\n{reasoning}\n\n\nAnswer: {answer}<|im_end|>) +# +# Batch job: +# qsub applications/bioreason/scripts/training/submit_bioreason_sft.sh +# qsub -v MAX_STEPS=2000 applications/bioreason/scripts/training/submit_bioreason_sft.sh +# # optional: start from a pretrained projector instead of random init +# qsub -v STAGE1_CKPT=/path/to/checkpoints/step_N applications/bioreason/scripts/training/submit_bioreason_sft.sh +# +# Interactive node (after `qsub -I -l select=1,walltime=01:00:00,filesystems=home:flare -q debug -A `): +# bash applications/bioreason/scripts/training/submit_bioreason_sft.sh +# Single-tile debug run: +# NPERNODE=1 MAX_STEPS=20 bash applications/bioreason/scripts/training/submit_bioreason_sft.sh + +set -eo pipefail + +PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/null && pwd)}" +# Under qsub the script runs from a spool copy, so BASH_SOURCE points there. +if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then + PROJ_DIR="${PBS_O_WORKDIR}" +fi +RUN_ID="${RUN_ID:-BIOREASON-SFT-KEGG}" +RUN_VARIANT="1B" +RUN_TAG="${RUN_TAG:-$(date +%Y%m%d-%H%M%S)}" +# Hyperparameters come from src/conf/training/bioreason_sft.yaml; set MAX_STEPS to override max_steps. +TRAIN_OVERRIDES="" +[ -n "${MAX_STEPS:-}" ] && TRAIN_OVERRIDES+=" training.max_steps=${MAX_STEPS}" +NPERNODE="${NPERNODE:-12}" # 6 PVC GPUs x 2 tiles per Aurora node (FLAT mode) + +# Site roots: datasets_config.json resolves dna_bioreason's local_path as +# ${PRISM_DATA_ROOT}/dna_local_shards/kegg. +export PRISM_DATA_ROOT="${PRISM_DATA_ROOT:-/lus/flare/projects//${USER}/cache}" +export PRISM_OUTPUT_ROOT="${PRISM_OUTPUT_ROOT:-/lus/flare/projects//${USER}/output}" +# Checkpoints (~13 GB each) go to flare, not Hydra's default outputs/ under the +# repo in /home, which overflows its quota. +export PRISM_CKPT_ROOT="${PRISM_CKPT_ROOT:-/lus/flare/projects//${USER}/modcon/checkpoints}" +HYDRA_RUN_DIR="${HYDRA_RUN_DIR:-${PRISM_CKPT_ROOT}/${RUN_ID}/${RUN_TAG}}" + +# Optional: Stage 1 (bioreason_projector, model=prism_olmo_1b) checkpoint. Unset +# (default) = projector trains from random init, as in BioReason. If set, its +# output width must match OLMo-1B's d_model (2048). +STAGE1_CKPT="${STAGE1_CKPT:-}" +RESUME_ARG="" +if [ -n "${STAGE1_CKPT}" ]; then + RESUME_ARG="training.resume_weights_only=${STAGE1_CKPT}" +fi + +cd "${PROJ_DIR}" + +# frameworks module + a --system-site-packages venv adding only what the module +# lacks (wandb, torchtune, webdataset, braceexpand). Not the repo's .venv, +# which carries a CUDA torch build. Build it once (from a UAN) with: +# module load frameworks/2026.1.0 +# uv venv --system-site-packages -p "$(which python)" $PRISM_VENV && source $PRISM_VENV/bin/activate +# uv pip install wandb braceexpand==0.1.7 && uv pip install --no-deps webdataset==1.0.2 torchao==0.5.0 +# # torchtune comes from the n-getty fork (the PyPI wheel lacks torchtune.dev.rl, which GRPO needs): +# git clone https://github.com/n-getty/torchtune-aurora.git $(dirname $PRISM_VENV)/torchtune-aurora +# uv pip install --no-deps -e $(dirname $PRISM_VENV)/torchtune-aurora +export PRISM_VENV="${PRISM_VENV:-/flare//${USER}/prism-envs/bioreason-fw2026.1}" +module use /soft/modulefiles +module load frameworks/2026.1.0 +module load hdf5 +if [ ! -f "${PRISM_VENV}/bin/activate" ]; then + echo "ERROR: PRISM_VENV not found: ${PRISM_VENV} (see build steps above)" + exit 1 +fi +source "${PRISM_VENV}/bin/activate" + +if [[ ! "${HOSTNAME}" =~ aurora-uan ]]; then + export HTTP_PROXY="http://proxy.alcf.anl.gov:3128" + export HTTPS_PROXY="http://proxy.alcf.anl.gov:3128" + export http_proxy="http://proxy.alcf.anl.gov:3128" + export https_proxy="http://proxy.alcf.anl.gov:3128" +fi + +# OLMo-1B-0724-hf, nucleotide-transformer-2.5b-multi-species and +# datasets--wanglab--kegg are all pre-cached here; run fully offline. +export HF_HOME="/flare//${USER}/cache/model_weights" +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export HF_DATASETS_OFFLINE=1 +export WANDB_MODE="${WANDB_MODE:-offline}" +export WANDB_SILENT=true +# PBS's default TMPDIR overflows the 108-byte AF_UNIX socket path limit that +# torch's DataLoader worker IPC relies on -- force a short one. +export TMPDIR=/tmp +# DDP side-copy streams index torch.accelerator.device_count(), which is 0 +# under ZE_AFFINITY_MASK on Aurora -> IndexError on first forward. +export PYTORCH_DDP_USE_SIDE_STREAM=0 +export PYTHONUNBUFFERED=1 +export PYTHONFAULTHANDLER=1 +export PYTHONWARNINGS=ignore +export NUMEXPR_MAX_THREADS=512 +export NUMEXPR_NUM_THREADS=128 +export ZE_FLAT_DEVICE_HIERARCHY=FLAT +export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 +# Text + 2 x max_dna_length (1024) DNA slots is ~2.8-3.6k tokens for KEGG; the +# default 2048 merged-length guard (model.py, issue #123) would reject it. +export MAX_SEQ_LENGTH="${MAX_SEQ_LENGTH:-4096}" + +# Falls back to the current host on an interactive node without PBS_NODEFILE. +if [ -n "${PBS_NODEFILE:-}" ]; then + MASTER_ADDR=$(head -n 1 "$PBS_NODEFILE") +else + MASTER_ADDR=$(hostname) +fi +export MASTER_ADDR +export MASTER_PORT=$((20000 + RANDOM % 20000)) + +REQUIRED=("${PRISM_DATA_ROOT}/dna_local_shards/kegg/dataset_info.json" \ + "${HF_HOME}/hub/models--allenai--OLMo-1B-0724-hf" \ + "${HF_HOME}/hub/models--InstaDeepAI--nucleotide-transformer-2.5b-multi-species") +[ -n "${STAGE1_CKPT}" ] && REQUIRED+=("${STAGE1_CKPT}/model.safetensors") +for p in "${REQUIRED[@]}"; do + if [ ! -e "$p" ]; then + echo "ERROR: missing required path: $p" + exit 1 + fi +done + +echo "================================================================" +echo " BIOREASON SFT (KEGG only, OLMo-1B + LoRA + projector, Aurora)" +echo " Node: $(hostname)" +echo " Master: ${MASTER_ADDR}:${MASTER_PORT}" +echo " Ranks: ${NPERNODE}" +echo " Projector init: ${STAGE1_CKPT:-random (no Stage 1)}" +echo " KEGG shard: ${PRISM_DATA_ROOT}/dna_local_shards/kegg" +echo " Overrides: ${TRAIN_OVERRIDES:- (none; using bioreason_sft.yaml)}" +echo " Run dir: ${HYDRA_RUN_DIR}" +echo "================================================================" + +mpiexec -n "${NPERNODE}" -ppn "${NPERNODE}" --cpu-bind depth --depth 8 bash -lc ' + module use /soft/modulefiles + module load frameworks/2026.1.0 + source '"${PRISM_VENV}"'/bin/activate + + export CCL_PROCESS_LAUNCHER=none + export CCL_ATL_TRANSPORT=ofi + export CCL_OP_SYNC=1 + export CCL_WORKER_COUNT=1 + export CCL_ALLREDUCE=ring + export CCL_OFI_ENABLE_HOSTNAME_SHARING=0 + export FI_CXI_RX_MATCH_MODE=hybrid + + export WORLD_SIZE=${PMI_SIZE:-${PALS_SIZE:-'"${NPERNODE}"'}} + export RANK=${PMI_RANK:-${PALS_RANKID:-0}} + export LOCAL_RANK=${PMI_LOCAL_RANK:-${PALS_LOCAL_RANKID:-0}} + export LOCAL_WORLD_SIZE=${PMI_LOCAL_SIZE:-${PALS_LOCAL_SIZE:-'"${NPERNODE}"'}} + export ZE_AFFINITY_MASK=$LOCAL_RANK + # Each rank now sees exactly one tile, as xpu:0. Accelerate/DDP build the + # device from LOCAL_RANK, so leaving it at 0..11 targets xpu: -> + # "device index is out of range". Same normalization as tools/launch_aurora_web.py. + export ACCELERATE_ORIG_LOCAL_RANK=$LOCAL_RANK + export LOCAL_RANK=0 PMI_LOCAL_RANK=0 PMIX_LOCAL_RANK=0 PALS_LOCAL_RANKID=0 OMPI_COMM_WORLD_LOCAL_RANK=0 + export MASTER_ADDR='"${MASTER_ADDR}"' + export MASTER_PORT='"${MASTER_PORT}"' + + cd '"${PROJ_DIR}"' + python -u src/train.py \ + training=bioreason_sft \ + model=prism_olmo_1b \ + exp.id='"${RUN_ID}"' \ + exp.variant='"${RUN_VARIANT}"' \ + '"${RESUME_ARG}"' \ + training.bioreason_dataset=wanglab/kegg \ + '"${TRAIN_OVERRIDES}"' \ + wandb.mode='"${WANDB_MODE}"' \ + hydra.run.dir='"${HYDRA_RUN_DIR}"' +' diff --git a/applications/bioreason/submit_bioreason_projector.sh b/applications/bioreason/submit_bioreason_projector.sh deleted file mode 100755 index 49ad8e5..0000000 --- a/applications/bioreason/submit_bioreason_projector.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env bash -# Polaris-specific launcher (PBS/mpiexec/cudatoolkit-standalone/argonne_tpc -# allocation) -- unlike this repo's other, Aurora-first launchers. -#PBS -N bioreason_projector_kegg -#PBS -l select=1 -#PBS -l walltime=04:00:00 -#PBS -j oe -#PBS -l filesystems=home:eagle -#PBS -A argonne_tpc -#PBS -q preemptable - -set -euo pipefail - -PROJ_DIR="/home/abalaji/projects/modcon/genome/BaseMM_PRISM" -OUTPUT_BASE="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output" -RUN_ID="GENOME-PROJECTOR-KEGG" -RUN_VARIANT="1B" - -# Set to a checkpoint directory to resume this same stage (optimizer/scheduler/step -# included), e.g. after preemption. Leave empty to start a fresh run. -# RESUME_CKPT="${OUTPUT_BASE}/GENOME-PROJECTOR-1B/2026-07-01/12-00-00/checkpoints/step_2500" -RESUME_CKPT="" - -cd "${PROJ_DIR}" - -# --- Cuda and NCCL (Polaris) --- -ml use /soft/modulefiles -ml --ignore_cache load cudatoolkit-standalone/12.6.1 - -# --- Conda env --- -ml --ignore_cache load conda -conda activate /lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism - -# --- Proxy --- -export HTTP_PROXY="http://proxy.alcf.anl.gov:3128" -export HTTPS_PROXY="http://proxy.alcf.anl.gov:3128" -export http_proxy="http://proxy.alcf.anl.gov:3128" -export https_proxy="http://proxy.alcf.anl.gov:3128" -export no_proxy="127.0.0.1,admin,polaris-adminvm-01,localhost,*.cm.polaris.alcf.anl.gov,polaris-*,*.polaris.alcf.anl.gov,*.alcf.anl.gov" - -# --- HuggingFace cache --- -export HF_HOME="/lus/eagle/projects/argonne_tpc/abalaji/model_weights" -export HF_DATASETS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights" -export TRANSFORMERS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub" -export HF_HUB_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub" - -# --- W&B --- -# No WANDB_API_KEY here -- a plaintext key was committed here previously -# and has been removed; rotate that key on wandb.ai if it hasn't been -# already. wandb picks up credentials from ~/.netrc (machine api.wandb.ai) -# or an already-exported WANDB_API_KEY in the calling shell automatically -# -- nothing to set here. - -# --- NCCL (Polaris) --- -export NCCL_NET_GDR_LEVEL=PHB -export NCCL_CROSS_NIC=1 -export NCCL_COLLNET_ENABLE=1 -export LD_LIBRARY_PATH=/soft/libraries/aws-ofi-nccl/v1.9.1-aws/lib:${LD_LIBRARY_PATH:-} - -# --- General --- -export PYTHONUNBUFFERED=1 -export PYTHONFAULTHANDLER=1 -export PYTHONWARNINGS=ignore -export OMP_NUM_THREADS=8 -export NUMEXPR_MAX_THREADS=64 - -# --- Distributed --- -# Falls back to the current host when run directly on an already-allocated -# interactive node (no PBS_NODEFILE, e.g. after `qsub -I`). -if [ -n "${PBS_NODEFILE:-}" ]; then - NNODES=$(sort -u "$PBS_NODEFILE" | wc -l) - MASTER_ADDR=$(sort -u "$PBS_NODEFILE" | head -n 1) -else - NNODES=1 - MASTER_ADDR=$(hostname) -fi -NPERNODE=4 # 4 GPUs per Polaris node -NPROCS=$((NNODES * NPERNODE)) - -export MASTER_ADDR -export MASTER_PORT=$((20000 + RANDOM % 20000)) -export WORLD_SIZE=${NPROCS} - -echo "=== BioReason Projector Training (Stage 1) ===" -echo "Node: $(hostname)" -echo "PBS_NODEFILE: ${PBS_NODEFILE:-}" -echo "NNODES=${NNODES} NPERNODE=${NPERNODE} NPROCS=${NPROCS}" -echo "MASTER_ADDR=${MASTER_ADDR} MASTER_PORT=${MASTER_PORT}" -echo "Output base: ${OUTPUT_BASE}" -echo "Resume ckpt: ${RESUME_CKPT:-}" -echo "" - -# Unquoted splice below: when empty, this contributes zero words to the command -# line (not an empty-string arg), so train.py/Hydra sees a normal 3-override -# invocation when not resuming. -RESUME_OVERRIDE="" -if [ -n "${RESUME_CKPT}" ]; then - RESUME_OVERRIDE="training.resume_from_checkpoint=${RESUME_CKPT}" -fi - -mpiexec -n "${NPROCS}" -ppn "${NPERNODE}" \ - --envall \ - bash -lc ' - export RANK=${PMI_RANK:-${PALS_RANKID:-0}} - export WORLD_SIZE=${PMI_SIZE:-${PALS_SIZE:-'"${NPROCS}"'}} - export LOCAL_RANK=${PMI_LOCAL_RANK:-${PALS_LOCAL_RANKID:-0}} - export LOCAL_WORLD_SIZE=${PMI_LOCAL_SIZE:-${PALS_LOCAL_SIZE:-'"${NPERNODE}"'}} - export MASTER_ADDR="'"${MASTER_ADDR}"'" - export MASTER_PORT="'"${MASTER_PORT}"'" - - cd '"${PROJ_DIR}"' - python -u src/train.py \ - training=bioreason_projector \ - exp.id='"${RUN_ID}"' \ - exp.variant='"${RUN_VARIANT}"' \ - '"${RESUME_OVERRIDE}"' - ' diff --git a/applications/bioreason/submit_bioreason_sft.sh b/applications/bioreason/submit_bioreason_sft.sh deleted file mode 100644 index e69e99e..0000000 --- a/applications/bioreason/submit_bioreason_sft.sh +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env bash -# Polaris-specific launcher (PBS/mpiexec/cudatoolkit-standalone/argonne_tpc -# allocation) -- unlike this repo's other, Aurora-first launchers. -#PBS -N prism-sft -#PBS -l select=1 -#PBS -l walltime=07:00:00 -#PBS -j oe -#PBS -l filesystems=home:eagle -#PBS -A argonne_tpc -#PBS -q preemptable - -set -euo pipefail - -PROJ_DIR="/home/abalaji/projects/modcon/genome/BaseMM_PRISM" -OUTPUT_BASE="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output" -RUN_ID="GENOME-SFT" -RUN_VARIANT="1B" -STAGE1_CKPT="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-PROJECTOR-1B/2026-07-02/17-50-47/checkpoints/step_1000" -#/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-28/18-18-57/checkpoints/step_3900 -#"/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-PROJECTOR-1B/2026-06-18/16-57-40/checkpoints/step_1500" -cd "${PROJ_DIR}" - -# --- Cuda and NCCL (Polaris) --- -ml use /soft/modulefiles -ml --ignore_cache load cudatoolkit-standalone/12.6.1 - -# --- Conda env --- -ml --ignore_cache load conda -conda activate /lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism - -# --- Proxy --- -export HTTP_PROXY="http://proxy.alcf.anl.gov:3128" -export HTTPS_PROXY="http://proxy.alcf.anl.gov:3128" -export http_proxy="http://proxy.alcf.anl.gov:3128" -export https_proxy="http://proxy.alcf.anl.gov:3128" -export no_proxy="127.0.0.1,admin,polaris-adminvm-01,localhost,*.cm.polaris.alcf.anl.gov,polaris-*,*.polaris.alcf.anl.gov,*.alcf.anl.gov" - -# --- HuggingFace cache --- -export HF_HOME="/lus/eagle/projects/argonne_tpc/abalaji/model_weights" -export HF_DATASETS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights" -export TRANSFORMERS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub" -export HF_HUB_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub" - -# --- W&B --- -# No WANDB_API_KEY here -- a plaintext key was committed here previously -# and has been removed; rotate that key on wandb.ai if it hasn't been -# already. wandb picks up credentials from ~/.netrc (machine api.wandb.ai) -# or an already-exported WANDB_API_KEY in the calling shell automatically -# -- nothing to set here. - -# --- NCCL (Polaris) --- -export NCCL_NET_GDR_LEVEL=PHB -export NCCL_CROSS_NIC=1 -export NCCL_COLLNET_ENABLE=1 -export LD_LIBRARY_PATH=/soft/libraries/aws-ofi-nccl/v1.9.1-aws/lib:${LD_LIBRARY_PATH:-} - -# --- General --- -export PYTHONUNBUFFERED=1 -export PYTHONFAULTHANDLER=1 -export PYTHONWARNINGS=ignore -export OMP_NUM_THREADS=8 -export NUMEXPR_MAX_THREADS=64 - -# --- Distributed --- -# Falls back to the current host when run directly on an already-allocated -# interactive node (no PBS_NODEFILE, e.g. after `qsub -I`). -if [ -n "${PBS_NODEFILE:-}" ]; then - NNODES=$(sort -u "$PBS_NODEFILE" | wc -l) - MASTER_ADDR=$(sort -u "$PBS_NODEFILE" | head -n 1) -else - NNODES=1 - MASTER_ADDR=$(hostname) -fi -NPERNODE=4 # 4 GPUs per Polaris node -NPROCS=$((NNODES * NPERNODE)) - -export MASTER_ADDR -export MASTER_PORT=$((20000 + RANDOM % 20000)) -export WORLD_SIZE=${NPROCS} - -echo "=== BioReason SFT Training (Zone C / Stage 2) ===" -echo "Node: $(hostname)" -echo "PBS_NODEFILE: ${PBS_NODEFILE:-}" -echo "NNODES=${NNODES} NPERNODE=${NPERNODE} NPROCS=${NPROCS}" -echo "MASTER_ADDR=${MASTER_ADDR} MASTER_PORT=${MASTER_PORT}" -echo "Stage 1 ckpt: ${STAGE1_CKPT}" -echo "Output base: ${OUTPUT_BASE}" -echo "" - -mpiexec -n "${NPROCS}" -ppn "${NPERNODE}" \ - --envall \ - bash -lc ' - export RANK=${PMI_RANK:-${PALS_RANKID:-0}} - export WORLD_SIZE=${PMI_SIZE:-${PALS_SIZE:-'"${NPROCS}"'}} - export LOCAL_RANK=${PMI_LOCAL_RANK:-${PALS_LOCAL_RANKID:-0}} - export LOCAL_WORLD_SIZE=${PMI_LOCAL_SIZE:-${PALS_LOCAL_SIZE:-'"${NPERNODE}"'}} - export MASTER_ADDR="'"${MASTER_ADDR}"'" - export MASTER_PORT="'"${MASTER_PORT}"'" - - cd '"${PROJ_DIR}"' - python -u src/train.py \ - training=bioreason_sft \ - exp.id='"${RUN_ID}"' \ - exp.variant='"${RUN_VARIANT}"' \ - training.resume_weights_only='"${STAGE1_CKPT}"' - ' diff --git a/docs/applications/bioreason-eval.md b/docs/applications/bioreason-eval.md index ccec64b..9037870 100644 --- a/docs/applications/bioreason-eval.md +++ b/docs/applications/bioreason-eval.md @@ -4,7 +4,7 @@ Evaluates a BioReason SFT or GRPO checkpoint against the KEGG test split. > [!IMPORTANT] > `eval_kegg.py` is **not yet on `main`** — it ships with the eval-pipeline PR -> (#133). `applications/bioreason/submit_eval_kegg.sh` already invokes it, so both this page +> (#133). `applications/bioreason/scripts/eval/submit_eval_kegg.sh` already invokes it, so both this page > and that script only work once #133 lands. The invocation below is recorded > so the arguments are documented; it will not run against a clean checkout > today. diff --git a/docs/applications/bioreason_grpo.md b/docs/applications/bioreason_grpo.md index ecb127d..b20f520 100644 --- a/docs/applications/bioreason_grpo.md +++ b/docs/applications/bioreason_grpo.md @@ -29,9 +29,9 @@ the Stage 3 GRPO reinforcement learning implementation. Each stage loads weights from the previous via `training.resume_weights_only`. Submit scripts: -- Stage 1: `applications/bioreason/submit_bioreason_projector.sh` -- Stage 2: `applications/bioreason/submit_bioreason_sft.sh` -- Stage 3: `applications/bioreason/submit_bioreason_grpo.sh` +- Stage 1: `applications/bioreason/scripts/training/submit_bioreason_projector.sh` +- Stage 2: `applications/bioreason/scripts/training/submit_bioreason_sft.sh` +- Stage 3: `applications/bioreason/scripts/training/submit_bioreason_grpo.sh` --- diff --git a/src/conf/data/bioreason_demo_mix.yaml b/src/conf/data/bioreason_demo_mix.yaml new file mode 100644 index 0000000..4794d9a --- /dev/null +++ b/src/conf/data/bioreason_demo_mix.yaml @@ -0,0 +1,17 @@ +# Written by applications/bioreason/bioreason_demo.ipynb (section 5). +# KEGG + variant-effect mix (BioReason weights). Use with: data=bioreason_demo_mix +dataset_overrides: + dna_bioreason: + skip: false + preferred_source: local + weight: 0.5 + dna_bioreason_variant_effect_coding: + skip: false + preferred_source: local + weight: 0.25 + local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_coding + dna_bioreason_variant_effect_non_snv: + skip: false + preferred_source: local + weight: 0.25 + local_path: ${oc.env:PRISM_DATA_ROOT}/dna_local_shards/variant_effect_non_snv diff --git a/src/conf/model/prism_olmo_1b.yaml b/src/conf/model/prism_olmo_1b.yaml index 9a3bfcc..1e67355 100644 --- a/src/conf/model/prism_olmo_1b.yaml +++ b/src/conf/model/prism_olmo_1b.yaml @@ -51,6 +51,22 @@ modalities: # - "geometry" # - "graph" +# Interleaved DNA splice tokens. Authored as literal special-token strings +# (unlike time_series's pre-baked fixed integer ids) — UnifiedTransformer.__init__ +# registers any missing ones as new special tokens and resolves this to integer +# ids before the merge path runs. Must match _render_dna_prompt_text +# (src/data/multimodal.py) exactly. +# +# Required for correctness, not just for GRPO: without is_interleaved_qa=true, +# multimodal.py's DNA branch falls through to the "[dna_bioreason]" full- +# sequence-loss sentinel (the same one the projector-alignment stage uses) +# instead of computing the "P T" prompt/target split SFT needs to mask loss to +# just the assistant span. ZoneDTrainer (GRPO) also asserts this unconditionally. +is_interleaved_qa: true +modality_start_end_token_indices: + dna_reference: ["", ""] + dna_variant: ["", ""] + # IsoFLOP projector capacity knobs (defaults match ModelConfig dataclass) -- # Hydra struct mode rejects overrides for keys not declared in the YAML, # so callers can pass `model.projector_hidden_mult=N` without `+` prefix. diff --git a/src/conf/training/bioreason_grpo.yaml b/src/conf/training/bioreason_grpo.yaml index 35e4178..d4306ff 100644 --- a/src/conf/training/bioreason_grpo.yaml +++ b/src/conf/training/bioreason_grpo.yaml @@ -56,7 +56,7 @@ grpo_reward_functions: - correctness # Data -data_num_workers: 4 +data_num_workers: 1 # KEGG local shard is a single Arrow file; extra workers get no shard and the stream looks empty bioreason_dataset: "wanglab/kegg" max_dna_length: 1024 max_text_length: 1024 diff --git a/src/conf/training/bioreason_projector.yaml b/src/conf/training/bioreason_projector.yaml index be83035..f5a4eb3 100644 --- a/src/conf/training/bioreason_projector.yaml +++ b/src/conf/training/bioreason_projector.yaml @@ -4,12 +4,14 @@ # Based on: https://arxiv.org/abs/2505.23579 task: "bioreason_projector" -batch_size: 8 -max_steps: 5000 # ~5 epochs over KEGG (1450 samples) +batch_size: 1 # batch 8 at ~3k merged tokens (1024 DNA slots x2 + text) OOMs an Aurora tile +max_steps: 100 # demo-sized run; raise for a full alignment (the SFT-stage runs used 500) device: "auto" learning_rate: 5e-5 -gradient_accumulation_steps: 8 -eval_enabled: true # ZoneATrainer's periodic-eval block is opt-in; must be true for the KEGG eval below to fire +gradient_accumulation_steps: 1 +# Off: ZoneATrainer.run_kegg_eval imports eval_kegg.py, which is not in this repo, +# so the first eval step would raise ImportError and kill the run. +eval_enabled: false eval_every_n_steps: 200 # Also drives the KEGG validation eval below viz_every_n_steps: 1000 @@ -19,13 +21,13 @@ viz_every_n_steps: 1000 kegg_eval_max_examples: null # null = full val split # Checkpointing -save_every_n_steps: 500 +save_every_n_steps: 50 resume_from_checkpoint: null resume_weights_only: null output_dir: ${hydra:run.dir} # Optimization -warmup_steps: 500 # 10% of max_steps=5000 +warmup_steps: 10 # 10% of max_steps=100 weight_decay: 0.01 scheduler_type: "cosine" min_lr_ratio: 0.0 @@ -50,7 +52,7 @@ use_reasoning_traces: false data_num_workers: 1 # BioReason-specific -bioreason_dataset: "wanglab/kegg" # or variant_effect_coding, variant_effect_non_snv +bioreason_dataset: "wanglab/kegg" max_dna_length: 1024 max_text_length: 1024 dna_truncation_per_side: 1024 diff --git a/src/conf/training/bioreason_sft.yaml b/src/conf/training/bioreason_sft.yaml index e6432a0..a439b8d 100644 --- a/src/conf/training/bioreason_sft.yaml +++ b/src/conf/training/bioreason_sft.yaml @@ -1,20 +1,37 @@ -# BioReason Stage 2: SFT with LoRA -# Loads projector weights from Stage 1 (bioreason_projector.yaml) and freezes them. -# Trains only LoRA adapters on the LLM backbone. DNA encoder remains frozen. +# BioReason SFT with LoRA on KEGG (wanglab/kegg), 1B OLMo base. +# Single-stage, matching the BioReason repo (no projector pre-training stage): +# the DNA projector starts from its fresh random init and trains jointly with +# the LoRA adapters on the frozen LLM backbone. DNA encoder (NT-2.5B) remains +# frozen. Pass training.resume_weights_only= only to opt in to +# initializing from a pretrained projector (bioreason_projector.yaml). # Based on: https://arxiv.org/abs/2505.23579 # -# Before running: set resume_weights_only to the Stage 1 checkpoint path, e.g.: -# resume_weights_only: /path/to/stage1/outputs/.../checkpoints/step_final +# Must be composed with the 1B model (KEGG-only comes from datasets_config.json, where only dna_bioreason is enabled): +# python src/train.py training=bioreason_sft model=prism_olmo_1b +# (config.yaml's default model is prism_olmo3_7b, so model=prism_olmo_1b is required.) +# +# Loss: labels are -100 everywhere except the assistant span +# <|assistant|>\n[ \n{reasoning}\n\n\nAnswer: {answer}<|endoftext|> ] +# (OLMo turn format: plain-text <|user|>/<|assistant|> headers, OLMo eos <|endoftext|> ends the target; +# no Qwen <|im_start|>/<|im_end|>.) +# (prompt/target split via the "P T" _metadata, requires model.is_interleaved_qa=true). +# Launcher: applications/bioreason/scripts/training/submit_bioreason_sft.sh task: "bioreason_sft" +# Per-rank batch 1 x accumulation 1 x 12 ranks = 12 samples/step (BioReason: 1 x 8 on 1 GPU). +# batch_size 8 at ~2.8-3.6k merged tokens/sample ran out of XPU memory (~40 s/micro-batch). +# accumulation 1 also keeps the scheduler units right: the trainer sizes the LR schedule as +# max_steps x num_ranks optimizer steps, which only matches when step == optimizer step. batch_size: 1 -max_steps: 10000 # ~10 epochs over KEGG (1450 train samples); previous 5000 steps showed no convergence trend +max_steps: 500 # ~5 epochs over the local KEGG shard (1159 train samples / 12 per step = ~97 steps/epoch), as in BioReason device: "auto" learning_rate: 5e-5 -gradient_accumulation_steps: 8 -eval_enabled: true # ZoneATrainer's periodic-eval block is opt-in; must be true for the KEGG eval below to fire +gradient_accumulation_steps: 1 +# Off: ZoneATrainer.run_kegg_eval imports eval_kegg.py, which is not in this repo, +# so the first eval step would raise ImportError and kill the run. +eval_enabled: false eval_every_n_steps: 1000 # Also drives the KEGG validation eval below -viz_every_n_steps: 1000 +viz_every_n_steps: 100 # KEGG validation eval (wanglab/kegg 'val' split), run inline during training on # the eval_every_n_steps cadence above. See ZoneATrainer.run_kegg_eval / @@ -22,31 +39,33 @@ viz_every_n_steps: 1000 kegg_eval_max_examples: null # null = full val split # Checkpointing -save_every_n_steps: 500 +save_every_n_steps: 100 resume_from_checkpoint: null -# Set this to the Stage 1 checkpoint directory to load projector weights. -# Loads model weights only — optimizer and step counter reset for the new stage. -resume_weights_only: ${oc.env:PRISM_OUTPUT_ROOT,}/GENOME-PROJECTOR-1B/2026-06-18/16-57-40/checkpoints/step_1500 +# Optional model-weights-only init (optimizer and step counter reset). null = +# projector starts from random init, as in BioReason. Set to a +# bioreason_projector (same model=prism_olmo_1b) checkpoint to start from a +# pretrained projector instead. +resume_weights_only: null output_dir: ${hydra:run.dir} # Optimization -warmup_steps: 300 # 3% of max_steps=10000; reduced from 500 — projector already pretrained +warmup_steps: 50 # 10% of max_steps=500, as in BioReason (get_cosine_schedule_with_warmup, 0.1 * total); projector is untrained weight_decay: 0.01 scheduler_type: "cosine" min_lr_ratio: 0.0 # Differential Learning Rates -lr_connector: 5e-5 # Projector LR +lr_connector: 5e-5 # Projector LR (random init, trains jointly with LoRA; same as BioReason's single lr 5e-5) lr_dna: 5e-6 # DNA encoder LR — only active if freeze_vit: false (frozen by default) -lr_llm: null # LLM frozen by default +lr_llm: null # Also the LoRA LR (null -> learning_rate); base LLM is frozen # Freezing # freeze_llm and lora_enabled are independent knobs: freeze_llm freezes the # base backbone, lora_enabled adds trainable LoRA adapters on top. Both true -# here -- base backbone frozen, only the LoRA adapters train. -freeze_llm: true # base backbone frozen; LoRA adapters (below) are the only trainable part +# here -- base backbone frozen, LoRA adapters + DNA projector train. +freeze_llm: true # base backbone frozen; LoRA adapters (below) train on top freeze_vit: true # DNA encoder frozen (matches BioReason default) -freeze_connector: true # projector loaded from Stage 1 and frozen; only LoRA trains +freeze_connector: false # projector randomly initialized and trained (lr_connector) # LoRA (applied to LLM backbone) lora_enabled: true @@ -61,14 +80,17 @@ use_reasoning_traces: true # Class-balanced loss: inverse-frequency weighting so rare diseases get proportional gradient. # Weights computed from wanglab/kegg train split at dataset init. # w_c = min(N / (C * n_c), class_weight_max) — capped to avoid n=1 instability. -use_class_weights: true +# Disabled: plain unweighted CE over the assistant span. (When true, the weights +# are counted via load_dataset("wanglab/kegg") from the Hub, which fails on +# offline compute nodes and silently falls back to uniform weights anyway.) +use_class_weights: false class_weight_max: 10.0 # Data data_num_workers: 1 # BioReason-specific -bioreason_dataset: "wanglab/kegg" # or variant_effect_coding, variant_effect_non_snv +bioreason_dataset: "wanglab/kegg" max_dna_length: 1024 max_text_length: 1024 dna_truncation_per_side: 1024 diff --git a/src/config.py b/src/config.py index e9b6952..bbc8a6a 100644 --- a/src/config.py +++ b/src/config.py @@ -208,6 +208,10 @@ class ModelConfig: # DNA encoder settings dna_model_name: str = "InstaDeepAI/nucleotide-transformer-2.5b-multi-species" dna_is_evo2: bool = False # If True, DNAEncoder loads an Evo2 checkpoint instead of NT + # Fixed DNA span length in tokens. The data path pads every DNA span to exactly + # this (multimodal.py) and DNAEncoder sizes each interleaved slot from it, so + # both must read the same value. Set from training.max_dna_length in train.py. + max_dna_length: int = 1024 # Image encoder / processor image_encoder_id: str = "google/siglip2-base-patch16-224" diff --git a/src/data/datasets_config.json b/src/data/datasets_config.json index 76cef73..0d1de26 100644 --- a/src/data/datasets_config.json +++ b/src/data/datasets_config.json @@ -196,7 +196,7 @@ "output" ], "fallback_dummy": false, - "skip": false, + "skip": true, "size_formatted": "~105K samples" }, "ts_instruction": { @@ -427,7 +427,7 @@ "variant_sequence" ], "fallback_dummy": false, - "skip": false, + "skip": true, "size_formatted": "48,850 samples (train split)" }, "dna_bioreason_variant_effect_non_snv": { @@ -446,7 +446,7 @@ "mutated_sequence" ], "fallback_dummy": false, - "skip": false, + "skip": true, "size_formatted": "35,215 samples (train split)" } } diff --git a/src/data/multimodal.py b/src/data/multimodal.py index c1acfc8..64edce3 100644 --- a/src/data/multimodal.py +++ b/src/data/multimodal.py @@ -210,14 +210,22 @@ def __init__( # NT tokenizer for DNA-LLM mode — loaded once here, not per item. # LLM mode inlines sequences as plain text and never needs this. + # Loaded from the same checkpoint as the DNA encoder: NT v1 (e.g. 2.5B) + # and NT v2 vocabularies order their 6-mers differently, so a tokenizer + # from another NT release feeds the encoder the wrong k-mer ids. self.dna_tokenizer = None if self.model_name == "dna-llm": + dna_tokenizer_id = getattr( + model_config, + "dna_model_name", + "InstaDeepAI/nucleotide-transformer-2.5b-multi-species", + ) try: self.dna_tokenizer = AutoTokenizer.from_pretrained( - "InstaDeepAI/nucleotide-transformer-v2-250m-multi-species", + dna_tokenizer_id, trust_remote_code=True, ) - logger.info("DNA tokenizer loaded: nucleotide-transformer-v2-250m-multi-species") + logger.info(f"DNA tokenizer loaded: {dna_tokenizer_id}") except Exception as e: logger.warning(f"Could not load DNA tokenizer: {e}") @@ -2663,17 +2671,22 @@ def _process_ts_time_mmd(self, item): def _render_dna_prompt_text(self, prompt: list) -> tuple: """ Renders the chat-format prompt list produced by _process_dna_bioreason into - two flat strings that match the Qwen/ChatML format expected by the tokenizer - and the interleaved merge machinery. + two flat strings in OLMo's own (Tulu-style) turn format -- NOT Qwen/ChatML. + The tokenizer here is OLMo's, which has no <|im_start|>/<|im_end|>: those + would be split into plain subword pieces, so the model could never emit a + real stop token. The only OLMo special token used is its eos + (self.tokenizer.eos_token, "<|endoftext|>"), which ends the assistant turn. + OLMo has no start/BOS token, so none is added; turns are introduced by the + plain-text headers "<|user|>" and "<|assistant|>". The label-masking logic in _merge_text_input_ids_with_modality_embeds locates the boundary between question and answer via the "P T" _metadata - token counts, while the collate fn scans for the literal substring - "<|im_end|>\\n<|im_start|>assistant\\n" to find where the assistant turn - begins. This renderer must produce exactly that substring. + token counts: P covers everything up to and including "<|assistant|>\\n", + T covers the assistant turn through eos (so eos is supervised and the model + learns to stop). User turn rendering: - <|im_start|>user\\n + <|user|>\\n [for each content slot] type=="dna_reference" → Reference sequence: \\n type=="dna_variant" → Variant sequence: \\n @@ -2681,22 +2694,41 @@ def _render_dna_prompt_text(self, prompt: list) -> tuple: modality_start_end_token_indices, NOT the chat-template <|dna_start|><|dna_pad|><|dna_end|> variant) type=="text" → the question text - <|im_end|>\\n + \\n Assistant turn rendering (SFT only): - <|im_start|>assistant\\n + <|assistant|>\\n [if reasoning] \\n{reasoning}\\n\\n\\n - {answer text} - <|im_end|>\\n + {answer text} (KEGG: "Answer: {answer}") + <|endoftext|> (OLMo eos; end of the loss span) Returns: full_text — user turn + assistant turn (use for example["text"]) - prompt_text — user turn only + "<|im_start|>assistant\\n" suffix + prompt_text — user turn only + "<|assistant|>\\n" suffix (use to measure prompt_tokens for _metadata) """ modality_tags = self.model_config.modality_start_end_token_indices or {} - dna_ref_tags = modality_tags.get("dna_reference", ("", "")) - dna_var_tags = modality_tags.get("dna_variant", ("", "")) + + # UnifiedTransformer.__init__ resolves these tag strings to integer ids + # *in place* on the shared ModelConfig, and train.py builds the dataset + # after the model. Rendering an int here would emit its digits as plain + # text ("Reference sequence: 5029250293"), so the merge never finds a + # splice point and the DNA embeddings are silently dropped. Map ids back + # to their token strings instead. + def _tag_str(tag): + if isinstance(tag, str): + return tag + return self.tokenizer.convert_ids_to_tokens(int(tag)) + + dna_ref_tags = tuple( + _tag_str(t) + for t in modality_tags.get("dna_reference", ("", "")) + ) + dna_var_tags = tuple( + _tag_str(t) + for t in modality_tags.get("dna_variant", ("", "")) + ) + eos = getattr(self.tokenizer, "eos_token", None) or "<|endoftext|>" # OLMo EOS dna_ref_placeholder = f"Reference sequence: {dna_ref_tags[0]}{dna_ref_tags[1]}\n" dna_var_placeholder = f"Variant sequence: {dna_var_tags[0]}{dna_var_tags[1]}\n" @@ -2720,25 +2752,27 @@ def _render_dna_prompt_text(self, prompt: list) -> tuple: parts.append(dna_var_placeholder) elif c.get("type") == "text": parts.append(c["text"]) - user_body = f"<|im_start|>user\n{''.join(parts)}<|im_end|>\n" + user_body = f"<|user|>\n{''.join(parts)}\n" elif role == "assistant": + # The sequence ends at eos with nothing after it: the SFT loss span is + # everything after prompt_text, so labels are exactly + # (<|assistant|>\n, eos] and nothing beyond. reasoning = msg.get("reasoning_content", "") answer = msg["content"][0]["text"] if reasoning: assistant_body = ( - f"<|im_start|>assistant\n" + f"<|assistant|>\n" f"\n{reasoning}\n\n\n" - f"{answer}<|im_end|>\n" + f"{answer}{eos}" ) else: - assistant_body = f"<|im_start|>assistant\n{answer}<|im_end|>\n" + assistant_body = f"<|assistant|>\n{answer}{eos}" # prompt_text ends exactly at the assistant marker so that: # len(tokenize(prompt_text)) == P (the P in the "P T" _metadata string) - # The marker "<|im_end|>\n<|im_start|>assistant\n" is the literal string - # relied on elsewhere to find where the assistant turn begins. - prompt_text = user_body + "<|im_start|>assistant\n" + # The marker "\n<|assistant|>\n" is the literal string that ends the prompt. + prompt_text = user_body + "<|assistant|>\n" # SFT: full_text = user turn + complete assistant turn (answer in the sequence). # RL: full_text = user turn + generation prompt only (model generates the answer). @@ -2933,14 +2967,15 @@ def _process_dna_bioreason(self, item): if self.is_sft: # SFT: append full assistant turn so collate fn can mask prompt tokens from loss. + # Target rendered by _render_dna_prompt_text: + # \n{reasoning}\n\n\nAnswer: {answer}<|endoftext|> + answer_text = f"Answer: {answer.strip()}" # Optionally truncate reasoning to stay within token budget. if self.use_reasoning_traces and reasoning and self.tokenizer is not None: MAX_SEQ = 2048 prompt_text = question_text prompt_len = len(self.tokenizer.encode(prompt_text, add_special_tokens=False)) - answer_overhead = len(self.tokenizer.encode( - f"{answer.strip()}", add_special_tokens=False - )) + answer_overhead = len(self.tokenizer.encode(answer_text, add_special_tokens=False)) reasoning_budget = MAX_SEQ - prompt_len - answer_overhead - 16 if reasoning_budget > 0: reasoning_ids = self.tokenizer.encode(reasoning, add_special_tokens=False) @@ -2954,7 +2989,7 @@ def _process_dna_bioreason(self, item): prompt.append({ "role": "assistant", "reasoning_content": reasoning.strip() if self.use_reasoning_traces else "", - "content": [{"type": "text", "text": f"{answer.strip()}"}], + "content": [{"type": "text", "text": answer_text}], }) # RL: no assistant turn — answer is kept as the reward signal only. @@ -3397,6 +3432,7 @@ def __iter__(self): "time_series": [], "geometry": [], "text": [], + "dna": [], } for name in list(self.datasets_objs.keys()): diff --git a/src/train.py b/src/train.py index 42579b1..451175e 100644 --- a/src/train.py +++ b/src/train.py @@ -255,71 +255,12 @@ def analyze_vla_dataset(dataset, train_config): logger.info("=" * 50 + "\n") -@hydra.main(config_path="conf", config_name="config", version_base="1.2") -def main(cfg: DictConfig): - if _is_main_env(): - print_prism_banner("Foundational Multimodal Training") - logger.info(f"--- Starting PRISM Training [{cfg.exp.id}] ---") - logger.info(OmegaConf.to_yaml(cfg)) - - # 0. Determine execution path - use_native_fsdp = os.environ.get("USE_NATIVE_FSDP", "0") == "1" - use_native_ddp = os.environ.get("USE_NATIVE_DDP", "0") == "1" - - # Defaults for Accelerate path (setup_distributed is native-only). - def _env_int(keys, default): - for key in keys: - val = os.environ.get(key, "") - if val and val.strip(): - try: - return int(val) - except ValueError: - continue - return default - - rank = _env_int(["RANK", "PMI_RANK", "PALS_RANKID"], 0) - world_size = _env_int(["WORLD_SIZE", "PMI_SIZE", "PALS_SIZE"], 1) - local_rank = _env_int(["LOCAL_RANK", "PMI_LOCAL_RANK", "PALS_LOCAL_RANKID"], 0) - device = cfg.training.get("device", "cpu") - - # Only call setup_distributed() for native DDP/FSDP paths. - # For Accelerate path, Accelerator handles distributed init internally. - # Calling both causes "Fatal error in internal_Init_thread" (double MPI init). - if use_native_fsdp or use_native_ddp: - # 0. Setup Distributed Environment (mpi4py-based for multi-node) - rank, world_size, local_rank, device = setup_distributed() - if rank == 0: - logger.info( - f"[Distributed] Rank {rank}/{world_size}, Local Rank {local_rank}, Device: {device}" - ) - - # Seed plumbing: `system.seed` lives in src/conf/config.yaml (default 42) but - # no code read it before. Stage A IsoFLOP's variance-floor measurement needs - # the BASE@3e17 seed-replicas to actually diverge, so seed every RNG the - # trainer touches. Set the same base seed on every rank — downstream - # samplers/collators add `rank` themselves where they need rank-specific - # streams (e.g. DistributedSampler), and starting from identical RNG state - # makes runs reproducible across world-size changes. - _seed_cfg = cfg.get("system", None) - seed = int(_seed_cfg.get("seed", 42)) if _seed_cfg is not None else 42 - random.seed(seed) - np.random.seed(seed) - torch.manual_seed(seed) - if hasattr(torch, "xpu") and torch.xpu.is_available(): - torch.xpu.manual_seed_all(seed) - if torch.cuda.is_available(): - torch.cuda.manual_seed_all(seed) - if rank == 0: - logger.info(f"[Seed] Set RNG state from system.seed={seed}") - - # 1. Configuration Mapping - from hydra.core.hydra_config import HydraConfig - - try: - hydra_out = HydraConfig.get().runtime.output_dir - except Exception: - hydra_out = os.getcwd() +def build_model_config(cfg: DictConfig, rank: int = 0) -> ModelConfig: + """Map the Hydra ``cfg.model`` / ``cfg.training`` tree onto a ``ModelConfig``. + Shared by ``main()`` and the notebooks/tools that need the exact model + ``src/train.py`` would build (e.g. applications/bioreason/bioreason_demo.ipynb). + """ ts_projector = str(cfg.model.get("ts_projector", "linear")) timeomni_patch_lens = _coerce_int_list(cfg.model.get("timeomni_patch_len", 16)) timeomni_strides = _resolve_timeomni_strides( @@ -371,6 +312,7 @@ def _env_int(keys, default): "dna_model_name", "InstaDeepAI/nucleotide-transformer-2.5b-multi-species" ), dna_is_evo2=cfg.model.get("dna_is_evo2", False), + max_dna_length=cfg.training.get("max_dna_length", 1024), image_encoder_id=cfg.model.get( "image_encoder_id", "google/siglip2-base-patch16-224" ), @@ -440,11 +382,14 @@ def _env_int(keys, default): if cfg.model.get("decoder_configs", None) is not None else {}, ) + return model_config - # Allow override of dimensions if specified in cfg (not implemented fully in hydra yaml yet, relying on defaults) - - +def build_training_config( + cfg: DictConfig, model_config: ModelConfig, output_dir: str, seed: int +) -> TrainingConfig: + """Map ``cfg.training`` onto a ``TrainingConfig``; checkpoints go to ``output_dir/checkpoints``.""" + hydra_out = output_dir train_config = TrainingConfig( batch_size=cfg.training.batch_size, max_steps=cfg.training.max_steps, @@ -526,10 +471,6 @@ def _env_int(keys, default): grpo_beta=cfg.training.get("grpo_beta", 0.0), grpo_reward_functions=cfg.training.get("grpo_reward_functions", None), ) - if train_config.wandb_mode: - os.environ["WANDB_MODE"] = train_config.wandb_mode - if train_config.wandb_entity: - os.environ["WANDB_ENTITY"] = train_config.wandb_entity # Inject eval_interval logic if getattr(cfg.training, "debug_mode", False) or cfg.training.max_steps < 100: @@ -545,6 +486,180 @@ def _env_int(keys, default): # gate never fires no matter what the YAML / CLI says. Surfaced during # the Smoke 4 replay on 2026-05-27. train_config.eval_enabled = getattr(cfg.training, "eval_enabled", False) + return train_config + + +def build_bio_dataset_kwargs( + task_type: str, train_config: TrainingConfig, model_config: ModelConfig +) -> dict: + """Extra ``StreamingMultimodalDataset`` kwargs for the BioReason DNA/KEGG tasks.""" + is_bio_task = task_type in ("bioreason_sft", "bioreason_grpo", "bioreason_projector") + # StreamingMultimodalDataset kwargs specific to BioReason DNA/KEGG runs. + # is_sft=False for the GRPO stage (no assistant turn appended; answer kept + # for reward computation), True for SFT/projector (assistant turn + # appended, loss masked to the answer). is_projector_only selects the + # "[dna_bioreason]" full-sequence-loss sentinel for the projector-only + # alignment stage, distinct from is_sft — see multimodal.py's + # _finalize_dna_item / model.py's label-masking branch. + bio_dataset_kwargs = ( + { + "use_reasoning_traces": train_config.use_reasoning_traces, + "model_name": "dna-llm" if Modality.DNA in model_config.modalities else "llm", + "is_sft": task_type != "bioreason_grpo", + "task": task_type, + "use_class_weights": train_config.use_class_weights, + "class_weight_max": train_config.class_weight_max, + "is_projector_only": task_type == "bioreason_projector", + } + if is_bio_task + else {} + ) + return bio_dataset_kwargs + + +def build_collator( + tokenizer, model_config: ModelConfig, use_bucketed_collator: bool, collator_max_seq: int +): + """Pick the Bucketed/Multimodal collator for the non-VLA streaming path.""" + _is_dynamic_length_ts = model_config.ts_projector in DYNAMIC_LENGTH_TS_PROJECTORS + collator: BucketedCollator | MultimodalCollator + if use_bucketed_collator and not _is_dynamic_length_ts: + collator = BucketedCollator( + tokenizer, + sort_within_batch=True, + log_efficiency=True, + max_seq_length=collator_max_seq, + ) + logger.info(f"Using BucketedCollator with max_seq_length={collator_max_seq}") + else: + if use_bucketed_collator and _is_dynamic_length_ts: + logger.info( + f"Disabling BucketedCollator for ts_projector={model_config.ts_projector}; " + "using MultimodalCollator with time_series passthrough for dynamic batching." + ) + # Interleaved-QA samples carry / token pairs and + # prompt/target metadata that must stay in sync; tail-truncating + # them at collate time would desync those invariants and crash the + # merge in model.py. That path is capped at the variate level in + # _process_ts_qa instead, so pass max_seq_length=None here. For + # plain (non-interleaved) batches, apply the same cap as the + # bucketed collator (issue #120). + collator_cap = None if model_config.is_interleaved_qa else collator_max_seq + collator = MultimodalCollator( + tokenizer, + max_seq_length=collator_cap, + passthrough_time_series=_is_dynamic_length_ts, + ) + logger.info( + f"Using MultimodalCollator with max_seq_length={collator_cap} " + f"(is_interleaved_qa={model_config.is_interleaved_qa}, " + f"passthrough_time_series={_is_dynamic_length_ts})" + ) + return collator + + +def build_streaming_dataset( + cfg: DictConfig, + tokenizer, + model_config: ModelConfig, + train_config: TrainingConfig, + bio_dataset_kwargs: dict, + max_seq_length: int, + allow_dummy_data: bool = False, +) -> StreamingMultimodalDataset: + """Build the HF/local-Arrow ``StreamingMultimodalDataset`` (the BioReason DNA path).""" + return StreamingMultimodalDataset( + tokenizer=tokenizer, + batch_size=train_config.batch_size, + max_steps=train_config.max_steps, + model_config=model_config, + hf_token=os.environ.get("HF_TOKEN"), + allow_dummy_data=allow_dummy_data, + verbosity=getattr( + logging, cfg.training.get("verbosity", "INFO").upper(), logging.INFO + ), + dataset_overrides=_resolve_dataset_overrides(cfg), + max_seq_length=max_seq_length, + **bio_dataset_kwargs, + ) + + +@hydra.main(config_path="conf", config_name="config", version_base="1.2") +def main(cfg: DictConfig): + if _is_main_env(): + print_prism_banner("Foundational Multimodal Training") + logger.info(f"--- Starting PRISM Training [{cfg.exp.id}] ---") + logger.info(OmegaConf.to_yaml(cfg)) + + # 0. Determine execution path + use_native_fsdp = os.environ.get("USE_NATIVE_FSDP", "0") == "1" + use_native_ddp = os.environ.get("USE_NATIVE_DDP", "0") == "1" + + # Defaults for Accelerate path (setup_distributed is native-only). + def _env_int(keys, default): + for key in keys: + val = os.environ.get(key, "") + if val and val.strip(): + try: + return int(val) + except ValueError: + continue + return default + + rank = _env_int(["RANK", "PMI_RANK", "PALS_RANKID"], 0) + world_size = _env_int(["WORLD_SIZE", "PMI_SIZE", "PALS_SIZE"], 1) + local_rank = _env_int(["LOCAL_RANK", "PMI_LOCAL_RANK", "PALS_LOCAL_RANKID"], 0) + device = cfg.training.get("device", "cpu") + + # Only call setup_distributed() for native DDP/FSDP paths. + # For Accelerate path, Accelerator handles distributed init internally. + # Calling both causes "Fatal error in internal_Init_thread" (double MPI init). + if use_native_fsdp or use_native_ddp: + # 0. Setup Distributed Environment (mpi4py-based for multi-node) + rank, world_size, local_rank, device = setup_distributed() + if rank == 0: + logger.info( + f"[Distributed] Rank {rank}/{world_size}, Local Rank {local_rank}, Device: {device}" + ) + + # Seed plumbing: `system.seed` lives in src/conf/config.yaml (default 42) but + # no code read it before. Stage A IsoFLOP's variance-floor measurement needs + # the BASE@3e17 seed-replicas to actually diverge, so seed every RNG the + # trainer touches. Set the same base seed on every rank — downstream + # samplers/collators add `rank` themselves where they need rank-specific + # streams (e.g. DistributedSampler), and starting from identical RNG state + # makes runs reproducible across world-size changes. + _seed_cfg = cfg.get("system", None) + seed = int(_seed_cfg.get("seed", 42)) if _seed_cfg is not None else 42 + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + if hasattr(torch, "xpu") and torch.xpu.is_available(): + torch.xpu.manual_seed_all(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + if rank == 0: + logger.info(f"[Seed] Set RNG state from system.seed={seed}") + + # 1. Configuration Mapping + from hydra.core.hydra_config import HydraConfig + + try: + hydra_out = HydraConfig.get().runtime.output_dir + except Exception: + hydra_out = os.getcwd() + + model_config = build_model_config(cfg, rank=rank) + + + # Allow override of dimensions if specified in cfg (not implemented fully in hydra yaml yet, relying on defaults) + + + train_config = build_training_config(cfg, model_config, hydra_out, seed) + if train_config.wandb_mode: + os.environ["WANDB_MODE"] = train_config.wandb_mode + if train_config.wandb_entity: + os.environ["WANDB_ENTITY"] = train_config.wandb_entity # Debug print _rprint( @@ -691,26 +806,7 @@ def _env_int(keys, default): raise RuntimeError("training.task=vla_calvin requires model.is_vla=true") is_bio_task = task_type in ("bioreason_sft", "bioreason_grpo", "bioreason_projector") - # StreamingMultimodalDataset kwargs specific to BioReason DNA/KEGG runs. - # is_sft=False for the GRPO stage (no assistant turn appended; answer kept - # for reward computation), True for SFT/projector (assistant turn - # appended, loss masked to the answer). is_projector_only selects the - # "[dna_bioreason]" full-sequence-loss sentinel for the projector-only - # alignment stage, distinct from is_sft — see multimodal.py's - # _finalize_dna_item / model.py's label-masking branch. - bio_dataset_kwargs = ( - { - "use_reasoning_traces": train_config.use_reasoning_traces, - "model_name": "dna-llm" if Modality.DNA in model_config.modalities else "llm", - "is_sft": task_type != "bioreason_grpo", - "task": task_type, - "use_class_weights": train_config.use_class_weights, - "class_weight_max": train_config.class_weight_max, - "is_projector_only": task_type == "bioreason_projector", - } - if is_bio_task - else {} - ) + bio_dataset_kwargs = build_bio_dataset_kwargs(task_type, train_config, model_config) # 3. Tokenizer / Data _rprint("Initializing Tokenizer...", flush=True) @@ -922,39 +1018,9 @@ def __call__(self, text, **kwargs): # from outlier-length sequences at epoch boundaries. # For projector training, 512 is sufficient; for E2E, match max_seq_length. collator_max_seq = int(os.environ.get("MAX_SEQ_LENGTH", "2048")) - _is_dynamic_length_ts = model_config.ts_projector in DYNAMIC_LENGTH_TS_PROJECTORS - if use_bucketed_collator and not _is_dynamic_length_ts: - collator = BucketedCollator( - tokenizer, - sort_within_batch=True, - log_efficiency=True, - max_seq_length=collator_max_seq, - ) - logger.info(f"Using BucketedCollator with max_seq_length={collator_max_seq}") - else: - if use_bucketed_collator and _is_dynamic_length_ts: - logger.info( - f"Disabling BucketedCollator for ts_projector={model_config.ts_projector}; " - "using MultimodalCollator with time_series passthrough for dynamic batching." - ) - # Interleaved-QA samples carry / token pairs and - # prompt/target metadata that must stay in sync; tail-truncating - # them at collate time would desync those invariants and crash the - # merge in model.py. That path is capped at the variate level in - # _process_ts_qa instead, so pass max_seq_length=None here. For - # plain (non-interleaved) batches, apply the same cap as the - # bucketed collator (issue #120). - collator_cap = None if model_config.is_interleaved_qa else collator_max_seq - collator = MultimodalCollator( - tokenizer, - max_seq_length=collator_cap, - passthrough_time_series=_is_dynamic_length_ts, - ) - logger.info( - f"Using MultimodalCollator with max_seq_length={collator_cap} " - f"(is_interleaved_qa={model_config.is_interleaved_qa}, " - f"passthrough_time_series={_is_dynamic_length_ts})" - ) + collator = build_collator( + tokenizer, model_config, use_bucketed_collator, collator_max_seq + ) # Check if we should use direct MultiWebDataset mode # This bypasses StreamingMultimodalDataset which has integration issues # with multi-dataset loading (Issue 14 in the Feb-2026 bring-up debug @@ -1201,35 +1267,15 @@ def __call__(self, text, **kwargs): f"WARNING: LOCAL_SHARDS_DIR={local_shards_dir} exists but has no .tar files, falling back to StreamingMultimodalDataset", flush=True, ) - dataset = StreamingMultimodalDataset( - tokenizer=tokenizer, - batch_size=train_config.batch_size, - max_steps=train_config.max_steps, - model_config=model_config, - hf_token=os.environ.get("HF_TOKEN"), - allow_dummy_data=_enable_all_modalities, - verbosity=getattr( - logging, cfg.training.get("verbosity", "INFO").upper(), logging.INFO - ), - dataset_overrides=_resolve_dataset_overrides(cfg), - max_seq_length=collator_max_seq, - **bio_dataset_kwargs, + dataset = build_streaming_dataset( + cfg, tokenizer, model_config, train_config, bio_dataset_kwargs, + max_seq_length=collator_max_seq, allow_dummy_data=_enable_all_modalities, ) else: # Pure legacy mode: Use StreamingMultimodalDataset - dataset = StreamingMultimodalDataset( - tokenizer=tokenizer, - batch_size=train_config.batch_size, - max_steps=train_config.max_steps, - model_config=model_config, - hf_token=os.environ.get("HF_TOKEN"), - allow_dummy_data=_enable_all_modalities, - verbosity=getattr( - logging, cfg.training.get("verbosity", "INFO").upper(), logging.INFO - ), - dataset_overrides=_resolve_dataset_overrides(cfg), - max_seq_length=collator_max_seq, - **bio_dataset_kwargs, + dataset = build_streaming_dataset( + cfg, tokenizer, model_config, train_config, bio_dataset_kwargs, + max_seq_length=collator_max_seq, allow_dummy_data=_enable_all_modalities, ) # Compare the modalities the dataloader will actually emit against diff --git a/src/training/distributed.py b/src/training/distributed.py index f096359..9158330 100644 --- a/src/training/distributed.py +++ b/src/training/distributed.py @@ -1127,7 +1127,9 @@ def load_model_weights_only(model, checkpoint_path, device="cpu"): ) print(f"[Resume] Loading model weights from {safetensors_path}") - state_dict = load_file(safetensors_path, device=str(device)) + # Load on CPU: `device` may be the string "auto" (training.device), which + # safetensors rejects; load_state_dict copies onto the model's device anyway. + state_dict = load_file(safetensors_path, device="cpu") # Strip "module." prefix if present (from DDP-saved checkpoints) clean_state_dict = {} diff --git a/src/training/trainer_zone_a.py b/src/training/trainer_zone_a.py index cb76e45..22ed975 100644 --- a/src/training/trainer_zone_a.py +++ b/src/training/trainer_zone_a.py @@ -394,15 +394,25 @@ def __init__(self, model: UnifiedTransformer, config: TrainingConfig, train_load sched_type = getattr(config, "scheduler_type", "cosine") min_lr_ratio = getattr(config, "min_lr_ratio", 0.0) + # accelerator.prepare(scheduler) below wraps it in AcceleratedScheduler, + # which calls the inner scheduler.step() num_processes times per + # optimizer step (it assumes the dataloader batch was scaled by world + # size). Express every schedule length in those units so warmup/decay + # are counted in optimizer steps; otherwise on 12 ranks warmup ends 12x + # early and the HF cosine passes its end at max_steps/12 and rises again. + n_sched = self.accelerator.num_processes + sched_total = config.max_steps * n_sched + sched_warmup = config.warmup_steps * n_sched + if sched_type == "molmo_layered": from src.utils.scheduler import get_molmo_scheduler self.accelerator.print("Using Molmo Layered Scheduler (Separate Warmups)") self.scheduler = get_molmo_scheduler( self.optimizer, - config.max_steps, - warmup_connector=getattr(config, "warmup_steps_connector", 200), - warmup_main=getattr(config, "warmup_steps_main", 2000), + sched_total, + warmup_connector=getattr(config, "warmup_steps_connector", 200) * n_sched, + warmup_main=getattr(config, "warmup_steps_main", 2000) * n_sched, min_lr_ratio=min_lr_ratio, ) elif sched_type == "cosine_with_min_lr": @@ -411,8 +421,8 @@ def __init__(self, model: UnifiedTransformer, config: TrainingConfig, train_load self.accelerator.print(f"Using Cosine Scheduler with Floor {min_lr_ratio}") self.scheduler = get_cosine_with_min_lr( self.optimizer, - num_warmup_steps=config.warmup_steps, - num_training_steps=config.max_steps, + num_warmup_steps=sched_warmup, + num_training_steps=sched_total, min_lr_ratio=min_lr_ratio, ) elif sched_type == "wsd": @@ -427,18 +437,18 @@ def __init__(self, model: UnifiedTransformer, config: TrainingConfig, train_load ) self.scheduler = get_wsd_scheduler( self.optimizer, - num_warmup_steps=config.warmup_steps, - num_training_steps=config.max_steps, + num_warmup_steps=sched_warmup, + num_training_steps=sched_total, min_lr_ratio=min_lr_ratio, decay_ratio=getattr(config, "wsd_decay_ratio", 0.1), - decay_steps=wsd_decay_steps, + decay_steps=wsd_decay_steps * n_sched if wsd_decay_steps else wsd_decay_steps, ) else: # Default HF self.scheduler = get_cosine_schedule_with_warmup( self.optimizer, - num_warmup_steps=config.warmup_steps, - num_training_steps=config.max_steps, + num_warmup_steps=sched_warmup, + num_training_steps=sched_total, ) # 3. Prepare via Accelerator @@ -603,6 +613,33 @@ def add_evaluator(name, registry_key): self._kegg_eval_logger = None self._kegg_eval_nt_tokenizer = None + # Parameter-name keys -> label, shared by the gradient watchdog and its snapshot. + _GRAD_COMPONENTS = ( + ("projector", "Projector"), + ("lora_", "LoRA"), + ("encoder", "Encoder/ViT"), + ("backbone", "Backbone/LLM"), + ) + + def _snapshot_grad_norms(self): + """Record the largest grad norm per component (label -> (norm, param name)). + + Called just before ``optimizer.zero_grad()`` on steps whose *next* iteration runs + ``check_parameter_status``: that check runs at the top of the loop, after + ``zero_grad()`` has already set every ``.grad`` to None, so it can't see them. + """ + snap = {} + for key, label in self._GRAD_COMPONENTS: + best, name = 0.0, None + for n, p in self.model.named_parameters(): + if key in n and p.requires_grad and p.grad is not None: + g = p.grad.norm().item() + if g > best: + best, name = g, n + if name is not None: + snap[label] = (best, name) + self._grad_snapshot = snap + def check_parameter_status(self, step): """Watchdog: Fail fast if projectors are seemingly frozen.""" if not self.accelerator.is_main_process: @@ -672,12 +709,8 @@ def check_parameter_status(self, step): # 2. Check Gradients (Only if step > 0) # Sample from each component type if step > 0: - components = [ - ("projector", "Projector"), - ("lora_", "LoRA"), - ("encoder", "Encoder/ViT"), - ("backbone", "Backbone/LLM"), - ] + components = self._GRAD_COMPONENTS + snapshot = getattr(self, "_grad_snapshot", None) or {} for key, label in components: found_grad = False @@ -692,9 +725,17 @@ def check_parameter_status(self, step): sample_name = n found_grad = True + from_snapshot = False + if not found_grad and label in snapshot: + # Live grads were already cleared by zero_grad(); use the norms + # recorded just before it on the previous optimizer step. + max_norm, sample_name = snapshot[label] + found_grad, from_snapshot = True, True + if found_grad and max_norm > 0: + src = " [last optimizer step]" if from_snapshot else "" print( - f" ✅ {label}: {sample_name.split('.')[-2]}.{sample_name.split('.')[-1]} (Norm: {max_norm:.6f})" + f" ✅ {label}: {sample_name.split('.')[-2]}.{sample_name.split('.')[-1]} (Norm: {max_norm:.6f}){src}" ) elif found_grad: print(f" ⚠️ {label}: Gradients exist but are zero!") @@ -1059,6 +1100,10 @@ def train(self): t3 = time.perf_counter() self.optimizer.step() self.scheduler.step() + # The watchdog runs at the top of iteration step+1 (step 1 and every + # 100th), after zero_grad(); capture the grad norms it will report. + if (step + 1) == 1 or (step + 1) % 100 == 0: + self._snapshot_grad_norms() self.optimizer.zero_grad() # Skipped under PRISM_PRODUCTION_MODE if not _production_mode and hasattr(torch, "xpu") and torch.xpu.is_available(): @@ -1428,8 +1473,9 @@ def run_kegg_eval(self, step): if dna_encoder is not None and getattr(dna_encoder, "tokenizer", None) is not None: self._kegg_eval_nt_tokenizer = dna_encoder.tokenizer else: + # Same checkpoint as the encoder (see StreamingMultimodalDataset). self._kegg_eval_nt_tokenizer = AutoTokenizer.from_pretrained( - "InstaDeepAI/nucleotide-transformer-v2-250m-multi-species", + unwrapped.config.dna_model_name, trust_remote_code=True, ) diff --git a/tests/data/test_dna_bioreason_datasets_config.py b/tests/data/test_dna_bioreason_datasets_config.py index 465c84a..2879748 100644 --- a/tests/data/test_dna_bioreason_datasets_config.py +++ b/tests/data/test_dna_bioreason_datasets_config.py @@ -47,9 +47,13 @@ def test_handler_field_matches_key(zone_a_datasets): assert zone_a_datasets[key]["handler"] == key -def test_none_are_skipped(zone_a_datasets): - for key in DNA_HANDLER_KEYS: - assert zone_a_datasets[key]["skip"] is False, f"{key} is marked skip=true" +def test_bioreason_demo_default_is_kegg_only(zone_a_datasets): + # The bioreason demo trains on KEGG only: dna_bioreason is enabled and the two + # variant-effect datasets are skipped by default (flip skip to false, or use + # data.dataset_overrides, to mix them back in). + assert zone_a_datasets["dna_bioreason"]["skip"] is False + for key in ("dna_bioreason_variant_effect_coding", "dna_bioreason_variant_effect_non_snv"): + assert zone_a_datasets[key]["skip"] is True, f"{key} should be skip=true in the demo default" def test_all_dna_handlers_contain_dna_substring(zone_a_datasets): @@ -119,10 +123,8 @@ def test_modality_group_normalization_matches_multimodal_py(): modality_groups[target].append((name, w)) dna_group = dict(modality_groups["dna"]) - assert set(dna_group.keys()) == set(DNA_HANDLER_KEYS.keys()) + assert set(dna_group.keys()) == {"dna_bioreason"} total = sum(dna_group.values()) normalized = {k: v / total for k, v in dna_group.items()} - assert normalized["dna_bioreason"] == pytest.approx(0.50) - assert normalized["dna_bioreason_variant_effect_coding"] == pytest.approx(0.25) - assert normalized["dna_bioreason_variant_effect_non_snv"] == pytest.approx(0.25) + assert normalized["dna_bioreason"] == pytest.approx(1.0) diff --git a/tests/data/test_variant_effect_coding_handler.py b/tests/data/test_variant_effect_coding_handler.py index 3c01340..68d2b48 100644 --- a/tests/data/test_variant_effect_coding_handler.py +++ b/tests/data/test_variant_effect_coding_handler.py @@ -147,12 +147,12 @@ def test_sft_true_appends_answer_tagged_assistant_turn_with_no_reasoning(): # never "Answer: X" (BioReason's literal format) — see handler docstring. assert "" not in caption assert "pathogenic" in caption - assert "<|im_start|>assistant" in caption + assert "<|assistant|>" in caption # The "promptlen targetlen" split is computed in __iter__ from these two # renderings, so assert the property it depends on: the prompt stops at the # assistant marker and the answer lands strictly after it. full_text, prompt_text = ds._render_dna_prompt_text(result["prompt"]) - assert prompt_text.endswith("<|im_start|>assistant\n") + assert prompt_text.endswith("<|assistant|>\n") assert len(full_text) > len(prompt_text) assert "pathogenic" not in prompt_text @@ -163,7 +163,7 @@ def test_sft_false_omits_assistant_turn_for_grpo(): caption = render(ds, result) # The generation marker is still emitted (that is what makes it a generation # prompt); what must be absent is the answer text after it. - assert "<|im_start|>assistant" in caption + assert "<|assistant|>" in caption assert "" not in caption # generation-prompt only, no answer text # The answer still surfaces for reward computation, but as a real key -- # trainer_grpo.py reads batch["answer"] rather than parsing a metadata tag. @@ -191,7 +191,7 @@ def test_none_item_returns_safe_fallback(): # fallback ref/var = ("CTGA", "CTGA") in dna-llm mode, so there is still a # real pair for __iter__ to encode. assert result["dna_sequences"] == ["CTGA", "CTGA"] - assert "<|im_start|>assistant" in caption # is_sft=True still appends fallback assistant turn + assert "<|assistant|>" in caption # is_sft=True still appends fallback assistant turn def test_none_item_with_is_sft_false_omits_assistant_turn(): diff --git a/tests/data/test_variant_effect_non_snv_handler.py b/tests/data/test_variant_effect_non_snv_handler.py index 56a4c60..63ecf83 100644 --- a/tests/data/test_variant_effect_non_snv_handler.py +++ b/tests/data/test_variant_effect_non_snv_handler.py @@ -153,7 +153,7 @@ def test_sft_false_omits_assistant_turn_for_grpo(): caption = render(ds, ds._process_dna_bioreason_variant_effect_non_snv(REAL_SAMPLE_ITEM_WITH_TERMS)) # The generation marker is still emitted (the renderer always opens the # assistant turn); what is absent is the answer the model must produce. - assert "<|im_start|>assistant" in caption + assert "<|assistant|>" in caption assert "" not in caption @@ -177,7 +177,7 @@ def test_none_item_returns_safe_fallback(): assert "No variant information available." in caption # dna-llm still needs a well-formed pair for __iter__ to tokenize. assert result["dna_sequences"] == ["CTGA", "CTGA"] - assert "<|im_start|>assistant" in caption + assert "<|assistant|>" in caption def test_none_item_with_is_sft_false_omits_assistant_turn(): diff --git a/tests/multimodal/test_modality_contracts.py b/tests/multimodal/test_modality_contracts.py index 90922f3..405b2da 100644 --- a/tests/multimodal/test_modality_contracts.py +++ b/tests/multimodal/test_modality_contracts.py @@ -411,7 +411,7 @@ def decode(self, token_ids): ds = StreamingMultimodalDataset( tokenizer=_Tok(), batch_size=1, - max_steps=4, + max_steps=32, model_config=cfg, allow_dummy_data=True, # fallback so missing local data doesn't hard-fail ) @@ -426,13 +426,13 @@ def decode(self, token_ids): try: for i, sample in enumerate(ds): seen_keys.update(sample.keys()) - if i >= 3: + if i >= 31: break except Exception as exc: pytest.skip(f"StreamingMultimodalDataset iter failed (likely HF offline): {exc}") assert modality in seen_keys, ( - f"Pulled 4 samples; requested modality {modality!r} never appeared. " + f"Pulled 32 samples; requested modality {modality!r} never appeared. " f"Saw keys: {sorted(seen_keys)}" ) diff --git a/tools/ci/site_path_allowlist.json b/tools/ci/site_path_allowlist.json index 89db34d..35ee1f2 100644 --- a/tools/ci/site_path_allowlist.json +++ b/tools/ci/site_path_allowlist.json @@ -1,11 +1,9 @@ { "_comment": "Per-file counts of hardcoded site paths, as a ratchet. See tools/ci/check_site_paths.py. Counts may only decrease; regenerate with `python tools/ci/check_site_paths.py --write`.", "files": { - "applications/bioreason/submit_bioreason_grpo.sh": 8, - "applications/bioreason/submit_bioreason_projector.sh": 7, - "applications/bioreason/submit_bioreason_sft.sh": 10, - "applications/bioreason/submit_eval_kegg.sh": 8, - "applications/bioreason/verify_dna_dataset_mix.sh": 7, + "applications/bioreason/scripts/eval/submit_eval_kegg.sh": 7, + "applications/bioreason/scripts/test/verify_dna_dataset_mix.sh": 6, + "applications/bioreason/scripts/training/submit_bioreason_grpo.sh": 7, "applications/material_science/stage_materials_text_model.py": 1, "applications/text/convert_arxiv_streaming.py": 6, "applications/text/convert_nemotron_to_webdataset.py": 4,