Repository navigation
Feat root pe - #789
Open
anna-grim wants to merge 9 commits into
Open
Feat root pe#789anna-grim wants to merge 9 commits into
anna-grim wants to merge 9 commits into
Conversation
…ts GCS calls - models/joint_model.py: JointProofreader with shared CNN3D backbone, Arborist merge head, and HGAT split head; routes batches by task at forward time - models/new_gnn_models.py: NewVisionHGAT for standalone split correction - split_datasets.py: replace list_gcs_subdirectories(bucket, prefix) calls with list_gcs_subprefixes(full_gcs_path) to match the current util API Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… fixes - models/joint_model.py: add JointProofreader (joint split+merge model with shared CNN3D backbone, Arborist merge head, and 3-layer HGAT split head) - models/new_gnn_models.py: add gat3 (3rd HGAT layer), bump gnn_hidden_dim and img_embed_dim defaults 64→96, add ProposalTreeEncoder - split_proofreading/new_split_feature_extraction.py: add ArboristSplitDataset (prefetched iterator with Arborist tree features alongside image patches) - split_proofreading/split_inference.py: route to ArboristSplitDataset when use_tree_features=True (required by JointProofreader split head) - configs.py: add fov_merge to ImageConfig (load at fov, resize to patch_shape for merge detection); add modality to MergeInferenceConfig; add model_config_path and use_tree_features to SplitInferenceConfig; add keys/__getitem__ to Config base class - machine_learning/image_dataloader.py: DetectionPatchLoader loads at fov_merge when set, then resizes to patch_shape before returning - machine_learning/train.py: add n_mistake_mips reservoir-sampled mistake MIP saving per val epoch - utils/img_util.py: cast img to float32 before np.percentile to fix NaN on float16 arrays (all-white MIP bug) - merge_proofreading/search_datasets.py: fix subgraph_to_tree_sample root arg - proofreading_pipeline.py: pass modality and use_tree_features from configs - proposal_graph.py: accept **kwargs in __init__ for forward-compat - utils/geometry_util.py: guard is_double against empty query_nodes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…mask new_split_feature_extraction.py: replaces old _build_line_graph_edge_index + manual root detection with build_tree_sample, removes per-curve self.transform (graph_transform already covers rotation/flip before decomposition), and switches to in-place backward differencing. image_dataloader.py: removes dead get_foreground_nodes call in create_mask — the result only fed annotate_foreground which was commented out (TEMP). At inference scale this was a full kdtree query per proposal with no effect on output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
proposal_graph.py + groundtruth_generation.py: groundtruth run() now also returns on_gt_ids (fragments that have GT alignment); proposals where either fragment is off-GT are discarded before training so their unknown labels don't corrupt the loss. split_feature_extraction.py: stub create_segment_mask to zeros (old implementation used graph.kdtree which is being removed from the critical path); bump branch fill_val from 0.5 to 1. ml_util.py: BinaryMetricAccumulator.compute now accepts min_precision alongside min_recall and reports f1_at_precision from the PR curve. merge_datamodules.py, point_cloud_models.py: remove unused code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CNN3D with use_output_head=True already provides the same capability; callers can import it directly from new_vision_models. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The eval path constructed a bare TreeSample, so edge_type, hop_dist, heat basis, curve_endpoints and curve_geom were missing. Route through the arborist factory so inference sees the same structural fields as training. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- JointProofreader merge path concatenates the Arborist CLS embedding with the mean of the root-adjacent curve embeddings. The merge site is the root, so the local geometry of the curves meeting there is read out directly instead of only through CLS attention. merge_head input grows by arborist_latent_dim. - Mask channel: img_util.soften_mask renders exp(-d^2 / 2 sigma^2) of the distance to the skeleton after resizing to patch_shape, controlled by ImageConfig.mask_sigma (default 1.5 voxels; 0/None keeps binary). A one-voxel skeleton was nearly erased by the first pooling layer. Applied in both DetectionPatchLoader (merge) and PatchFeatureExtractor (split); mask_sigma is threaded through FragmentsDataset -> ImageExtractor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FragmentsDataset stored img_config.transform but never applied it, so split training ran without any image augmentation. Thread `transform` through FeaturePipeline -> ImageFeatureExtractor -> PatchFeatureExtractor and apply it to the stacked (2, D, H, W) patch after resizing and mask softening, before the float16 cast. Also fixes the previous commit: split_datasets passed mask_sigma to FeaturePipeline, which did not accept it yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each ts.open without an explicit context created its own 8 GB cache pool, so memory scaled linearly with the number of brains and blocks loaded. A single module-level context bounds the decoded-chunk cache and GCS request concurrency globally. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.