Skip to content

Conversation

@arnobock
Copy link

@arnobock arnobock commented Jan 30, 2026

This PR implements a first version of a builtin realising independent subnet training [1] as DML script.
[1] https://www.vldb.org/pvldb/vol15/p1581-wolfe.pdf

The implementation provides:

  • a training routine for IST-based subnet optimization
  • a disjoint masking/partitioning function to construct independent subnet models
  • test infrastructure using LeNet + MNIST to validate correctness

As hinted this version needs further runtime optimisation to enhance performance.
A current determined bottleneck is the gradient computation, which (even though executed through a parfor-loop) remains rather slow. One potential reason for this could be the sparse masking of disjoint subnets since dimensionalities of the model matrices are effectively not reduced.

The code contains some TODO's which hint at potential performance boosts.

Note: A dense masking which reduces subnet matrices size is currently in progress and could solve the remaining issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant