-
Notifications
You must be signed in to change notification settings - Fork 16
Consistency checks for weight initialization #438
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
The model weights are currently initialised by matching regexes against the FQNs.
For each regex, we define a routine specifying how the weights are supposed to be initialised.
This can be prone to errors if FQNs change, as is the case for torch.compile wich introduces a _orig_mod. to FQNs.
Since side-effects like these can easily go unnoticed we should introduce some consistency checks:
- Each FQN must match with exactly one regex.
- If a FQN is not supposed to be initialized via in a initialisation approach (e.g., some norms), the regex should call a "no-op" instead of an init routine.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request