Skip to content

[Feature] Added Sparse Initialization#1498

Merged
angeloskath merged 8 commits into
ml-explore:mainfrom
cvnad1:issue-1459
Oct 24, 2024
Merged

[Feature] Added Sparse Initialization#1498
angeloskath merged 8 commits into
ml-explore:mainfrom
cvnad1:issue-1459

Conversation

@cvnad1

@cvnad1 cvnad1 commented Oct 18, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Adding Feature Sparse Initialization as proposed in the issue #1459

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

@Saanidhyavats

Copy link
Copy Markdown
Contributor

@awni, @cvnad1 and I have added sparse init method with documentation and tests. Let us know if any change needs to be done

@angeloskath angeloskath left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the addition!

I think this implementation is unnecessarily complicated. The same can be achieved via a mask. Something as simple as

v = mx.random.normal(shape=a.shape, scale=std, loc=mean, dtype=dtype)
m = mx.random.bernoulli(1-sparsity, shape=a.shape)
return m * v

Comment thread python/mlx/nn/init.py
@angeloskath angeloskath merged commit 430ffef into ml-explore:main Oct 24, 2024
@zcbenz

zcbenz commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

@cvnad1 Sorry for pinging you on an old PR but can you take a look at #3725 to verify whether its claim is correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants