Don't clone submodules for tools that aren't being built

[jyn514]

Right now, this is the first thing you see on a fresh clone:

$ python x.py check
Updating only changed submodules
Updating submodule src/tools/rust-installer
Submodule 'src/rust-installer' (https://github.com/rust-lang/rust-installer.git) registered for path 'src/tools/rust-installer'
Cloning into '/home/joshua/src/rust/rust/src/tools/rust-installer'...
remote: Enumerating objects: 30, done.        
remote: Counting objects: 100% (30/30), done.        
remote: Compressing objects: 100% (26/26), done.        
remote: Total 799 (delta 9), reused 11 (delta 3), pack-reused 769        
Receiving objects: 100% (799/799), 246.22 KiB | 4.83 MiB/s, done.
Resolving deltas: 100% (486/486), done.
Submodule path 'src/tools/rust-installer': checked out 'd66f476b4d5e7fdf1ec215c9ac16c923dc292324'
Updating submodule src/doc/nomicon
...

This can go on for many minutes, especially on a slow connection. Instead, x.py should only clone the submodules when they're actually needed.

I think cargo build requires having the Cargo.toml of the submodules to work, which might be tricky ... maybe x.py could default to a shallow clone? The main thing I'd love to have is for x.py check to not have to download LLVM.

Thanks to @Lokathor and @thomcc for encouraging me to throw away my clone and start from scratch ;)

[Mark-Simulacrum]

Many of these must be cloned with the current global workspace, because otherwise cargo will fail to resolve the workspace. I think that may be true for all of them - or at least all cargo projects. Once downloadable LLVM lands we can skip that though.

[Mark-Simulacrum]

@ehuss @Eh2406 -- I think you've both been working on Cargo's resolver recently. How feasible would it be to explicitly tell Cargo "this crate, and its dependencies, should be presumed unchanged" when re-resolving a lockfile? My guess is it's quite hard.

Currently we get something like this whenever a submodule is just not yet initialized:

error: failed to read `/home/mark/Build/rust/src/tools/cargo/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

One "solution" to this perhaps is to duplicate submodule's Cargo.toml's in-tree but stubbed out and pointing at empty crates (basically just so resolver works). But I'd really prefer to avoid doing that, and it does seem like Cargo should have all the information already available. We could plausibly arrange for x.py to fetch just the Cargo.toml's for these submodules, I guess, though that also seems painful.

[ehuss]

I think it will be quite difficult, since there are a few other things in Cargo that expect the manifests and files to be there.

Do the cargo-based projects really matter performance-wise? It seems like submodule stuff is completely dominated by llvm. On my system, it looks like it is 3.5 times larger than all the others added together.

Also, rust-by-example is unnecessarily huge. I think squashing the gh-pages branch in that repo would shrink it dramatically.

The next largest is rust-analyzer. That's more feasible to ignore, since it is in its own little world. Unfortunately it has node-modules checked in, which is quite huge.

What is the status of using git subtree?

[mati865]

What is the status of using git subtree?

AFAIK patch to subtree is still not upstream so you need to patch and build it locally to create subtree. cc @flip1995

[jyn514]

I agree LLVM is the main issue - it would be nice to have the rest downloaded on-demand, but it's not dramatically hurting setup times I think.

Also, rust-by-example is unnecessarily huge. I think squashing the gh-pages branch in that repo would shrink it dramatically.

👍 sounds easy enough to do

[flip1995]

git-subtree is only really feasible, when the git PR (gitgitgadget/git#493) gets merged and released. Also Clippy is waiting for this rustup feature to get released: rust-lang/rustup#2438

[jyn514]

Also, rust-by-example is unnecessarily huge. I think squashing the gh-pages branch in that repo would shrink it dramatically.

Do you have suggestions for how to do this? I tried https://stackoverflow.com/a/1661283/7669110 locally, which worked in the sense that the history is now one commit, but didn't decrease the size of .git at all even after running git gc --aggressive.

[bjorn3]

You probably still have the remote branch. Try removing .git/refs/remotes/origin/gh-pages and then running git gc --aggressive.

[ehuss]

I think the issue you created looks correct. It needs someone with write permissions to that repo to do it directly. As with anything git-related, there are many ways to accomplish the same task. If it is wrong, worst case we can rebuild the branch. I don't think there is any history there that is important, and at least I have a local backup copy.

I did this before with some repo (I can't remember which). I think GitHub will gc pretty quick (probably within 24 hours), though I imagine they have heuristics on how often that happens.