Skip to content

fix: vendor hosted patches directly from lockfiles #1648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
base: main
Choose a base branch
from
+381 −153
Draft

fix: vendor hosted patches directly from lockfiles #1648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions crates/socket-patch-cli/CLI_CONTRACT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -720,31 +720,39 @@ Not covered: the agent-mode blob cache and manifest are checked from `.socket` d

## Vendor command contract

`vendor` is `apply`'s committable sibling: instead of patching installed packages in place
(machine-local state), it ejects each patched package into `.socket/vendor/` and rewires the
`vendor` converts the hosted patches pinned in the project's lockfiles into committable
`.socket/vendor/` artifacts, without installing dependencies first. It rewires the
ecosystem's lockfile/config so the project consumes the vendored copy. After committing
`.socket/vendor/` + the lockfile edits, a fresh checkout builds with the patched dependency on
machines with **no socket-patch installed and no Socket API access** (registry access for other,
unvendored dependencies may still be needed). Every mechanism below was validated against the real
package managers (`spikes/PHASE0-FINDINGS.txt`).

**Eject a hosted project (v5.0)**: standalone `vendor` with NO manifest in a project whose lockfiles
**Eject a hosted project (v5.0)**: standalone `vendor` in a project whose lockfiles
pin hosted patches (not under `--global`; `--ecosystems` narrows the pins) takes its patch set from
those pins — each pin's purl plus the patch uuid in its hosted URL — fetches each record from the
patch API (`GET …/patches/view/<uuid>`, the same client and public-proxy fallback as `get`), vendors
into `.socket/vendor/` exactly like `scan --mode vendored`, and rewires each package from hosted to
vendored (the upstream registry entry is restored first, so a later `vendor --revert` returns the
project to upstream, never to hosted). The human output opens with
`Ejecting N hosted package(s) into .socket/vendor/...` (`Would eject …` under `--dry-run`); the
JSON is the vendor envelope. It needs no installed `node_modules` / site-packages: the sources are
fetched from the upstream registry, so a fresh hosted checkout ejects.
JSON is the vendor envelope. Hosted pins take precedence even when `.socket/manifest.json`
exists: the manifest is neither used to select patches nor rewritten. The conversion skips
installed-package discovery and variant probes, so an unrelated virtualenv or global cache
cannot reject a lockfile-selected patch. It uses the shared lockfile readers, without VEX
attestation or installed-byte verification; downloaded artifacts still pass integrity and
patched-member verification. Upstream registry metadata is resolved for reversal, while the
patched archives come from the artifact service. No installed `node_modules`, site-packages,
or upstream source download is needed. When no hosted pins are in scope, legacy manifest-driven
vendoring remains available.

The eject is ONE planned transition, all-or-nothing: (1) every record is fetched first — a failed
(or 404) view fetch is a `failed` event with `errorCode: "patch_fetch_failed"` and the run stops
with `status: "error"`, `error.code: "eject_refused"`, exit 1, nothing touched; (2) the upstream
restore of every pin is resolved against staged copies — a refused pin (offline registry, a
non-derivable field; a binary `bun.lockb` record is rebuilt like the takeover's) is `eject_refused` with the `git checkout -- <lockfile>`
remedy, nothing touched; (3) `--dry-run` stops here and reports each pin as an `applied` event with
remedy, nothing touched. Backend refusals are also previewed against the restored text in memory
(for example, unsupported pnpm catalog entries); (3) `--dry-run` stops here and reports each pin as an `applied` event with
reason `eject_planned` — no file is written and no `.socket/` is created; (4) the wet run snapshots
every file the eject may touch under one `apply.lock`, restores upstream, then vendors. If any
package then fails, exactly the files the eject wrote are put back (the pins' files, the upstream
Expand Down
171 changes: 105 additions & 66 deletions crates/socket-patch-cli/src/commands/vendor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! `socket-patch vendor` — committable vendoring of patched dependencies.
//!
//! Works like `apply`, but instead of patching installed packages in place it
//! ejects each patched package into `.socket/vendor/<eco>/<patch-uuid>/…` and
//! Converts the hosted patches pinned in the project's lockfiles into
//! `.socket/vendor/<eco>/<patch-uuid>/…` without an installed package tree, and
//! rewires the ecosystem's lockfile/config so the project consumes the
//! vendored copy. After committing `.socket/vendor/` + the lockfile edits, a
//! fresh checkout builds with the patched dependency on machines with no
Expand All @@ -15,7 +15,8 @@
//! vendored` drive this module's [`vendor_records`] engine directly in
//! DETACHED mode: every entry they write carries its patch record embedded
//! in the ledger and no `.socket/manifest.json` is ever written — this
//! command is the one manifest-driven writer (`detached: false`). See
//! command also supports legacy manifest-driven vendoring when no hosted
//! pins are in scope (`detached: false`). See
//! CLI_CONTRACT.md "Ownership, state, and reversal".

use clap::Args;
Expand Down Expand Up @@ -987,15 +988,10 @@ pub async fn run(args: VendorArgs) -> i32 {
let manifest_path = args.common.resolved_manifest_path();
let socket_dir = crate::args::socket_dir_of(&manifest_path, &args.common.cwd);

// `--revert` derives everything from state.json + the vendor tree; it
// must work after the manifest was deleted. Plain vendor needs the
// manifest and exits clean without one (same contract as apply). This
// is a MANIFEST check, not a `.socket/` check: `scan`/`get --mode
// vendored` projects have `.socket/` but never a manifest. Nothing is
// locked or written on this path.
if !args.revert && tokio::fs::metadata(&manifest_path).await.is_err() {
// A hosted project (no manifest, hosted pins in its lockfiles)
// ejects: its patch set is the lockfiles' hosted pins.
// Hosted pins are the input, even when an agent manifest also exists.
// The raw lockfile inventory does not require VEX eligibility or an
// installed copy of a patch. Revert needs only the vendor ledger.
if !args.revert {
let inventory = crate::commands::hosted_inventory(&args.common, &args.common.cwd).await;
// Contested hosted wiring: the patch set cannot be read off the
// lockfiles, and a "nothing to vendor" answer would hide it.
Expand All @@ -1020,6 +1016,12 @@ pub async fn run(args: VendorArgs) -> i32 {
}
return run_eject(&args, pins).await;
}
}

// Preserve legacy manifest-only vendoring when there are no hosted
// pins. A missing manifest is a no-op; detached vendored projects keep
// their records in the ledger instead.
if !args.revert && tokio::fs::metadata(&manifest_path).await.is_err() {
// A requested `--vex` still attests what the `.socket/vendor`
// ledgers and lockfiles already wire. Same contract as `apply --vex`
// with no manifest: nothing referenced anywhere keeps exit 0; any
Expand Down Expand Up @@ -1719,9 +1721,9 @@ impl EjectSnapshot {
}
}

/// Standalone `vendor` in a hosted project — no manifest, hosted pins in the
/// lockfiles: EJECT. The patch set is the pins themselves (purl + the uuid
/// in each hosted URL); each record is fetched from the API, vendored into
/// Standalone `vendor` in a hosted project: EJECT. The patch set comes only
/// from the lockfile pins (purl + the uuid in each hosted URL); each record
/// is fetched from the API, vendored into
/// `.socket/vendor/` exactly like `scan --mode vendored`, and the lock is
/// rewired from hosted to vendored (the engine's takeover restores each
/// pin's upstream registry entry first, so `vendor --revert` later returns
Expand All @@ -1731,6 +1733,7 @@ async fn run_eject(args: &VendorArgs, pins: Vec<HostedPin>) -> i32 {
let (client, use_public_proxy) =
get_api_client_with_overrides(common.api_client_overrides()).await;
let telemetry = TelemetryAuth::for_client(&client);
let service = common.vendor_service_config(Some(client.clone()), use_public_proxy);
if !common.json && !common.silent {
println!(
"{} {} into .socket/vendor/...",
Expand Down Expand Up @@ -1827,11 +1830,39 @@ async fn run_eject(args: &VendorArgs, pins: Vec<HostedPin>) -> i32 {
.refused()
.map(|(pin, why)| (pin.purl.clone(), "redirect_revert_failed", why.to_string()))
.collect();
// An sbt / scala-cli pin is vendored only past the build-evidence gate
// (no `target/` evidence in a fresh clone, stale evidence, …). Run it
// now, while the hosted pin still serves the patch: a pin the gate
// would stop after the restore would end neither hosted nor vendored.
// Reuse the takeover's read-only backend preview over the restored
// lockfile text. A manifest's presence must not change whether a dry
// run catches an unsupported lock entry (for example a pnpm catalog).
let uninstalled = common.cwd.join(".socket/vendor/.uninstalled");
let blobs = common.socket_dir().join("blobs");
let sources = PatchSources {
blobs_path: &blobs,
mem_blobs: None,
};
let pipenv_version = tokio::sync::OnceCell::new();
let installed_sites = vendor::pypi::InstalledSiteListings::default();
let vendored_at = now_rfc3339();
for pin in &pins {
if let Some((code, detail)) = takeover_dry_refusal(
&plan,
&pin.purl,
PackageSource::Installed(&uninstalled),
&common.cwd,
&records[&pin.purl],
&sources,
&vendored_at,
args.force,
Some(&service),
&pipenv_version,
&installed_sites,
)
.await
{
refused.push((pin.purl.clone(), code, detail));
}
// sbt / scala-cli also require build evidence. The backend treats
// that gate as a benign skip, but an eject must retain the hosted
// pin if it cannot vendor it.
if let Err((code, detail)) =
socket_patch_core::vendor::maven_repo::jvm_gate_preflight(&common.cwd, &pin.purl).await
{
Expand Down Expand Up @@ -1912,11 +1943,10 @@ async fn run_eject(args: &VendorArgs, pins: Vec<HostedPin>) -> i32 {
}

// One transaction under one apply lock: snapshot what the eject can
// touch, restore every pin upstream (so the vendor engine resolves the
// pristine registry package even in a fresh checkout with nothing
// installed), vendor, and on ANY failure put back what the eject wrote
// — a failed eject leaves the project hosted, exactly as it was, and
// every file it never wrote untouched.
// touch, restore upstream metadata for the vendor ledger's reversal,
// download and wire the prebuilt patches, and on ANY failure put back
// what the eject wrote. A failed eject leaves the project hosted,
// exactly as it was, and every file it never wrote untouched.
let socket_dir = common.socket_dir();
let timeout = Duration::from_secs(common.lock_timeout.unwrap_or(0));
let guard = match crate::commands::lock_cli::acquire_with_status(&socket_dir, timeout) {
Expand Down Expand Up @@ -1987,17 +2017,16 @@ async fn run_eject(args: &VendorArgs, pins: Vec<HostedPin>) -> i32 {
}
let manifest = crate::commands::vendored_backend::records_manifest(records);
// The same vendored apply as `scan --mode vendored`: detached (the
// eject is manifest-free) over this run's client and the shared
// service-config assembler.
let service = common.vendor_service_config(Some(client.clone()), use_public_proxy);
// eject does not use the agent manifest) over this run's client
// and the shared service-config assembler.
let applied = VendoredBackend::new(common, Some(&service))
.apply(
ApplyRequest {
manifest: &manifest,
socket_dir: &socket_dir,
ledger: load_state(&common.cwd).await,
detached: true,
force: false,
force: args.force,
prior: None,
eject: Some(&mut capture),
},
Expand Down Expand Up @@ -2082,7 +2111,7 @@ async fn run_eject(args: &VendorArgs, pins: Vec<HostedPin>) -> i32 {

// Embedded VEX: same contract as the manifest-driven arm — only on
// success, never on a dry run, and a requested-but-failed VEX flips the
// exit code. The ejected project has no manifest.
// exit code. The ejected patches are recorded in the vendor ledger.
if exit == 0 {
if let Some(vex_path) = args.vex.vex.as_ref() {
if common.dry_run {
Expand Down Expand Up @@ -2769,41 +2798,48 @@ pub(crate) async fn vendor_records_reusing(
vendor::prestage::sweep_stale(&common.cwd).await;
}

let crawler_options = common.crawler_options();
// Resolve installed packages with the qualified-purl-aware resolver, never
// a base-keyed one: the manifest keys release-variant ecosystems (gem
// `?platform=`, pypi `?artifact_id=`, maven `?classifier=&ext=`) by
// *qualified* purls, but the crawler only knows the *base* purl, so a
// base-keyed map would misclassify every installed qualified purl as
// "not installed". The rollback variant fans each base path back out to
// every qualified manifest purl.
let mut all_packages: HashMap<String, StagedSource> = find_packages_for_rollback_reusing(
&vendorable_partition,
&crawler_options,
common.silent || common.json,
prior,
)
.await
.into_iter()
.map(|(purl, dir)| (purl, StagedSource::Installed(dir)))
.collect();

// An npm alias is installed under its dependency key, not its actual
// package name. The targeted resolver probes canonical paths; before
// fetching a supposedly missing source, resolve aliases by the installed
// package.json identity. This also permits offline binary Bun vendoring.
let missing_npm: Vec<&String> = vendorable_partition
.get(&Ecosystem::Npm)
// An eject already selected exact patch UUIDs from the lockfiles. Its
// artifacts come from the service: installed copies (including an
// unrelated global Python/Ruby environment) cannot select or reject
// them. Leave their sources missing and let the backends read the locks.
let mut all_packages: HashMap<String, StagedSource> = HashMap::new();
if eject.is_none() {
let crawler_options = common.crawler_options();
// Resolve installed packages with the qualified-purl-aware resolver, never
// a base-keyed one: the manifest keys release-variant ecosystems (gem
// `?platform=`, pypi `?artifact_id=`, maven `?classifier=&ext=`) by
// *qualified* purls, but the crawler only knows the *base* purl, so a
// base-keyed map would misclassify every installed qualified purl as
// "not installed". The rollback variant fans each base path back out to
// every qualified manifest purl.
all_packages = find_packages_for_rollback_reusing(
&vendorable_partition,
&crawler_options,
common.silent || common.json,
prior,
)
.await
.into_iter()
.flatten()
.filter(|p| !all_packages.contains_key(*p))
.map(|(purl, dir)| (purl, StagedSource::Installed(dir)))
.collect();
let by_identity = match prior.and_then(|p| p.packages_for(&crawler_options)) {
Some(installed) => npm_paths_by_identity_in(installed, &missing_npm),
None => npm_paths_by_identity(&crawler_options, &missing_npm).await,
};
for (purl, paths) in by_identity {
all_packages.insert(purl, StagedSource::Installed(paths[0].clone()));

// An npm alias is installed under its dependency key, not its actual
// package name. The targeted resolver probes canonical paths; before
// fetching a supposedly missing source, resolve aliases by the installed
// package.json identity. This also permits offline binary Bun vendoring.
let missing_npm: Vec<&String> = vendorable_partition
.get(&Ecosystem::Npm)
.into_iter()
.flatten()
.filter(|p| !all_packages.contains_key(*p))
.collect();
let by_identity = match prior.and_then(|p| p.packages_for(&crawler_options)) {
Some(installed) => npm_paths_by_identity_in(installed, &missing_npm),
None => npm_paths_by_identity(&crawler_options, &missing_npm).await,
};
for (purl, paths) in by_identity {
all_packages.insert(purl, StagedSource::Installed(paths[0].clone()));
}
}
let vendored_installs =
drop_vendored_installs_by(&common.cwd, &mut all_packages, |source| match source {
Expand Down Expand Up @@ -2884,10 +2920,13 @@ pub(crate) async fn vendor_records_reusing(
// the lockfiles still pin hosted must restore its upstream registry
// entry FIRST (see the dispatch loop below). Discovered once, before any
// write of this run.
let hosted_pins: Vec<socket_patch_core::patch::redirect::upstream::HostedPin> =
socket_patch_core::patch::redirect::upstream::HostedPin::all(
&crate::commands::discover_wiring(common, &common.cwd).await,
);
// Eject restored all its pins in one transaction before calling us;
// there is no per-package takeover or second discovery to perform.
let hosted_pins = if eject.is_some() {
Vec::new()
} else {
HostedPin::all(&crate::commands::discover_wiring(common, &common.cwd).await)
};
let hosted_pin_of = |purl: &str| {
hosted_pins
.iter()
Expand Down
3 changes: 3 additions & 0 deletions crates/socket-patch-cli/src/commands/vendored_backend/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ pub(crate) struct ApplyRequest<'a> {
/// rollback undoes exactly these), and the wired flavors. With it, the
/// run's human summary and "Next steps:" are left to the eject, which
/// prints them only once it knows the eject stands.
/// The records are exact lockfile-selected patches with their hosted
/// wiring already restored upstream; skip installed-package discovery
/// and per-package hosted takeover.
pub(crate) eject: Option<&'a mut EjectCapture>,
}

Expand Down
9 changes: 5 additions & 4 deletions crates/socket-patch-cli/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,12 @@ pub enum Commands {
/// Generate OpenVEX for vulnerabilities addressed by verified patches
Vex(commands::vex::VexArgs),

/// Store patched dependencies in .socket/vendor/ for offline installs
/// Download hosted patches from lockfiles into .socket/vendor/
///
/// Rewire dependency files to use the committed artifacts. Unpatched
/// dependencies still need their normal registry or cache. Use --revert
/// to undo vendoring.
/// Works before installing dependencies and rewires dependency files to
/// use the committed artifacts. Falls back to the agent manifest when no
/// hosted patches are in scope. Unpatched dependencies still need their
/// normal registry or cache. Use --revert to undo vendoring.
Vendor(commands::vendor::VendorArgs),

/// Reapply agent-mode patches after installing dependencies
Expand Down
16 changes: 15 additions & 1 deletion crates/socket-patch-cli/tests/in_process_vendor_pnpm_takeover.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -637,7 +637,21 @@ async fn vendor_dry_run_over_hosted_pnpm_catalog_dep_previews_the_refusal() {
.unwrap();
let hosted = snapshot(root);
let vendor = |extra: &[&str]| {
let mut args = vec!["vendor", "--json", "--cwd", root.to_str().unwrap()];
let uri = server.uri();
let mut args = vec![
"vendor",
"--json",
"--cwd",
root.to_str().unwrap(),
"--api-url",
&uri,
"--api-token",
"fake",
"--org",
ORG,
"--patch-server-url",
&uri,
];
args.extend_from_slice(extra);
run_json(root, &server.uri(), &args)
};
Expand Down
Loading
Loading