From 671cbc7c7ff4f9e01b41ecb86ad49e3372e906f0 Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Sat, 7 Nov 2026 11:33:54 +0000 Subject: [PATCH 1/3] Give each package its own version and release only what changed Every published module is a package in ci/packages.json, named by its artifactId and tagged /. A publish takes its plan from common-ci's steps/release-plan.ps1, which releases a package when a file under its folders changed since its last tag, and every package that depends on it, each at its last tag with the patch number moved. build-package.ps1 writes each module's own version into its pom.xml, with the versions of its parent and of the packages it depends on, and deploys only the planned modules. publish-package.ps1 tags and releases each one. check-version-bump.ps1 and check-release-interval.ps1 check each planned package against its own tags. build-project.ps1 fails a pull request whose poms do not match ci/packages.json. Step 3.5 of https://github.com/51Degrees/common-ci/issues/306. --- AGENTS.md | 22 +++-- README.md | 7 +- ci/README.md | 29 +++++-- ci/build-package.ps1 | 57 +++++-------- ci/build-project.ps1 | 8 ++ ci/check-release-interval.ps1 | 42 ++++++++++ ci/check-version-bump.ps1 | 36 +++++++++ ci/get-next-package-version.ps1 | 38 ++++----- ci/package-update-required.ps1 | 23 ++++++ ci/packages.json | 47 +++++++++++ ci/packages.ps1 | 139 +++++++++++++++++++++++++++++++- ci/publish-package.ps1 | 43 +++++++++- ci/release.ps1 | 125 ++++++++++++++++++++++++++++ ci/run-integration-tests.ps1 | 21 ++--- ci/run-performance-tests.ps1 | 13 +-- ci/update-packages.ps1 | 2 +- pom.xml | 12 +-- 17 files changed, 559 insertions(+), 105 deletions(-) create mode 100644 ci/check-release-interval.ps1 create mode 100644 ci/check-version-bump.ps1 create mode 100644 ci/package-update-required.ps1 create mode 100644 ci/packages.json create mode 100644 ci/release.ps1 diff --git a/AGENTS.md b/AGENTS.md index 971eb5d05..bd5ab1aab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,14 +226,20 @@ example. The IP Intelligence cloud examples read their key from The root `pom.xml` lists the six package trees as modules but is not their parent, so nothing from it reaches the published poms. Each tree keeps its own -parent pom. They all carry the same version, and the libraries reference each -other through `${project.version}`, so Maven resolves those dependencies from -the reactor. The examples reference published versions unless the `dev` -profile is active. Keep it that way: a reader of the examples should see the -package versions a project of their own would use. - -Because the root is not a parent, `versions:set` has to run in each tree. CI -does that. When changing versions by hand, run it in each of the six folders. +parent pom. In the repository they all carry the same development version, +and the libraries reference each other through `${project.version}`, so Maven +resolves those dependencies from the reactor. The examples reference published +versions unless the `dev` profile is active. Keep it that way, so that a +reader of the examples sees the package versions a project of their own would +use. + +Each published module is released at its own version, tagged +`/`. The publish writes that version into the module's +`pom.xml`, with the versions of its parent and of the packages it depends on, +so leave the versions in the poms as they are. When a module is added, or one +module starts to depend on another, add it to `ci/packages.json` as well, or +the pull request build fails. `ci/README.md` describes how a release is +planned. ## Data files and `project.root` diff --git a/README.md b/README.md index 479f1ff36..125c2c612 100644 --- a/README.md +++ b/README.md @@ -132,9 +132,10 @@ does the same. ## Versions -The repository has one version, and every package is released with it. CI -sets the version on each package tree before a release and tags the commit -with the plain version, such as `4.5.127`. +Each package has its own version, and a release publishes only the packages +that changed since their last release and the packages that depend on them. +Each is tagged with its name and version, such as `pipeline.core/4.5.131`. +[ci/README.md](ci/README.md) describes how CI works out what to release. ## History diff --git a/ci/README.md b/ci/README.md index bf6b2a62b..e6e1dc7ee 100644 --- a/ci/README.md +++ b/ci/README.md @@ -11,10 +11,18 @@ repositories used to do for one package tree each. publish, the two on-premise engines that build a native library, and the generated accessors. The other scripts loop over it rather than naming folders themselves. -- The root `pom.xml` builds every tree in one reactor but is not their parent, - so [build-package.ps1](build-package.ps1) sets the version in each tree - before it deploys from the root. The examples are built and tested but left - out of the deploy. +- [packages.json](packages.json) lists every published module as a package, + named by its artifactId, with its folder, the other folders its build takes + files from, and the packages it depends on, its parent included. A parent's + folder is its `pom.xml` alone, because the rest of the folder is its + modules. [build-project.ps1](build-project.ps1) fails a pull request whose + poms refer to a package that `DependsOn` does not name, or add a module + that has no entry. +- The root `pom.xml` builds every tree in one reactor but is not their parent. + [build-package.ps1](build-package.ps1) writes each published module's own + version into its `pom.xml`, with the versions of its parent and of the + packages it depends on, and deploys only the modules being released. The + examples are built and tested but left out of the deploy. - [build-project.ps1](build-project.ps1) and [run-unit-tests.ps1](run-unit-tests.ps1) use the `dev` profile, so a pull request tests the examples against the libraries it changes. The publish @@ -31,9 +39,16 @@ repositories used to do for one package tree each. API, Device Detection and IP Intelligence and writes one results file per configuration. The Device Detection and IP Intelligence metrics are prefixed with the product, as both use the same names. -- The repository has one version, tagged as a plain version such as `4.5.127`. - [get-next-package-version.ps1](get-next-package-version.ps1) moves only the - patch number. +- Each package has its own version, tagged `/`, such as + `pipeline.core/4.5.131`. A publish releases a package when a file under its + folders changed since its last tag, and every package that depends on it, + each at its last tag with the patch number moved. + [release.ps1](release.ps1) takes that plan from common-ci's + `steps/release-plan.ps1`, and the publish scripts build, check and tag from + it. [check-version-bump.ps1](check-version-bump.ps1) and + [check-release-interval.ps1](check-release-interval.ps1) check each package + against its own tags, for a major or minor move and for a release less than + a week after its last. ## Secrets diff --git a/ci/build-package.ps1 b/ci/build-package.ps1 index 38de86fe9..d456c15fb 100644 --- a/ci/build-package.ps1 +++ b/ci/build-package.ps1 @@ -7,7 +7,15 @@ param( $ErrorActionPreference = "Stop" $PSNativeCommandUseErrorActionPreference = $true -. "$PSScriptRoot/packages.ps1" +. "$PSScriptRoot/release.ps1" + +# Builds the packages this publish releases, each at its own version from the +# release plan (see release.ps1). $Version is common-ci's label for the run and +# is not used. +$plan = @(Get-BuildPlan $RepoName) +Write-Host "Building $($plan.Count) packages:" +$plan | Format-Table Name, Previous, Version, Reason -AutoSize | Out-String -Width 200 | + Write-Host # Put the native libraries built on every platform into the engine they came # from, so each engine's jar carries all of its own and none of the other's. @@ -21,46 +29,25 @@ foreach ($package in $NativePackages) { } } -# The root pom.xml is not the parent of the package trees, so setting the -# version at the root does not reach them. Set it in each tree, the examples -# included, so the whole repository carries the version being released. -foreach ($package in $Packages) { - Write-Host "Setting the version of '$($package.Path)' to '$Version'" - Push-Location "$RepoName/$($package.Path)" - try { - mvn versions:set @MavenOptions "-DnewVersion=$Version" -DgenerateBackupPoms=false - } finally { - Pop-Location - } -} - -# The examples are never published. They are left out of the deploy so that -# the last project in the reactor is one the publishing plugin is configured -# for. Maven 3 does not exclude a folder's modules with the folder, so each of -# them is named. -$repoPath = (Resolve-Path $RepoName).Path -$examples = foreach ($package in $Packages | Where-Object { -not $_.Publish }) { - Get-ChildItem -Path "$RepoName/$($package.Path)" -Recurse -Filter pom.xml | - Where-Object { $_.FullName -notmatch '[\\/](target|node_modules)[\\/]' } | - ForEach-Object { '!' + [IO.Path]::GetRelativePath($repoPath, $_.DirectoryName).Replace('\', '/') } -} +# Every published module gets its own version, and the versions of its parent +# and of the packages it depends on, including those this publish does not +# release, which stay at their last release. +Set-PomVersions $RepoName (Get-BuildVersions $RepoName $plan) -# The publishing plugin writes the one bundle for the whole reactor under the -# first tree that publishes. common-ci copies the bundle from where a single -# tree puts it, below target at the root, so a link at that path leads to the -# bundle. The link is made before the deploy, which Linux allows. -$firstPublishing = ($Packages | Where-Object { $_.Publish } | Select-Object -First 1).Path -New-Item -ItemType Directory -Force -Path "$RepoName/target" | Out-Null -New-Item -ItemType SymbolicLink -Force -Path "$RepoName/target/central-publishing" -Target "../$firstPublishing/target/central-publishing" | Out-Null +# Only the modules in the plan are deployed, so the one publishing bundle holds +# only them. A package they depend on that is not released is taken from Maven +# Central at its last release, the version their poms now require. +$modules = @($plan | ForEach-Object { Get-ModulePath $_ }) -# common-ci sets the version of the root pom.xml, deploys from the root to a -# local staging folder with signing, and copies the result and the publishing -# bundle into the package folder. +# common-ci deploys from the root to a local staging folder with signing, and +# copies the result and the publishing bundle into the package folder. The +# versions are already set, so it is given no folder to set them in. ./java/build-package.ps1 ` -RepoName $RepoName ` -Name $Name ` -Version $Version ` - -ExtraArgs "-DskipNativeBuild=true", "-pl", ($examples -join ',') ` + -VersionPaths @() ` + -ExtraArgs "-DskipNativeBuild=true", "-pl", ($modules -join ',') ` -JavaGpgKeyPassphrase $Keys['JavaGpgKeyPassphrase'] ` -JavaPGP $Keys['JavaPGP'] ` -CodeSigningKeyVaultName: $Keys['CodeSigningKeyVaultName'] ` diff --git a/ci/build-project.ps1 b/ci/build-project.ps1 index d8524aa3e..eea288f3f 100644 --- a/ci/build-project.ps1 +++ b/ci/build-project.ps1 @@ -9,6 +9,14 @@ $PSNativeCommandUseErrorActionPreference = $true Write-Host "JAVA_HOME: $ENV:JAVA_HOME" +# A publish releases a package's dependents only when packages.json names +# them, so a module or dependency it leaves out fails here, before the change +# is merged, rather than at the publish. +$gaps = @(Get-PackagesJsonGaps $RepoName) +if ($gaps.Count -gt 0) { + throw "ci/packages.json does not match the poms:`n$($gaps -join "`n")" +} + # One build of the whole reactor, native libraries included. The dev profile # builds the examples against the libraries here rather than the versions # published to Maven Central, so a change to a library is tested by the diff --git a/ci/check-release-interval.ps1 b/ci/check-release-interval.ps1 new file mode 100644 index 000000000..305e15512 --- /dev/null +++ b/ci/check-release-interval.ps1 @@ -0,0 +1,42 @@ +param ( + [Parameter(Mandatory)][string]$RepoName, + [string]$Repository, + [string]$HowToAllow +) +$ErrorActionPreference = "Stop" +. "$PSScriptRoot/release.ps1" + +# The shared publish runs this in place of its own check, because each package +# has its own version and tags (see release.ps1). Every package about to be +# published goes through the shared steps/check-release-interval.ps1, run from +# the common-ci root, against its own '/' tags, so each +# package gets at most one new version a week. That refuses a package +# released less than a week ago unless ALLOW_EARLY_RELEASE is 'true', from the +# date in WEEKLY_RELEASES_FROM. + +$plan = @(Get-ReleasePlan $RepoName) +if ($plan.Count -eq 0) { + Write-Host "No package is to be published, so there is nothing to check." + exit 0 +} + +$arguments = @{ RepoName = $RepoName; Repository = $Repository } +if ($HowToAllow) { + $arguments.HowToAllow = $HowToAllow +} + +$refused = @() +foreach ($release in $plan) { + Write-Host "$($release.Name):" + $global:LASTEXITCODE = 0 + ./steps/check-release-interval.ps1 @arguments -TagPrefix $release.Name + if ($LASTEXITCODE -ne 0) { + $refused += $release.Name + } +} + +if ($refused) { + Write-Host "Nothing is released, because these packages were released less than a week ago: $($refused -join ', ')." + exit 1 +} +exit 0 diff --git a/ci/check-version-bump.ps1 b/ci/check-version-bump.ps1 new file mode 100644 index 000000000..f53fae22d --- /dev/null +++ b/ci/check-version-bump.ps1 @@ -0,0 +1,36 @@ +param ( + [Parameter(Mandatory)][string]$RepoName, + [string]$Version +) +$ErrorActionPreference = "Stop" +. "$PSScriptRoot/release.ps1" + +# The shared publish runs this in place of its own check, because each package +# has its own version (see release.ps1). Every package about to be published +# goes through the shared steps/check-version-bump.ps1, run from the common-ci +# root, against its own '/' tags. That refuses a major or +# minor move unless ALLOW_VERSION_BUMP is 'true'. $Version is common-ci's label +# for the run. + +Write-Host "The publish worked out $Version, its highest. Each package is checked on its own." + +$plan = @(Get-ReleasePlan $RepoName) +if ($plan.Count -eq 0) { + Write-Host "No package is to be published, so there is nothing to check." + exit 0 +} + +$refused = @() +foreach ($release in $plan) { + $global:LASTEXITCODE = 0 + ./steps/check-version-bump.ps1 -Version $release.Version -RepoName $RepoName -TagPrefix $release.Name + if ($LASTEXITCODE -ne 0) { + $refused += $release.Name + } +} + +if ($refused) { + Write-Host "Nothing is published, because these packages were refused: $($refused -join ', ')." + exit 1 +} +exit 0 diff --git a/ci/get-next-package-version.ps1 b/ci/get-next-package-version.ps1 index de0e6c291..ddfc48d64 100644 --- a/ci/get-next-package-version.ps1 +++ b/ci/get-next-package-version.ps1 @@ -3,31 +3,21 @@ param ( [Parameter(Mandatory)][string]$VariableName ) $ErrorActionPreference = "Stop" +. "$PSScriptRoot/release.ps1" -# The whole repository has one version, tagged as a plain version such as -# 4.5.127. Only the patch number moves here. The major and minor numbers -# change only when someone edits FirstVersion before the first release or -# pushes a tag by hand. - -# The version used while the repository has no version tag. It is above every -# version the absorbed repositories published, so the first release from here -# is an ordinary upgrade for each package. -$FirstVersion = '4.5.127' - -$tags = @(git -C $RepoName tag --list | - Where-Object { $_ -match '^\d+\.\d+\.\d+$' } | - Sort-Object { [version]$_ }) -$headTags = @(git -C $RepoName tag --points-at HEAD | - Where-Object { $_ -match '^\d+\.\d+\.\d+$' }) - -if ($headTags) { - # Already released, so the publish sees the tag and does nothing. - $next = $headTags | Sort-Object { [version]$_ } | Select-Object -Last 1 -} elseif ($tags) { - $highest = [version]$tags[-1] - $next = "$($highest.Major).$($highest.Minor).$($highest.Build + 1)" +# common-ci's publish carries one version through its jobs. This repository +# releases each package at its own version (see release.ps1), so that one +# version is only a label for the run, being the highest version this publish +# releases. Whether anything is released is decided by +# package-update-required.ps1, and each package's version and tag by +# build-package.ps1 and publish-package.ps1, all from the same release plan. +$plan = @(Get-ReleasePlan $RepoName) +if ($plan.Count -gt 0) { + $label = ($plan.Version | Sort-Object { [version]$_ })[-1] } else { - $next = $FirstVersion + # Nothing to release. Any valid version will do, because + # package-update-required.ps1 stops the publish before it is used. + $label = '4.5.0' } -Set-Variable -Scope Global -Name $VariableName -Value $next +Set-Variable -Scope Global -Name $VariableName -Value $label diff --git a/ci/package-update-required.ps1 b/ci/package-update-required.ps1 new file mode 100644 index 000000000..da231faad --- /dev/null +++ b/ci/package-update-required.ps1 @@ -0,0 +1,23 @@ +param ( + [Parameter(Mandatory)][string]$RepoName, + [string]$Version +) +$ErrorActionPreference = "Stop" +. "$PSScriptRoot/release.ps1" + +# Called by common-ci's publish in place of its own check, because a plain tag +# lookup cannot tell this repository's packages apart. Exits 0 when the release +# plan has something in it and 1 when it is empty, as common-ci's check does. +# $Version is common-ci's label for the run and is not used here. Each +# package's version is checked by check-version-bump.ps1. + +$plan = @(Get-ReleasePlan $RepoName) +if ($plan.Count -eq 0) { + Write-Output "No package changed since its last release, so there is nothing to publish" + exit 1 +} + +Write-Output "This publish releases $($plan.Count) packages, in this order:" +$plan | Format-Table Name, Previous, Version, Reason -AutoSize | Out-String -Width 200 | + Write-Output +exit 0 diff --git a/ci/packages.json b/ci/packages.json new file mode 100644 index 000000000..427227219 --- /dev/null +++ b/ci/packages.json @@ -0,0 +1,47 @@ +[ + {"Name": "pipeline", "Path": "pipeline/pom.xml", "DependsOn": [], "Publish": true}, + {"Name": "pipeline.common", "Path": "pipeline/pipeline.common", "DependsOn": ["pipeline"], "Publish": true}, + {"Name": "pipeline.core", "Path": "pipeline/pipeline.core", "DependsOn": ["pipeline", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.did", "Path": "pipeline/pipeline.did", "ExtraPaths": ["pipeline/owid-java"], "DependsOn": ["pipeline"], "Publish": true}, + {"Name": "pipeline.engines", "Path": "pipeline/pipeline.engines", "DependsOn": ["pipeline", "pipeline.core", "pipeline.caching", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.caching", "Path": "pipeline/pipeline.caching", "DependsOn": ["pipeline", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.cloudrequestengine", "Path": "pipeline/pipeline.cloudrequestengine", "DependsOn": ["pipeline", "pipeline.engines", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.engines.fiftyone", "Path": "pipeline/pipeline.engines.fiftyone", "DependsOn": ["pipeline", "pipeline.engines", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.translation", "Path": "pipeline/pipeline.translation", "DependsOn": ["pipeline", "pipeline.engines", "pipeline.core"], "Publish": true}, + {"Name": "pipeline.javascriptbuilder", "Path": "pipeline/pipeline.javascriptbuilder", "DependsOn": ["pipeline", "pipeline.jsonbuilder", "pipeline.engines", "pipeline.common"], "Publish": true}, + {"Name": "pipeline.jsonbuilder", "Path": "pipeline/pipeline.jsonbuilder", "DependsOn": ["pipeline", "pipeline.engines.fiftyone", "pipeline.common", "pipeline.engines"], "Publish": true}, + {"Name": "pipeline.web.packages", "Path": "pipeline/pipeline.web.packages/pom.xml", "DependsOn": ["pipeline"], "Publish": true}, + {"Name": "pipeline.web.shared", "Path": "pipeline/pipeline.web.packages/pipeline.web.shared", "DependsOn": ["pipeline.web.packages", "pipeline.javascriptbuilder"], "Publish": true}, + {"Name": "pipeline.web", "Path": "pipeline/pipeline.web.packages/pipeline.web", "DependsOn": ["pipeline.web.packages", "pipeline.web.shared", "pipeline.core"], "Publish": true}, + {"Name": "pipeline.web.mvc", "Path": "pipeline/pipeline.web.packages/pipeline.web.mvc", "DependsOn": ["pipeline.web.packages", "pipeline.web", "pipeline.core"], "Publish": true}, + {"Name": "pipeline-aggregate", "Path": "pipeline/pipeline.aggregate", "DependsOn": ["pipeline", "pipeline.web", "pipeline.javascriptbuilder", "pipeline.cloudrequestengine"], "Publish": true}, + {"Name": "pipeline.developer-examples", "Path": "pipeline/pipeline.developer-examples/pom.xml", "DependsOn": ["pipeline"], "Publish": true}, + {"Name": "pipeline.developerexamples.web.shared", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.web.shared", "DependsOn": ["pipeline.developer-examples"], "Publish": true}, + {"Name": "pipeline.developerexamples.flowelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement", "DependsOn": ["pipeline.developer-examples", "pipeline.core"], "Publish": true}, + {"Name": "pipeline.developerexamples.onpremiseengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true}, + {"Name": "pipeline.developerexamples.cloudengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.cloudrequestengine"], "Publish": true}, + {"Name": "pipeline.developerexamples.clientsideelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true}, + {"Name": "pipeline.developerexamples.clientsideelement.mvc", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element-mvc", "DependsOn": ["pipeline.developer-examples", "pipeline.developerexamples.clientsideelement", "pipeline.web.mvc", "pipeline.developerexamples.web.shared"], "Publish": true}, + {"Name": "pipeline.developerexamples.usagesharing", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing", "DependsOn": ["pipeline.developer-examples", "pipeline.engines.fiftyone"], "Publish": true}, + {"Name": "pipeline.developerexamples.fodid", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.fodid", "DependsOn": ["pipeline.developer-examples", "pipeline.did"], "Publish": true}, + {"Name": "pipeline.device-detection", "Path": "device-detection/pom.xml", "DependsOn": ["pipeline-aggregate"], "Publish": true}, + {"Name": "device-detection.shared", "Path": "device-detection/device-detection.shared", "DependsOn": ["pipeline.device-detection", "pipeline.engines.fiftyone"], "Publish": true}, + {"Name": "device-detection.hash.engine.on-premise", "Path": "device-detection/device-detection.hash.engine.on-premise", "ExtraPaths": ["cxx"], "DependsOn": ["pipeline.device-detection", "device-detection.shared"], "Publish": true}, + {"Name": "device-detection.cloud", "Path": "device-detection/device-detection.cloud", "DependsOn": ["pipeline.device-detection", "device-detection.shared", "pipeline.cloudrequestengine"], "Publish": true}, + {"Name": "device-detection", "Path": "device-detection/device-detection", "DependsOn": ["pipeline.device-detection", "device-detection.hash.engine.on-premise", "device-detection.cloud", "pipeline-aggregate", "device-detection.shared"], "Publish": true}, + {"Name": "pipeline.ip-intelligence", "Path": "ip-intelligence/pom.xml", "DependsOn": ["pipeline-aggregate"], "Publish": true}, + {"Name": "ip-intelligence.shared", "Path": "ip-intelligence/ip-intelligence.shared", "DependsOn": ["pipeline.ip-intelligence", "pipeline.engines.fiftyone"], "Publish": true}, + {"Name": "ip-intelligence.translation", "Path": "ip-intelligence/ip-intelligence.translation", "ExtraPaths": ["cxx"], "DependsOn": ["pipeline.ip-intelligence", "ip-intelligence.shared", "pipeline.translation", "ip-intelligence.engine.on-premise"], "Publish": true}, + {"Name": "ip-intelligence.engine.on-premise", "Path": "ip-intelligence/ip-intelligence.engine.on-premise", "ExtraPaths": ["cxx"], "DependsOn": ["pipeline.ip-intelligence", "ip-intelligence.shared"], "Publish": true}, + {"Name": "ip-intelligence.cloud", "Path": "ip-intelligence/ip-intelligence.cloud", "DependsOn": ["pipeline.ip-intelligence", "ip-intelligence.shared", "pipeline.cloudrequestengine"], "Publish": true}, + {"Name": "ip-intelligence", "Path": "ip-intelligence/ip-intelligence", "DependsOn": ["pipeline.ip-intelligence", "ip-intelligence.engine.on-premise", "ip-intelligence.cloud", "pipeline-aggregate", "ip-intelligence.shared"], "Publish": true}, + {"Name": "pipeline.geo-location", "Path": "location/pom.xml", "DependsOn": [], "Publish": true}, + {"Name": "geo-location.core", "Path": "location/geo-location.core", "DependsOn": ["pipeline.geo-location", "pipeline.cloudrequestengine"], "Publish": true}, + {"Name": "geo-location.cloud", "Path": "location/geo-location.cloud", "DependsOn": ["pipeline.geo-location", "geo-location.core", "pipeline.cloudrequestengine", "pipeline.engines.fiftyone"], "Publish": true}, + {"Name": "geo-location", "Path": "location/geo-location", "DependsOn": ["pipeline.geo-location", "geo-location.cloud", "pipeline.cloudrequestengine", "pipeline.common"], "Publish": true}, + {"Name": "geo-location.examples", "Path": "location/geo-location.shell.examples", "DependsOn": ["pipeline.geo-location", "geo-location", "device-detection", "pipeline.engines"], "Publish": true}, + {"Name": "geo-location.web.examples", "Path": "location/geo-location.web.examples/pom.xml", "DependsOn": ["pipeline.geo-location"], "Publish": true}, + {"Name": "geo-location.web.examples.mvc", "Path": "location/geo-location.web.examples/geo-location.web.examples.mvc", "DependsOn": ["geo-location.web.examples", "pipeline.web.mvc", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true}, + {"Name": "geo-location.web.examples.servlet", "Path": "location/geo-location.web.examples/geo-location.web.examples.servlet", "DependsOn": ["geo-location.web.examples", "pipeline.web", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true}, + {"Name": "geo-location.web.examples.shared", "Path": "location/geo-location.web.examples/geo-location.web.examples.shared", "DependsOn": ["geo-location.web.examples"], "Publish": true} +] diff --git a/ci/packages.ps1 b/ci/packages.ps1 index e1319727e..62c7ff54f 100644 --- a/ci/packages.ps1 +++ b/ci/packages.ps1 @@ -2,10 +2,10 @@ # relative to the repository root. # The package trees in this repository. The root pom.xml builds them all in -# one reactor but is not their parent, so a version change has to be made in -# each of them. Publish says whether the tree's packages go to Maven Central; -# the examples are built and tested but never published. -$Packages = @( +# one reactor but is not their parent. Publish says whether the tree's +# packages go to Maven Central. The examples are built and tested but never +# published. Each published module is a package of its own in packages.json. +$Trees = @( @{ Path = 'pipeline'; Publish = $true } @{ Path = 'device-detection'; Publish = $true } @{ Path = 'ip-intelligence'; Publish = $true } @@ -40,3 +40,134 @@ $Accessors = @( # Arguments every Maven call in these scripts passes. $MavenOptions = '--batch-mode', '--no-transfer-progress' + +$PomNamespace = 'http://maven.apache.org/POM/4.0.0' + +# The packages in packages.json that are published, one per Maven module, +# each named by its artifactId. +function Get-PublishedPackages([string]$RepoPath) { + return @(Get-Content -Raw (Join-Path $RepoPath 'ci/packages.json') | + ConvertFrom-Json | Where-Object { $_.Publish -eq $true }) +} + +# A parent's Path is its pom.xml alone, because its folder also holds its +# modules, whose changes are not changes to the parent. +function Get-PomPath($Package) { + if ($Package.Path.EndsWith('/pom.xml') -eq $true) { + return $Package.Path + } + return "$($Package.Path)/pom.xml" +} + +# The folder Maven builds a package in. +function Get-ModulePath($Package) { + return (Get-PomPath $Package) -replace '/pom\.xml$', '' +} + +# A pom.xml read with its whitespace kept, so that writing it back changes +# only what was set. +function Read-Pom([string]$Path) { + # .NET resolves a relative path from the process's folder, which is not + # the folder PowerShell is in after a Set-Location. + $Path = (Resolve-Path -LiteralPath $Path).ProviderPath + $xml = [Xml.XmlDocument]::new() + $xml.PreserveWhitespace = $true + $xml.Load($Path) + $namespaces = [Xml.XmlNamespaceManager]::new($xml.NameTable) + $namespaces.AddNamespace('m', $PomNamespace) + return [pscustomobject]@{ Path = $Path; Xml = $xml; Ns = $namespaces } +} + +function Save-Pom($Pom) { + $settings = [Xml.XmlWriterSettings]::new() + $settings.Encoding = [Text.UTF8Encoding]::new($false) + $settings.NewLineHandling = [Xml.NewLineHandling]::None + $writer = [Xml.XmlWriter]::Create($Pom.Path, $settings) + try { + $Pom.Xml.Save($writer) + } finally { + $writer.Dispose() + } +} + +function Get-PomText($Node, [string]$Name, $Ns) { + $child = $Node.SelectSingleNode("m:$Name", $Ns) + if ($null -eq $child) { + return $null + } + return $child.InnerText.Trim() +} + +# The other packages a pom.xml refers to, being its parent and its +# dependencies on them in any scope, wherever they are declared. Version is the element that +# holds the version, or $null where a parent pom manages it. +function Get-PomReferences($Pom, [string[]]$Names) { + $self = Get-PomText $Pom.Xml.DocumentElement 'artifactId' $Pom.Ns + $parent = $Pom.Xml.SelectSingleNode('/m:project/m:parent', $Pom.Ns) + if ($null -ne $parent) { + $name = Get-PomText $parent 'artifactId' $Pom.Ns + if ($Names -contains $name) { + [pscustomobject]@{ + Name = $name + Version = $parent.SelectSingleNode('m:version', $Pom.Ns) + } + } + } + foreach ($dependency in $Pom.Xml.SelectNodes('//m:dependency', $Pom.Ns)) { + $name = Get-PomText $dependency 'artifactId' $Pom.Ns + $group = Get-PomText $dependency 'groupId' $Pom.Ns + if ($Names -contains $name -and $name -ne $self -and + $group -in 'com.51degrees', '${project.groupId}') { + [pscustomobject]@{ + Name = $name + Version = $dependency.SelectSingleNode('m:version', $Pom.Ns) + } + } + } +} + +# The modules of the published trees, as folders, read from each pom's +# modules. +function Get-PublishedModules([string]$RepoPath) { + $folders = [Collections.Generic.Queue[string]]::new() + foreach ($tree in $Trees | Where-Object { $_.Publish -eq $true }) { + $folders.Enqueue($tree.Path) + } + while ($folders.Count -gt 0) { + $folder = $folders.Dequeue() + $folder + $pom = Read-Pom (Join-Path $RepoPath $folder 'pom.xml') + foreach ($module in $pom.Xml.SelectNodes('/m:project/m:modules/m:module', $pom.Ns)) { + $folders.Enqueue("$folder/$($module.InnerText.Trim())") + } + } +} + +# What packages.json leaves out, as one line each. Every published module must +# be a package, named by its artifactId, and each package's DependsOn must name +# every other package its pom.xml refers to. A dependency it leaves out would +# not be released when the package it depends on is, so its users would not +# get the fix. +function Get-PackagesJsonGaps([string]$RepoPath) { + $packages = Get-PublishedPackages $RepoPath + $names = @($packages.Name) + $folders = @($packages | ForEach-Object { Get-ModulePath $_ }) + foreach ($folder in Get-PublishedModules $RepoPath) { + if ($folders -notcontains $folder) { + "$folder/pom.xml is a published module with no entry in ci/packages.json" + } + } + foreach ($package in $packages) { + $pom = Read-Pom (Join-Path $RepoPath (Get-PomPath $package)) + $artifactId = Get-PomText $pom.Xml.DocumentElement 'artifactId' $pom.Ns + if ($artifactId -ne $package.Name) { + "$(Get-PomPath $package) is $artifactId, but ci/packages.json names it $($package.Name)" + } + $missing = @(Get-PomReferences $pom $names | + Where-Object { $package.DependsOn -notcontains $_.Name } | + ForEach-Object Name | Select-Object -Unique) + foreach ($name in $missing) { + "$(Get-PomPath $package) refers to $name, which the DependsOn of $($package.Name) in ci/packages.json does not name" + } + } +} diff --git a/ci/publish-package.ps1 b/ci/publish-package.ps1 index f2ee093f3..321cc12df 100644 --- a/ci/publish-package.ps1 +++ b/ci/publish-package.ps1 @@ -1,12 +1,49 @@ param( [Parameter(Mandatory)][string]$RepoName, + [Parameter(Mandatory)][string]$OrgName, [Parameter(Mandatory)][string]$Version, [Parameter(Mandatory)][Hashtable]$Keys, [bool]$DryRun = $false ) +$ErrorActionPreference = "Stop" +$PSNativeCommandUseErrorActionPreference = $true -# Uploads the one publishing bundle build-package.ps1 made, which holds the -# packages of every tree that publishes. +. "$PSScriptRoot/release.ps1" + +# The plan is read before anything is tagged, because the tags made below +# change what the plan would say. +$plan = @(Get-ReleasePlan $RepoName) + +# Uploads the one publishing bundle build-package.ps1 made, which holds every +# package this publish releases. ./java/publish-package-maven.ps1 -RepoName $RepoName -MavenSettings $Keys['MavenSettings'] -Version $Version -DryRun $DryRun -exit $LASTEXITCODE +# One tag and one GitHub release per package, '/', in place +# of common-ci's single tag, which would name no package. The notes list the +# commits that touched the package's files since its last release. +foreach ($release in $plan) { + $tag = "$($release.Name)/$($release.Version)" + git -C $RepoName tag $tag + if ($DryRun -eq $true) { + Write-Output "Dry run - tagged '$tag' locally, not pushed, no release made" + continue + } + git -C $RepoName push origin $tag + + $notes = "$($release.Name) $($release.Version): $($release.Reason)." + if ($null -ne $release.Previous) { + $package = Get-PublishedPackages $RepoName | + Where-Object { $_.Name -eq $release.Name } + $range = "$($release.Name)/$($release.Previous)..HEAD" + $commits = git -C $RepoName log --format='- %s (%h)' $range ` + -- (Get-PackagePaths $package) + if ($commits) { + $notes += "`n`n" + ($commits -join "`n") + } + } + gh -R "$OrgName/$RepoName" release create $tag --verify-tag ` + --title "$($release.Name) $($release.Version)" --notes $notes +} + +# Tells common-ci's publish not to add its own single tag and release. +$global:SkipUpdateTag = $true diff --git a/ci/release.ps1 b/ci/release.ps1 new file mode 100644 index 000000000..13bfa8501 --- /dev/null +++ b/ci/release.ps1 @@ -0,0 +1,125 @@ +# Dot-sourced by the scripts that publish. Each package in packages.json has +# its own version, tagged '/'. A publish releases a package +# when a file under its folders changed since its last tag, and every package +# that depends on it, with the patch number moved. common-ci's +# steps/release-plan.ps1 works out the plan, and the functions here build it. +# +# Kept apart from packages.ps1, which every script loads, so that building and +# testing a pull request does not need the release plan. +. "$PSScriptRoot/packages.ps1" + +# common-ci runs these scripts from its own root, with this repository checked +# out inside it. A checkout of common-ci beside this repository is found too, +# or one elsewhere can be named with COMMON_CI. +$releasePlan = @( + $env:COMMON_CI, + (Join-Path $PSScriptRoot '..' '..'), + (Join-Path $PSScriptRoot '..' '..' 'common-ci')) | + Where-Object { $_ } | + ForEach-Object { Join-Path $_ 'steps' 'release-plan.ps1' } | + Where-Object { Test-Path $_ } | + Select-Object -First 1 +if ($null -eq $releasePlan) { + throw "common-ci's steps/release-plan.ps1 was not found. Set COMMON_CI " + + "to a checkout of common-ci." +} +. $releasePlan +Remove-Variable releasePlan + +# The examples' properties that name the version of a package they use, which +# the publish sets to the version it built. +$ExampleVersionProperties = @{ + 'device-detection.version' = 'device-detection' + 'ip-intelligence.version' = 'ip-intelligence' + 'pipeline.jsonbuilder.version' = 'pipeline.jsonbuilder' +} + +# The packages a publish builds. When nothing is to be released, every package +# is built at its released version, so that the tests have packages to install +# and the nightly publish passes without publishing anything. +function Get-BuildPlan([string]$RepoPath) { + $plan = @(Get-ReleasePlan $RepoPath) + if ($plan.Count -gt 0) { + return $plan + } + $published = Get-PublishedPackages $RepoPath + foreach ($name in (Sort-ByDependency $published @($published.Name))) { + $package = $published | Where-Object { $_.Name -eq $name } + $last = @(Get-ReleasedVersions $RepoPath $name)[-1] + [pscustomobject]@{ + Name = $name + Path = $package.Path + Version = "$last" + Previous = $last + Reason = 'already released, built for the tests' + } + } +} + +# The version of every published package in a build, which is its version in +# the plan, or its last release when the plan leaves it out. +function Get-BuildVersions([string]$RepoPath, $Plan = $null) { + if ($null -eq $Plan) { + $Plan = @(Get-BuildPlan $RepoPath) + } + $versions = @{} + foreach ($package in Get-PublishedPackages $RepoPath) { + $release = $Plan | Where-Object { $_.Name -eq $package.Name } + if ($null -ne $release) { + $versions[$package.Name] = $release.Version + continue + } + $released = @(Get-ReleasedVersions $RepoPath $package.Name) + if ($released.Count -eq 0) { + throw "$($package.Name) has never been released and is not in the plan." + } + $versions[$package.Name] = "$($released[-1])" + } + return $versions +} + +# Writes into every published pom.xml its own version, its parent's and those +# of the packages it depends on, so each module is built at its own version and +# its published pom requires the versions it was built with. The poms in the +# repository keep one development version for every module, which this +# replaces only in the checkout being built. +function Set-PomVersions([string]$RepoPath, [hashtable]$Versions) { + $gaps = @(Get-PackagesJsonGaps $RepoPath) + if ($gaps.Count -gt 0) { + throw "ci/packages.json does not match the poms:`n$($gaps -join "`n")" + } + $packages = Get-PublishedPackages $RepoPath + $names = @($packages.Name) + foreach ($package in $packages) { + $pom = Read-Pom (Join-Path $RepoPath (Get-PomPath $package)) + $project = $pom.Xml.DocumentElement + $version = $project.SelectSingleNode('m:version', $pom.Ns) + if ($null -eq $version) { + # The module took its version from its parent. It gets an element + # of its own after its artifactId, indented as the artifactId is. + $artifactId = $project.SelectSingleNode('m:artifactId', $pom.Ns) + $version = $pom.Xml.CreateElement('version', $PomNamespace) + $null = $project.InsertAfter($version, $artifactId) + $indent = $artifactId.PreviousSibling + if ($null -ne $indent -and + $indent.NodeType -eq [Xml.XmlNodeType]::Whitespace) { + $null = $project.InsertAfter($indent.Clone(), $artifactId) + } + } + $version.InnerText = $Versions[$package.Name] + foreach ($reference in Get-PomReferences $pom $names) { + if ($null -ne $reference.Version) { + $reference.Version.InnerText = $Versions[$reference.Name] + } + } + Save-Pom $pom + } +} + +# The arguments that point the examples at the versions a publish built. +function Get-ExampleVersionArguments([string]$RepoPath) { + $versions = Get-BuildVersions $RepoPath + foreach ($property in $ExampleVersionProperties.Keys | Sort-Object) { + "-D$property=$($versions[$ExampleVersionProperties[$property]])" + } +} diff --git a/ci/run-integration-tests.ps1 b/ci/run-integration-tests.ps1 index ae705f41a..00b1f0e39 100644 --- a/ci/run-integration-tests.ps1 +++ b/ci/run-integration-tests.ps1 @@ -1,8 +1,9 @@ param( [Parameter(Mandatory)][string]$RepoName, [string]$Name, - # Set by the publish workflow to the version it built and installed. Not - # set when a pull request is tested, which builds the reactor itself. + # Set by the publish workflow, which built and installed the packages, to + # its label for the run. Not set when a pull request is tested, which + # builds the reactor itself. [string]$Version, [string]$TestResourceKey, [string]$DeviceDetection, @@ -20,13 +21,15 @@ $results = "$RepoName/test-results/integration" $failed = @() # The examples take the libraries from Maven Central unless told otherwise. -# Pointing them at the version the publish workflow installed tests the -# packages as a project of your own would use them. For a pull request the -# dev profile points them at the reactor build-project.ps1 installed. -[string[]]$versionArgs = if ($Version) { - "-Ddevice-detection.version=$Version", "-Dip-intelligence.version=$Version" +# Pointing them at the versions the publish workflow built and installed, each +# package's own, tests the packages as a project of your own would use them. +# For a pull request the dev profile points them at the reactor +# build-project.ps1 installed. +if ($Version) { + . "$PSScriptRoot/release.ps1" + [string[]]$versionArgs = Get-ExampleVersionArguments $RepoName } else { - , '-Pdev' + [string[]]$versionArgs = '-Pdev' } # The Pipeline API developer examples, as the pipeline repository ran them. @@ -55,7 +58,7 @@ if ($Version) { Write-Host "::warning title=Examples skipped::Skipping '$($example.Path)', as IpIntelligenceUrl was not provided." continue } - Write-Host "::group::$($example.Path) against $Version" + Write-Host "::group::$($example.Path) against $($versionArgs -join ' ')" Push-Location "$RepoName/$($example.Path)" try { mvn clean test @MavenOptions @versionArgs ` diff --git a/ci/run-performance-tests.ps1 b/ci/run-performance-tests.ps1 index 9b24f7024..a9d57bca5 100644 --- a/ci/run-performance-tests.ps1 +++ b/ci/run-performance-tests.ps1 @@ -1,8 +1,9 @@ param( [Parameter(Mandatory)][string]$RepoName, [Parameter(Mandatory)][string]$Name, - # Set by the publish workflow to the version it built and installed. Not - # set when a pull request is tested, which builds the reactor itself. + # Set by the publish workflow, which built and installed the packages, to + # its label for the run. Not set when a pull request is tested, which + # builds the reactor itself. [string]$Version, [string]$IpIntelligenceUrl ) @@ -21,10 +22,12 @@ $summary = @{ HigherIsBetter = [ordered]@{}; LowerIsBetter = [ordered]@{} } $results = "$RepoName/test-results/performance" $failed = @() -[string[]]$versionArgs = if ($Version) { - "-Ddevice-detection.version=$Version", "-Dip-intelligence.version=$Version" +# A publish measures the versions it built, each package's own. +if ($Version) { + . "$PSScriptRoot/release.ps1" + [string[]]$versionArgs = Get-ExampleVersionArguments $RepoName } else { - , '-Pdev' + [string[]]$versionArgs = '-Pdev' } # Pipeline API. The test writes its figures as JSON itself. diff --git a/ci/update-packages.ps1 b/ci/update-packages.ps1 index adec719d1..5bc798f62 100644 --- a/ci/update-packages.ps1 +++ b/ci/update-packages.ps1 @@ -13,7 +13,7 @@ $PSNativeCommandUseErrorActionPreference = $true # pipeline.version is left alone. The libraries set it to ${project.version} # so that they build against the Pipeline API in this repository, and it must # not be replaced by the latest published version. -foreach ($package in $Packages) { +foreach ($package in $Trees) { $allowSnapshots = (-not $package.Publish).ToString().ToLower() Write-Host "Updating dependencies of '$($package.Path)'. Patch version only. Snapshots allowed: $allowSnapshots." Push-Location "$RepoName/$($package.Path)" diff --git a/pom.xml b/pom.xml index 567875602..f638fd9df 100644 --- a/pom.xml +++ b/pom.xml @@ -9,13 +9,13 @@ nothing here reaches Maven Central and the published poms are the same shape as before the packages shared a repository. - Every module carries the same version. CI sets it on all of them - before a release with versions:set and processAllModules. The + Every module carries the same development version, and the libraries reference each other through ${project.version}, so a - change in pipeline-java is seen by device-detection-java in the same - build. The examples reference the packages published to Maven Central - unless the dev profile is active, which points them at the libraries - here instead. --> + change to the Pipeline API is seen by Device Detection in the same + build. A release gives each published module its own version, which + ci/build-package.ps1 writes into the poms. The examples reference the + packages published to Maven Central unless the dev profile is + active, which points them at the libraries here instead. --> com.51degrees 51degrees-java 4.5.127-SNAPSHOT From 70901e7ffbe7f4b4282070fbe507bbfcecbe4e7d Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Sat, 7 Nov 2026 17:16:02 +0000 Subject: [PATCH 2/3] Release nothing for a pom.xml change only to its metadata common-ci's release plan calls ci/release-plan-filter.ps1 with the files under a package's folders that changed since its last tag, once its Ignore globs and the default globs for documentation, tests and examples have left theirs out. The filter leaves out a pom.xml whose only change is to its name, description, url, licenses, developers, scm or issueManagement, or to its comments and layout, so a new link does not release the package. A version, dependency, plugin or build change still counts, as does every other file. Its Pester tests are ci/release-plan-filter.Tests.ps1. --- README.md | 2 + ci/README.md | 9 +- ci/release-plan-filter.Tests.ps1 | 161 +++++++++++++++++++++++++++++++ ci/release-plan-filter.ps1 | 70 ++++++++++++++ 4 files changed, 241 insertions(+), 1 deletion(-) create mode 100644 ci/release-plan-filter.Tests.ps1 create mode 100644 ci/release-plan-filter.ps1 diff --git a/README.md b/README.md index 125c2c612..9cc6185a6 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,8 @@ does the same. Each package has its own version, and a release publishes only the packages that changed since their last release and the packages that depend on them. +A change only to documentation, tests, examples or the descriptive parts of a +`pom.xml`, such as its links, releases nothing. Each is tagged with its name and version, such as `pipeline.core/4.5.131`. [ci/README.md](ci/README.md) describes how CI works out what to release. diff --git a/ci/README.md b/ci/README.md index e6e1dc7ee..a5cfc6e11 100644 --- a/ci/README.md +++ b/ci/README.md @@ -42,7 +42,14 @@ repositories used to do for one package tree each. - Each package has its own version, tagged `/`, such as `pipeline.core/4.5.131`. A publish releases a package when a file under its folders changed since its last tag, and every package that depends on it, - each at its last tag with the patch number moved. + each at its last tag with the patch number moved. Changes only to Markdown, + `LICENSE` files, or files under folders named `docs`, `tests`, `test` or + `examples`, which takes in each module's `src/test`, release nothing, and + [release-plan-filter.ps1](release-plan-filter.ps1) leaves out a `pom.xml` + whose only change is to its `name`, `description`, `url`, `licenses`, + `developers`, `scm` or `issueManagement`, or to its comments and layout. + Its Pester 5 tests are + [release-plan-filter.Tests.ps1](release-plan-filter.Tests.ps1). [release.ps1](release.ps1) takes that plan from common-ci's `steps/release-plan.ps1`, and the publish scripts build, check and tag from it. [check-version-bump.ps1](check-version-bump.ps1) and diff --git a/ci/release-plan-filter.Tests.ps1 b/ci/release-plan-filter.Tests.ps1 new file mode 100644 index 000000000..c3f5dd8c4 --- /dev/null +++ b/ci/release-plan-filter.Tests.ps1 @@ -0,0 +1,161 @@ +# Pester 5 tests of release-plan-filter.ps1. Each test makes a repository +# whose pom.xml is tagged as released, changes it in a second commit, and asks +# the filter whether the change counts. + +BeforeAll { + $Filter = Join-Path $PSScriptRoot 'release-plan-filter.ps1' + + $Released = @' + + + 4.0.0 + com.51degrees + pipeline.core + 4.5.130 + 51Degrees :: Pipeline :: Core + Pipeline core + https://51degrees.com + + EUPL-1.2 + + + 51Degrees + + + https://github.com/51Degrees/pipeline-java + + + https://github.com/51Degrees/pipeline-java/issues + + + + central + https://repo1.maven.org/maven2 + + + + + org.json + json + 20240303 + + + + + + maven-jar-plugin + 3.3.0 + + + + +'@ + + # A repository holding $Released at the tag 'pkg/4.5.130', then the given + # files in a second commit. Returns the repository's folder. + function New-ChangedRepo([hashtable]$Files, [string[]]$Remove = @()) { + $repo = Join-Path $TestDrive ([Guid]::NewGuid().ToString('N')) + $null = New-Item -ItemType Directory -Path (Join-Path $repo 'pkg') + Set-Content -LiteralPath (Join-Path $repo 'pkg/pom.xml') -Value $Released -NoNewline + Set-Content -LiteralPath (Join-Path $repo 'pkg/Core.java') -Value 'class Core {}' -NoNewline + git -C $repo init -q + git -C $repo config user.email 'test@example.com' + git -C $repo config user.name 'Test' + git -C $repo config core.autocrlf false + git -C $repo add -A + git -C $repo commit -q -m 'released' + git -C $repo tag 'pkg/4.5.130' + foreach ($path in $Files.Keys) { + $full = Join-Path $repo $path + $null = New-Item -ItemType Directory -Force -Path (Split-Path $full) + Set-Content -LiteralPath $full -Value $Files[$path] -NoNewline + } + foreach ($path in $Remove) { + git -C $repo rm -q $path + } + git -C $repo add -A + git -C $repo commit -q --allow-empty -m 'changed' + return $repo + } + + # The released pom with one piece of text replaced, which must be there. + function Edit-Released([string]$Old, [string]$New) { + if ($Released.Contains($Old) -eq $false) { + throw "The released pom has no '$Old'." + } + return $Released.Replace($Old, $New) + } + + function Invoke-Filter([string]$Repo, [string[]]$Files) { + return @(& $Filter -RepoPath $Repo -Package 'pkg' -From 'pkg/4.5.130' -Files $Files) + } +} + +Describe 'release-plan-filter.ps1' { + It 'passes a file that is not a pom.xml through' { + $repo = New-ChangedRepo @{ 'pkg/Core.java' = 'class Core { int a; }' } + Invoke-Filter $repo @('pkg/Core.java') | Should -Be @('pkg/Core.java') + } + + It 'leaves out a pom.xml whose changed' -ForEach @( + @{ Metadata = 'name'; Old = 'Pipeline :: Core'; New = 'Pipeline :: Core API' } + @{ Metadata = 'description'; Old = 'Pipeline core'; New = 'The Pipeline core' } + @{ Metadata = 'url'; Old = 'https://51degrees.com'; New = 'https://51degrees.com/documentation' } + @{ Metadata = 'licenses'; Old = 'EUPL-1.2'; New = 'EUPL 1.2' } + @{ Metadata = 'developers'; Old = '51Degrees'; New = '51Degrees Mobile Experts' } + @{ Metadata = 'scm'; Old = 'pipeline-java'; New = '51degrees-java' } + @{ Metadata = 'issueManagement'; Old = 'pipeline-java/issues'; New = '51degrees-java/issues' } + ) { + $repo = New-ChangedRepo @{ 'pkg/pom.xml' = (Edit-Released $Old $New) } + Invoke-Filter $repo @('pkg/pom.xml') | Should -BeNullOrEmpty + } + + It 'leaves out a pom.xml whose only change is a comment or its layout' { + $pom = Edit-Released '4.0.0' ` + "`n 4.0.0`n" + $repo = New-ChangedRepo @{ 'pkg/pom.xml' = $pom } + Invoke-Filter $repo @('pkg/pom.xml') | Should -BeNullOrEmpty + } + + It 'keeps a pom.xml whose changed' -ForEach @( + @{ Change = 'version'; Old = '4.5.130'; New = '4.5.131' } + @{ Change = 'dependency version'; Old = '20240303'; New = '20250107' } + @{ Change = 'dependencies'; Old = ''; New = 'org.slf4jslf4j-api' } + @{ Change = 'plugin'; Old = '3.3.0'; New = '3.4.1' } + @{ Change = 'build'; Old = ''; New = 'core' } + @{ Change = 'repository url'; Old = 'https://repo1.maven.org/maven2'; New = 'https://repo.maven.apache.org/maven2' } + ) { + $repo = New-ChangedRepo @{ 'pkg/pom.xml' = (Edit-Released $Old $New) } + Invoke-Filter $repo @('pkg/pom.xml') | Should -Be @('pkg/pom.xml') + } + + It 'keeps a pom.xml whose metadata and dependencies both changed' { + $pom = (Edit-Released 'Pipeline core' 'The Pipeline core'). + Replace('20240303', '20250107') + $repo = New-ChangedRepo @{ 'pkg/pom.xml' = $pom } + Invoke-Filter $repo @('pkg/pom.xml') | Should -Be @('pkg/pom.xml') + } + + It 'keeps a pom.xml that is new since the release' { + $repo = New-ChangedRepo @{ 'pkg/sub/pom.xml' = $Released } + Invoke-Filter $repo @('pkg/sub/pom.xml') | Should -Be @('pkg/sub/pom.xml') + } + + It 'keeps a pom.xml that was removed since the release' { + $repo = New-ChangedRepo @{} -Remove @('pkg/pom.xml') + Invoke-Filter $repo @('pkg/pom.xml') | Should -Be @('pkg/pom.xml') + } + + It 'judges each file on its own' { + $repo = New-ChangedRepo @{ + 'pkg/pom.xml' = (Edit-Released 'Pipeline core' 'The Pipeline core') + 'pkg/Core.java' = 'class Core { int a; }' + } + Invoke-Filter $repo @('pkg/pom.xml', 'pkg/Core.java') | Should -Be @('pkg/Core.java') + } + + It 'gives nothing back for no files' { + $repo = New-ChangedRepo @{} + Invoke-Filter $repo @() | Should -BeNullOrEmpty + } +} diff --git a/ci/release-plan-filter.ps1 b/ci/release-plan-filter.ps1 new file mode 100644 index 000000000..7561498f1 --- /dev/null +++ b/ci/release-plan-filter.ps1 @@ -0,0 +1,70 @@ +# Called by common-ci's steps/release-plan.ps1 with the files under a +# package's folders that changed since its last tag, once its Ignore globs are +# applied. Writes out the files whose change releases the package. +# +# A pom.xml counts only when something other than the project's descriptive +# metadata changed, so a new link, description or developer does not release +# the package, while a version, dependency, plugin or build change does. +# Comments and layout do not count either. Every other file counts. +param( + [Parameter(Mandatory)][string]$RepoPath, + # The package the files belong to. Not needed here, because a pom.xml is + # judged the same way whichever package it belongs to. + $Package, + # The revision the package was last released from, normally its tag. + [Parameter(Mandatory)][string]$From, + [string[]]$Files = @() +) +$ErrorActionPreference = 'Stop' +# 'git show' fails for a file that is not in a revision, which is an answer +# here rather than an error. +$PSNativeCommandUseErrorActionPreference = $false + +# The children of that describe the project rather than build it. +$MetadataElements = @( + 'name', 'description', 'url', 'licenses', 'developers', 'scm', + 'issueManagement') + +# The pom.xml at a revision with its comments, its layout and its metadata +# taken out, or $null when the revision has no such file or it is not XML. +function Get-PomBuildText([string]$Revision, [string]$File) { + # git's message for a missing file is not an error of this script. + $ErrorActionPreference = 'Continue' + $text = & git -C $RepoPath show "${Revision}:$File" 2>$null + $found = $LASTEXITCODE -eq 0 + $global:LASTEXITCODE = 0 + if ($found -eq $false) { + return $null + } + $xml = [Xml.XmlDocument]::new() + $xml.PreserveWhitespace = $false + try { + $xml.LoadXml(($text -join "`n")) + } catch { + return $null + } + foreach ($comment in @($xml.SelectNodes('//comment()'))) { + $null = $comment.ParentNode.RemoveChild($comment) + } + $project = $xml.DocumentElement + foreach ($child in @($project.ChildNodes)) { + if ($child.NodeType -eq [Xml.XmlNodeType]::Element -and + $MetadataElements -contains $child.LocalName) { + $null = $project.RemoveChild($child) + } + } + return $project.OuterXml +} + +foreach ($file in $Files) { + if (($file -replace '\\', '/') -notmatch '(^|/)pom\.xml$') { + $file + continue + } + $before = Get-PomBuildText $From $file + $after = Get-PomBuildText 'HEAD' $file + # A pom.xml that is new, removed or unreadable on either side counts. + if ($null -eq $before -or $null -eq $after -or $before -ne $after) { + $file + } +} From 8e0c2a3cc9583acb36d898939bfed7b8b8186e4c Mon Sep 17 00:00:00 2001 From: James Rosewell Date: Mon, 9 Nov 2026 19:44:21 +0000 Subject: [PATCH 3/3] Release nothing for a change to an example module The example modules inside the published trees are packages of their own, but only the files under a folder named examples were ignored, so a change to the pom.xml, the web resources or the launcher of one released that module and every module depending on it. Each of the fourteen now ignores its whole folder. A release of a package an example depends on still releases the example, as before. --- ci/packages.json | 28 ++++++++++++++-------------- ci/packages.ps1 | 2 ++ 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/ci/packages.json b/ci/packages.json index 427227219..153c48995 100644 --- a/ci/packages.json +++ b/ci/packages.json @@ -15,15 +15,15 @@ {"Name": "pipeline.web", "Path": "pipeline/pipeline.web.packages/pipeline.web", "DependsOn": ["pipeline.web.packages", "pipeline.web.shared", "pipeline.core"], "Publish": true}, {"Name": "pipeline.web.mvc", "Path": "pipeline/pipeline.web.packages/pipeline.web.mvc", "DependsOn": ["pipeline.web.packages", "pipeline.web", "pipeline.core"], "Publish": true}, {"Name": "pipeline-aggregate", "Path": "pipeline/pipeline.aggregate", "DependsOn": ["pipeline", "pipeline.web", "pipeline.javascriptbuilder", "pipeline.cloudrequestengine"], "Publish": true}, - {"Name": "pipeline.developer-examples", "Path": "pipeline/pipeline.developer-examples/pom.xml", "DependsOn": ["pipeline"], "Publish": true}, - {"Name": "pipeline.developerexamples.web.shared", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.web.shared", "DependsOn": ["pipeline.developer-examples"], "Publish": true}, - {"Name": "pipeline.developerexamples.flowelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement", "DependsOn": ["pipeline.developer-examples", "pipeline.core"], "Publish": true}, - {"Name": "pipeline.developerexamples.onpremiseengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true}, - {"Name": "pipeline.developerexamples.cloudengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.cloudrequestengine"], "Publish": true}, - {"Name": "pipeline.developerexamples.clientsideelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true}, - {"Name": "pipeline.developerexamples.clientsideelement.mvc", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element-mvc", "DependsOn": ["pipeline.developer-examples", "pipeline.developerexamples.clientsideelement", "pipeline.web.mvc", "pipeline.developerexamples.web.shared"], "Publish": true}, - {"Name": "pipeline.developerexamples.usagesharing", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing", "DependsOn": ["pipeline.developer-examples", "pipeline.engines.fiftyone"], "Publish": true}, - {"Name": "pipeline.developerexamples.fodid", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.fodid", "DependsOn": ["pipeline.developer-examples", "pipeline.did"], "Publish": true}, + {"Name": "pipeline.developer-examples", "Path": "pipeline/pipeline.developer-examples/pom.xml", "DependsOn": ["pipeline"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.web.shared", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.web.shared", "DependsOn": ["pipeline.developer-examples"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.flowelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement", "DependsOn": ["pipeline.developer-examples", "pipeline.core"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.onpremiseengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.cloudengine", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine", "DependsOn": ["pipeline.developer-examples", "pipeline.cloudrequestengine"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.clientsideelement", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element", "DependsOn": ["pipeline.developer-examples", "pipeline.engines"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.clientsideelement.mvc", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.clientside-element-mvc", "DependsOn": ["pipeline.developer-examples", "pipeline.developerexamples.clientsideelement", "pipeline.web.mvc", "pipeline.developerexamples.web.shared"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.usagesharing", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing", "DependsOn": ["pipeline.developer-examples", "pipeline.engines.fiftyone"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, + {"Name": "pipeline.developerexamples.fodid", "Path": "pipeline/pipeline.developer-examples/pipeline.developer-examples.fodid", "DependsOn": ["pipeline.developer-examples", "pipeline.did"], "Publish": true, "Ignore": ["pipeline/pipeline.developer-examples/**"]}, {"Name": "pipeline.device-detection", "Path": "device-detection/pom.xml", "DependsOn": ["pipeline-aggregate"], "Publish": true}, {"Name": "device-detection.shared", "Path": "device-detection/device-detection.shared", "DependsOn": ["pipeline.device-detection", "pipeline.engines.fiftyone"], "Publish": true}, {"Name": "device-detection.hash.engine.on-premise", "Path": "device-detection/device-detection.hash.engine.on-premise", "ExtraPaths": ["cxx"], "DependsOn": ["pipeline.device-detection", "device-detection.shared"], "Publish": true}, @@ -39,9 +39,9 @@ {"Name": "geo-location.core", "Path": "location/geo-location.core", "DependsOn": ["pipeline.geo-location", "pipeline.cloudrequestengine"], "Publish": true}, {"Name": "geo-location.cloud", "Path": "location/geo-location.cloud", "DependsOn": ["pipeline.geo-location", "geo-location.core", "pipeline.cloudrequestengine", "pipeline.engines.fiftyone"], "Publish": true}, {"Name": "geo-location", "Path": "location/geo-location", "DependsOn": ["pipeline.geo-location", "geo-location.cloud", "pipeline.cloudrequestengine", "pipeline.common"], "Publish": true}, - {"Name": "geo-location.examples", "Path": "location/geo-location.shell.examples", "DependsOn": ["pipeline.geo-location", "geo-location", "device-detection", "pipeline.engines"], "Publish": true}, - {"Name": "geo-location.web.examples", "Path": "location/geo-location.web.examples/pom.xml", "DependsOn": ["pipeline.geo-location"], "Publish": true}, - {"Name": "geo-location.web.examples.mvc", "Path": "location/geo-location.web.examples/geo-location.web.examples.mvc", "DependsOn": ["geo-location.web.examples", "pipeline.web.mvc", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true}, - {"Name": "geo-location.web.examples.servlet", "Path": "location/geo-location.web.examples/geo-location.web.examples.servlet", "DependsOn": ["geo-location.web.examples", "pipeline.web", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true}, - {"Name": "geo-location.web.examples.shared", "Path": "location/geo-location.web.examples/geo-location.web.examples.shared", "DependsOn": ["geo-location.web.examples"], "Publish": true} + {"Name": "geo-location.examples", "Path": "location/geo-location.shell.examples", "DependsOn": ["pipeline.geo-location", "geo-location", "device-detection", "pipeline.engines"], "Publish": true, "Ignore": ["location/geo-location.shell.examples/**"]}, + {"Name": "geo-location.web.examples", "Path": "location/geo-location.web.examples/pom.xml", "DependsOn": ["pipeline.geo-location"], "Publish": true, "Ignore": ["location/geo-location.web.examples/**"]}, + {"Name": "geo-location.web.examples.mvc", "Path": "location/geo-location.web.examples/geo-location.web.examples.mvc", "DependsOn": ["geo-location.web.examples", "pipeline.web.mvc", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true, "Ignore": ["location/geo-location.web.examples/**"]}, + {"Name": "geo-location.web.examples.servlet", "Path": "location/geo-location.web.examples/geo-location.web.examples.servlet", "DependsOn": ["geo-location.web.examples", "pipeline.web", "geo-location.cloud", "geo-location.web.examples.shared"], "Publish": true, "Ignore": ["location/geo-location.web.examples/**"]}, + {"Name": "geo-location.web.examples.shared", "Path": "location/geo-location.web.examples/geo-location.web.examples.shared", "DependsOn": ["geo-location.web.examples"], "Publish": true, "Ignore": ["location/geo-location.web.examples/**"]} ] diff --git a/ci/packages.ps1 b/ci/packages.ps1 index 62c7ff54f..e3e153f2d 100644 --- a/ci/packages.ps1 +++ b/ci/packages.ps1 @@ -5,6 +5,8 @@ # one reactor but is not their parent. Publish says whether the tree's # packages go to Maven Central. The examples are built and tested but never # published. Each published module is a package of its own in packages.json. +# The example modules inside the published trees are packages too, and give +# an Ignore glob over their whole folder, so a change to one releases nothing. $Trees = @( @{ Path = 'pipeline'; Publish = $true } @{ Path = 'device-detection'; Publish = $true }