Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions .github/instructions/gradle.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ All `src/*` Gradle projects share two repo config files: **`eng/gradle/plugin-re
pluginManagement {
apply from: "${rootDir}/../../eng/gradle/plugin-repositories.gradle", to: pluginManagement
}
if (System.getenv('ANDROID_MIRROR_MAVEN_DEPENDENCIES') == 'true') {
apply from: "${rootDir}/../../eng/gradle/credential-provider.gradle"
}
dependencyResolutionManagement {
apply from: "${rootDir}/../../eng/gradle/dependency-repositories.gradle", to: dependencyResolutionManagement
}
Expand All @@ -31,17 +28,17 @@ Both files switch on `System.getenv('RUNNINGONCI')`. Azure DevOps exports the
- **`RUNNINGONCI=true`** (Azure DevOps, sourced from `RunningOnCI` in `build-tools/automation/yaml-templates/variables.yaml`) → dnceng `dotnet-public-maven` feed (CFSClean isolation, https://aka.ms/1es/netiso/CFS). Anonymous read of cached packages.
- **unset** (local, Dependabot, GitHub Actions) → `google()` + `mavenCentral()` + `gradlePluginPortal()` for plugins, `google()` + `mavenCentral()` for deps. No credentials needed.

CI reads cached packages from the mirror anonymously. The Azure Artifacts
credential provider is loaded only when `ANDROID_MIRROR_MAVEN_DEPENDENCIES=true`;
`mirror-dependencies.ps1` sets this while seeding uncached packages.
CI reads cached packages from the mirror anonymously. `mirror-dependencies.ps1`
runs the same anonymous Gradle resolution, then seeds each missing URL with an
authenticated HTTP request until the build succeeds.

Test the CI path locally: `$env:RUNNINGONCI='true'` (PowerShell) or `RUNNINGONCI=true ...` (bash).

## When CI fails 401 on a Dependabot bump

The new package isn't cached in the dnceng `dotnet-public-maven` feed yet. CI agents only do anonymous reads, so someone has to authenticate once locally to make the feed pull the package (and its transitive deps) from upstream.

Use the helper script — it runs the build, parses any 401 URLs out of the log, re-fetches each one with an Azure DevOps bearer token (so the feed mirrors it), and loops until the build succeeds:
Use the helper script — it runs the build, parses any 401 URLs out of the log, re-fetches each one with an Azure DevOps OAuth token using Basic authentication (so the feed mirrors it), and loops until the build succeeds:

```powershell
az login # one-time, corp account with MFA satisfied
Expand All @@ -57,6 +54,16 @@ The mirror must run in the project that actually needs the new package — a sib

After it succeeds, just re-run the failed CI job. No PR edits needed — the packages are now anonymous-readable forever.

Tests that resolve Maven files without Gradle can seed coordinates directly:

```powershell
pwsh ./eng/gradle/mirror-dependencies.ps1 `
-MavenArtifact 'androidx.core:core:1.12.0'
```

This attempts the coordinate's POM, JAR, AAR, and Gradle module metadata. Append
the exact filename as a fourth segment for a nonstandard payload.

## Don'ts

- Don't hard-code Maven repo URLs in `build.gradle` / `settings.gradle`; use the shared file.
Expand Down
15 changes: 7 additions & 8 deletions build-tools/scripts/Prepare.proj
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,14 @@
Text="The specified `%24(AndroidToolchainDirectory)` '$(AndroidToolchainDirectory)' contains a space. Android NDK commands do not support this. Please create a Configuration.Override.props file that sets the AndroidToolchainDirectory property to a different path."
Condition=" '$(HostOS)' == 'Windows' And $(AndroidToolchainDirectory.Contains (' '))"
/>
<MSBuild
Projects="$(_TopDir)\build-tools\Xamarin.Android.Tools.BootstrapTasks\Xamarin.Android.Tools.BootstrapTasks.csproj"
Targets="Restore;Build"
Properties="Configuration=$(Configuration)"
<!-- Build out of process so NuGet-generated props are evaluated after restore. -->
<Exec
Command="&quot;$(DotNetPreviewTool)&quot; build &quot;$(_TopDir)\build-tools\Xamarin.Android.Tools.BootstrapTasks\Xamarin.Android.Tools.BootstrapTasks.csproj&quot; -p:Configuration=$(Configuration) -bl:$(_BinlogPathPrefix)-bootstrap-tasks.binlog"
WorkingDirectory="$(_TopDir)"
/>
<MSBuild
Projects="$(_TopDir)\src\workloads\workloads.csproj"
Targets="Restore;Build"
Properties="Configuration=$(Configuration)"
<Exec
Command="&quot;$(DotNetPreviewTool)&quot; build &quot;$(_TopDir)\src\workloads\workloads.csproj&quot; -p:Configuration=$(Configuration) -bl:$(_BinlogPathPrefix)-workloads.binlog"
WorkingDirectory="$(_TopDir)"
/>
<CallTarget Targets="PrepareJavaInterop" />
<MSBuild
Expand Down
17 changes: 0 additions & 17 deletions eng/gradle/credential-provider.gradle

This file was deleted.

94 changes: 77 additions & 17 deletions eng/gradle/mirror-dependencies.ps1
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env pwsh
<#
.SYNOPSIS
Mirrors a gradle project's dependencies into the dnceng dotnet-public-maven
Azure Artifacts feed so CI can resolve them anonymously.
Mirrors Gradle dependencies or explicit Maven artifacts into the dnceng
dotnet-public-maven Azure Artifacts feed so CI can resolve them anonymously.

.DESCRIPTION
When Dependabot bumps a gradle dependency (or its transitive graph changes),
Expand All @@ -12,10 +12,11 @@

This script does that by running the requested gradle build in a loop:
1. Run gradle with RUNNINGONCI=true so it points at the dnceng feed.
2. Parse any 'Could not GET' URLs out of the build log.
3. Re-fetch each failing URL with an Azure DevOps OAuth bearer token
(obtained via `az account get-access-token`). The feed's upstream
connector then pulls the package and caches it for anonymous reads.
2. Parse any 'Could not GET/HEAD' URLs out of the build log.
3. Re-fetch each failing URL with an Azure DevOps OAuth token using Basic
authentication (obtained via `az account get-access-token`). The
feed's upstream connector then pulls the package and caches it for
anonymous reads.
4. Repeat until the build succeeds or no more 401s appear.

After the loop converges, no PR edits are needed — just re-run the failing
Expand All @@ -30,6 +31,12 @@
Gradle task(s) to run. Should be one that resolves the new dependency
graph (e.g. 'assembleDebug', 'build', 'extractProguardFiles').

.PARAMETER MavenArtifact
Maven coordinates to mirror directly, for tests that do not use Gradle.
Each value is group:artifact:version, which attempts the POM, JAR, AAR, and
Gradle module metadata files. Append an exact filename as a fourth segment
when a test requests a nonstandard payload.

.PARAMETER GradleWrapper
Optional path to the Gradle wrapper used by CI for this project, relative
to the repository root or absolute. Defaults to build-tools/gradle/gradlew.
Expand Down Expand Up @@ -59,19 +66,30 @@
-ProjectDir external/Java.Interop/tests/Xamarin.Android.Tools.Bytecode-Tests/kotlin-gradle `
-Task classes `
-GradleWrapper external/Java.Interop/build-tools/gradle/gradlew.bat

.EXAMPLE
pwsh ./eng/gradle/mirror-dependencies.ps1 `
-MavenArtifact 'androidx.core:core:1.12.0', `
'com.facebook.react:react-android:0.76.1:react-android-0.76.1.module'
#>
[CmdletBinding()]
[CmdletBinding(DefaultParameterSetName='Gradle')]
param(
[Parameter(Mandatory=$true)]
[string] $ProjectDir,
[Parameter(Mandatory=$true, ParameterSetName='Gradle')]
[string] $ProjectDir = '.',

[Parameter(Mandatory=$true)]
[Parameter(Mandatory=$true, ParameterSetName='Gradle')]
[string] $Task,

[Parameter(Mandatory=$true, ParameterSetName='MavenArtifact')]
[string[]] $MavenArtifact,

[Parameter(ParameterSetName='Gradle')]
[string] $GradleWrapper,

[Parameter(ParameterSetName='Gradle')]
[string] $AndroidHome = $env:ANDROID_HOME,

[Parameter(ParameterSetName='Gradle')]
[int] $MaxIterations = 15
)

Expand Down Expand Up @@ -104,13 +122,14 @@ function Get-AzDevOpsToken {
}

function Invoke-Mirror($logPath) {
$urls = Select-String -Path $logPath -Pattern "Could not GET 'https://pkgs\.dev\.azure\.com/dnceng/[^']+'" -AllMatches |
$urls = Select-String -Path $logPath -Pattern "Could not (?:GET|HEAD) '(https://pkgs\.dev\.azure\.com/dnceng/[^']+)'" -AllMatches |
ForEach-Object { $_.Matches } |
ForEach-Object { $_.Value -replace "^Could not GET '", "" -replace "'$", "" } |
ForEach-Object { $_.Groups[1].Value } |
Sort-Object -Unique
if ($urls.Count -eq 0) { return 0 }
$token = Get-AzDevOpsToken
$headers = @{ Authorization = "Bearer $token" }
$basicCredential = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$token"))
$headers = @{ Authorization = "Basic $basicCredential" }
$ok = 0; $fail = 0
foreach ($u in $urls) {
try {
Expand All @@ -125,18 +144,59 @@ function Invoke-Mirror($logPath) {
return $urls.Count
}

function Get-MavenArtifactUrls($artifacts) {
$feedBaseUrl = 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-maven/maven/v1'
foreach ($artifact in $artifacts) {
$parts = $artifact.Split(':', 4)
if ($parts.Count -lt 3) {
throw "Invalid Maven artifact '$artifact'. Expected group:artifact:version[:filename]."
}
$group = $parts[0].Replace('.', '/')
$name = $parts[1]
$version = $parts[2]
$filenames = if ($parts.Count -eq 4) {
@($parts[3])
} else {
@(
"$name-$version.pom"
"$name-$version.jar"
"$name-$version.aar"
"$name-$version.module"
)
}
foreach ($filename in $filenames) {
"$feedBaseUrl/$group/$name/$version/$filename"
}
}
}

# Verify az is available and authenticated up front so we fail fast.
Get-AzDevOpsToken | Out-Null

if ($PSCmdlet.ParameterSetName -eq 'MavenArtifact') {
Write-Host "Mirroring Maven artifacts directly:"
$MavenArtifact | ForEach-Object { Write-Host " $_" }
$log = Join-Path ([IO.Path]::GetTempPath()) 'maven-artifact-mirror.log'
try {
Get-MavenArtifactUrls $MavenArtifact |
ForEach-Object { "Could not GET '$_'" } |
Set-Content $log
Invoke-Mirror $log | Out-Null
}
finally {
Remove-Item $log -ErrorAction SilentlyContinue
}
return
}

Write-Host "Repo root: $repoRoot"
Write-Host "Project: $projectDirAbs"
Write-Host "Task: $Task"
Write-Host "Gradle: $gradlew"
if ($AndroidHome) { Write-Host "ANDROID_HOME: $AndroidHome" }

# Verify az is available and authenticated up front so we fail fast.
Get-AzDevOpsToken | Out-Null

if ($AndroidHome) { $env:ANDROID_HOME = $AndroidHome }
$env:RUNNINGONCI = 'true'
$env:ANDROID_MIRROR_MAVEN_DEPENDENCIES = 'true'

Push-Location $projectDirAbs
try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ namespace Java.Interop.Tools.Maven_Tests.Extensions;

class MavenProjectResolver : IProjectResolver
{
const string DotNetPublicMaven = "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-maven/maven/v1";
readonly IMavenRepository repository;

public MavenProjectResolver (IMavenRepository repository)
Expand All @@ -19,10 +20,15 @@ static MavenProjectResolver ()
{
var cache_path = Path.Combine (Environment.GetFolderPath (Environment.SpecialFolder.LocalApplicationData), "dotnet-android", "MavenCacheDirectory");

Central = new MavenProjectResolver (new CachedMavenRepository (cache_path, MavenRepository.Central));
Google = new MavenProjectResolver (new CachedMavenRepository (cache_path, MavenRepository.Google));
Central = new MavenProjectResolver (new CachedMavenRepository (cache_path, GetRepository (MavenRepository.Central, "central")));
Google = new MavenProjectResolver (new CachedMavenRepository (cache_path, GetRepository (MavenRepository.Google, "google")));
}

static MavenRepository GetRepository (MavenRepository localRepository, string name) =>
string.Equals (Environment.GetEnvironmentVariable ("RUNNINGONCI"), "true", StringComparison.OrdinalIgnoreCase)
? new MavenRepository (DotNetPublicMaven, name)
: localRepository;

public Project Resolve (Artifact artifact)
{
if (repository.TryGetFile (artifact, $"{artifact.Id}-{artifact.Version}.pom", out var stream)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@

<!--
Shared MSBuild target for compiling the real Kotlin/Gradle fixture in
tests/Xamarin.Android.Tools.Bytecode-Tests/kotlin-gradle/ using the shared
Gradle wrapper from build-tools/gradle. The wrapper downloads Gradle +
Kotlin on first run, so the resulting .class files do not need to be
committed to source control.
tests/Xamarin.Android.Tools.Bytecode-Tests/kotlin-gradle/ using the Gradle
executable configured by GradleWPath. Gradle downloads Kotlin on first run,
so the resulting .class files do not need to be committed to source control.

This file is imported by any test project that embeds the resulting
.class files as resources (Bytecode-Tests and generator-Tests). Each
Expand All @@ -24,6 +23,7 @@
<PropertyGroup>
<_KotlinGradleProjectDir>$(MSBuildThisFileDirectory)kotlin-gradle</_KotlinGradleProjectDir>
<_TestKotlinGradleOutputDir>$([System.IO.Path]::GetFullPath('$(MSBuildProjectDirectory)\$(IntermediateOutputPath)kotlin-gradle\classes'))</_TestKotlinGradleOutputDir>
<_TestKotlinGradleBuildDir>$([System.IO.Path]::GetFullPath('$(MSBuildProjectDirectory)\$(IntermediateOutputPath)kotlin-gradle\build'))</_TestKotlinGradleBuildDir>
<!--
This target is imported by two projects (Xamarin.Android.Tools.Bytecode-Tests
and generator-Tests) that build in parallel during the solution build. Gradle is
Expand All @@ -41,7 +41,7 @@
BeforeTargets="BeforeCompile"
Inputs="@(TestKotlinGradleSource)"
Outputs="@(TestKotlinGradleOutput)">
<Exec Command="&quot;$(GradleWPath)&quot; $(GradleArgs) -p &quot;$(_KotlinGradleProjectDir)&quot; -PkotlinClassesDir=&quot;$(_TestKotlinGradleOutputDir)&quot; classes"
<Exec Command="&quot;$(GradleWPath)&quot; $(GradleArgs) -p &quot;$(_KotlinGradleProjectDir)&quot; -PkotlinBuildDir=&quot;$(_TestKotlinGradleBuildDir)&quot; -PkotlinClassesDir=&quot;$(_TestKotlinGradleOutputDir)&quot; classes"
EnvironmentVariables="JAVA_HOME=$(JavaSdkDirectory);APP_HOME=$(GradleHome);GRADLE_USER_HOME=$(_KotlinGradleUserHome)" />
</Target>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ plugins {
kotlin("jvm") version "2.4.10"
}

layout.buildDirectory.set(file((findProperty("kotlinBuildDir") as String?) ?: "$rootDir/build"))

// Don't pin a jvmToolchain -- it would force Gradle to auto-provision a
// matching JDK and fail in CI environments without download repositories
// configured. Use whatever JDK the caller already set in JAVA_HOME (the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@
pluginManagement {
apply(from = "$rootDir/../../../../../eng/gradle/plugin-repositories.gradle", to = this)
}

if (System.getenv("ANDROID_MIRROR_MAVEN_DEPENDENCIES") == "true") {
apply(from = "$rootDir/../../../../../eng/gradle/credential-provider.gradle")
}

dependencyResolutionManagement {
apply(from = "$rootDir/../../../../../eng/gradle/dependency-repositories.gradle", to = this)
}
Expand Down
6 changes: 0 additions & 6 deletions external/Java.Interop/tools/java-source-utils/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,6 @@ java {
targetCompatibility = ext.javaTargetVer
}

repositories {
// Use maven central for resolving dependencies.
// You can declare any Maven/Ivy/file repository here.
mavenCentral()
}

dependencies {
// This dependency is used by the application.
implementation 'com.github.javaparser:javaparser-core:3.28.2'
Expand Down
15 changes: 7 additions & 8 deletions external/Java.Interop/tools/java-source-utils/settings.gradle
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
/*
* This file was generated by the Gradle 'init' task.
*
* The settings file is used to specify which projects to include in your build.
*
* Detailed information about configuring a multi-project build in Gradle can be found
* in the user manual at https://docs.gradle.org/6.3/userguide/multi_project_builds.html
*/
// See: eng/gradle/plugin-repositories.gradle, eng/gradle/dependency-repositories.gradle
pluginManagement {
apply from: "${rootDir}/../../../../eng/gradle/plugin-repositories.gradle", to: pluginManagement
}
dependencyResolutionManagement {
apply from: "${rootDir}/../../../../eng/gradle/dependency-repositories.gradle", to: dependencyResolutionManagement
}

rootProject.name = 'java-source-utils'
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ public void BuildApp ([Values (AndroidRuntime.CoreCLR, AndroidRuntime.NativeAOT)
}

var gradleProject = AndroidGradleProject.CreateDefault (GradleTestProjectDir, isApplication: true);
FileAssert.Exists (Path.Combine (GradleTestProjectDir, TestEnvironment.IsWindows ? "gradlew.bat" : "gradlew"));
var moduleName = gradleProject.Modules.First ().Name;

var proj = new XamarinAndroidApplicationProject {
Expand Down Expand Up @@ -510,7 +511,7 @@ namespace = ""{gradleModule.PackageName}""
dependencies {{
implementation(""androidx.appcompat:appcompat:1.6.1"")
implementation(""com.google.android.material:material:1.11.0"")
implementation(""com.facebook.android:facebook-android-sdk:latest.release"")
implementation(""com.facebook.android:facebook-android-sdk:18.3.0"")
}}
";
gradleModule.JavaSources.Add (new AndroidItem.AndroidJavaSource ("FacebookSdk.java") {
Expand Down Expand Up @@ -598,6 +599,9 @@ static IEnumerable<object[]> GetAgpGradleVersionTestData ()
[TestCaseSource (nameof (GetAgpGradleVersionTestData))]
public void BindLibraryWithMultipleGradleVersions (string agpVersion, string gradleVersion, int compileSdk)
{
if (TestEnvironment.IsRunningOnCI)
Assert.Ignore ("Alternate Gradle distributions are not downloaded from public services in CI.");

var gradleProject = AndroidGradleProject.CreateDefault (GradleTestProjectDir, agpVersion, gradleVersion, compileSdk: compileSdk);
var gradleModule = gradleProject.Modules.First ();
var moduleName = gradleModule.Name;
Expand Down
Loading
Loading