[Feature]: Added artifacts for ai by pulk17 · Pull Request #1059 · CCExtractor/sample-platform

pulk17 · 2026-03-11T16:52:23Z

[FEATURE] AI-Accessible Test Artifacts & Structured Test Report Endpoint

In raising this pull request, I confirm the following (please check boxes):

I have read and understood the contributors guide.
I have checked that another pull request for this purpose does not exist.
I have considered, and confirmed that this submission will be valuable to others.
I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

I have never used the project.
I have used the project briefly.
I have used the project extensively, but have not contributed previously.
I am an active contributor to the project.

What does this PR do?

This PR introduces artifact capture and upload from CI test VMs to GCS, along with a set of new download endpoints and a structured JSON report endpoint designed to make test results fully accessible — including to AI agents.

A demo video is attached below showcasing all the new features end to end.

PR.Showcase.mp4

Objective

Right now, when a test run fails, debugging is hard. You can see that something went wrong, but you can't easily grab the binary that was tested, see what it actually printed, or get the coredump if it crashed. Everything lives ephemerally on the VM and disappears. This PR fixes that by persisting the important artifacts to GCS and making them accessible via clean download URLs.

The ai.json endpoint was also something I wanted to add to make it possible for AI agents (or any automated tooling) to consume a full structured report of a test run — with direct download links to every artifact — without needing to scrape HTML.

Changes

`install/ci-vm/ci-linux/ci/runCI`

Added a wrapper script that intercepts every ccextractor invocation during testing, tees all stdout/stderr into a combined log at /tmp/combined_stdout.log, and preserves the original exit code faithfully. This means you get a single unified log of everything ccextractor printed across all test cases in a run.
After tests complete, the following artifacts are uploaded to GCS under test_artifacts/{test_id}/:
- The ccextractor binary itself
- The combined stdout/stderr log
- The first coredump found in /tmp/coredumps/, if one was produced
testID is now passed as VM instance metadata so the CI script knows where to namespace the uploads.

`mod_ci/controllers.py`

testID is now included in the metadata passed to both Linux and Windows VM instances at creation time.
Fixed two instances of os.rename() → os.replace() in upload handlers. os.rename can fail silently or raise an error when source and destination are on different filesystems (which can happen with temp directories). os.replace handles this correctly.

`mod_test/controllers.py`

Refactored download_build_log_file to use send_from_directory directly instead of going through serve_file_download, which was doing unnecessary GCS signed URL generation for a file that's already on disk.
Added a shared _artifact_redirect helper that looks up a blob in GCS, generates a short-lived signed URL, and redirects to it — or returns 404 if the blob doesn't exist. All new download endpoints use this.
Added the following new routes:
- /<test_id>/binary — downloads the ccextractor binary used in the test (tries Linux name first, then Windows)
- /<test_id>/coredump — downloads the coredump if one was captured
- /<test_id>/combined-stdout — downloads the unified stdout/stderr log
- /<test_id>/regression/<regression_test_id>/<output_id>/output-got — downloads the actual output file for a specific test case
- /<test_id>/regression/<regression_test_id>/<output_id>/output-expected — downloads the expected output file for a specific test case
- /<test_id>/sample/<sample_id> — downloads the sample file used in a regression test
- /<test_id>/ai.json — returns a full structured JSON report of the test run (see below)

`utility.py`

Fixed a path construction bug where os.path.join was used to build GCS blob paths. Since GCS paths are always forward-slash separated and os.path.join behaves differently on Windows, this is now done with a manual /.join` with empty-part filtering.
Fixed the default value for GCS_SIGNED_URL_EXPIRY_LIMIT from '' (an empty string that would crash timedelta) to 30.

The `ai.json` Endpoint

GET /<test_id>/ai.json returns a JSON object that looks roughly like:

{
  "test_id": 123,
  "commit": "abc123",
  "platform": "linux",
  "branch": "master",
  "status": "completed",
  "binary_url": "https://...",
  "coredump_url": null,
  "log_url": "https://...",
  "combined_stdout_url": "https://...",
  "summary": { "total": 50, "passed": 48, "failed": 2 },
  "test_cases": [
    {
      "regression_test_id": 7,
      "category": "...",
      "sample_filename": "sample.ts",
      "sample_url": "https://...",
      "arguments": "-out=ttxt",
      "result": "Fail",
      "exit_code": 1,
      "expected_exit_code": 0,
      "runtime_ms": 4231,
      "how_to_reproduce": "./ccextractor -out=ttxt sample.ts",
      "outputs": [
        {
          "output_id": 3,
          "correct_extension": ".ttxt",
          "expected_url": "https://...",
          "got_url": "https://...",
          "diff_url": "https://..."
        }
      ]
    }
  ],
  "how_to_reproduce": "Download the binary and sample, then run: ./ccextractor {arguments} {sample_filename}"
}

Every URL in the response is a signed GCS download link valid for 30 minutes. The intent is that this endpoint gives an AI agent (or any automated system) everything it needs to fully investigate a test failure — the binary, the sample, the expected and actual outputs, the diff, and the full stdout log — without any additional context.

Notes

The new artifact download routes do not require authentication. This is intentional — the signed URLs are short-lived and the artifacts contain no sensitive data, and requiring auth would prevent AI agents from being able to use them directly.
The wrapper script uses a hardcoded /tmp/combined_stdout.log path, which is fine given each test run gets its own VM.

sonarqubecloud · 2026-03-18T18:06:02Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

canihavesomecoffee · 2026-05-31T07:33:14Z

+        gcs_bucket=$(curl -s "http://metadata/computeMetadata/v1/instance/attributes/bucket" -H "Metadata-Flavor: Google")
+        test_id=$(curl -s "http://metadata/computeMetadata/v1/instance/attributes/testID" -H "Metadata-Flavor: Google")
+        token=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google" | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")


I'm not in favour of directly uploading to a bucket; I'd rather have it stored through the SP, where we can exercise more control and do extra checks. This direct upload would allow someone with malicious intentions to hijack a CCExtractor PR and start uploading other things.

canihavesomecoffee · 2026-05-31T07:39:03Z



 @mod_test.route('/log-files/<test_id>')
-@login_required


A login should still be required.

canihavesomecoffee · 2026-05-31T07:40:45Z

+    """Download the ccextractor binary used in a test (linux or windows)."""
+    from run import storage_client_bucket
+    # Try linux name first, then windows
+    for name in ['ccextractor', 'ccextractor.exe']:


Based on the test ID, you can exactly know what platform it was for, and thus check the direct extension rather than "guessing".

canihavesomecoffee · 2026-05-31T07:40:59Z

+
+
+@mod_test.route('/<int:test_id>/binary', methods=['GET'])
+def download_binary(test_id):


This needs a logged-in user.

canihavesomecoffee · 2026-05-31T07:41:04Z

+
+
+@mod_test.route('/<int:test_id>/coredump', methods=['GET'])
+def download_coredump(test_id):


This needs a logged-in user.

canihavesomecoffee · 2026-05-31T07:41:10Z

+
+
+@mod_test.route('/<int:test_id>/combined-stdout', methods=['GET'])
+def download_combined_stdout(test_id):


This needs a logged-in user.

canihavesomecoffee · 2026-05-31T07:43:03Z

+    )
+
+
+@mod_test.route('/<int:test_id>/regression/<int:regression_test_id>/<int:output_id>/output-got', methods=['GET'])


This needs a logged-in user.

canihavesomecoffee · 2026-05-31T07:43:11Z

+
+
+@mod_test.route('/<int:test_id>/regression/<int:regression_test_id>/<int:output_id>/output-expected', methods=['GET'])
+def download_output_expected(test_id, regression_test_id, output_id):


This needs a logged-in user.

canihavesomecoffee · 2026-05-31T07:44:30Z

+        filename=f'output_expected_{regression_test_id}_{output_id}{ext}'
+    )
+@mod_test.route('/<int:test_id>/sample/<int:sample_id>', methods=['GET'])
+def download_sample_ai(test_id, sample_id):


Even with AI workflow, there should be auth. We could perfectly issue a long(er) living access token for programmatic access. If we don't, a scraper might find this and still cause huge fees.

canihavesomecoffee · 2026-05-31T07:44:54Z

+    )
+
+
+def _process_test_case(test_id, category_name, t_data):


Please fix the code smell here.

pulk17 added 2 commits March 11, 2026 20:03

added artifacts for ai

6d95999

minor cleanup

0e183d4

pulk17 requested review from canihavesomecoffee and thealphadollar as code owners March 11, 2026 16:52

Code refactor for sonarqube

889d0d1

pulk17 changed the title ~~added artifacts for ai~~ [Feature]! Added artifacts for ai Mar 23, 2026

pulk17 changed the title ~~[Feature]! Added artifacts for ai~~ [Feature]: Added artifacts for ai Mar 23, 2026

canihavesomecoffee reviewed May 31, 2026

View reviewed changes

canihavesomecoffee requested changes May 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]: Added artifacts for ai#1059

[Feature]: Added artifacts for ai#1059
pulk17 wants to merge 3 commits into
CCExtractor:masterfrom
pulk17:feat/test-artifacts

pulk17 commented Mar 11, 2026

Uh oh!

sonarqubecloud Bot commented Mar 18, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

canihavesomecoffee May 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		@mod_test.route('/<int:test_id>/binary', methods=['GET'])
		def download_binary(test_id):



		@mod_test.route('/<int:test_id>/coredump', methods=['GET'])
		def download_coredump(test_id):



		@mod_test.route('/<int:test_id>/combined-stdout', methods=['GET'])
		def download_combined_stdout(test_id):

		)


		@mod_test.route('/<int:test_id>/regression/<int:regression_test_id>/<int:output_id>/output-got', methods=['GET'])



		@mod_test.route('/<int:test_id>/regression/<int:regression_test_id>/<int:output_id>/output-expected', methods=['GET'])
		def download_output_expected(test_id, regression_test_id, output_id):



		@mod_test.route('/log-files/<test_id>')
		@login_required

Conversation

pulk17 commented Mar 11, 2026

What does this PR do?

Objective

Changes

install/ci-vm/ci-linux/ci/runCI

mod_ci/controllers.py

mod_test/controllers.py

utility.py

The ai.json Endpoint

Notes

Uh oh!

sonarqubecloud Bot commented Mar 18, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`install/ci-vm/ci-linux/ci/runCI`

`mod_ci/controllers.py`

`mod_test/controllers.py`

`utility.py`

The `ai.json` Endpoint