Skip to content

docs: clarify what the webhook event covers and warn about SCA duplicate deliveries - #2774

Merged
claudiacodacy merged 3 commits into
masterfrom
docs-webhooks-coverage-scope
Oct 6, 2026
Merged

claudiacodacy merged 3 commits into
masterfrom
docs-webhooks-coverage-scope

Conversation

@claudiacodacy

@claudiacodacy claudiacodacy commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Two clarifications on the webhooks page, both from the Slack thread with Amy on 2026-10-06 (https://codacy.slack.com/archives/C0B5EJJAVB3/p1791272301555039).

1. The event covers quality analysis only

The page says Codacy sends an event whenever it "finishes analyzing" a branch or pull request. That reads as all analysis. The event is quality.analysis.completed, and coverage analysis is a separate analysis that sends nothing yet.

  • The intro and the page description now say "the quality analysis of a branch or a pull request".
  • New section What the event doesn't cover, after "Events sent to your endpoint". It says the event reports quality analysis including SAST and SCA, and doesn't report coverage, DAST or container scanning. It says there's no webhook event for coverage analysis yet, and points to getPullRequestCoverageReports and the existing "identifying commits without coverage data" example for checking coverage in the meantime.

2. Organizations with SCA must review how they handle duplicate deliveries

Under Delivery behavior: a new row in the "why you received another delivery" table, and an important note.

An SCA run on a commit sends its own quality.analysis.completed delivery. It has the same repository.name, target and commitSha as the commit's regular delivery, a new X-Codacy-Delivery and a later timestamp, and nothing in the payload says it came from SCA. Its status reflects only the SCA run. The page's current advice, keep the delivery with the latest timestamp, can therefore replace the quality result with the SCA one. The note tells SCA organizations to review their handling of several deliveries for one commit and to use the API when the difference matters.

Sourcing

  • "Including SAST and SCA" and "doesn't report DAST or container scanning" come from Cláudia's status list in the Slack thread (SAST ✅, SCA ✅, Coverage ❌, DAST ❌, Container ❌).
  • Coverage is not in the event: outbound-hooks @13db735 builds quality.analysis.completed only from PullRequest.Finished and Commit.RunningTasksEnded, and neither reads coverage.
  • SCA deliveries, read from code, not observed live:
    • repository-listener @291e9e6: a Project.SCA message (sent by analysis-scheduler @801ed37) schedules an SCA job for the head commit of the main branch (RepositoryListener.scheduleSCA).
    • codacy-worker @43d5fd98b: an SCA job runs SCATaskCreator, AnalysisStatusStore and Terminator (TaskPayloadFactory.scala:92). TaskTerminator.run publishes Commit.RunningTasksEnded with no job-type check, and computes status from the tasks of that job only (TaskTerminator.scala, filter(_.payload.jobType.contains(requestContext.jobType))).
    • outbound-hooks sends a delivery for every such event that has branch names.
  • I don't know how often SCA runs are triggered, so the note doesn't say.
  • getPullRequestCoverageReports and the example page already exist in these docs.

Follow-up

When a coverage event ships (Linear project "Coverage analysis webhook event", OD-838 covers the docs), the coverage section is removed or rewritten and the new event documented.

Checks run

  • mkdocs build --strict exit 0.
  • Vale: 0 errors, 0 warnings, 0 suggestions.

👀 Live preview

🚧 To do

  • If relevant, include the Jira issue key at the end of the pull request title (no issue for this change)
  • Perform a self-review of the changes
  • Fix any issues reported by the CI/CD

🤖 Generated with Claude Code

quality.analysis.completed reports quality analysis (including SAST and SCA).
It does not report coverage, DAST or container scanning, and there is no
coverage webhook yet. Adds a "What the event doesn't cover" section and says
"quality analysis" in the intro and description.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@claudiacodacy
claudiacodacy requested a review from a team as a code owner October 6, 2026 08:20
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Overall readability score: 54.3 (🟢 +0)

File Readability
webhooks.md 76.42 (🔴 -0.45)
View detailed metrics

🟢 - Shows an increase in readability
🔴 - Shows a decrease in readability

File Readability FRE GF ARI CLI DCRS
webhooks.md 76.42 48.3 7.42 9 9.68 5.85
  🔴 -0.45 🔴 -0.3 🔴 -0.07 🔴 -0.2 🔴 -0.06 🟢 +0.03

Averages:

  Readability FRE GF ARI CLI DCRS
Average 54.3 43.05 10.88 12.31 12.26 7.98
  🟢 +0 🟢 +0 🟢 +0 🟢 +0 🟢 +0 🟢 +0
View metric targets
Metric Range Ideal score
Flesch Reading Ease 100 (very easy read) to 0 (extremely difficult read) 60
Gunning Fog 6 (very easy read) to 17 (extremely difficult read) 8 or less
Auto. Read. Index 6 (very easy read) to 14 (extremely difficult read) 8 or less
Coleman Liau Index 6 (very easy read) to 17 (extremely difficult read) 8 or less
Dale-Chall Readability 4.9 (very easy read) to 9.9 (extremely difficult read) 6.9 or less

@github-actions
github-actions Bot temporarily deployed to Netlify October 6, 2026 08:21 Inactive
@codacy-production

Copy link
Copy Markdown
Contributor

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

No merge-blocking issues were identified by the code or quality reviews. Codacy is up to standards with no new issues.

The requested documentation behaviors lack automated validation, and the claims about scanner coverage rely on an external status list. Acceptance criteria should be verified before merging, particularly the API links and generated anchor.

About this PR

  • Add documentation validation covering the updated metadata, introduction, exclusions, API/example links, and generated anchor.

Test suggestions

  • Verify the page metadata and introduction say the webhook fires when quality analysis finishes.
  • Verify the new section documents SAST/SCA coverage and excludes coverage, DAST, and container scanning.
  • Verify the coverage API and identifying-commits example links are present and valid.
  • Build the documentation with strict validation and verify the new anchor is generated.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify the page metadata and introduction say the webhook fires when quality analysis finishes.
2. Verify the new section documents SAST/SCA coverage and excludes coverage, DAST, and container scanning.
3. Verify the coverage API and identifying-commits example links are present and valid.
4. Build the documentation with strict validation and verify the new anchor is generated.
Low confidence findings
  • Confirm the SAST, SCA, DAST, and container-scanning claims against an authoritative source before relying on them in the documentation.

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

…ommit

For organizations with SCA, the SCA run on the head commit of the main branch
sends its own quality.analysis.completed delivery. Adds a row to the delivery
table and an important note under Delivery behavior, and links to it from the
"What the event doesn't cover" section.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@claudiacodacy claudiacodacy changed the title docs: clarify that the webhook event covers quality analysis only docs: clarify what the webhook event covers and warn about SCA duplicate deliveries Oct 6, 2026
@github-actions
github-actions Bot temporarily deployed to Netlify October 6, 2026 08:27 Inactive
@claudiacodacy
claudiacodacy enabled auto-merge (squash) October 6, 2026 08:42
@github-actions
github-actions Bot temporarily deployed to Netlify October 6, 2026 08:44 Inactive
@claudiacodacy
claudiacodacy merged commit 8d53a8e into master Oct 6, 2026
5 checks passed
@claudiacodacy
claudiacodacy deleted the docs-webhooks-coverage-scope branch October 6, 2026 08:44

This branch was previously deployed

1 inactive deployment
Netlify — 91d1f077 Deployed Oct 6, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants