A scanner grades a finding from what its engine said and what this installation knows about the repository. It cannot read the code the way a reviewer does: notice that the redirect target came out of the server's own database, or that the broken half of a package is a WebSocket client nothing constructs. That is the question somebody brings to a finding before they fix it or argue an exception, and Validate with AI asks it of a language model.
Sign in as an administrator, open the sidebar, expand Administration and
choose AI analysis, between Notifications and Triage rules. The address is
/ai on your installation. The section only appears for administrators: a
developer or reader sees neither the menu entry nor the page.
Pick one provider:
| Provider | What it needs |
|---|---|
| Anthropic (Claude) | An API key. The model defaults to claude-opus-5-5, and Effort to high. |
| OpenAI | An API key and a model name. The address can point at anything with the same chat completions API: Azure OpenAI, a gateway, a self hosted server. |
| Custom | An address, and the headers and body the way curl writes them. The word SAMPLE in the body is replaced with the question. |
A custom template looks like this:
-H 'Content-Type: application/json'
-H 'x-app-id: your-app-id'
--data-raw '{"model_class":"strong-fast","input_text":"SAMPLE"}'
Put the secret in the Secret header fields rather than the template. The
template is shown back on the page; the secret is stored encrypted and never
is. If the answer is not at output_text, choices.0.message.content or one
of the other common places, say where it is in Where the answer is.
List models asks the saved key what it can use. Send a test question asks one short question with nothing from any repository in it. Both use what is saved, not what is typed, so save first.
Every address has to be https and cannot be a link-local address, because it is where source code goes.
A finding shows Validate with AI once a provider is switched on and complete. To validate many, tick them on the findings list, or use Select all matching these filters, and press Validate with AI in the bar that appears. A bulk run works in the background, and the Learned rules page shows how far it has got, how many findings went to the model, how many a learned rule answered for nothing, and the tokens spent. Two questions run at once at most.
The answer is kept on the finding, newest first, with the model, the person who asked and when. What it was given lists every piece of evidence that went out.
Auto LLM review on the AI page validates every finding a scan turns up
for the first time, the same way as pressing the button on each one. It runs
in the background after the scan is saved. The answer is kept on the finding
under the name auto-review. A finding a learned rule already covers is
answered by the rule and sends nothing. Credentials and known exploited
packages are not sent, the same as a bulk run, and nothing goes out while the
provider is off.
Two settings keep it inside what the provider allows. Requests per minute
spaces the questions out so no more than that many go in a minute, even when
two scans finish at once. Set it below the provider's limit to leave room for
people and for the security report. New findings per scan reads the worst
first and stops there, so a first scan of a large estate, where everything is
new, does not turn into a very large bill. 0 means no limit. What is left
over can be found on the findings list and validated by hand. Each run is in
the audit log as ai.auto_review, with how many were queued and how many were
over the limit.
A finding a model has read, or a learned rule has graded, has an AI reviewed badge on the findings list. Hovering it says who asked and what the model said, or which rule graded it. A folded row where only some of its findings were read says how many, such as AI reviewed 2/7. The AI review filter shows only the ones read, or only the ones not read yet, and works with Select all matching these filters, so the leftovers can be sent in one go.
A reading lower than the finding's grade lowers it, and the finding says the grade came from AI validation and what it was before. It never raises a grade. It never lowers a credential, because nothing a model reads in the code shows a secret is dead, and never a package on the known exploited list. Those get the answer as advice and keep their grade. A bulk run does not send them at all.
A person's own regrade always has the last word. An AI grade under it is kept, and stands again if the regrade is cleared.
Every answer also says which other findings deserve the same grade for the same reason: the same rule in the same files with the same call on the flagged line, or every advisory on a package only a build tool pulls in. That is kept as a learned rule, with a synthetic test beside it. The finding it came from must match the rule, and anything a model later refused to lower must not.
A new rule grades the other findings it covers straight away, and from then on it answers instead of the model: pressing Validate with AI on a finding a rule covers costs nothing, and so does every later scan. A bulk run puts findings about the same thing next to each other, so the first answer usually settles the rest. Tick Ask the model anyway on a finding to go past the rule.
A rule applies in the repository it was learned in. Apply everywhere on the Learned rules page lets it answer in every repository, and is an administrator's decision. Retire stops a rule and puts back every grade it gave. A model asked again that disagrees with a rule retires it on its own, and keeps the finding it got wrong as a case the rule must not match. Run the synthetic tests checks every live rule and retires any that fail.
A grade decision, a person's regrade or an AI grade, records the code it was made against: the flagged lines for code and configuration, the version and advisory for a package. A later scan of the same code keeps it, so nobody is asked twice. When the flagged lines change, the decision is lifted, the finding goes back to the engine's grade, and its page says who decided what and that the code it was about has changed. A credential's regrade is never lifted this way, because a secret's identity is the secret itself.
Always:
- the finding, the reasons behind its grade, and the line the engine flagged, masked the way the scanner stores it (never for a credential);
- what is recorded about the repository: visibility, what it holds, an administrator's description;
- up to twelve findings nearby (same file, package or rule) with any exception or regrade reason somebody gave;
- this tool's own definition of the rule, when it is one of ours.
When ticked:
- the code: the lines around a code or configuration finding, or the places a vulnerable package is used, read from the branch at that moment and not stored. Never for a credential, whatever the box says, because the lines around a credential contain the credential;
- the advisory from osv.dev for a vulnerable package. Only the public id goes to osv.dev.
Never: who can reach the repository, who wrote the commit, or the credential itself.
Sending code means your source leaves this installation for the provider you picked. Check that is somewhere your organization allows it to go before ticking the box.
Only administrators can validate a finding, one at a time or in bulk, and see the Learned rules page, because every question is paid for, sends the finding outside this installation, and can lower grades for everyone. Developers and readers see the answers and the grades they gave, but have no button. Only administrators can change the configuration, apply a rule everywhere, retire one or run the synthetic tests.
What has been learned keeps applying with AI switched off: every scan still grades from the learned rules and the AI grades already given, and with no provider set up the bulk button on the findings list becomes Apply learned rules, which answers from the rules alone and sends nothing anywhere. Every validation, bulk run, save, test and rule decision is in the audit log.