Skip to content

Add Codex code review workflow#61

Draft
liobrasil wants to merge 1 commit intomainfrom
chore/codex-code-review
Draft

Add Codex code review workflow#61
liobrasil wants to merge 1 commit intomainfrom
chore/codex-code-review

Conversation

@liobrasil
Copy link
Collaborator

Summary

  • add a GitHub Actions workflow for Codex PR reviews
  • post/update a sticky PR comment tagged with <!-- codex-code-review -->
  • include model resolution with fallback from gpt-5.3-codex to gpt-5.2-codex
  • add a reusable review prompt file under .github/codex/prompts/

Notes

  • requires repository secret: OPENAI_API_KEY

@claude
Copy link

claude bot commented Feb 23, 2026

Claude Code Review

PR: Add Codex code review workflow
Files: .github/workflows/codex-code-review.yml, .github/codex/prompts/review.md


Summary

This PR adds a GitHub Actions workflow that runs OpenAI Codex on each PR and posts a sticky review comment. The overall structure is sound, but there are several issues worth addressing before merging.


Findings

High

1. Model names appear to be non-existent
File: .github/workflows/codex-code-review.yml (env vars CODEX_MODEL / CODEX_FALLBACK_MODEL)

gpt-5.3-codex and gpt-5.2-codex do not correspond to any known OpenAI API model identifiers. The model resolution step uses a curl probe to the /v1/models/{model} endpoint — if neither model exists, the primary probe fails silently (falling to the fallback), and if the fallback also fails, the codex-action step itself will error. This likely means the workflow will never succeed as written.

Recommendation: Verify the model identifiers against the OpenAI API docs and update to valid model names (e.g., gpt-4o, o4-mini, or whatever is currently available for code review tasks). If these are forward-looking model names, add a comment explaining the intent.


2. openai/codex-action@v1 is pinned to a mutable tag
File: .github/workflows/codex-code-review.yml (step Run Codex Code Review)

Using @v1 (a mutable tag) on a third-party action is a supply chain risk. If the action's repository is compromised or the tag is force-pushed, all future runs execute the attacker's code with write access to the PR.

Recommendation: Pin to a specific commit SHA:

uses: openai/codex-action@<full-sha>

The same applies to actions/checkout@v4 and actions/github-script@v7, though those are from actions/ org which is lower risk. Best practice is still to pin all third-party actions.


Medium

3. Model probe curl conflates all errors with "model unavailable"
File: .github/workflows/codex-code-review.yml (step Resolve Codex model)

The curl -fsS call falls back to CODEX_FALLBACK_MODEL on any non-2xx response, including transient network errors, 429 rate-limit responses, and 500 server errors. A rate-limit hit during model resolution would silently switch the model rather than retrying or failing loudly.

Recommendation: Check the HTTP status code more precisely and only fall back on 404 (model not found):

http_status=$(curl -o /dev/null -w "%{http_code}" -sS \
  "https://api.openai.com/v1/models/${CODEX_MODEL}" \
  -H "Authorization: Bearer ${OPENAI_API_KEY}")
if [ "${http_status}" = "200" ]; then
  selected_model="${CODEX_MODEL}"
else
  echo "::warning::${CODEX_MODEL} returned ${http_status}; using ${CODEX_FALLBACK_MODEL}."
  selected_model="${CODEX_FALLBACK_MODEL}"
fi

4. output-file: codex-output.md is created but never used
File: .github/workflows/codex-code-review.yml (step Run Codex Code Review)

The output-file parameter writes the review to codex-output.md, but the "Post or update" step reads from steps.codex_review.outputs['final-message'] instead. The file artifact is generated but effectively dead.

Recommendation: Either remove output-file if it's not needed, or use it as the source of truth for the comment body (e.g., read the file and post its contents), which would also sidestep potential env-var size limits for large reviews.


5. Verification step gives a misleading error if earlier steps failed
File: .github/workflows/codex-code-review.yml (step Verify Codex sticky comment exists)

The verification step runs if: always(), meaning it executes even if the Codex action or post step failed. If Codex itself errors, the comment was never posted, but the verification step will emit "No Codex sticky review comment found" — hiding the actual root cause.

Recommendation: Change the condition to only run verification when the post step succeeded:

if: steps.codex_review.conclusion == 'success'

Low

6. No cost/rate guardrails
File: .github/workflows/codex-code-review.yml

The workflow triggers on every synchronize event (every push to a PR branch). For an active repo, this can produce significant OpenAI API costs with no cap or opt-out mechanism.

Recommendation: Consider adding a label-based opt-in (e.g., only run if a codex-review label is present) or limiting the trigger to specific branches or PR sizes.

7. Review prompt lacks repository context
File: .github/codex/prompts/review.md

The prompt says "You are an expert reviewer for this repository" but provides no information about what the repository does, its conventions, or key architectural patterns. Codex may produce generic feedback that misses domain-specific issues (e.g., the cross-VM Cadence/EVM request lifecycle or sentinel value conventions described in CLAUDE.md).

Recommendation: Add a brief repository description paragraph to the prompt, or reference the CLAUDE.md content to give the model relevant context.


Checklist

  • Confirm model identifiers are valid OpenAI API models
  • Pin openai/codex-action to a commit SHA
  • Fix curl model-probe to distinguish 404 from other errors
  • Remove or use output-file
  • Fix verification step condition
  • Consider adding cost controls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant