Skip to content

feat: add evaluation api call (CM-1168)#4140

Open
ulemons wants to merge 2 commits into
mainfrom
feat/add-evaluation-api-call
Open

feat: add evaluation api call (CM-1168)#4140
ulemons wants to merge 2 commits into
mainfrom
feat/add-evaluation-api-call

Conversation

@ulemons
Copy link
Copy Markdown
Contributor

@ulemons ulemons commented May 21, 2026

Summary

Wires the projects evaluation worker to the external LFX AI onboarding-evaluation API, replacing the placeholder evaluator. Each pending project in projectCatalog is sent to the API, and the response is persisted as the project's action, evaluationResult, and evaluationReason.

Changes

  • Evaluator implementation (evaluator/evaluator.ts): real HTTP call to CROWD_PROJECT_EVALUATION_API_ENDPOINT with Bearer auth, application/x-www-form-urlencoded body (message, stream=false, user_id). All failure modes (missing config, network error, non-2xx, JSON parse error) resolve to outcome: 'unsure' with evaluationResult: 'error' and a descriptive evaluationReason, so a single bad call never crashes the workflow.
  • New skip outcome: extended EvaluationOutcome and ProjectCatalogAction to include 'skip' (when the API returns onboard: false). 'onboard' | 'skip' | 'unsure' are now treated as terminal in upsertProjectCatalog / bulkUpsertProjectCatalog a re-discovered project will not be moved back to evaluate.
  • Result shape: IEvaluationResult now carries evaluationResult (stringified onboard flag) and evaluationReason (the API's non_onboard_reason) instead of a single reason field, matching the new columns persisted on projectCatalog.
  • Idempotency guard (activities/activities.ts): before calling the API, refetch the project via findProjectCatalogById and short-circuit if evaluatedAt is already set. Protects against double-billing if Temporal retries an activity after a successful API call but a failed DB write.
  • Worker config (main.ts): declares the three required env vars (CROWD_PROJECT_EVALUATION_API_ENDPOINT, CROWD_PROJECT_EVALUATION_API_USER_ID, CROWD_PROJECT_EVALUATION_API_SECRET) so the service fails fast on boot if any are missing.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Performance improvement
  • Chore / dependency update
  • Documentation

JIRA ticket

ticket


Note

Medium Risk
Introduces an external HTTP dependency into the evaluation workflow and changes project action semantics (new terminal skip), which could affect queue behavior and data correctness if the API misbehaves or responses change.

Overview
Replaces the placeholder project evaluator with a real call to an external onboarding-evaluation API (env-configured endpoint/user/secret), mapping responses to onboard vs the new terminal skip, and returning unsure with evaluationResult='error' on all failure modes.

Updates the evaluation activity to be idempotent by reloading the project on the writer DB and skipping the API call if evaluatedAt is already set, and persists evaluationResult/evaluationReason alongside action when writing back.

Extends ProjectCatalogAction/EvaluationOutcome and DAL upsert logic to treat skip as a terminal state (preventing reversion to evaluate on rediscovery), and declares the required API env vars for the worker to fail fast on boot.

Reviewed by Cursor Bugbot for commit 0b5f491. Bugbot is set up for automated code reviews on this repo. Configure here.

@ulemons ulemons self-assigned this May 21, 2026
@ulemons ulemons added the Feature Created by Linear-GitHub Sync label May 21, 2026
Copilot AI review requested due to automatic review settings May 21, 2026 08:39
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
@ulemons ulemons force-pushed the feat/add-evaluation-api-call branch from 0e24f35 to f22ee1d Compare May 21, 2026 08:39
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f22ee1d. Configure here.

Comment thread services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR wires the projects evaluation worker to an external onboarding-evaluation API, persisting the returned decision into projectCatalog and extending the catalog “terminal” actions to include skip.

Changes:

  • Replace the placeholder evaluator with an authenticated HTTP POST to CROWD_PROJECT_EVALUATION_API_ENDPOINT, mapping results to onboard | skip | unsure and persisting evaluationResult/evaluationReason.
  • Treat skip as a terminal projectCatalog.action in DAL upsert/bulk-upsert logic so projects aren’t moved back into evaluate.
  • Add an activity-level idempotency guard and require the evaluation API env vars at worker startup.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
services/libs/data-access-layer/src/project-catalog/types.ts Extends ProjectCatalogAction with skip.
services/libs/data-access-layer/src/project-catalog/projectCatalog.ts Treats skip as terminal in upsert/bulk-upsert action resolution.
services/apps/projects_evaluation_worker/src/main.ts Declares required evaluation API env vars for fail-fast boot.
services/apps/projects_evaluation_worker/src/evaluator/types.ts Updates evaluator result/outcome types to support skip and new persisted fields.
services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts Implements the external API call and response-to-outcome mapping.
services/apps/projects_evaluation_worker/src/activities/activities.ts Adds a pre-call re-fetch guard and persists evaluation fields on update.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts Outdated
Comment thread services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts
Comment thread services/apps/projects_evaluation_worker/src/activities/activities.ts Outdated
Comment thread services/apps/projects_evaluation_worker/src/evaluator/evaluator.ts Outdated
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Created by Linear-GitHub Sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants