Devbase

Devbase helps developers review risks before making changes in code, by querying repo history from GitHub issues and pull requests in a RAG system.

It uses eval gates in a workflow runner so that teams can continue to update + refer to repo data in a continuously reliable RAG.

Built using:

Demo

🡳 Click image to watch demonstration + instructions for setup

Tech Stack

AI Orchestration	API & Web	Eval & Monitoring
LangGraph -> workflow orchestration Hugging Face -> LLM and embedding models LightRAG -> RAG storage/retrieval Presidio -> prompt-injection screening	Python -> FastAPI REST React -> Vite, Chakra UI Docker ->runs n8n container	n8n -> eval workflow Braintrust -> observability

Architecture

flowchart TB
    User["Developer"]
    CLI["Local setup CLI<br/>(set builder, n8n setup)"]
    UI["Devbase UI"]
    API["Backend API<br/>(FastAPI, LangGraph)"]
    GitHub["Repo Data<br/>(GitHub issues + PRs)"]
    RAG["RAG Storage<br/>(LightRAG, Hugging Face, Presidio)"]
    Gate["Quality Gate<br/>(n8n workflow)"]

    User --> UI
    UI --> API
    API --> RAG
    RAG --> API
    API --> UI

    User -.-> CLI
    CLI -.-> Gate

    API ==>|fetch| GitHub
    GitHub ==>|ingest| RAG
    API ==>|evaluate| Gate
    Gate ==>|promote new data| RAG

    linkStyle 0,1,2,3,4 stroke:#2563eb,stroke-width:2px
    linkStyle 5,6 stroke:#6b7280,stroke-width:2px,stroke-dasharray:5 5
    linkStyle 7,8,9,10 stroke:#16a34a,stroke-width:3px

Loading

Setup Instructions

Configure Environment

Create your local .env file from the template:

copy .env.example .env

Required

• HF_TOKEN: required for LightRAG to call Hugging Face LLM and embedding models.

Optional

• GITHUB_TOKEN: needed for private repos or higher GitHub API rate limits
• BRAINTRUST_API_KEY: enables eval logging/monitoring

Modifiable

• HF_LLM_MODEL: Hugging Face chat model used for report generation
• HF_EMBEDDING_MODEL: Hugging Face embedding model used by LightRAG
• HF_PROVIDER: Hugging Face inference provider. Keep auto unless you need a specific provider

Install Dependencies

From the project root, install dependencies:

pip install -r requirements.txt

Build Golden Set

Golden set cases are repo-scoped. The file: golden_test_set.jsonl contains cases for each repo that has been fetched using Devbase.

• The eval process only uses cases for the repo being updated.
• Rebuilding for the same repo replaces that repo's cases and keeps cases for other repos.

Note

A specific repo's golden set contains data pairs, each consisting of:

1. example case of your proposed code change
2. the sources that the RAG should cite

Run the builder:

python -m scripts.build_golden_set

The CLI asks for:

• Repository owner/repo: target repo, for example micattoc/devbase
• Issues to fetch: number of issue records
• Pull requests to fetch: number of PR records

For each generated candidate (records that have a score of 3+):

• y: approve case
• n: reject case
• e: edit prompt before saving
• q: stop review

Note

Candidate score is a heuristic (calculated when record is fetched):

• +1 when title/body contains high-signal words like bug, regression, breaking, compatibility, fix, route, request, or body
• +2 when labels contain those high-signal words
• +1 when the record is an issue or pull request
• +3 when the text links to another GitHub record with fixes, closes, or resolves #...
• +1 when the record has a URL

Higher score means the record is more likely to become a useful eval case.

The CLI shows up to 10 candidates, ranked by score, after dropping records below the minimum score threshold of 3.

Setup n8n Workflow

n8n runs the eval process: golden set evalution, promote staging RAG data only if eval passes, then return the result to Devbase.

Important

Prerequisites:

• Docker is running
• Golden set exists for the repo you want to update
• FastAPI is available at http://host.docker.internal:8000 when the workflow runs
• Registered an account in n8n

Run setup from the project root:

bash scripts/setup_n8n.sh

The script will start devbase_n8n Docker container with an imported n8n workflow that is automatically published as a production webhook at:

http://localhost:5678/webhook/kb-updated

You can view and modify the n8n workflow executions + stages at:

http://localhost:5678

Run Devbase

Run backend

From the project root:

uvicorn api.rest_api:app --reload --port 8000

Run the UI

In a second terminal:

cd web
npm run dev

Open on browser

http://localhost:5173

---

Developed by