ContributionPulse

Privacy-first contribution verification for developers with private repositories.

ContributionPulse connects to GitLab, Azure DevOps, and GitHub using read-only tokens, computes daily contribution aggregates, and exposes proof-style dashboards/reports without storing source-level metadata.

Product goals

• Verify contribution activity from private repositories.
• Keep user data privacy-safe by design.
• Support shareable public reports and PDF export.
• Scale sync via manual sync, scheduled sync, background jobs, and retries.

Core principles

• Store only day-level aggregate counts (commitCount, mergeCount, prCount, pipelineCount).
• Never store code, diffs, commit messages, repository names, or raw provider payloads.
• Encrypt provider credentials at rest with AES-256-GCM.
• Keep all secret handling server-side only.

---

Tech stack

• Frontend/App shell: Next.js 14 (App Router), TypeScript, Tailwind, shadcn/ui
• Auth: Supabase Auth (email magic link)
• Database: PostgreSQL + Prisma
• Queue/worker: Inngest free worker/pubsub backend, with Supabase DB queue fallback
• Charts: Recharts
• Client data/state: React Query (server communication), Zustand (UI state)

---

System architecture

High-level component diagram

flowchart LR
  U["User Browser"] --> WEB["Next.js App - App Router"]
  WEB --> SA["Supabase Auth - magic link"]
  WEB --> API[Next.js Route Handlers]
  API --> DB[("Postgres - Prisma")]
  API --> IJ["SyncJob rows - source of truth"]
  IJ --> DB
  API --> ING["Inngest Events - free pubsub"]
  ING --> HOOK["/api/inngest - node runtime"]
  HOOK --> FN["Inngest Functions - worker and scheduler"]
  FN --> DB
  FN --> GL[GitLab APIs]
  FN --> AZ[Azure DevOps APIs]
  FN --> GH[GitHub APIs]
  API <-->|status polling + SSE| U

  subgraph Optional queue backend
    SQ["Supabase queue processor - internal sync process"]
  end

  SQ -.uses SyncJob rows.-> DB

Loading

Runtime boundaries

• Web app process: UI rendering, API route handling, auth checks, report generation.
• Inngest functions: primary free production worker/scheduler path for provider sync, aggregation, and upserts.
• SyncJob table: inspectable queue state and user-facing job history; remains the source of truth even when Inngest delivers events.
• Inngest: event delivery, worker invocation, scheduled sync fanout, concurrency, and retry orchestration.
• Supabase queue processor: fallback that processes SyncJob rows by cron or manual processor calls.
• Redis pub/sub: optional realtime notification channel for the SSE sync events endpoint.
• Postgres: tenant-scoped persistent storage.

---

End-to-end flow

1) Auth + tenant resolution

1. User signs in via Supabase magic link.
2. Server calls requireAppUser() for protected pages/routes.
3. App user record is upserted by supabaseUserId.
4. All operations are scoped by appUser.id.

2) Onboarding

1. User submits provider credentials (GitLab/GitHub token, Azure token + org).
2. API encrypts token using AES-256-GCM (MASTER_KEY).
3. Integration is upserted per unique (userId, provider).
4. Optional author-email aliases are stored for commit matching.

3) Sync

1. User clicks Sync now, queues historical backfill, or enables scheduled sync in Settings.
2. API writes inspectable SyncJob rows and sends sync/job.requested events.
3. Inngest invokes /api/inngest; the worker function locks and processes each job row.
4. For each integration:
   • mark syncState=RUNNING
   • decrypt token in memory
   • fetch provider data with pagination + retries + pacing
   • clear existing aggregate rows for the provider/date range
   • aggregate to UTC day buckets
   • upsert into DailyActivity
   • set syncState=IDLE, lastSyncedAt=now
5. On failure: mark syncState=FAILED, emit sanitized error logs.

4) Realtime job status

1. Worker updates SyncJob rows and integration syncState.
2. Browser listens through SSE endpoint /api/sync/events and polls /api/sync/status while a sync is active.
3. Dashboard shows toasts and refreshes updated data.

5) Dashboard/report

1. Dashboard reads only aggregate rows (DailyActivity).
2. React Query handles all API calls (sync, backfill, shares, highlights, settings actions).
3. Public report uses tokenized, read-only route; no private metadata is exposed.
4. PDF export renders aggregate-only proof.

---

Detailed sync design

Sync sequence diagram

sequenceDiagram
  autonumber
  participant UI as Browser UI
  participant API as Next.js API
  participant DB as Postgres
  participant I as Inngest
  participant W as Inngest Worker
  participant P as Provider APIs

  UI->>API: request sync or backfill
  API->>DB: create SyncJob row per provider and date range
  API->>I: publish sync job requested event
  API-->>UI: ok true

  I->>W: invoke process-sync-job
  W->>DB: recover stale jobs, lock SyncJob
  W->>DB: load user + integrations
  loop selected provider integration
    W->>DB: set syncState=RUNNING
    W->>P: fetch paginated provider data with retry and pacing
    W->>W: aggregate to UTC day counts
    W->>DB: clear date range for provider
    W->>DB: upsert daily activity aggregate
    W->>DB: set sync state idle and last synced time
  end

  W->>DB: mark SyncJob completed/failed
  UI->>API: poll sync status and listen for sync events
  API-->>UI: latest job state and dashboard refresh

Loading

Retry, pagination, and rate limiting

• Retries transient failures (429, 5xx) with incremental backoff.
• Supports both page-based and continuation-token pagination.
• Applies per-host minimum interval (minIntervalMs) between outbound requests.

Backfill

• Backfill submits a year/provider pair and stores inspectable SyncJob rows.
• Past years cover Jan 1 through Dec 31; the current year is capped at today.
• Provider-specific date ranges are split for safer concurrency:
  • GitLab: 90-day ranges
  • GitHub: 31-day ranges
  • Azure DevOps: 14-day ranges
• Jobs are listed, expanded by date range, retried, deleted, or cleaned from the dashboard.

Scheduled sync

• Users can enable daily or weekly scheduled sync in Settings.
• Inngest deployments use the enqueue-scheduled-syncs cron function.
• Supabase-queue deployments should call POST /api/internal/sync/scheduled with CRON_SECRET to enqueue due users, then call POST /api/internal/sync/process to process queued jobs.

---

Data model

ER diagram

erDiagram
  User ||--o{ Integration : has
  User ||--o{ DailyActivity : has
  User ||--o{ ManualHighlight : has
  User ||--o{ PublicShare : has
  User ||--o{ SyncJob : has

  User {
    string id PK
    string supabaseUserId UK
    string email
  }

  Integration {
    string id PK
    string userId FK
    enum provider
    string encryptedToken
    string tokenIv
    string tokenTag
    string gitlabBaseUrl
    string azureOrg
    string[] authorEmails
    enum syncState
    datetime lastSyncedAt
  }

  DailyActivity {
    string id PK
    string userId FK
    enum provider
    datetime date
    int commitCount
    int mergeCount
    int prCount
    int pipelineCount
  }

  ManualHighlight {
    string id PK
    string userId FK
    datetime date
    string note
  }

  PublicShare {
    string id PK
    string userId FK
    string token UK
    datetime expiresAt
    datetime revokedAt
  }

  SyncJob {
    string id PK
    string userId FK
    enum provider NULL
    datetime from NULL
    datetime to NULL
    int backfillYear NULL
    enum status
    int attemptCount
    int maxAttempts
    datetime availableAt
    datetime lockedAt NULL
    datetime startedAt NULL
    datetime finishedAt NULL
    string errorMessage NULL
    datetime createdAt
    datetime updatedAt
  }

Loading

Privacy boundaries in data model

• Integration keeps encrypted secrets only.
• DailyActivity is aggregate-only.
• SyncJob stores orchestration metadata only (no provider raw payloads, no repo names, no commit messages).
• No model stores repository names or commit messages.

Supabase sync job lifecycle (when SYNC_QUEUE_BACKEND=supabase)

• QUEUED: job waiting for processing (availableAt <= now means runnable).
• RUNNING: worker has lock and is executing provider sync.
• COMPLETED: sync finished successfully, finishedAt set.
• FAILED: retries exhausted or non-retryable failure, errorMessage recorded.

Retry/backoff behavior:

• attemptCount increments when a worker locks a queued job.
• if attemptCount < maxAttempts, job is re-queued with future availableAt.
• backoff is exponential and capped (current implementation caps at 60 seconds).

---

API architecture

App route groups

• /api/integrations/* -> connect/update/disconnect providers
• /api/sync + /api/sync/backfill* -> queue operations
• /api/sync/events -> SSE stream for realtime sync status
• /api/share -> create/revoke public report links
• /api/highlights -> manual highlight CRUD (currently create)
• /api/report/pdf/[token] -> PDF export
• /api/account/delete -> account/data deletion

Client communication pattern

• React Query handles all client-initiated API communication.
• Mutations update local UI state and trigger selective refresh.
• Zustand handles non-server UI state (chart filters/year, etc.).

---

Security architecture

Credential encryption

• Algorithm: AES-256-GCM
• Key source: MASTER_KEY env var (base64, 32 bytes)
• Token encryption on write, decryption only in worker sync path.

Logging safety

• All structured logs go through sanitization.
• Token-like values are redacted before output.

Secret exposure controls

• Provider secrets are never sent to browser.
• Sensitive operations run in server routes/worker only.

Public report controls

• Tokenized URL
• Optional expiration timestamp
• Revocation support

---

Multi-tenant design

• Primary tenant key is userId.
• Uniques enforce per-tenant isolation (userId + provider, userId + provider + date).
• Every query/mutation path uses authenticated appUser.id.

---

Provider integration notes

GitLab

• Supports gitlab.com and self-hosted base URL.
• Uses GitLab events to discover activity/projects and merge/pipeline activity.
• Uses repository commits API with author aliases to count commits by authored date, preventing old commits pushed later from appearing in the wrong month.

Azure DevOps

• Requires PAT + organization name.
• Traverses projects/repositories/commits.
• Skips inaccessible repos (403/404) without failing whole sync.

GitHub

• Uses GitHub search APIs for commits and pull requests across accessible repositories, avoiding a repo-by-repo scan for the common metrics.
• GitHub Actions workflow runs are disabled by default because they require per-repo API calls. Set GITHUB_SYNC_INCLUDE_WORKFLOWS=true to include GitHub pipeline counts.

---

Local development

Prerequisites

• Node.js 20+ (project currently targets modern runtime)
• PostgreSQL
• Inngest account for free worker/pubsub production
• Supabase project (URL + anon + service role key)

Environment variables

Copy .env.example to .env.local and set:

• DATABASE_URL
• DIRECT_URL
• NEXT_PUBLIC_SUPABASE_URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY
• SUPABASE_SERVICE_ROLE_KEY
• MASTER_KEY (base64-encoded 32-byte key)
• SYNC_QUEUE_BACKEND (inngest or supabase)
• INNGEST_EVENT_KEY (required when SYNC_QUEUE_BACKEND=inngest in production)
• INNGEST_SIGNING_KEY (required when SYNC_QUEUE_BACKEND=inngest in production)
• INNGEST_DEV=1 (local Inngest dev server mode)
• REDIS_URL (required for Redis-backed realtime SSE sync events)
• CRON_SECRET (required when SYNC_QUEUE_BACKEND=supabase)
• SYNC_QUEUE_PROCESS_LIMIT (optional, Supabase queue jobs picked per processor call)
• SYNC_QUEUE_MAX_CONCURRENT_JOBS (optional, Supabase queue jobs processed in parallel)
• GITHUB_SYNC_INCLUDE_WORKFLOWS (optional, enables slower GitHub Actions pipeline sync)
• APP_URL
• NEXT_PUBLIC_AUTH_REDIRECT_URL (optional, production-only canonical origin for magic links)
• NEXT_PUBLIC_APP_NAME (optional, default ContributionPulse)
• NEXT_PUBLIC_APP_SLUG (optional, default derived from app name)

Generate MASTER_KEY:

openssl rand -base64 32

Install and run

npm install
npm run prisma:generate
npx prisma migrate deploy
npm run dev

Magic links use the current browser origin in local development, so http://localhost:3000/auth/callback stays local even if a production redirect URL exists in the environment. In production, set NEXT_PUBLIC_AUTH_REDIRECT_URL to the deployed origin if you want a canonical callback domain.

In Supabase Auth settings, add both redirect URLs:

• http://localhost:3000/auth/callback
• https://your-production-domain.com/auth/callback

Inngest worker/pubsub mode (recommended free production)

Set:

SYNC_QUEUE_BACKEND=inngest
INNGEST_EVENT_KEY=<event-key-from-inngest>
INNGEST_SIGNING_KEY=<signing-key-from-inngest>
SYNC_QUEUE_MAX_CONCURRENT_JOBS=4

Expose /api/inngest in production. Inngest Cloud discovers the functions served there:

• process-sync-job: event-driven provider sync worker
• enqueue-scheduled-syncs: nightly scheduled sync fanout

For local development, run the Next app and Inngest dev server:

INNGEST_DEV=1 npm run dev
inngest dev

Supabase queue mode

Use this if you do not want Inngest to invoke workers:

1. Set:

SYNC_QUEUE_BACKEND=supabase
CRON_SECRET=<strong-random-secret>
SYNC_QUEUE_PROCESS_LIMIT=6
SYNC_QUEUE_MAX_CONCURRENT_JOBS=4

2. Trigger queue processing by calling:

POST /api/internal/sync/process
Authorization: Bearer <CRON_SECRET>

Process manually in dev:

npm run queue:process

3. Schedule this endpoint with Supabase Cron (example every minute):

select
  cron.schedule(
    'process-contribution-sync-queue',
    '* * * * *',
    $$
    select
      net.http_post(
        url := 'https://contribution-pulse.vercel.app/api/internal/sync/process?limit=10',
        headers := jsonb_build_object(
          'Content-Type', 'application/json',
          'Authorization', 'Bearer CRON_SECRET'
        ),
        body := '{}'::jsonb
      );
    $$
  );

For faster historical backfills, keep GitHub Actions workflow sync disabled unless you need pipeline counts. GitHub and Azure DevOps backfills are split into smaller date ranges so the queue can process them concurrently. Start with SYNC_QUEUE_PROCESS_LIMIT=6 and SYNC_QUEUE_MAX_CONCURRENT_JOBS=4; raise carefully if provider rate limits and your database connection pool stay healthy.

---

Testing

npm test

Current automated tests include:

• encryption helper correctness (AES-256-GCM)
• daily aggregation/upsert logic
• auth redirect origin handling
• scheduled sync eligibility
• backfill current-year window capping
• GitLab author/query helpers
• sync window replay behavior
• email avatar URL generation

---

Deployment topology

Recommended free production topology:

1. Web service: Next.js app on Vercel or similar.
2. Worker/scheduler: Inngest Cloud invoking /api/inngest.
3. Database: managed Postgres, for example Supabase Postgres.
4. Auth: Supabase Auth.
5. Redis for the current SSE realtime notification channel.

Alternative topologies:

• Supabase queue mode: schedule /api/internal/sync/process with Supabase Cron.

---

Required GitHub repository secrets

Set these in GitHub repo settings -> Secrets and variables -> Actions:

• VERCEL_TOKEN
• VERCEL_ORG_ID
• VERCEL_PROJECT_ID
• DATABASE_URL
• DIRECT_URL
• NEXT_PUBLIC_SUPABASE_URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY
• SUPABASE_SERVICE_ROLE_KEY
• MASTER_KEY
• INNGEST_EVENT_KEY
• INNGEST_SIGNING_KEY
• REDIS_URL (for Redis-backed realtime SSE sync events)
• APP_URL
• NEXT_PUBLIC_AUTH_REDIRECT_URL

---

Docker deployment

The included Docker Compose file runs the web service plus Redis for local realtime SSE events. Inngest production deployments usually only need the Next.js web service plus the hosted /api/inngest endpoint.

Files added

• Dockerfile -> production image for Next.js web app
• docker-compose.yml -> web + Redis local/prod-like stack
• .dockerignore -> optimized build context

Docker architecture diagram

flowchart LR
  U["Browser"] --> WEB["web container - Next.js"]
  WEB --> DB[(Postgres)]
  WEB --> SA[Supabase Auth]
  WEB --> ING["Inngest Cloud - recommended"]
  ING --> WEBHOOK["/api/inngest"]
  WEBHOOK --> DB
  WEBHOOK --> GL[GitLab API]
  WEBHOOK --> AZ[Azure DevOps API]
  WEBHOOK --> GH[GitHub API]
  WEB --> R[(Redis - realtime SSE pub/sub)]

Loading

Run with Docker Compose

docker compose up --build -d

Services:

• web -> app on http://localhost:3000
• redis -> realtime SSE pub/sub broker

Stop:

docker compose down

Notes

• docker-compose.yml uses .env for app secrets.
• Keep DATABASE_URL pointing to your managed Postgres (or add a Postgres service if desired).

---