Go Task Queue (Mini‑Celery in Go)

A fully featured distributed task queue inspired by Celery — built with Go, Redis, and Docker.

This system supports:

• Multiple workers
• Automatic retries with exponential backoff
• Dead Letter Queue (DLQ)
• Result backend
• Job scheduling (like Celery Beat)
• Prometheus metrics
• REST API
• Dashboard endpoints
• Handler registry system (plug‑and‑play job types)

Designed for production-level reliability, but simple enough for learning and extension.

---

Features

Core Messaging System

• Redis-backed distributed queue
• Main Queue (jobs)
• Dead Letter Queue (DLQ) (jobs:dead)
• Reliable Pop/Push

Workers

• Concurrent worker pool (N workers per process)
• Job handler registry (no switch-case, plugin-style extensibility)
• Exponential backoff retry system
• Automatic DLQ routing
• Stores results in Redis backend
• Graceful shutdown support

Scheduler (Cron-Based)

• Uses robfig/cron
• Works like Celery Beat
• Produces jobs at specific times
• Independent of workers (decoupled)

Monitoring

• Prometheus metrics:
  • jobs_processed_total
  • jobs_failed_total

API Server

• Submit jobs via REST
• Get job results
• Get queue status

Dashboard

• Shows queue size
• Shows DLQ size
• List jobs in queues

Deployment

• Full Docker Compose setup:
  • Redis
  • API
  • Worker pool
  • Scheduler

---

Architecture

                 ┌─────────────────────────┐
                 │         Scheduler        │
                 │      (Cron Producer)     │
                 └─────────────┬───────────┘
                               │
                               ▼
                      ┌────────────────┐
                      │ Redis Queue    │
                      │ jobs + jobs:dead
                      └────────────────┘
                               │
                               ▼
               ┌─────────────────────────────────┐
               │             Workers              │
               │   - handler registry             │
               │   - retry w/ backoff             │
               │   - DLQ routing                  │
               │   - result backend               │
               └─────────────────────────────────┘
                               │
                               ▼
                      ┌────────────────┐
                      │ Result Backend │
                      │  result:<id>   │
                      └────────────────┘

API → Push Jobs → Workers → Save Results → Query Results
Dashboard → Inspect Queues + DLQ
Metrics → Prometheus

---

Project Structure

go-task-queue/
├── cmd/
│   ├── api/          # API server entry point
│   ├── worker/       # Worker entry point
│   └── scheduler/    # Cron-based scheduler entry
│
├── internal/
│   ├── jobs/         # Job model + types
│   ├── queue/        # Redis queue + DLQ logic
│   ├── worker/       # Worker pool + handlers + retry
│   ├── scheduler/    # Cron scheduler logic
│   ├── results/      # Redis result backend
│   ├── metrics/      # Prometheus metrics
│   └── dashboard/    # HTTP dashboard
│
├── docker-compose.yml
├── Dockerfile
├── go.mod
└── README.md

---

Prerequisites

• Docker & Docker Compose
• Go 1.19+ (for local development)
• Redis (auto-launched via docker-compose)

---

Getting Started

Run Everything With Docker Compose

git clone <repository-url>
cd go-task-queue
docker-compose up --build

This will start:

• Redis (6379)
• API server (8080)
• Worker pool
• Scheduler
• Prometheus metrics endpoint
• Dashboard endpoint

---

API Endpoints

Submit a job

POST /jobs
{
  "type": "email",
  "payload": { ... },
  "max_retries": 5
}

Check job result

GET /jobs/{id}

Dashboard

GET /dashboard/queue
GET /dashboard/dead

---

Configuration

Environment variables:

REDIS_ADDR       default: localhost:6379
REDIS_QUEUE      default: jobs
WORKER_COUNT     (per worker process)

Configured inside docker-compose.yml.

---

Scaling Workers

Add more workers:

docker-compose up --scale worker=5

Workers will automatically load-balance because they all pop from the same Redis queue.

---

Development

Install dependencies

go mod download

Run Redis

docker-compose up redis

Run API

go run ./cmd/api

Run worker

go run ./cmd/worker

Run scheduler

go run ./cmd/scheduler

---

Tests

go test ./...

---

License

Add your preferred license.

---

Contributing

Pull requests welcome!
Feel free to add new job types, new queue strategies, or new transports.