Executive Architecture Overview
Concurrent Distributed Job Queue is a high-reliability background task execution engine written in Go and PostgreSQL. Designed to coordinate asynchronous background work across multiple concurrent worker goroutines without requiring external broker infrastructure like RabbitMQ or Redis.
Database-backed queues suffer row lock contention when multiple workers poll for pending tasks at the same time: without SKIP LOCKED, workers serialize behind each other on the same candidate rows instead of working in parallel.
System Architecture & Data Pipeline
Producers enqueue tasks into PostgreSQL via HTTP POST. Concurrent worker goroutines poll pending tasks using SELECT FOR UPDATE SKIP LOCKED, atomically claiming jobs with zero lock contention.
HTTP Producer API
Enqueues background jobs with priority levels, JSON payloads, and scheduled run_at timestamps.
Queue Engine (SKIP LOCKED)
Executes SELECT FOR UPDATE SKIP LOCKED queries to claim pending jobs atomically without blocking other worker goroutines.
Worker Pool Dispatcher
Manages N worker goroutines, job context timeout cancellation, and graceful shutdown signal handling.
Orphan Reaper
Periodically sweeps jobs left in 'processing' past a configurable timeout and returns them to 'pending', recovering work abandoned by crashed workers.
System Design Trade-Offs
PostgreSQL FOR UPDATE SKIP LOCKED vs External Broker (RabbitMQ/Redis)
Engineering Rationale: Using PostgreSQL directly enables transactional enqueuing (job inserted in the exact same DB transaction as business domain data), preventing phantom job execution.
In-Process Worker Pools vs Separate Worker Binaries
Engineering Rationale: Embedding worker pools directly in Go with channels and sync primitives simplifies container orchestration and local development.
Database Schema & Indexing Strategy
Task state machine schema with composite partial indexes on status, priority, and run_at timestamp for sub-millisecond dequeue lookups.
CREATE TABLE jobs (id UUID PRIMARY KEY, payload JSONB NOT NULL, status VARCHAR(20) NOT NULL DEFAULT 'pending', priority INT NOT NULL DEFAULT 0, attempts INT NOT NULL DEFAULT 0, max_attempts INT NOT NULL DEFAULT 5, locked_by UUID, locked_until TIMESTAMPTZ, run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), created_at TIMESTAMPTZ DEFAULT NOW());CREATE INDEX idx_jobs_dequeue ON jobs (status, priority DESC, run_at) WHERE status = 'pending';Technology Stack Justifications
Go (Golang)
Native channels, select statements, and lightweight goroutines provide robust worker pool primitives.
PostgreSQL 9.5+
Native support for FOR UPDATE SKIP LOCKED enabling lock-free concurrent row selection.
REST API Specification
| Method | Endpoint Path | Semantics & Status |
|---|---|---|
| POST | /jobs | Enqueue new background job with payload, priority, and schedule |
| GET | /jobs/:id | Query job execution status, attempts count, and result |
| GET | /jobs | List jobs with status/kind filtering and keyset pagination |
| GET | /health | Liveness probe |
Production Takeaways & Next Steps
Engineering Lessons
- •SELECT ... FOR UPDATE SKIP LOCKED lets each worker claim a row without blocking on rows another worker already holds, so adding workers adds throughput instead of lock contention. An integration test asserts that exactly one worker claims a given job.
- •Recovering crashed workers by sweeping stale claims on a timeout is far simpler than a heartbeat protocol, and it degrades safely: the worst case is a job running twice, which is why processors must be idempotent.
Future Improvements
- •Retry scheduling: increment retry_count and reschedule via next_run_at with exponential backoff and full jitter (currently Fail is terminal).
- •Dead-letter queue for jobs that exhaust their retry budget (MoveToDLQ is defined in the repository interface but not yet implemented).
- •Prometheus /metrics endpoint exposing queue depth and processing rates.