Refactor: Improve queue and redis client robustness

Commit notes

This commit introduces significant improvements to the queue system and Redis client.

Queue System Enhancements: - **v2 Architecture**: Rebuilt the queue system with a more robust and efficient architecture using Redis LISTs for waiting jobs and ZSETs for delayed/processing/dead jobs. This eliminates polling and uses BRPOP for efficient blocking. - **Improved Job Lifecycle**: Implemented proper lifecycle management for jobs, including atomic operations for creation, completion, and failure. - **TTL for Job Data**: Added Time-To-Live (TTL) for completed, failed, and dead job data to prevent Redis key accumulation. - **Dedicated Blocking Connection**: Introduced a dedicated Redis connection for blocking operations (BRPOP) to prevent blocking other Redis commands. - **Enhanced Options**: Added new queue options like `completedJobTTL`, `failedJobTTL`, and `blockTimeout`. - **Stalled Job Detection**: Improved stalled job detection to be more accurate. - **Housekeeping**: Added a `runQueueHousekeeping` function to clean up old job data. - **Singleton Queues**: Updated singleton queue configurations with new options and defaults.

Redis Client Enhancements: - **Circuit Breaker**: Implemented a circuit breaker pattern to gracefully handle Redis connection issues and prevent cascading failures. - **Improved Retry Strategy**: Enhanced the retry strategy with exponential backoff and jitter for more resilient reconnections. - **Connection Settings**: Tuned connection settings (timeouts, keep-alive, auto-pipelining) for better performance and stability. - **Reduced Log Spam**: Minimized log output for routine reconnection events and errors. - **Health Stats**: Added more detailed Redis health statistics, including circuit breaker status. - **`withRedisRetry`**: Introduced a utility function for executing Redis operations with built-in retry logic and circuit breaker awareness. - **`getRedisInfo`**: Added a function to retrieve Redis server information.

Worker Process Improvements: - **Health Check**: Enhanced the health check endpoint to include Redis status, circuit breaker status, and more detailed queue worker information. - **Graceful Shutdown**: Improved graceful shutdown handling for queues and connections. - **Metrics**: Added interval for updating queue metrics and running housekeeping. - **Error Handling**: Refined error handling for uncaught exceptions and unhandled rejections to prevent worker crashes.

Co-authored-by: armin.naimi <[email redacted]>