Refactor: Improve worker health checks and Redis SCAN usage

Commit notes

This commit updates the worker health check to use an HTTP endpoint and replaces Redis KEYS with SCAN for better performance and safety.

Co-authored-by: armin.naimi <[email redacted]>