Every step,
from the first commit.

This commit updates the worker health check to use an HTTP endpoint and replaces Redis KEYS with SCAN for better performance and safety. Co-authored-by: armin.naimi <[email redacted]>

This commit introduces significant improvements to the queue system and Redis client. Queue System Enhancements: - **v2 Architecture**: Rebuilt the queue system with a more robust and efficient architecture using Redis LISTs for waiting jobs and ZSETs for delayed/processing/dead jobs. This eliminates polling and uses BRPOP for efficient blocking. - **Improved Job Lifecycle**: Implemented proper lifecycle management for jobs, including atomic operations for creation, completion, and failure. - **TTL for Job Data**: Added Time-To-Live (TTL) for completed, failed, and dead job data to prevent Redis key accumulation. - **Dedicated Blocking Connection**: Introduced a dedicated Redis connection for blocking operations (BRPOP) to prevent blocking other Redis commands. - **Enhanced Options**: Added new queue options like `completedJobTTL`, `failedJobTTL`, and `blockTimeout`. - **Stalled Job Detection**: Improved stalled job detection to be more accurate. - **Housekeeping**: Added a `runQueueHousekeeping` function to clean up old job data. - **Singleton Queues**: Updated singleton queue configurations with new options and defaults. Redis Client Enhancements: - **Circuit Breaker**: Implemented a circuit breaker pattern to gracefully handle Redis connection issues and prevent cascading failures. - **Improved Retry Strategy**: Enhanced the retry strategy with exponential backoff and jitter for more resilient reconnections. - **Connection Settings**: Tuned connection settings (timeouts, keep-alive, auto-pipelining) for better performance and stability. - **Reduced Log Spam**: Minimized log output for routine reconnection events and errors. - **Health Stats**: Added more detailed Redis health statistics, including circuit breaker status. - **`withRedisRetry`**: Introduced a utility function for executing Redis operations with built-in retry logic and circuit breaker awareness. - **`getRedisInfo`**: Added a function to retrieve Redis server information. Worker Process Improvements: - **Health Check**: Enhanced the health check endpoint to include Redis status, circuit breaker status, and more detailed queue worker information. - **Graceful Shutdown**: Improved graceful shutdown handling for queues and connections. - **Metrics**: Added interval for updating queue metrics and running housekeeping. - **Error Handling**: Refined error handling for uncaught exceptions and unhandled rejections to prevent worker crashes. Co-authored-by: armin.naimi <[email redacted]>

This commit introduces several improvements to the worker and queue modules, including: - **Eager Redis Connection:** The worker now connects to Redis eagerly at startup, ensuring that it can detect connection issues early. - **Worker Health Updates:** Added a periodic health update mechanism for workers in Redis, allowing for better monitoring of active jobs. - **Error Handling Enhancements:** Implemented backoff strategies for job processing in the queue, reducing pressure on Redis during errors and improving resilience. - **Polling Adjustments:** Adjusted polling intervals and concurrency settings for various queues to optimize performance and resource usage. These changes aim to improve the reliability and efficiency of the worker and queue systems.

This commit updates the `SimpleUploadButton` component to incorporate a disclosure mechanism for better UI interaction. Key changes include: - **Disclosure Hook:** Added `useDisclosure` to manage the open/close state of the menu. - **Menu State Management:** Updated the `Menu` component to reflect the disclosure state, allowing for improved user experience. - **File Handling Improvements:** Ensured that the menu closes after file selection or folder change, enhancing usability. These changes aim to streamline the file upload process and improve the overall user interface.

This commit introduces several improvements to the worker module, including: - **Health Check Endpoint:** Added a new HTTP endpoint (`/health`) to monitor the worker's status and health. - **Graceful Shutdown:** Implemented a mechanism for graceful shutdown, allowing the worker to finish processing current jobs before stopping. - **Heartbeat Mechanism:** Introduced a heartbeat-based lock extension for long-running jobs to prevent job timeouts. - **Automatic Stalled Job Recovery:** Enhanced job recovery capabilities to automatically handle stalled jobs, ensuring they are re-queued or marked as dead if necessary. Additionally, various code refactorings were made for consistency and clarity, including updates to import statements and logging improvements.

This commit refactors the `SimpleUploadButton` component to improve file selection and handling for audio uploads. Key changes include: - **File Selection Logic:** Introduced a new method to filter audio files based on their type and extension, ensuring only valid audio files are processed. - **Folder Upload Support:** Enhanced the component to handle folder uploads, allowing users to select folders containing audio files. - **UI Improvements:** Updated the UI to use a `Menu` for file and folder selection, replacing the previous `Popover` and `Dropzone` implementation for a more streamlined experience. - **Removed Unused Code:** Cleaned up the component by removing unnecessary state management and upload logic, focusing on the file selection aspect. Additionally, the `LibraryPage` was updated to reflect the new `onFilesSelected` prop for handling selected files.

This commit downgrades the lockfile version in `pnpm-lock.yaml` for compatibility and introduces a job recovery mechanism in the queue system. Key changes include: - **Lockfile Update:** Downgraded `lockfileVersion` to '6.0' in `pnpm-lock.yaml`. - **Job Recovery:** Implemented `recoverStaleJobs` method in the queue class to handle stale jobs on startup, ensuring that jobs stuck in processing are moved back to the waiting queue or marked as dead if max attempts are reached. - **Logging Enhancements:** Added logging for recovered jobs to improve visibility during job recovery. Additionally, several unused SQL files and snapshots have been removed to streamline the codebase.

This commit refactors various components to improve code structure, enhance UI elements, and fix minor issues. Key changes include: - **Component Refactoring:** Several components like `ErrorBoundary`, `CreatorAnalytics`, `AccountPopover`, `AuthSection`, `BarControls`, `BpmControl`, `StorageDashboard`, `GainMeter`, `GrooveIndicator`, `Header`, `KeyboardShortcuts`, `FilterBar`, `PacksPanel`, `SimpleUploadButton`, `TracksList`, `WaveformPlayer`, `PackManager`, `SampleTreeView`, `PatternControls`, `PatternManager`, `UserPatternManager`, `PitchControl`, `DivisionIcons`, `GrooveControls`, `PatternMini`, `SampleSelectorPopover`, `SequencerHotkeys`, and `SongArranger` have been refactored. This includes updating imports, adjusting component logic, and improving rendering efficiency. - **UI Enhancements:** Minor UI improvements have been made across several components, such as adjusting styling, improving accessibility, and refining user interactions. - **Bug Fixes:** Addressed minor bugs and inconsistencies within components to ensure smoother operation. - **Dependency Updates:** Updated imports and dependencies where necessary to align with project standards. Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

This commit introduces several new features related to collaboration and notifications, including workspaces, comments, collections, activity feeds, and notifications. It also includes updates to audio analysis and file generation. Co-authored-by: armin.naimi <[email redacted]>

This commit downgrades the lockfile version in `pnpm-lock.yaml` to ensure compatibility with existing dependencies. It also modifies the `docker-compose.yml` to use more descriptive environment variable names for PostgreSQL configuration. Additionally, several unused imports and components have been removed from the frontend code, streamlining the codebase and improving maintainability.

This commit updates various dependencies, including Mantine and TanStack libraries, to their latest versions. It also includes significant refactoring to normalize data structures fetched from the API, ensuring consistency across the application. Key changes include renaming properties to camelCase, improving type safety, and updating component logic to reflect these changes. Additionally, several minor bug fixes and UI improvements have been implemented. Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

Removes unused Mantine packages, route definitions, and Vite configurations. Co-authored-by: armin.naimi <[email redacted]>

This commit deletes the UsersDashboard component from the frontend, which was responsible for displaying user metrics and a searchable user table. The removal is part of a larger refactor to streamline the user interface and improve overall code maintainability.

- Updated various dependencies in `bun.lock` and `pnpm-lock.yaml` for better compatibility and performance. - Modified `docker-compose.yml` to use more generic environment variable names for database configuration. - Refactored Dockerfiles to streamline the installation process by removing unnecessary lockfile copies. - Added new SQL migration file to enhance the `samples` table with additional columns for audio analysis. - Implemented cleanup logic for stale chunked upload sessions in the backend. - Enhanced error handling and validation in audio processing functions. This commit aims to improve the overall stability and maintainability of the project.

Co-authored-by: armin.naimi <[email redacted]>

This commit introduces several enhancements: - **Similarity Search:** - Added a new endpoint `/samples/similar/:id` to find similar samples using vector embeddings. - Integrated pgvector for efficient similarity search, with a JavaScript fallback. - Added database schema changes for the `embedding_vector` column and indexing. - **Audio Analysis:** - Improved audio analysis accuracy by implementing DC offset removal and a more robust true peak calculation. - Enhanced confidence scoring for BPM and musical key detection. - Added spectral feature calculations (centroid, bandwidth, rolloff, flatness, flux, ZCR). - Normalized embeddings to unit length for cosine similarity. - Introduced content hashing (SHA-256) for sample deduplication and optimized analysis. - Added file size and duration limits for audio uploads. - Improved worker startup checks for FFmpeg availability. - Added comprehensive unit tests for audio analysis algorithms. Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

This commit introduces several improvements: - **Startup Verification:** Added a `verifyDependencies` function to ensure critical services like the database and Redis are available before the server starts. This prevents the server from running in a degraded state. - **Graceful Shutdown:** Implemented timeouts for both startup verification and graceful shutdown to prevent the server from hanging indefinitely. - **Scheduled Cleanup:** Introduced a new module `scheduled-cleanup.ts` for managing periodic cleanup tasks. - `cleanupExpiredOAuthStates`: Removes expired OAuth state entries from the database. - `cleanupExpiredRefreshTokens`: Removes expired refresh tokens. - `runAllCleanupTasks`: A utility to run all cleanup tasks. - **OAuth State Cleanup:** The server now schedules `cleanupExpiredOAuthStates` to run every 5 minutes. - **Refresh Token Limit:** Enforced a `MAX_REFRESH_TOKENS_PER_USER` limit (set to 10) during user login. If a user exceeds this limit, the oldest tokens are automatically revoked. - **File Upload Limit:** Added a `MAX_CHUNK_SIZE` check for multipart/form-data requests to prevent excessively large file uploads. - **Configuration Checks:** Added checks for necessary Google OAuth configuration variables before initiating the OAuth callback. - **Code Organization:** Refactored the main `index.ts` file to better organize startup, shutdown, and dependency verification logic. - **Exported Constant:** Exported `MAX_REFRESH_TOKENS_PER_USER` from `refresh-token.ts` to be used in `auth.ts`. Co-authored-by: armin.naimi <[email redacted]>

This commit includes several improvements: - **Dockerfiles:** Optimized Dockerfiles for both the API and worker, using alpine images and multi-stage builds for smaller image sizes. Added non-root users for enhanced security. - **docker-compose.yml:** Refactored docker-compose to use environment variables for configuration, making it more flexible and secure. Added health checks for all services. - **Auth:** - Implemented secure refresh token generation with a limit per user. - Improved Google OAuth flow to use transactions for user creation/linking and token generation. - Added secure cookie handling for tokens in production and URL parameters in development for OAuth callbacks. - **Error Handling:** Enhanced error handling in the main app to gracefully manage PayloadTooLargeError and RateLimitError. - **Logging:** Improved logging for requests, responses, and errors. - **Rate Limiting:** Integrated a rate limiting middleware. - **CSRF Protection:** Added CSRF protection middleware. - **Database Migrations:** Updated Drizzle metadata to reflect schema changes. - **Queue:** Corrected type parsing for queue job data. - **Redis:** Fixed type inference for rate limiting. - **Admin Routes:** Improved error handling for preview and sequencer generation. Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

This commit introduces comprehensive logging with Pino, request correlation, and metrics endpoints. It also enhances the health check and adds optional OpenTelemetry export. Database connection pooling is also configured for better performance. Co-authored-by: armin.naimi <[email redacted]>

Co-authored-by: armin.naimi <[email redacted]>

Fix background analysis and filter

This commit introduces audio analysis features, including BPM, key, genre, and category extraction. It also enhances the filtering system in the library to utilize these new attributes, improving sample discoverability. The analysis runs in the background to avoid blocking the upload process. Co-authored-by: armin.naimi <[email redacted]>

- Created a README.md file for the backend, detailing the Bun + Elysia API setup, database configuration, and migration instructions. - Added initial SQL migration scripts for database schema, including the creation of essential tables and types for analytics and audio management. - Introduced metadata journal and snapshot files for tracking database migrations. - Updated Redis upload limits to enhance concurrent upload capabilities. This commit establishes foundational documentation and database structure for the backend, improving developer onboarding and system organization.

- Upgraded `biome.json` schema version and enabled version control system support. - Modified `package.json` to update dependencies and adjust service configurations for the backend. - Introduced new Docker and Docker Compose files for backend services, including PostgreSQL and MinIO. - Added `.dockerignore` and `.gitignore` files to optimize build processes and exclude unnecessary files. - Created initial backend structure with essential configuration files, including `bun.lock`, `drizzle.config.ts`, and environment setup. - Implemented various routes for user authentication, sample management, and audio processing, enhancing the backend API functionality. This commit establishes a comprehensive backend setup, improving development workflows and expanding feature capabilities.

- Introduced a new `Dockerfile` for building and running the Bun application, including multi-stage builds for dependencies and production runtime. - Created `docker-compose.yml` to define services for PostgreSQL, Redis, and MinIO, with health checks and environment configurations for each service. - Added `.dockerignore` to exclude unnecessary files from the Docker context, improving build efficiency and security. - Included a `pnpm-lock.yaml` file for dependency management, ensuring consistent installations across environments. This commit establishes a robust backend infrastructure for the Bun application, facilitating development and deployment through containerization.

- Deleted various markdown files related to deployment, authentication, and integration guides that are no longer relevant to the current project structure. - Removed outdated configuration files and environment examples to streamline the codebase and reduce clutter. - Updated `package.json` and `pnpm-workspace.yaml` to reflect changes in workspace structure, removing references to deleted components. This commit aims to clean up the repository by eliminating unnecessary files and ensuring that the remaining documentation is relevant and up-to-date.