Horizon queue processing challenge

Hey Mason Moore, I completely get where you're coming from. The scenario you're describing is a classic for any growing SaaS application, and it's something many of us have wrestled with. Ah, the classic 'nightmare' scenario – though I prefer to call it a 'challenging optimization puzzle' myself, makes it sound less like something from a horror movie and more like a fun weekend project, right? You're right to focus on this, as inefficient queue management can quickly lead to resource waste or, worse, critical jobs being delayed. Your specific concern:

I'm finding it tough to ensure high-priority jobs don't get starved while also avoiding over-provisioning resources for jobs that aren't always active.

This is the core challenge, and Laravel Horizon provides excellent tools to address it, primarily through strategic supervisor and queue configuration. Here’s a breakdown of how we typically approach this for diverse workloads:

1. Implement a Granular Queue Separation Strategy

This is foundational. Instead of a single `default` queue, segment your jobs into logical queues based on their priority, execution time, and criticality. * High-Priority / Time-Sensitive: `high`, `notifications`, `webhooks` * Examples: User registration emails, password resets, real-time data processing, immediate server postbacks. These need to be processed ASAP. * Medium-Priority / General: `default`, `emails` * Examples: Marketing emails, less critical data synchronization, general background tasks. * Low-Priority / Long-Running: `reports`, `exports`, `data-imports` * Examples: Monthly report generation, large data exports, image processing. These can take minutes or even hours and should not block high-priority tasks. By separating them, you prevent a long-running report job from holding up an urgent password reset email.

2. Configure Horizon Supervisors Per Queue Group

Once you have your queues, you define specific supervisors in your `config/horizon.php` file. Each supervisor can manage one or more queues with distinct process settings. Here’s an example structure:

'supervisors' => [
    'default' => [
        'connection' => 'redis',
        'queue' => ['high', 'default', 'notifications'], // Process high first, then default, then notifications
        'balance' => 'auto', // Smart balancing for varied job lengths
        'min_processes' => 1,
        'max_processes' => 10, // Scale up for burst notifications/general tasks
        'tries' => 3,
        'timeout' => 60, // Shorter timeout for quick jobs
        'memory' => 128,
        'max_time' => 3600, // Restart workers every hour to prevent memory leaks
    ],

    'reports' => [
        'connection' => 'redis',
        'queue' => ['reports'],
        'balance' => 'false', // Or 'simple'. Don't use 'auto' for long-running jobs.
        'min_processes' => 1,
        'max_processes' => 2, // Keep this low to avoid over-provisioning for sporadic tasks
        'tries' => 1, // Often you don't want to retry failed reports aggressively
        'timeout' => 600, // Longer timeout for complex reports (10 minutes)
        'memory' => 512, // More memory for heavy tasks
        'max_time' => 7200, // Restart workers less frequently (2 hours)
    ],

    // Add more supervisors as needed, e.g., 'exports', 'data-sync'
],

Key Configuration Parameters: * `queue` array order: Horizon processes queues in the order they are listed. So, `['high', 'default']` means jobs in `high` will be picked before `default`. * `balance`: * `auto`: Ideal for queues with varied job lengths where you want Horizon to dynamically adjust worker assignments to keep throughput high. Great for `default` and `notifications`. * `simple`: Round-robin distribution. Good for queues with consistently similar job lengths. * `false`: Workers process jobs strictly FIFO. Use this for queues where order is critical or if you have very long-running jobs that could skew `auto` balancing. * `min_processes` & `max_processes`: This is your primary control for resource allocation within a supervisor. * For `high` priority queues, ensure `min_processes` is at least 1, perhaps 2-3, to guarantee immediate processing. `max_processes` can be higher for bursts. * For `reports` or `exports`, `min_processes` can be 0 (if you want workers to only spin up when jobs are present) or 1. `max_processes` should be low (1-2) to prevent these resource-intensive jobs from consuming too many server resources. * `timeout`: Crucial for long-running jobs. Set it appropriately for each job type. A notification email might timeout in 30 seconds, a report in 10 minutes. * `max_time`: Restarts workers after a certain time, preventing memory leaks, which is common in long-running PHP processes.

3. Auto-Scaling at the Server Level

While Horizon intelligently manages processes *within* a server, it doesn't auto-scale the underlying server instances. For true efficiency with fluctuating SaaS growth, you'll need to combine Horizon with cloud-provider auto-scaling solutions: * AWS: Use Auto Scaling Groups for EC2 instances, scaling based on SQS queue length metrics or CPU utilization. Alternatively, AWS Fargate/ECS services can scale based on queue depth. * Kubernetes: Leverage Horizontal Pod Autoscalers (HPA) to scale your worker pods based on queue metrics (e.g., from KEDA) or CPU/memory usage. * Other Clouds: Most major cloud providers offer similar auto-scaling capabilities for their compute instances or container services. This ensures you're not paying for idle servers when queue activity is low, but can quickly provision more when a large batch of jobs comes in.

4. Monitoring and Iteration

Horizon's dashboard (`/horizon`) is invaluable for monitoring queue lengths, job throughput, and failed jobs. Regularly review these metrics, especially during peak times, to identify bottlenecks or over-provisioned resources. Adjust your `min_processes`, `max_processes`, and queue assignments based on real-world data. This approach allows high-priority jobs to be processed quickly while giving long-running tasks the resources they need without starving other processes or breaking the bank. What kind of cloud infrastructure are you currently running your Laravel application on? Knowing that might help tailor further advice on auto-scaling.