Struggling with Horizon worker management after Laravel deployments

Hey everyone, I was here a couple of weeks ago asking for help with some persistent Laravel queue failures after a recent deployment. Thanks to some great advice, we managed to iron out most of those initial bugs related to job serialization and environment caching. Things were looking good for a bit, but now we're hitting a new wall, and it's specifically around Horizon and how it manages our workers post-deployment.

The issue now is that after almost every deployment, our Horizon worker management becomes incredibly inconsistent. It's like the workers just don't want to play nice. We're seeing situations where they don't scale correctly, jobs get stuck in a pending state even when there's capacity, or conversely, we get unexpected resource spikes with workers seemingly doing nothing useful. It's really impacting our background processing and user experience because critical tasks are either delayed or fail silently.

We've tried a bunch of things to get a handle on this. We've gone through our Horizon configuration multiple times, experimenting with different balance strategies, adjusting max_processes and max_time settings, and even trying tries and timeout values more aggressively. We're constantly checking server metrics – CPU, memory, I/O – to see if there's an underlying infrastructure bottleneck, but usually, the server itself seems fine; it's just Horizon misbehaving. We've also resorted to manual restarts of the Horizon supervisor after deployments, which sometimes helps, but it's not a reliable or automated solution, and definitely not ideal for a CI/CD pipeline.

The symptoms are pretty clear: in the Horizon dashboard, we'll see jobs accumulating rapidly in the 'Pending' queue, while the worker list shows many workers as 'inactive' or 'paused' despite having plenty of jobs to process. Other times, we'll see high CPU usage from the Horizon process itself, but the throughput of actual job completion is minimal. It's like the workers are busy doing nothing, or they're just not picking up new tasks efficiently. Sometimes, a full server reboot or a complete re-provisioning of the supervisor process is the only thing that kicks it back into gear, which is obviously a last resort.

So, my main question is, what are the best practices for robust Laravel worker management with Horizon, especially concerning deployment strategies? How do you guys ensure your Horizon workers come up gracefully and consistently after a new code push without these kinds of hiccups? Are there specific deployment scripts or supervisor configurations that are more resilient?

I'm really looking for any insights or proven methodologies here. Anyone faced this before and found a solid, repeatable solution?