Experiencing critical cPanel daemon failures after recent WHM server administration updates, seeking diagnostic strategies.

Hey Alejandro Hernandez,

I understand how frustrating these types of intermittent daemon failures can be, especially after a critical update to your server administration systems. I've certainly dealt with similar head-scratching issues post-update on client servers, where the logs just abruptly end, leaving you guessing. Given your thorough initial diagnostics, it's clear we need to dig into some deeper layers.

Here’s a more advanced diagnostic strategy for these cPanel/WHM v110.0.x daemon failures on CentOS 7 and AlmaLinux 8 systems:

1. Leverage journalctl More Aggressively: While you've checked dmesg, journalctl is the central logging facility for systemd. Instead of just general checks, try correlating timestamps.
   • Monitor actively for specific service units: journalctl -fu cpdavd.service, journalctl -fu cpanellogd.service, journalctl -fu httpd.service, journalctl -fu mysql.service.
   • Look for messages immediately preceding the "terminated" event. Pay close attention to any systemd messages about the service failing or being killed, as well as any kernel messages that might not make it to dmesg immediately.
   • Use journalctl -p err -b -1 to review errors from the previous boot, or journalctl --since "1 hour ago" to focus on recent events.
2. Enable cPanel Internal Debugging: cPanel has internal debug levels.
   • Edit /var/cpanel/cpanel.config and look for debug_mode. If it's not there or set to 0, add/change it to debug_mode=1.
   • You can also adjust the error logging level. For instance, to increase the verbosity of cpanellogd, you might need to find its specific configuration (often in /usr/local/cpanel/etc/cpanellogd.conf or similar) and set a higher log level if available. For cpdavd, check its configuration for a log level option.
   • After modifying cpanel.config, run /usr/local/cpanel/scripts/restartsrv_cpanel to apply changes.
3. Check systemd Service Unit Files and Resource Limits:
   • Examine the actual systemd unit files for the affected services. For example, systemctl status httpd will show you the path to its unit file (e.g., /usr/lib/systemd/system/httpd.service).
   • Look for directives like LimitNOFILE, LimitNPROC, MemoryLimit, or TimeoutStopSec. If these were overridden locally (e.g., in /etc/systemd/system/httpd.service.d/*.conf) or if the cPanel update changed the default values, it could be causing services to be killed.
   • Verify the User and Group directives in these unit files match what cPanel expects, especially for services like cpdavd.
4. strace or lsof for Dying Daemons: This is more advanced but can be very telling.
   • If you can catch a daemon just before it dies, attach strace -p <PID> to it. This will show you the system calls it's making. Look for anything unusual, like failed file operations, memory allocation issues, or signals received.
   • Alternatively, run lsof -p <PID> to see all open files and network connections. Sometimes, a daemon might be trying to access a file or library that no longer exists or has incorrect permissions after an update.
5. Library Mismatches and cPanel's Perl Environment: Your intuition about compiler flags or library mismatches is valid.
   • cPanel heavily relies on its own bundled Perl environment. After a major WHM update, there could be an issue with how Perl modules are compiled or linked.
   • Try rebuilding cPanel's Perl environment: /usr/local/cpanel/scripts/build_cpanel_perl.
   • Use ldd /usr/local/cpanel/sbin/cpdavd (or the binary for other failing services) to check its shared library dependencies. Look for any "not found" or unusual library paths.
   • Ensure all cPanel-managed RPMs are consistent: /usr/local/cpanel/scripts/check_cpanel_rpms --fix.
6. SELinux Policy Issues: While you'd typically see specific audit logs, SELinux can sometimes silently block operations, leading to service failures.
   • Check audit.log for any AVC denials around the time of the daemon failures: grep AVC /var/log/audit/audit.log.
   • Temporarily set SELinux to permissive mode (setenforce 0) to see if the issue resolves. If it does, you'll need to generate a custom SELinux policy or report it to cPanel. Remember to set it back to enforcing (setenforce 1) afterward.
7. Review cPanel Update Logs: Sometimes the update process itself logs errors that don't make it to general error logs.
   • Check /var/cpanel/updatelogs/update.<TIMESTAMP>.log for the specific update that initiated these issues. Look for any warnings or errors during the update stages.

Given the complexity and the fact that this is affecting multiple client sites under your web hosting management, I highly recommend opening a direct support ticket with cPanel themselves. Provide them with all the diagnostic steps you've already taken. They have internal tools and insights into their specific release changes that are often invaluable for pinpointing these kinds of post-update regressions.

Hope this helps you get those services stabilized!

Oh nice! Tbh this is exactly what I needed, thanks a ton!

The systemd unit file check combined with ldd pointed me right to a library mismatch, everything's stable now.