Automated preventative maintenance failing

hey everyone, struggling with something kinda deep here on our dev ops side. we're running a pretty hefty stack, mostly kubernetes based microservices, serving high traffic, and naturally, we've got a whole suite of custom preventative maintenance scripts. these are mostly shell scripts and python services that handle stuff like log rotation, tmp file cleanup, database vacuuming, and various system health checks. they're all scheduled via cron or a custom python service we built for more complex orchestration across our clusters.

the issue is our automated preventative maintenance system is intermittently just... failing. we're seeing specific cleanup and log rotation scripts not completing, sometimes they just hang indefinitely, sometimes they exit silently mid-way without a trace. this is leading to resource issues, disk space bloat on worker nodes, memory accumulation from stale processes, and generally just makes our proactive maintenance unreliable.

i've gone through the usual troubleshooting steps, you know? first thing, verified cron logs, and yep, the jobs are definitely kicking off as scheduled. then i dove into the script logs, and they're weird – sometimes they show partial execution, other times absolutely nothing, like a silent failure, no stack trace, no error code dumped anywhere. i've been monitoring resource usage during the scheduled runs, thinking maybe a sudden spike, but there are no obvious cpu contention or memory spikes that would indicate a resource starvation issue. confirmed script permissions and user contexts, everything's spot on, the user running cron has all the necessary privileges. and this is the kicker, if i ssh into the exact same server and manually execute the very same script, it works perfectly, every single time, without fail. i've even implemented advanced error logging within the scripts themselves – python try-except blocks, shell scripts with set -e and trap ERR, redirecting all stderr to dedicated log files, but still, nothing conclusive when they fail automatically. i've also investigated potential race conditions, thinking maybe two processes hitting the same resource at the same microsecond, but i can't find any clear patterns or deadlocks. checked kernel logs, dmesg, journalctl, nothing about OOM killer or other system-level issues that would just kill a process without logging.

the specific technical block i'm hitting is that these scripts fail only when triggered by cron or our python scheduling service. this strongly points to some subtle environment difference, perhaps a transient resource lock that's just too quick to catch, or an elusive bug in our custom automation wrapper not passing specific environment variables, PATH settings, or perhaps even subtle differences in stdin/stdout handling. standard debugging tools like strace or lsof aren't helping much because these failures are so non-deterministic and incredibly hard to catch live. it's like a ghost in the machine.

i'm really scratching my head for unconventional debugging approaches or tools to expose these transient, non-deterministic failures in automated proactive maintenance scripts. anyone faced this before?