Kubernetes containerization headache

we're running a fairly complex microservices architecture, mostly stateless, but a few critical services are stateful, on a self-managed Kubernetes cluster. we're facing some perplexing performance dips and resource over-provisioning issues, especially with burst traffic. our goal was to leverage containerization for efficiency, but it's not quite working out.

• The Problem: Specifically, we're seeing HPA (Horizontal Pod Autoscaler) react too slowly or over-scale for short bursts, leading to either latency spikes or unnecessary cloud spend. for our stateful services (e.g., a custom caching layer, a message queue), we're struggling to find the right balance for resource requests and limits. we've tried various metrics for HPA (CPU, memory, custom metrics via Prometheus), but nothing feels optimal. the biggest headache is ensuring consistent performance without massive over-provisioning during off-peak hours, while still being able to handle sudden traffic surges smoothly. it's really making our container orchestration efforts a bit of a nightmare.
• What We've Explored:
  • Tuned HPA stabilizationWindowSeconds and targetAverageUtilization.
  • Experimented with different resource.requests and resource.limits values, including Guaranteed QoS classes for critical pods.
  • Looked into Vertical Pod Autoscaler (VPA) but that interferes with HPA.
  • Considered KEDA for event-driven autoscaling, but it feels like overkill for our current setup and adds complexity.
• Seeking Advice: Hoping someone with deep Kubernetes containerization experience has tackled similar challenges, especially with mixed stateless/stateful workloads. any insights on advanced HPA configurations, or alternative autoscaling strategies that play nice with stateful sets, would be immensely helpful. really waiting for an expert reply on this one.