Crawl Budget & Technical SEO

I'm facing a complex technical SEO challenge with my SaaS platform, which is growing rapidly but experiencing unexpected crawl budget allocation issues. My SaaS has a large, dynamic content base, including significant user-generated content (UGC), and we're trying to scale our content marketing efforts. The core problem is that Googlebot seems to be spending an inordinate amount of its crawl budget on low-value or outdated pages, despite our extensive efforts to guide it. This severely impacts the indexing speed and visibility of our new, high-priority content, creating a significant bottleneck in our overall technical SEO strategy. We've already meticulously optimized robots.txt, implemented dynamic XML sitemaps, used canonical tags extensively across relevant pages, and refined our internal linking structures to consolidate PageRank flow. Furthermore, we've diligently monitored Core Web Vitals and analyzed server logs for crawl patterns to identify any anomalies. However, despite these standard practices, we're still struggling with inefficient crawl budget distribution. How can we employ more advanced strategies to precisely re-allocate Googlebot's crawl budget? I'm particularly interested in techniques that go beyond the usual recommendations, especially for sites with a complex mix of dynamic and static content, and how to effectively prioritize the indexing of new content over older, less critical, but still indexed pages. Are there specific server-side configurations, header directives like X-Robots-Tag, or even machine learning approaches being used effectively by others to manage this at scale? Help a brother out please...