Struggling with Dynamic Sitemap Generation for Large Site Indexation – Pages Not Crawling?

Hey everyone,

We're running a SaaS platform with a ton of user-generated content and product pages, so we rely heavily on dynamic sitemaps to keep Google updated on our ever-growing catalog. It's been crucial for our organic visibility.

Lately, though, we've noticed new pages or updated content taking ages to get discovered and properly indexed, which is really affecting our overall visibility and growth. It feels like our dynamic sitemap isn't as effective for timely crawling as it should be, especially given our rapidly increasing page count. We're starting to get concerned about potential issues with lastmod dates not being respected, or maybe even pagination within the sitemap itself causing problems for Googlebot. We want to ensure optimal indexation for all our valuable content.

Here’s a snapshot from Google Search Console that kinda illustrates the problem:

Google Search Console - Crawl Stats (Sample)

Total crawled pages (last 90 days): 1,500,000
Average daily crawled: 16,666
Pages discovered but not yet crawled: 850,000
Pages with crawl anomalies: 5,000
Crawl requests per day: Fairly stable
Time spent downloading a page: Increasing slightly

Given this, I'm really scratching my head. What are the common pitfalls in dynamic sitemap generation for large, frequently updated sites like ours? Are there specific strategies or tools you'd recommend to ensure faster indexation and more efficient crawling? How do you guys handle lastmod for content that changes frequently but subtly without triggering unnecessary recrawls or losing freshness signals?

Any advice would be a lifesaver. Help a brother out please...