XML sitemap generation for huge sites?

I'm imagining what happens when we reach 100k, 500k, or even a million pages.

That's a very common scaling headache, and you're wise to address it now rather than when your servers are screaming. For truly large sites, relying on external online tools or simple WordPress plugins quickly becomes impractical for XML sitemap generation. The core strategy shifts from manual or plugin-based generation to programmatic, integrated solutions.

First, you absolutely need to leverage sitemap index files. Instead of one massive sitemap, you'll break your URLs into multiple sitemaps, each containing up to 50,000 URLs, and then reference them all in a single sitemap index file (e.g., sitemap.xml pointing to sitemap_1.xml, sitemap_2.xml, etc.). This allows search engines to process them efficiently and helps you manage updates.

For the generation itself, move away from external tools. The most robust approach is dynamic sitemap generation directly from your application. This typically involves:

• Dedicated Microservice or Serverless Function: This is highly recommended. Running your sitemap generation as a separate process (e.g., an AWS Lambda function, Google Cloud Function, or a dedicated containerized service) isolates the CPU and memory load from your primary application servers. This prevents performance bottlenecks on your main site during generation.
• Database-Driven Generation: Your application's database is the source of truth for your URLs. A script or service can query your database for all relevant URLs, structure them into sitemap files, and then upload them to your web server or a CDN.
• Open-Source Libraries: Most modern web frameworks (Python/Django/Flask, Ruby on Rails, Node.js/Express, PHP/Laravel) have battle-tested libraries or easy ways to integrate custom scripts for generating sitemaps from your data models. You're building this logic into your application stack, not relying on a third-party website to crawl yours.
• Handling Real-time Updates: For frequently changing content, trigger partial sitemap regeneration. When a page is published, updated, or deleted, an event can be queued. Your dedicated sitemap generation service then only needs to update the specific sitemap file(s) affected, rather than regenerating everything. A daily or weekly full regeneration can act as a catch-all, but event-driven updates are key for freshness.

Regarding hosting, the primary consideration is isolating the generation process, as mentioned with microservices. This means your main web servers remain focused on serving users, while a separate, scalable compute environment handles the heavy lifting of building sitemaps. While I can't recommend an internal tool here, many enterprise SEO platforms like Botify or DeepCrawl offer advanced crawling and sitemap analysis capabilities, but the actual generation of sitemaps for millions of URLs is typically an internal process driven by your own content management system or application logic.

Hope this helps your conversions!

Ah, got it. The sitemap index files and dynamic generation directly from our application's database makes a lot more sense than relying on external tools. And a dedicated microservice for generation, especially for those event-driven partial updates, seems like the most robust way to keep things fresh without hammering our main servers.