my free XML sitemap generator keeps showing 'file not found' errors, affecting website indexing
hey everyone, so i'm pretty new to this whole SaaS thing and just launched my little web tool, a free XML sitemap generator. it's supposed to help peaple with their website indexing by making sitemaps super easy.
but i'm running into a really frustrating problem. sometimes, when users try to generate a sitemap for their site, it just throws up a 'file not found' error. it's not consistent, which makes it even harder to debug. it's like it can't access the target website's root sometimes, or maybe it's something with specific URL structures?
here's what the console output often looks like:
GET /generate?url=https://example.com/sitemap.xml 404 (Not Found) Error: Could not retrieve resource at https://example.com/robots.txt at generateSitemap (sitemap-generator.js:123:19) at processRequest (server.js:45:7)i'm totally stumped. is this a common issue with sitemap generators? like, server permissions, or maybe how i'm handling external requests? anyone faced this before?
2 Answers
Salma Abdullah
Answered 1 week agoThat specific 'file not found' error, especially onsometimes, when users try to generate a sitemap for their site, it just throws up a 'file not found' error.
robots.txt, often points to your generator being blocked by the target server's firewall or encountering DNS resolution issues. Ensure your HTTP client handles network timeouts and varying user-agent strings to reliably crawl their site structure and manage your own crawl budget. Are you logging specific HTTP status codes beyond the 404 from the target servers?Valentina Martinez
Answered 1 week agoThat's a good call on the robots.txt error often being firewall or DNS related. And definitely going to dig into logging specific HTTP status codes more thoroughly. Cheers.