robots.txt and sitemaps solve different problems.
Separate discovery from access
A sitemap offers a list of URLs you want a crawler to discover. robots.txt gives cooperative crawlers instructions about fetching paths. Neither is a password system. A private invoice needs authentication, not just a robots rule. Start by classifying public content, internal utility pages and genuinely private resources.
Build a deliberate crawl rule
Suppose your public site has /guides/ and an internal search route /search/. A basic Disallow: /search/ under User-agent: * asks that group not to fetch that path. Check exact path prefixes before deploying. Blocking a directory that also contains CSS or images may interfere with rendering. An empty Disallow in the generated group places no restriction on crawling.
Do not confuse blocking with removal
A robots block can prevent a crawler from seeing a noindex directive on the blocked page. It is therefore not a reliable way to remove a URL from search results. If removal is the task, read the search engine’s current guidance and choose the appropriate indexing or authentication controls. Do not assume every bot will obey the same rules.
Curate the sitemap list
Export canonical public URLs from the system you actually publish with. Remove redirects, fragments, private pages and duplicate query variants. The builder requires a single origin and HTTPS and removes duplicate normalized strings. Those checks do not establish that each URL responds successfully. Test the deployed list separately. Avoid assigning today’s date to every page unless each page really changed.
Deploy and verify separately
Serve robots.txt from the host root and publish the XML at a stable address. Open both files as a visitor and confirm the server returns the intended content rather than a login screen or HTML error. Add the sitemap through your search engine’s webmaster tools where appropriate. Discovery is an invitation: a submitted sitemap never guarantees indexing. Revisit both files whenever routes or access rules change.