Plain-language explainer
What is robots.txt?
robots.txt is a file at the root of a site that tells crawlers which paths they may or may not crawl. It does not remove pages from search results by itself.
The work to plan together
A scope shaped around this requirement.
- Blocks crawling, not indexingScope
- Use noindex to keep pages out ofReview
- Check you are not blocking important pagesCheck
The useful starting point
robots.txt controls requests by compliant crawlers. It does not reliably hide a known URL from search results, and blocking a page can prevent the crawler from seeing a noindex instruction on that page.
What to review
- Blocks crawling, not indexing
- Use noindex to keep pages out of results
- Check you are not blocking important pages
- Can allow or block AI crawlers
What to avoid or interpret carefully
The file is public and is not a security barrier. Different AI crawlers have different purposes, so review each policy deliberately.
Practical detail
Separate crawl control from indexing and security
An illustrative robots.txt rule can keep compliant crawlers away from a low-value path. It is not a password or a reliable way to hide private information.
| Part of the work | Example output | How to review it |
|---|---|---|
| Crawl rule | User-agent: * / Disallow: /internal-search/ | The slash-separated text here represents separate lines in the file. |
| Index exclusion | A supported noindex directive readable to the crawler | Do not block crawling and then assume the noindex will be read. |
| Private data | Authentication and appropriate access control | robots.txt is public and does not protect the resource. |
| Validation | Check the root file, matching path and important pages | An unintended rule can block useful content. |
Example file content is two lines: User-agent: *, then Disallow: /internal-search/. A blocked URL may still be known from links. Apply the control that matches the actual goal.
Reference guidance: Google robots.txt guidance.
Use crawl rules for their intended purpose
Robots rules manage crawler access to paths. They are not a reliable way to keep sensitive information private, and a blocked URL can still appear in search without its content being crawled. Separate crawl control, indexing and access security.
Identify the target
Review the exact host, path and crawler affected. A broad rule can block useful pages unintentionally. Test representative URLs before changing a live file.
Choose the right control
If the issue is private data, use appropriate access protection. If the aim is indexing control, review the supported method and whether a crawler can see it. One file cannot solve every visibility problem.
Validate the change
Inspect the published rules and important pages after release. Include staging and production differences. Keep a rollback record so an accidental broad restriction can be corrected promptly.
How to interpret this guidance
A migration or redesign can accidentally carry restrictive staging rules into production. Include that check in release sign-off rather than assuming the file never changes.
Can robots.txt protect customer documents?
No. Sensitive material needs appropriate authentication and access controls. Crawl instructions are not a substitute for security. Review private files with the technical and data owner.
Can robots.txt remove a page already in Google?
Not by itself. Use the appropriate indexing/removal route and allow the crawler to observe the relevant instruction.
Can Tangensys help with the next step?
Yes. Share your website, goal and the issue you are investigating. We can discuss the relevant service and provide a custom quote within 2 working days.
Further reading
Consult the relevant platform documentation for current requirements: official documentation. Examples explain an approach; platform interfaces and policies can change.
Related reading and services
Let’s agree the right next step.
Tell us your goals and current setup. We’ll recommend a practical starting point and provide a custom quote.
Custom quote within 2 working days