Plain-language explainer
What is crawl budget?
Crawl budget is how many pages a search engine is willing and able to crawl on a site in a given time. It mostly matters for large sites.
The work to plan together
A scope shaped around this requirement.
- Small sites rarely need to worryScope
- Duplicate and low-value URLs waste itReview
- Fast servers allow more crawlingCheck
The useful starting point
Crawl budget concerns how much of a site a search engine crawls. It usually deserves more attention on large sites with many changing or redundant URLs than on a small business website.
What to review
- Small sites rarely need to worry
- Duplicate and low-value URLs waste it
- Fast servers allow more crawling
- Use internal links and sitemaps well
A practical example
A store’s filters may create many combinations with little unique value. Review discovery and canonical handling before assuming the server needs more crawl capacity.
The numbers or scenarios in this guide are illustrative. They are not claimed Tangensys client results.
What to avoid or interpret carefully
Do not block important pages simply to reduce URL counts. Understand which URLs are useful, duplicate or needed for customers first.
Turn the guide into a small action plan
Record the starting point
Use the example above to identify what is happening in your own website, campaign or process. Save the evidence before changing it.
Choose the first check
Start with the relevant items in the review list. Assign an owner, identify required access and write down what you expect to learn.
Review before expanding
Compare the result with the original evidence. Keep useful changes, explain uncertainty and investigate problems before increasing scope.
Check whether crawl capacity is really the constraint
Crawl budget is often raised before there is evidence it limits a site. A small business with a modest page inventory may have more immediate content, access or indexing issues. Start with observed patterns instead of importing concerns from very large sites.
Review the pattern
Look for repeated unnecessary URL spaces and important pages that remain unvisited. Use available crawl evidence rather than a single tool warning.
Reduce unnecessary paths
Filters, parameters or obsolete structures may create redundant destinations. Choose controls with the developer according to their real purpose; do not block useful content indiscriminately.
Reassess priorities
After a confirmed change, review important discovery and crawl behaviour. More crawling is not itself a business outcome. Useful indexed content and customer journeys still matter.
How to interpret this guidance
A large generated inventory deserves publication and maintenance controls. It does not justify making hundreds of low-value pages merely because a crawler can reach them.
Should every small website buy a crawl-budget audit?
No. First establish whether the issue exists. Important URL evidence and the site’s actual scale should guide the decision. Many small sites benefit more from clearer content, internal links or fixing a specific access problem.
Can Tangensys help with the next step?
Yes. Share your website, goal and the issue you are investigating. We can discuss the relevant service and provide a custom quote within 2 working days.
Further reading
Consult the relevant platform documentation for current requirements: official documentation. Examples explain an approach; platform interfaces and policies can change.
Related reading and services
Let’s agree the right next step.
Tell us your goals and current setup. We’ll recommend a practical starting point and provide a custom quote.
Custom quote within 2 working days