Crawl budget optimisation

[Hannah Gilbert]

The site I'm working on has TONS of old pages that we can't get rid of due to the nature of the business, but I'd like Google to stop crawling them. They are all indexed at at the moment, if I add them to robots.txt will they continue to be indexed?

 

[Community]

To be safe I always try and promote the usage of robots.txt, plus adding noindex/nofollow headers or meta data. A little more explained here :

https://blog.hubspot.com/marketing/how-to-unindex-pages-from-search-engines

 

[Bradwell]

This only really impacts sites with large numbers of pages and low numbers of links.