I would like to prevent Google / Search Engines from Crawling - Indexing some specific pages or an entire website. Could anyone guide me in the right direction or share some tips to do this the right way?
Thanks so much, and apologies in advance if this is not the place to ask this question, or it has been asked before (couldn’t find anything) in such a case I would delete or move where it belongs.
You can add the following to the meta data of your page:
<meta name="robots" content="noindex, nofollow">
This will only apply to the page that you have this in.
The robots.txt file you can specify directories to not be indexed such as an admin folder etc. as well as point to your sitemap.xml file. If you have a large site, you don’t really want lots of individual pages in the robots.txt file, the meta tag will be a better choice.