How can I setup my forum's Robots.Txt?

[aminur]

I need help to set up a perfect robot.txt for my website: themeshare.net
Please help me.
Thank you in advance.

 

[Subhodip]

First, you have to write a few codes as per your requirements. You can write in Notepad, Microsoft Word, or any text editor. After that save the file as ‘robots,’ and save it as “.txt” format and upload the file to the root directory of your site.

Now you have to add:
User-agent: *
Disallow:

Now you have to choose 2 options.

• If you want to block the search engine spider from your whole website. Then your code will be:

User-agent: *
Disallow: /

• If you want to block the search engine spider from certain areas of your site. Then your code will be:

User-agent: *
Disallow: /database/
Disallow: /scripts/

You can search for it as www. Yourdomain . com/ robots.txt

 

[bracknelson]

Here I want to share some terms of Setup my forums Roborts TXT:

1. Go to Google Webmaster Tools.
2. Choose the Site Configuration / Crawler Access on the menu.
3. Leave the text of robots. txt” box intact.
4. In the “URLs” text box, paste some URLs from your site like these: ...
5. Click the Test button, then scroll down to the results.