By using seoforum’s services you agree to our Cookies Use and Data Transfer outside the EU.
We and our partners operate globally and use cookies, including for analytics, personalisation, ads and Newsletters.

  • Join the best UK dedicated SEO Forum

    Provide or get advice on everything SEO, ask questions, gain confirmation or just become apart of a friendly, like minded community who love SEO and Online Marketing.

    Join 50,000 members!

How can I setup my forum's Robots.Txt?



New Member
I need help to set up a perfect robot.txt for my website:
Please help me.
Thank you in advance.


Established Memeber
First, you have to write a few codes as per your requirements. You can write in Notepad, Microsoft Word, or any text editor. After that save the file as ‘robots,’ and save it as “.txt” format and upload the file to the root directory of your site.

Now you have to add:
User-agent: *

Now you have to choose 2 options.
  • If you want to block the search engine spider from your whole website. Then your code will be:
User-agent: *
Disallow: /
  • If you want to block the search engine spider from certain areas of your site. Then your code will be:
User-agent: *
Disallow: /database/
Disallow: /scripts/

You can search for it as www. Yourdomain . com/ robots.txt


New Member
Here I want to share some terms of Setup my forums Roborts TXT:
  1. Go to Google Webmaster Tools.
  2. Choose the Site Configuration / Crawler Access on the menu.
  3. Leave the text of robots. txt” box intact.
  4. In the “URLs” text box, paste some URLs from your site like these: ...
  5. Click the Test button, then scroll down to the results.
Internal linking plugin

Latest Products

  • [Rivmedia] Lazy Loader XF2
    [Rivmedia] Lazy Loader XF2
    Load images asynchronously on your forum, allowing images to be loaded only when they are in view
    • Rivmedia
    • Updated:
  • [Rivmedia] Guest Redirect & Profile unlink
    [Rivmedia] Guest Redirect & Profile unlink
    Forums which prevent member profile access for guests, redirction and unlinking for profiles
    • Rivmedia
    • Updated:
  • [Rivmedia] Simple Redirects
    [Rivmedia] Simple Redirects
    Simple redirects allows forum admins to make simple 301 or 302 redirects via their admin panel
    • Rivmedia
    • Updated:
  • [Rivmedia] Minimum Message Length
    [Rivmedia] Minimum Message Length
    Eradicate short, pointless posts with a minimum message length and improving content worth on a thre
    • Rivmedia
    • Updated: