By using seoforum’s services you agree to our Cookies Use and Data Transfer outside the EU.
We and our partners operate globally and use cookies, including for analytics, personalisation, ads and Newsletters.

  • Join the best UK dedicated SEO Forum

    Provide or get advice on everything SEO, ask questions, gain confirmation or just become apart of a friendly, like minded community who love SEO and Online Marketing.


    Join 50,000 members!

robots.txt for xenforo

Mr Lucky

Mr Lucky

New Member
I notice the site here has nothing at all blocked by robots.txt. Is there a reason for that?
I'm asking because so many people recommend what they do on xenforo.com

Code:
User-agent: *
Disallow: /community/whats-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
Sitemap: https://xenforo.com/community/sitemap.xml
So I wonder about pros and cons either way?
 
Community

Community

Administrator
Staff member
I prefer to control what google is allowed to visit via canonical, redirects and no-index/nofollow tags or making specific pages less "thin". That said there are a few pages we could realistically remove/block but at the same time blocking access via robots.txt isn't always favorable.

We also rather heavily cut out the normal forum generated "thin" error pages, such as if you view the forum as a guest, you'll notice all user profiles url's are not only redirected, but all usernames are also "unlinked" so to search engines they are simply not in existance.

With the default xenforo install, usernames are still linked and URL's reachable, which is fine if its done from the outset, but if a forum has migrated to xenforo or has previously allowed the indexing of profiles. Throwing a no-index/nofollow login page isn't ideal. It essentially creates thousands of crawl errors and wasted crawl budget ( also debatable worthyness, but we've seen great results with the way we implement it ).

I'm not one for debating what forum owners should and shouldn't allow/disallow, we do what works for this particular site, assessing another forum or website the strategy maybe different and we may use robots.txt blocking fairly comprehensively. The best SEO strategy is never blanket coverage, it's tailored for a specific site and goals.
 
Mr Lucky

Mr Lucky

New Member
OK that makes sense.

Part of the reason I was asking is I also noticed that Yoast recommend allowing everything, so I was thinking of going that route if what they say makes sans (which is is maybe only a Wordpress thing though of course)

But on xenforo I suppose it at least makes sense to block /posts/ as those are just fragments of what is already indexed in threads.
 

Latest Products

  • [Rivmedia] Lazy Loader XF2
    [Rivmedia] Lazy Loader XF2
    Load images asynchronously on your forum, allowing images to be loaded only when they are in view
    • Rivmedia
    • Updated:
  • [Rivmedia] Guest Redirect & Profile unlink
    [Rivmedia] Guest Redirect & Profile unlink
    Forums which prevent member profile access for guests, redirction and unlinking for profiles
    • Rivmedia
    • Updated:
  • [Rivmedia] Simple Redirects
    [Rivmedia] Simple Redirects
    Simple redirects allows forum admins to make simple 301 or 302 redirects via their admin panel
    • Rivmedia
    • Updated:
  • [Rivmedia] Minimum Message Length
    [Rivmedia] Minimum Message Length
    Eradicate short, pointless posts with a minimum message length and improving content worth on a thre
    • Rivmedia
    • Updated:
Top