By using seoforum’s services you agree to our Cookies Use and Data Transfer outside the EU.
We and our partners operate globally and use cookies, including for analytics, personalisation, ads and Newsletters.

  • Join the best UK dedicated SEO Forum

    Provide or get advice on everything SEO, ask questions, gain confirmation or just become apart of a friendly, like minded community who love SEO and Online Marketing.


    Join 50,000 members!

robots.txt for xenforo

Mr Lucky

Mr Lucky

New Member
Established Memeber
I notice the site here has nothing at all blocked by robots.txt. Is there a reason for that?
I'm asking because so many people recommend what they do on xenforo.com

Code:
User-agent: *
Disallow: /community/whats-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
Sitemap: https://xenforo.com/community/sitemap.xml
So I wonder about pros and cons either way?
 
Community

Community

Administrator
Staff member
Established Memeber
I prefer to control what google is allowed to visit via canonical, redirects and no-index/nofollow tags or making specific pages less "thin". That said there are a few pages we could realistically remove/block but at the same time blocking access via robots.txt isn't always favorable.

We also rather heavily cut out the normal forum generated "thin" error pages, such as if you view the forum as a guest, you'll notice all user profiles url's are not only redirected, but all usernames are also "unlinked" so to search engines they are simply not in existance.

With the default xenforo install, usernames are still linked and URL's reachable, which is fine if its done from the outset, but if a forum has migrated to xenforo or has previously allowed the indexing of profiles. Throwing a no-index/nofollow login page isn't ideal. It essentially creates thousands of crawl errors and wasted crawl budget ( also debatable worthyness, but we've seen great results with the way we implement it ).

I'm not one for debating what forum owners should and shouldn't allow/disallow, we do what works for this particular site, assessing another forum or website the strategy maybe different and we may use robots.txt blocking fairly comprehensively. The best SEO strategy is never blanket coverage, it's tailored for a specific site and goals.
 
Mr Lucky

Mr Lucky

New Member
Established Memeber
OK that makes sense.

Part of the reason I was asking is I also noticed that Yoast recommend allowing everything, so I was thinking of going that route if what they say makes sans (which is is maybe only a Wordpress thing though of course)

But on xenforo I suppose it at least makes sense to block /posts/ as those are just fragments of what is already indexed in threads.
 
jgaulard

jgaulard

New Member
I prefer to control what google is allowed to visit via canonical, redirects and no-index/nofollow tags or making specific pages less "thin". That said there are a few pages we could realistically remove/block but at the same time blocking access via robots.txt isn't always favorable.
This is a great response. Let me ask you a question, since you're the admin here. How do you feel about all the 301 redirects in the XenForo install? For instance, take a look at this partial link from your homepage (sorry, I'm not allowed to post a full link yet):

threads/how-to-add-static-text-to-blog-page.5856/post-19824

See the "/post-19824" at the end of it? Also, there are tons of "/latest" endings on links too. What I'm finding is that Google isn't canonicalizing all of them. About 1/10th of those redirects on my sites forward, but never actually merge with the target page. I've seen this on many other forums as well. Even the XenForo community itself.

I just wanted to get your take on this as you seem to know what you're doing. I'm sure I'll have more questions for you as well.

Thanks!

Jay
 
Community

Community

Administrator
Staff member
Established Memeber
Hi Jay,

Two things to consider.

1. If this is a recent thing you've noticed ( like in the last 4 months ) then it could be related to the bugs Google's been having with their indexing and canonical following :

Example news report: https://www.seroundtable.com/google-index-bugs-canonicalization-mobile-indexing-30201.html

Although its much more wide spread than the dates outlined, these fluctuations have been happening for a few months at least and Google have claimed to have fixed them a few times, only yesterday claiming 99% of the indexing problem is now resolved ( hmm )

2. Canonicals are still only a "guide", if a particular URL has been linked to alot and therefore is considered to be more relevent or powerful, then a Canonicalised URL can still appear in search and can still take president.

With all that said, it could also indicate other problems, misconfigurations or structure issues. With Xenforo although it is fairly good as far as forums go, there is still alot that can be done to help with crawl budgets, thin and duplicate content pages etc. More specifically on larger forums with alot of existing traffic.

Also to note, Since the above posts, Google has made it clear that they no longer really pay attention to Robots.txt files for blocking purposes.
 
Internal linking plugin

Latest Products

  • [Rivmedia] Lazy Loader XF2
    [Rivmedia] Lazy Loader XF2
    Load images asynchronously on your forum, allowing images to be loaded only when they are in view
    • Rivmedia
    • Updated:
  • [Rivmedia] Guest Redirect & Profile unlink
    [Rivmedia] Guest Redirect & Profile unlink
    Forums which prevent member profile access for guests, redirction and unlinking for profiles
    • Rivmedia
    • Updated:
  • [Rivmedia] Simple Redirects
    [Rivmedia] Simple Redirects
    Simple redirects allows forum admins to make simple 301 or 302 redirects via their admin panel
    • Rivmedia
    • Updated:
  • [Rivmedia] Minimum Message Length
    [Rivmedia] Minimum Message Length
    Eradicate short, pointless posts with a minimum message length and improving content worth on a thre
    • Rivmedia
    • Updated:
Top