robots.txt for xenforo

[Mr Lucky]

I notice the site here has nothing at all blocked by robots.txt. Is there a reason for that?
I'm asking because so many people recommend what they do on xenforo.com

User-agent: *
Disallow: /community/whats-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

Sitemap: https://xenforo.com/community/sitemap.xml

So I wonder about pros and cons either way?

 

[Community]

I prefer to control what google is allowed to visit via canonical, redirects and no-index/nofollow tags or making specific pages less "thin". That said there are a few pages we could realistically remove/block but at the same time blocking access via robots.txt isn't always favorable.

We also rather heavily cut out the normal forum generated "thin" error pages, such as if you view the forum as a guest, you'll notice all user profiles url's are not only redirected, but all usernames are also "unlinked" so to search engines they are simply not in existance.

With the default xenforo install, usernames are still linked and URL's reachable, which is fine if its done from the outset, but if a forum has migrated to xenforo or has previously allowed the indexing of profiles. Throwing a no-index/nofollow login page isn't ideal. It essentially creates thousands of crawl errors and wasted crawl budget ( also debatable worthyness, but we've seen great results with the way we implement it ).

I'm not one for debating what forum owners should and shouldn't allow/disallow, we do what works for this particular site, assessing another forum or website the strategy maybe different and we may use robots.txt blocking fairly comprehensively. The best SEO strategy is never blanket coverage, it's tailored for a specific site and goals.

 

[Mr Lucky]

OK that makes sense.

Part of the reason I was asking is I also noticed that Yoast recommend allowing everything, so I was thinking of going that route if what they say makes sans (which is is maybe only a Wordpress thing though of course)

But on xenforo I suppose it at least makes sense to block /posts/ as those are just fragments of what is already indexed in threads.

 

[jgaulard]

I prefer to control what google is allowed to visit via canonical, redirects and no-index/nofollow tags or making specific pages less "thin". That said there are a few pages we could realistically remove/block but at the same time blocking access via robots.txt isn't always favorable.

This is a great response. Let me ask you a question, since you're the admin here. How do you feel about all the 301 redirects in the XenForo install? For instance, take a look at this partial link from your homepage (sorry, I'm not allowed to post a full link yet):

threads/how-to-add-static-text-to-blog-page.5856/post-19824

See the "/post-19824" at the end of it? Also, there are tons of "/latest" endings on links too. What I'm finding is that Google isn't canonicalizing all of them. About 1/10th of those redirects on my sites forward, but never actually merge with the target page. I've seen this on many other forums as well. Even the XenForo community itself.

I just wanted to get your take on this as you seem to know what you're doing. I'm sure I'll have more questions for you as well.

Thanks!

Jay

 

[Community]

Hi Jay,

Two things to consider.

1. If this is a recent thing you've noticed ( like in the last 4 months ) then it could be related to the bugs Google's been having with their indexing and canonical following :

Example news report: https://www.seroundtable.com/google-index-bugs-canonicalization-mobile-indexing-30201.html

Although its much more wide spread than the dates outlined, these fluctuations have been happening for a few months at least and Google have claimed to have fixed them a few times, only yesterday claiming 99% of the indexing problem is now resolved ( hmm )

2. Canonicals are still only a "guide", if a particular URL has been linked to alot and therefore is considered to be more relevent or powerful, then a Canonicalised URL can still appear in search and can still take president.

With all that said, it could also indicate other problems, misconfigurations or structure issues. With Xenforo although it is fairly good as far as forums go, there is still alot that can be done to help with crawl budgets, thin and duplicate content pages etc. More specifically on larger forums with alot of existing traffic.

Also to note, Since the above posts, Google has made it clear that they no longer really pay attention to Robots.txt files for blocking purposes.