By using seoforum’s services you agree to our Cookies Use and Data Transfer outside the EU.
We and our partners operate globally and use cookies, including for analytics, personalisation, ads and Newsletters.

  • Join the best UK dedicated SEO Forum

    Provide or get advice on everything SEO, ask questions, gain confirmation or just become apart of a friendly, like minded community who love SEO and Online Marketing.


    Join 50,000 members!
R

Reena

New Member
Likes
0
#1
Hi,

I need help in a robots.txt file.I am working on Collegesearch.in and found some of the tools are showing I have blocked my site for all crawlers.
Please find the below URL of robots.txt: https://www.collegesearch.in/robots.txt

I am planning to create robots.txt is
Sitemap: https://www.collegesearch.in/sitemaps_index.xml


User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: AdsBot-Google
User-agent: Alexabot
User-agent: Mail.Ru bot
User-agent: bingbot
User-agent: YandexBot
User-agent: Yahoo!
User-agent: Baiduspider
User-agent: archive.org_bot
User-agent: MSNBot
User-agent: ShopWiki
User-agent: ia_archiver
User-agent: netEstate Crawler
User-agent: bitlybot
User-agent: linkdexbot
User-agent: AhrefsBot
User-agent: SearchmetricsBot
User-agent: FacebookExternalHit
User-agent: SEOkicks-Robot
User-agent: Blekkobot
User-agent: HubSpot Connect
User-agent: BingPreview
User-agent: 360Spider
User-agent: yacybot
User-agent: Leikibot
User-agent: AboutUsBot
User-agent: OpenWebSpider
User-agent: BUbiNG
User-agent: backlink-check.de
User-agent: PayPal IPN
User-agent: Feedly
User-agent: WebCookies
User-agent: LinkedInBot
User-agent: BacklinkCrawler
User-agent: MetaGeneratorCrawler
User-agent: alexa site audit
User-agent: webmastercoffee
User-agent: MetaHeadersBot
User-agent: Open Web Analytics Bot
User-agent: Automattic Analytics Crawler
User-agent: YoudaoBot
User-agent: YodaoBot
User-agent: facebookplatform
User-agent: Whoismindbot
User-agent: gonzo
User-agent: KeywordDensityRobot
User-agent: pmoz.info ODP link checker
User-agent: YRSpider
User-agent: Twiceler
User-agent: XML Sitemaps Generator
User-agent: FeedFinder/bloggz.se
User-agent: DNS-Digger-Explorer
User-agent: RSSMicro.com RSS/Atom Feed Robot
Disallow: /ajax/
Disallow: /images/
Disallow: /upload/

It is fine or need some changes.
 
Mr Lucky

Mr Lucky

New Member
Likes
2
#2
I wonder why you list all those robots, wouldn't it work for you just have User-agent: *

Is this wordpress? IIs there are reason you are wanting to disallow images and upload?

I recently read the Yoast article on Wordpress robots.txt, they seem to think it's best to not block anything
 
hilleug61

hilleug61

Member
Likes
8
#4
I wonder why you list all those robots, wouldn't it work for you just have User-agent: *

Is this wordpress? IIs there are reason you are wanting to disallow images and upload?

I recently read the Yoast article on Wordpress robots.txt, they seem to think it's best to not block anything
I agree with you, there is no point in making the robots.txt file that difficult.
 

Latest posts

Latest Products

  • [Rivmedia] Lazy Loader XF2
    [Rivmedia] Lazy Loader XF2
    Load images asynchronously on your forum, allowing images to be loaded only when they are in view
    • Rivmedia
    • Updated:
  • [Rivmedia] Guest Redirect & Profile unlink
    [Rivmedia] Guest Redirect & Profile unlink
    Forums which prevent member profile access for guests, redirction and unlinking for profiles
    • Rivmedia
    • Updated:
  • [Rivmedia] Simple Redirects
    [Rivmedia] Simple Redirects
    Simple redirects allows forum admins to make simple 301 or 302 redirects via their admin panel
    • Rivmedia
    • Updated:
  • [Rivmedia] Minimum Message Length
    [Rivmedia] Minimum Message Length
    Eradicate short, pointless posts with a minimum message length and improving content worth on a thre
    • Rivmedia
    • Updated: