Robots.txt

[Reena]

Hi,

I need help in a robots.txt file.I am working on Collegesearch.in and found some of the tools are showing I have blocked my site for all crawlers.
Please find the below URL of robots.txt: https://www.collegesearch.in/robots.txt

I am planning to create robots.txt is
Sitemap: https://www.collegesearch.in/sitemaps_index.xml


User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: AdsBot-Google
User-agent: Alexabot
User-agent: Mail.Ru bot
User-agent: bingbot
User-agent: YandexBot
User-agent: Yahoo!
User-agent: Baiduspider
User-agent: archive.org_bot
User-agent: MSNBot
User-agent: ShopWiki
User-agent: ia_archiver
User-agent: netEstate Crawler
User-agent: bitlybot
User-agent: linkdexbot
User-agent: AhrefsBot
User-agent: SearchmetricsBot
User-agent: FacebookExternalHit
User-agent: SEOkicks-Robot
User-agent: Blekkobot
User-agent: HubSpot Connect
User-agent: BingPreview
User-agent: 360Spider
User-agent: yacybot
User-agent: Leikibot
User-agent: AboutUsBot
User-agent: OpenWebSpider
User-agent: BUbiNG
User-agent: backlink-check.de
User-agent: PayPal IPN
User-agent: Feedly
User-agent: WebCookies
User-agent: LinkedInBot
User-agent: BacklinkCrawler
User-agent: MetaGeneratorCrawler
User-agent: alexa site audit
User-agent: webmastercoffee
User-agent: MetaHeadersBot
User-agent: Open Web Analytics Bot
User-agent: Automattic Analytics Crawler
User-agent: YoudaoBot
User-agent: YodaoBot
User-agent: facebookplatform
User-agent: Whoismindbot
User-agent: gonzo
User-agent: KeywordDensityRobot
User-agent: pmoz.info ODP link checker
User-agent: YRSpider
User-agent: Twiceler
User-agent: XML Sitemaps Generator
User-agent: FeedFinder/bloggz.se
User-agent: DNS-Digger-Explorer
User-agent: RSSMicro.com RSS/Atom Feed Robot
Disallow: /ajax/
Disallow: /images/
Disallow: /upload/

It is fine or need some changes.

 

[Mr Lucky]

I wonder why you list all those robots, wouldn't it work for you just have User-agent: *

Is this wordpress? IIs there are reason you are wanting to disallow images and upload?

I recently read the Yoast article on Wordpress robots.txt, they seem to think it's best to not block anything

 

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

 

[hilleug61]

I wonder why you list all those robots, wouldn't it work for you just have User-agent: *

Is this wordpress? IIs there are reason you are wanting to disallow images and upload?

I recently read the Yoast article on Wordpress robots.txt, they seem to think it's best to not block anything

I agree with you, there is no point in making the robots.txt file that difficult.

 

[webxerossolutions]

You do not need to create this much longer robots.txt file. You simply can use user-agent * to disallow some of the pages and files.
The way you created the robots.txt file is much complicated.
If you want to stop all search engines to crawl the same pages then you can use user-agent: *

 

[mallikarjuna]

robot.txt is a standard exclusion protocol which is used to communicate with web crawlers or bots to crawl or not crawl your website pages. If you want to crawl your pages by all bots just place user-agent:*. It is enough to call allow all bots

 

[Paro]

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.