HubsInfo Network
Social PHP Linux Web Hosting Online Games Free Web Directory

Using proper robots txt file

0 Members and 1 Guest are viewing this topic.

Using proper robots txt file
« on: March 17, 2020, 01:01:10 PM »
Using proper robots txt file
__UPDATED on 17.03.2020


Code: [Select]
User-agent: *

Crawl-delay: 10

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: YandexImages
Disallow: /

User-agent: msnbot-media
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: W3C-checklink
Disallow: /

User-agent: W3C_Validator
Disallow: /

User-agent: W3C_CSS_Validator
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: FavIconizer
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: A6-Indexer/1.0
Disallow: /

User-agent: butterfly
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: grapeshot
Disallow: /

User-agent: nutch
Disallow: /

User-agent: pingdom
Disallow: /

User-agent: proximic
Disallow: /

User-agent: Python
Disallow: /

User-agent: SearchmetricsBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: spbot
Disallow: /

User-agent: Wget
Disallow: /

User-agent: yie8
Disallow: /

User-agent: 360Spider
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: hivaBot
Disallow: /

User-agent: NerdyBot
Disallow: /

User-agent: updown_tester
Disallow: /

User-agent: AndroidDownloadManager
Disallow: /

User-agent: voltron
Disallow: /

User-agent: oBot
Disallow: /

User-agent: Riddler
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: Mail.Ru
Disallow: /
« Last Edit: March 17, 2020, 01:03:33 PM by Slowloris »

 


Linux Ubuntu Guides Linux games - Lutris.net