0 Members and 1 Guest are viewing this topic.
User-agent: *Crawl-delay: 10User-agent: SemrushBotDisallow: /User-agent: SemrushBot-SADisallow: /User-agent: BLEXBotDisallow: /User-agent: YandexBotDisallow: /User-agent: YandexImagesDisallow: /User-agent: msnbot-mediaDisallow: /User-agent: MJ12botDisallow: /User-agent: AhrefsBotDisallow: /User-agent: DotBotDisallow: /User-agent: W3C-checklinkDisallow: /User-agent: W3C_ValidatorDisallow: /User-agent: W3C_CSS_ValidatorDisallow: /User-agent: YandexDisallow: /User-agent: BaiduspiderDisallow: /User-agent: archive.org_botDisallow: /User-agent: ia_archiverDisallow: /User-agent: FavIconizerDisallow: /User-agent: AhrefsBotDisallow: /User-agent: A6-Indexer/1.0Disallow: /User-agent: butterflyDisallow: /User-agent: ExabotDisallow: /User-agent: grapeshotDisallow: /User-agent: nutchDisallow: /User-agent: pingdomDisallow: /User-agent: proximicDisallow: /User-agent: PythonDisallow: /User-agent: SearchmetricsBotDisallow: /User-agent: SeznamBotDisallow: /User-agent: Sogou web spiderDisallow: /User-agent: sogou spiderDisallow: /User-agent: SogouDisallow: /User-agent: YoudaoBotDisallow: /User-agent: SosospiderDisallow: /User-agent: spbotDisallow: /User-agent: WgetDisallow: /User-agent: yie8Disallow: /User-agent: 360SpiderDisallow: /User-agent: TurnitinBotDisallow: /User-agent: hivaBotDisallow: /User-agent: NerdyBotDisallow: /User-agent: updown_testerDisallow: /User-agent: AndroidDownloadManagerDisallow: /User-agent: voltronDisallow: /User-agent: oBotDisallow: /User-agent: RiddlerDisallow: /User-agent: Mail.RU_BotDisallow: /User-agent: Mail.RuDisallow: /