There was a problem loading the comments.

Blocking Robots using .htaccess

Support Portal  »  Knowledgebase  »  Viewing Article

  Print
If you want to block the major backlink checker tools from accessing your site, add the following lines to your site’s .htaccess file.

Unlike WordPress plugins like SpyderSpanker, by adding these lines to your site’s .htaccess file, you can block bots from crawling a static HTML site (such as a Wayback Archive restore).

Using this method, it is also possible to enable caching plugins to speed up your WordPress site without it overriding your bot blocking plugin and allowing Majestic, Ahrefs and Open Site Explorer to index your backlinks.

To do this, start by logging in to your site’s cPanel, opening the File Manager, and enabling “dot (hidden) files”. Find the “.htaccess” file in your site’s public_html folder and add these lines at the end.

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
SetEnvIfNoCase User-Agent .*spbot.* bad_bot
SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
SetEnvIfNoCase User-Agent .*nutch.* bad_bot
SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

This will return a 403 error when any of the listed bots try to access your site.
To test if it is working, you can use our free BotChecker site.

Share via
Did you find this article useful?  

Related Articles


Comments

Add Comment

Replying to  

Captcha
© Bulk Buy Hosting