If you want to block the major backlink checker tools from accessing your site, add the following lines to your site’s .htaccess file.
Unlike WordPress plugins like SpyderSpanker, by adding these lines to your site’s .htaccess file, you can block bots from crawling a static HTML site (such as a Wayback Archive restore).
Using this method, it is also possible to enable caching plugins to speed up your WordPress site without it overriding your bot blocking plugin and allowing Majestic, Ahrefs and Open Site Explorer to index your backlinks.
To do this, start by logging in to your site’s cPanel, opening the
File Manager, and enabling “dot (hidden) files”. Find the “.htaccess”
file in your site’s public_html folder and add these lines at the end.
SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent .*semrushbot.* bad_bot
SetEnvIfNoCase User-Agent .*ia_archiver.* bad_bot
SetEnvIfNoCase User-Agent .*searchmetricsbot.* bad_bot
SetEnvIfNoCase User-Agent .*seokicks-robot.* bad_bot
SetEnvIfNoCase User-Agent .*sistrix.* bad_bot
SetEnvIfNoCase User-Agent .*lipperhey spider.* bad_bot
SetEnvIfNoCase User-Agent .*ncbot.* bad_bot
SetEnvIfNoCase User-Agent .*backlinkcrawler.* bad_bot
SetEnvIfNoCase User-Agent .*archive.org_bot.* bad_bot
SetEnvIfNoCase User-Agent .*meanpathbot.* bad_bot
SetEnvIfNoCase User-Agent .*pagesinventory.* bad_bot
SetEnvIfNoCase User-Agent .*aboundexbot.* bad_bot
SetEnvIfNoCase User-Agent .*spbot.* bad_bot
SetEnvIfNoCase User-Agent .*linkdexbot.* bad_bot
SetEnvIfNoCase User-Agent .*nutch.* bad_bot
SetEnvIfNoCase User-Agent .*blexbot.* bad_bot
SetEnvIfNoCase User-Agent .*ezooms.* bad_bot
SetEnvIfNoCase User-Agent .*scoutjet.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-12.* bad_bot
SetEnvIfNoCase User-Agent .*majestic-seo.* bad_bot
SetEnvIfNoCase User-Agent .*dsearch.* bad_bot
SetEnvIfNoCase User-Agent .*blekkobo.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
This will return a 403 error when any of the listed bots try to access your site.
To test if it is working, you can
use our free BotChecker site.