5
(1)

Last Updated on April 20, 2021 by Amit

Block search engine bots using htaccess

htaccess block search engines

With htaccess it is possible to block search crawler bots such as Google , yahoo and bing bot or other web crawlers from accessing your website.
Block search engine bots using htaccess

If you don’t want your website to be visible to search engines then you can easily deny access to web crawling bots using RewriteRule directive in an htaccess file .
In this article I will show you how to block search engines from crawling your website. We will use our htaccess file for this blocking.

htaccess code to block google, yahoo and bing bots

The following is a tested and 100% working code so you can just copy and paste it to your .htaccess file .

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} google|yahoo|bing [NC]
RewriteRule ^.*$ - [F]

This rule will show a forbidden (403) error for “search bots” when they try to visit any URLs on your website and won’t be able to crawl or index your website.
explanations of the code above :

  1. The first line RewriteEngine on turns the RewriteEngine on. This tells server to read RewriteRule(s) in htaccess file. You can remove this line if your htaccess already contains one and place the rule just bellow that because Only one RewriteEngine directive is needed per htaccess.
  2. The second line RewriteCond %{HTTP_USER_AGENT} google|yahoo|bing [NC] is a condition for the RewriteRule. Here we match against the “HTTP_USER_AGENT” string using a regular expression based pattern that matches those three bots. Since we dont know the exect user agent string so we can just use their name like “google” ,”yahoo” ,”bing” the pattern will automatically match the full user_agent string.
  3. The third line RewriteRule ^.*$ – [F] rewrites all requests to F it is short for forbidden or 403 error . If the condition above the rule is met (http_user_agent ===google OR yahoo OR bing) then this rule will get triggered.
    You can replace the F flag with R=403 or any other error status.
    The pattern (.*) means the entire site. If you want to block search crawler bots from reaching a specific file or path on your server then you can change the pattern accordingly ie ^demo.php$ applies only this specific file.

Block a specific user agent

To block a specific search bot for example googlebot,you can use the following:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} google [NC]
RewriteRule ^.*$ - [F]

How useful was this post?

Click on a star to rate it!

Average rating 5 / 5. Vote count: 1

No votes so far! Be the first to rate this post.