# Should you block AI as a blogger?

- Link: https://wpbarista.com/should-you-block-ai-as-a-blogger/
- Published: 2024-05-21T15:40:49-08:00
- Author: Cathy Mitchell

**Update for first quarter of 2026**: Instead of effecting your SEO, it turns out
that AI chats themselves are a traffic source, and at the very minimum a trust signal
to your audience. The advice is the same: do not block AI crawlers. Although the
reasons are a little different from discussed below.

If you have ads on your site or keep up on SEO news, you will have been presented
with the **option to block AI crawlers**. Let’s discuss what they are, and if you
should indeed block them or no?

First – this requires some thought and is an important decision that will affect
the life of your content. You need to decide whether it is **beneficial **for your
site to be used by AI.

Important Questions:

 * Should your blog be a source that AI considers when creating content?
 * Should you pay for the resources that the crawlers use?
 * Will it effect your privacy and intellectual property?
 * Will it change your SEO scores?
 * How will it affect your reputation and branding?

We have added this discussion in all our audits and maintenance upgrade services
as well. So if you come across those options and need some more info – here it is!

## What Are AI Crawlers?

AI crawlers are automated programs that scan websites to index content for search
engines and other online platforms .

There are so many AI platforms nowadays it is difficult to say for sure, what the
information is used for. Some examples would be Google crawling your site to provide
the data for it’s zero-placement results and information carousels. Another example
is when Feedly uses a crawler to gather relevant data to provide to interested readers–
they expand your audience this way.

Of course your ad companies use AI -based crawlers to scan your site for content,
analyze it, and let the companies bid to place an ad on your site and in specific
placements too. Email based crawlers will study your email list’s demographic data(
and more) to provide suggestions on how to personalize your emails.

An example of an AI-based crawler that uses artists’ published work, is DeepArt.
DeepArt analyzes and recreates images in the style of various artists, mimicking
the styles of famous painters. It is these sorts of crawlers that some have reservations
about. I’ll give you my opinion below. (so far – my opinion is evolving the more
I learn)

Basically – anything that crawls your website, nowadays would be silly not to use
AI to be faster, better and provide more relevant results.

## Pros of Allowing Ethical AI Crawlers

My biggest concern so far – that has yet to be addressed – is how blocking AI will
end up affecting your search engine exposure. If you do not allow big corporations
to use your content for ethical purposes, I believe you risk your search engine 
visibility. So allowing crawlers provides a huge opportunity for search engine growth
and visibility.

Note that Google has _said_ that blocking the Google-**Extended** crawler does not
influence the content in it’s results – it only blocks google AI from being trained
on it – namely Google Bard.

 * 301 Permanent redirects are not ‘counted against you’ – yet, sometimes, our experience
   tells us otherwise.

In 99% of content creator cases, I recommend you allow bots.

If you do not allow big corporations to use your content for ethical purposes, I
believe you risk your search engine visibility.

Plagiarism is not ethical. And stealing intellectual property is not legal. If we
allow crawling – we are absolutely only allowing the **ethical use** of our content.
See below for how to allow **ethical ai bots.**

A secondary concern is that you will be unable to use the programs you like if they
cannot crawl your site. AI is getting better and better at search engine research,
keyword suggestions and even writing in your own tone of voice. All these tools 
will be unavailable if you block all AI crawlers.

## Pros of Blocking Ethical AI Crawlers

The biggest issue we have with crawlers is **how they use the data**. For most of
you, creating and publishing is your full time income. Do you want people taking
and using that without permission? Or misusing it and affecting your good reputation?

**ADVICE **

If you believe your content to be worthy of intellectual property rights, and you
do not want anyone to use it to **inspire **their own work, then you should block
crawlers. If you, like me, believe that others can use my research, my thoughts 
and my conclusions to inspire their own work or conclusions – then do not block 
crawlers.

Secondly – if you are paying too much for your hosting resources and you are getting
far more crawlers than visitors – you can block crawlers to save money.

**EXAMPLE**

For today’s post, I did my own research and came to my own conclusions. However 
others are free to use this information and have the same or different conclusions.

Should crawlers or AI use my work and copy it verbatim? **Absolutely not – that 
is plagiarism. **

Whether to block crawlers or not to block, should be based on whether or not to 
block **ethical **AI crawlers. (the reasons are below – in how to block ethical 
crawlers).

**ART & Original Works**

The biggest impact AI will have in the short term, I believe, are on creatives that
publish their original artwork. These are the creators that publish photos of paintings
and sculptures and the like – original artwork. In this discussion, I am not talking
about authors.

Art is so subjective, that **ethical** use might still rub you the wrong way. And
I would encourage you to look up “derivative works, copy rights”. If you would like
your work NOT used for derivative works I would recommend you block AI crawlers.

**Stealing** or **Research?**

Is it ‘stealing’ to allow an AI large Language model to read your stuff and integrate
it into the world’s biggest database of knowledge? I do not think so. Others do.

**Content Creators & AI Bots**

News organizations are taking a hard stance against AI. According to research by
[Palewire](https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html), 
47% of the tracked news websites do not allow AI crawlers.

The publishers see their work as a product. Hopefully you do not – your work is 
the influence you have over your audience – In My Opinion, it is a LOT more than
just some words on a page.

## How to Block AI Crawlers

The Yoast SEO plugin includes a feature that allows users to block AI crawlers **
without **affecting search engine crawlers like Googlebot. They do this by targeting
specific AI bots while allowing others. This means we can only block crawlers – 
ALL of them (as curated by Yoast)- or Specifically one at a time.

![](https://i0.wp.com/wpbarista.com/wp-content/uploads/2024/05/yoast-seo-crawl-settings-
block-bots.webp?resize=1045%2C363&ssl=1)

Premium Version of Yoast SEO

Other plugins are coming up with easy methods to block AI bots too. I was alerted
today to Raptive’s new setting to block AI crawlers. Yes you can use that – but 
I haven’t looked up how they are doing it. It would have to be individually because
they will want to preserve their ad network and stats plugins too.

**Problems with AI Blocking Plugins**

You should know this is **not foolproof **– it will successfully block chatgpt and
other honest AI bots. But AI crawlers do not have to declare themselves honestly.
Most of the time, bad bots hide their identity which is fairly easy to do.

So Yoast, and other blocking plugins are blocking in one of two ways:

 1. a curated list (eg Yoast’s)
 2. individual ai bots (eg Yoast’s Premium Version)

Because we have to block crawlers individually, we will never be 100% sure we have
got them all. As I’ve said above, for 99% of you, my current recommendation is to
let the bots do their work. Hopefully you reap search engine visibility.

## Conclusion

Ultimately it is a personal decision that you – the creator must decide. But keep
the following points in mind:

 * IMO, it is likely that search engine visibility will be affected if not now, 
   then soon;
 * if your main concern is the unethical use of your content, that will happen regardless
   of this plugin’s settings
 * you are paying for the crawlers to crawl your site (as part of your bandwidth
   or pageviews)

Also – let this be the catalyst that pushes you to lean into the special sauce of
your brand – it is not words, or product descriptions. I bet that you have a loyal
following because of what you provide to your audience – and that cannot be crawled
or stolen or misused.

Cheering you on,

Cathy Mitchell

 * [  Email ](https://wpbarista.com/should-you-block-ai-as-a-blogger/?subject=%5BShared%20Post%5D%20Should%20you%20block%20AI%20as%20a%20blogger%3F&body=https%3A%2F%2Fwpbarista.com%2Fshould-you-block-ai-as-a-blogger%2F&share=email)
 * [  LinkedIn ](https://wpbarista.com/should-you-block-ai-as-a-blogger/?share=linkedin)
 * [  X ](https://wpbarista.com/should-you-block-ai-as-a-blogger/?share=x)
