If you have ads on your site or keep up on SEO news, you will have been presented with the option to block AI crawlers. Let’s discuss what they are, and if you should indeed block them or no?

First – this will need to be something you spend a few minutes on. You need to decide whether it is beneficial for your site to be used by AI.

  • Should your blog be a source that AI considers when creating content?
  • Should you pay for the resources that the crawlers use?
  • Will it effect your privacy and intellectual property?
  • Will it change your SEO scores?
  • How will it affect your reputation and branding?

We have added this discussion in all our audits and maintenance upgrade services as well. So if you come across those options and need some more info – here it is!

What Are AI Crawlers?

There are so many AI platforms nowadays it is difficult to say for sure, what the information is used for. Some examples would be Google crawling your site to provide the data for it’s zero-placement results and information carousels. Another example is when Feedly uses a crawler to gather relevant data to provide to interested readers – they expand your audience this way.

Of course your ad companies use AI -based crawlers to scan your site for content, analyze it, and let the companies bid to place an ad on your site and in specific placements too. Email based crawlers will study your email list’s demographic data (and more) to provide suggestions on how to personalize your emails.

An example of an AI-based crawler that uses artists’ published work, is DeepArt. DeepArt analyzes and recreates images in the style of various artists, mimicking the styles of famous painters. It is these sorts of crawlers that some have reservations about. I’ll give you my opinion below. (so far – my opinion is evolving the more I learn)

Basically – anything that crawls your website, nowadays would be silly not to use AI to be faster, better and provide more relevant results.

Pros of Allowing Ethical AI Crawlers

My biggest concern so far – that has yet to be addressed – is how blocking AI will end up affecting your search engine exposure. If you do not allow big corporations to use your content for ethical purposes, I believe you risk your search engine visibility. So allowing crawlers provides a huge opportunity for search engine growth and visibility.

Note that Google has said that blocking the Google-Extended crawler does not influence the content in it’s results – it only blocks google AI from being trained on it – namely Google Bard.

301 Permanent redirects are not ‘counted against you’ – yet, sometimes, our experience tells us otherwise.

In 99% of content creator cases, I recommend you allow bots.

If you do not allow big corporations to use your content for ethical purposes, I believe you risk your search engine visibility.

Plagiarism is not ethical. And stealing intellectual property is not legal. If we allow crawling – we are absolutely only allowing the ethical use of our content. See below for how to allow ethical ai bots.

A secondary concern is that you will be unable to use the programs you like if they cannot crawl your site. AI is getting better and better at search engine research, keyword suggestions and even writing in your own tone of voice. All these tools will be unavailable if you block all AI crawlers.

Pros of Blocking Ethical AI Crawlers

The biggest issue we have with crawlers is how they use the data. For most of you, creating and publishing is your full time income. Do you want people taking and using that without permission? Or misusing it and affecting your good reputation?

ADVICE

If you believe your content to be worthy of intellectual property rights, and you do not want anyone to use it to inspire their own work, then you should block crawlers. If you, like me, believe that others can use my research, my thoughts and my conclusions to inspire their own work or conclusions – then do not block crawlers.

Secondly – if you are paying too much for your hosting resources and you are getting far more crawlers than visitors – you can block crawlers to save money.

EXAMPLE

For today’s post, I did my own research and came to my own conclusions. However others are free to use this information and have the same conclusion.

Should crawlers or AI use my work and copy it verbatim? Absolutely not – that is plagiarism.

Whether to block crawlers or not to block, should be based on whether or not to block ethical AI crawlers. (the reasons are below – in how to block ethical crawlers).

ART & Original Works

The biggest impact AI will have in the short term, I believe, are on creatives that publish their original artwork. These are the creators that publish photos of paintings and sculptures and the like – original artwork. In this discussion, I am not talking about authors.

Art is so subjective, that ethical use might still rub you the wrong way. And I would encourage you to look up “derivative works, copy rights”. If you would like your work NOT used for derivative works I would recommend you block AI crawlers.

Stealing or Research?

Is it ‘stealing’ to allow an AI large Language model to read your stuff and integrate it into the world’s biggest database of knowledge? I do not think so. Others do.

Content Creators & AI Bots

News organizations are taking a hard stance against AI. According to research by Palewire, 47% of the tracked news websites do not allow AI crawlers.

The publishers see their work as a product. Hopefully you do not – your work is the influence you have over your audience – In My Opinion, it is a LOT more than just some words on a page.

How to Block AI Crawlers

The Yoast SEO plugin includes a feature that allows users to block AI crawlers without affecting search engine crawlers like Googlebot. They do this by targeting specific AI bots while allowing others. This means we can only block crawlers – ALL of them (as curated by Yoast)- or Specifically one at a time.

Premium Version of Yoast SEO

Other plugins are coming up with easy methods to block AI bots too. I was alerted today to Raptive’s new setting to block AI crawlers. Yes you can use that – but I haven’t looked up how they are doing it. It would have to be individually because they will want to preserve their ad network and stats plugins too.

Problems with AI Blocking Plugins

You should know this is not foolproof – it will successfully block chatgpt and other honest AI bots. But AI crawlers do not have to declare themselves honestly. Most of the time, bad bots hide their identity which is fairly easy to do.

So Yoast, and other blocking plugins are blocking in one of two ways:

  1. a curated list (eg Yoast’s)
  2. individual ai bots (eg Yoast’s Premium Version)

Because we have to block crawlers individually, we will never be 100% sure we have got them all. As I’ve said above, for 99% of you, my current recommendation is to let the bots do their work. Hopefully you reap search engine visibility.

Conclusion

Ultimately it is a personal decision that you – the creator must decide. But keep the following points in mind:

  • IMO, it is likely that search engine visibility will be affected if not now, then soon;
  • if your main concern is the unethical use of your content, that will happen regardless of this plugin’s settings
  • you are paying for the crawlers to crawl your site (as part of your bandwidth or pageviews)

Also – let this be the catalyst that pushes you to lean into the special sauce of your brand – it is not words, or product descriptions. I bet that you have a loyal following because of what you provide to your audience – and that cannot be crawled or stolen or misused.

Cheering you on,

Cathy Mitchell

Beginner Checklist

If you’re starting out, you’ll love our comprehensive 52 point checklist for your website! Read through once, and then work on items one at a time as it comes up!

52 Edits Checklist – beginners categories

Cathy Mitchell

Single Mom, Lifelong Learner, Jesus Follower, Founder and CEO at WPBarista.