With millions of sites built on Webflow, we've had a unique vantage point into the rapid rise of AI crawlers and how they are transforming the way websites are discovered and indexed. Over the past few months, we’ve seen firsthand how this surge in bot traffic is challenging the limits of the traditional tools that were not designed for the scale and complexity of today’s AI systems in mind.

In the first part of this year, AI and LLM crawler traffic to our hosted websites surged by over 125%, quickly becoming the second-largest category of bot traffic behind traditional SEO crawlers–highlighting a significant change in the nature of web traffic across the Internet.

As AI agents become increasingly sophisticated at crawling, understanding, and summarizing web content, businesses need to rethink how they structure and present information. The good news is that the fundamentals of creating useful, well-organized content remain just as important. The difference is that, now more than ever, you're optimizing for both human visitors and AI agents at the same time. 

That is why Webflow has provided clear guidance to help customers manage how AI crawlers and modern search engines interact with their sites. Whether you want to block specific bots, or improve how your content appears in AI-driven experiences, we are making it easier to take control. 

This is an exciting time for the future of the web, and while Webflow’s priority is to help you stay ahead, stay visible, and succeed in this new landscape–we are not tackling this challenge alone. Recently, our partner Cloudflare, the leading connectivity cloud company, announced a new evolution in how website owners can manage AI crawler access to their site content and determine how this information is used. 

Going forward, AI companies working with Cloudflare will verify their crawler activities and specify whether they are crawling content for training, content generation, search purposes, or assistant-related operations. This is a significant step forward in transparency and control over AI crawler access to websites and will be a key strategic lever for all website owners in the near future.

While there are no immediate changes for Webflow sites, we’re collaborating with Cloudflare to develop new functionality that will soon give our customers advanced control over how their work is accessed and reused online. This reinforces our commitment to providing best-in-class tools that help teams build, manage, and optimize their web experiences.

Managing search engines crawlers and AI bots in Webflow

Given all that is changing on the web with this announcement, it felt like an opportune moment to resurface some details about how Webflow customers can effectively manage access to, and usage of, their site content by AI bots today.

Webflow gives you direct control over how your content is indexed by both traditional search engines and AI crawlers and our SEO tools as well as our marketplace of AI SEO apps and partners give you a number of ways to prevent search engines and AI bots from indexing pages, folders, your entire site, CMS items, or just your webflow.io subdomain. This can be useful for hiding pages, like your site’s 404 page, from being indexed and listed in search results, or controlling which parts of your site an AI bot can use, and for which purpose.

For example, you can block your entire site from being indexed by enabling “Disable Webflow subdomain indexing” in the SEO section of your Site Settings on the Webflow Dashboard. Or you can get more targeted and disable indexing on specific pages by unchecking “Index this page” in the SEO section of your Page Settings. You can learn more about configuring your site’s settings here. 

Webflow also lets you define a custom robots.txt file under Site Settings > SEO, giving you the ability to block specific bots by user-agent. Your `robots.txt` file is crucial for controlling how AI tools interact with your website. It allows you to protect sensitive content, manage bandwidth, and guide the ethical use of your content by automated systems.

For example, to request that OpenAI’s GPTBot doesn’t crawl your any page on your site:

User-agent: GPTBot
Disallow: /

Another example would be if you want to prevent Google’s Gemini and AI model training services (not the traditional Search or Ads products that use User-agent: Googlebot) from crawling a specific part of your site, such as the /drafts/ directory: 

User-agent: Google-Extended
Disallow: /drafts/

These rules are respected by most well-behaved crawlers, including search engines and leading AI agents. While they don’t remove previously indexed content immediately, they are standard signals that prevent future indexing. 

With Webflow’s SEO and bot control tools, you have the flexibility to protect sensitive content, manage search visibility, and control how AI systems interact with your site. You can learn more about customizing your robots.txt rules in our user docs.

Looking forward

We are thrilled to continue working alongside Cloudflare to provide our customers with the tools and control they need as the web continues to evolve faster than ever before – so keep your eyes peeled for more updates as we continue shipping the features your team needs to excel in an increasingly AI-driven digital landscape.