University
Home
Search
⌘E
Dashboard
Try Webflow — it's free
Learn
Courses
Docs
Videos
Interactive Learning
Glossary
Certifications
Engage
Support
Community
Events
Forum
Accessibility
Switch to Light mode
Switch to Dark mode
High contrast
Glossary
Robots.txt

Robots.txt

A robots.txt file instructs search engine crawlers which webpages and sections they can or can't crawl and index. They exist in a website's root directory (the topmost folder in a file system) and regulate how search engine crawlers should behave while indexing a website.

Think of a robots.txt file as traffic signs for crawlers — steering them toward authorized areas while cautioning against others. Web developers create robots.txt files to regulate how search engines interact with websites, allowing site owners to protect sensitive information, prevent content duplication, and keep maintenance areas hidden.

However, robots.txt files have specific limitations. For example, not all search engines support robot.txt files — Google’s crawlers are likely to follow a file’s rules, but other crawlers may interpret syntax differently. As a result, understanding index-blocking methods, such as the “noindex” directive and the correct syntax, is vital for protecting website information.

Visit Webflow University to learn how to create a robots.txt file for your website.

Relevant courses

Web animations with After Effects & Lottie

Create animations in Adobe After Effects — the industry standard for motion graphics. Embed and animate them in your Webflow site.

Welcome to Webflow

Discover what Webflow offers, how it works behind-the-scenes, and tips and tricks for starting your learning journey.

Moving your business online

A free course on how to take your business online. Serve more clients and generate more revenue in this new environment.

Figma to Webflow

Learn the entire design process from idea to final output as we take you through Figma, Cinema 4D and Octane, and Webflow.

Freelance web design boot camp

The Freelancer’s Journey is a free, comprehensive course to help you build a successful, fulfilling freelance career in web design.

Getting started with Webflow Enterprise

Get familiar with Webflow’s website experience platform and Enterprise features.

Other glossary terms

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Select field

→
A form element that allows you to make a selection in a dropdown list.

Descendant selector

→
A descendant selector is a CSS (Cascading Style Sheets) selector that targets elements nested inside a parent element.

Algorithm

→
An algorithm is a set of rules or steps that help solve problems or carry out specific tasks.

Latency

→
Network latency is the time data needs to travel from one point to another, impacting the responsiveness and speed of web applications and services.

Domain-Specific Language (DSL)

→
A Domain-Specific Language (DSL) is computer script that solves software problems in a specific domain.

Pseudocode

→
Pseudocode is a simplified, informal representation of programming code that outlines a program’s structure and logic without strict syntax.

Shared Workspace template

→
Shared Workspace templates are templates made from your own sites. They can help you create sites with similar layouts and increase your efficiency when designing. Your Workspace members will also have access to your shared Workspace template(s).

Checkbox

→
Checkboxes are HTML structures that allow visitors to input data.

GIF

→
A GIF (Graphics Interchange Format) is a digital image format that uses multiple frames of an image or soundless video to create a short, looping animation.

Container

→
An element that contains content.

Modal window

→
A modal window (or modal) is a web element that displays above all content and restricts user interaction until it’s manually dismissed.

Behavioral targeting

→
Behavioral targeting is a marketing technique that leverages user data to deliver personalized ads to drive conversions.

Node title

→
The type of element you're working with, i.e., H1, link, button, etc.

SaaS

→
Software as a Service (SaaS) is a cloud-based approach to delivering software on a subscription basis without requiring local installation.

Typography section

→
A section in the Style panel that houses CSS properties related to how typography is displayed such as font, font size and height, and align.

Span text

→
The span tag is used to style inline-text.

Plain text field

→
A plain text field is an input field for short-form text without formatting or styling.

Collection field

→
An individual field within a Collection. You'll select which fields to display in dynamic Collection lists and Collection pages.

Content network

→
A content network is a collection of websites where advertisers can pay to display digital ads.

Date/Time field

→
A field available in CMS Collections. This field lets you display a date and time in a Collection and/or Collection item.

Attribute

→
An attribute in HTML is a term inside an opening tag of an element, such as a font or color, that provides additional information about the element.

Element settings panel

→
The second panel on the right side of the Designer. Here, you can change element-specific settings such as the element ID, Custom attributes, Search index settings, Editor settings, etc. You can learn more about the element settings for a specific element by searching for the element’s name on Webflow University.

DHTML

→
DHTML, or Dynamic HTML, is a web-development technique combining HTML, CSS, and JavaScript to create dynamic, interactive web pages and complex web applications.

Rich text field

→
A rich text field is an input tool for formatting or applying text styles, such as bold, colors, font sizes, italics, underline, and other visual modifications.

Language code

→
A language code is a standardized code of letters and numbers that represent different languages.

Object-oriented programming (OOP)

→
Object-oriented programming (OOP) is a programming approach that organizes code into self-contained units of data, functions, and behavior.

Cryptanalysis

→
Cryptanalysis examines cryptographic systems to find weaknesses that could allow attackers to uncover plaintext or break encryption.

Descriptor

→
A CSS descriptor sets specific properties and values of an at-rule (@-rule) element, dictating its appearance and behavior on a web page.

Last updated

→
The last time someone made a change to your site.

Em

→
An em in CSS is a scalable and relative unit of measurement that defines the size of an element relative to its parent's font size.

Serialization

→
Serialization is the process of converting data into an optimized format for storage, transmission, and sharing.

Password protection

→
Password protection is a security feature restricting certain website areas to authorized users, requiring a secret phrase or key to access them.

ALPN

→
ALPN stands for Application-Layer Protocol Negotiation. It’s a Transport Layer Security (TLS) extension that aids client and server identification during secure data transfers.

Publish site

→
The act of updating your live site to reflect new design changes.

Height: auto

→
The "height: auto" CSS property automatically adjusts an element’s height based on its content.

Google Universal Analytics Tracking ID

→
A unique ID you can use to link your Google Analytics account to your Webflow site.

CSS preprocessor

→
A CSS preprocessor is a program that helps developers write Cascading Style Sheets (CSS) by providing advanced features such as variables, functions, and nesting.

Display: block

→
Display properties tell browsers how an element should behave on the page. The most common display property keyword category is display: block.

Flex child

→
An element contained within a parent element that has display: flex set.

Header

→
A header is a section at the top of a webpage.

Cookie

→
Cookies are small text files sent by a website to a user’s web browser that enhance their browsing experience by remembering site preferences and activities.

Number field

→
Available in CMS Collections. It lets you insert an integer with or without decimals.

Left toolbar

→
The vertical bar of tools on the left side of the Webflow Designer. It holds the Add panel, Symbols panel, Navigator panel, Pages panel, CMS panel, Ecommerce panel, Assets panel, Settings panel, Site Activity log, Audits, Search, Video tutorials, and Help.

Canvas

→
The <canvas> tag in HTML5 creates a container for dynamic graphics and image rendering on a webpage.

Deprecated

→
Deprecated HTML tags are old tags replaced by updated alternatives in modern web design and development.

Closed-loop marketing

→
Closed-loop marketing is an ecommerce marketing strategy that uses customer data to create and improve targeted content.

Image field

→
Available in CMS Collections. With this field, you can upload images from your computer and use them as dynamic items in your individual Collection item.

Text area

→
A form element that allows site visitors to enter multi-line data.

Callback function

→
A callback function is passed as an argument to another function and executed upon completing the latter’s task.

Breadcrumb

→
A breadcrumb, or breadcrumb trail, is a navigational element representing a user’s current location within a website's hierarchical structure.

Email template

→
Email templates — or pre-designed HTML layouts — ensure consistent branding and visual aesthetic in marketing campaigns.

Parent

→
The element directly above in the hierarchy tree.

Adobe Fonts API token

→
An Adobe Fonts API token protects and supports your data from Adobe Fonts with other programs you may be using to personalize your website’s fonts.

Robots.txt

→
A robots.txt file instructs search engine crawlers which webpages and sections they can or can't crawl and index.

Abstraction

→
In object-oriented programming (OOP), abstraction conceals irrelevant details so they don’t interfere with the task at hand.

CRLF

→
On Windows-based systems, the sequence of two ASCII characters representing a line break in text files is called "Carriage Return, Line Feed," or CRLF.

Web server

→
A web server is a computer program or hardware device that delivers web content to clients using HTTP. It handles client requests for web pages, images, and videos.

Workspace

→
A space to hold and access sites, members, settings, and integrations. All users start with a Workspace that can become collaborative when and as needed.

Contextual advertising

→
Contextual advertising is a form of paid advertising that matches web content with ad intent.

Twitter button

→
A Webflow element that embeds a Twitter Follow or Tweet button into your site.

Graphical User Interface (GUI)

→
A Graphical User Interface (GUI) is a visual medium that enables visual interaction with software applications or electronic devices through elements like icons, buttons, and menus.

Optimizely

→
A web app that helps you measure how different site designs impact user behavior.

Collection item

→
An individual item within a Collection. For example, in a “Blog Posts” Collection, the Collection items are individual blog posts.

Text block

→
A typography element that is used when no other element is specifically suitable.

Unpublished change

→
A change that has been made in the Webflow Editor but will not appear on the live website until it’s published. You can track any unpublished changes on the bottom right of the Editor toolbar.

Percent

→
A measurement unit relative to an element's parent.

Pagination settings

→
Through the pagination settings, you can preview Collection list pages by choosing the page number from the page dropdown list, or you can switch to the previous or next page using the arrow keys.

Favicon

→
A 16x16 px image that displays in browser tabs, bookmarks, and other browser areas. It's typically a simplified version of the site's logo.

Document type definition (DTD)

→
A document type definition (DTD) defines the structure, validity, and attributes of standard generalized markup language (SGML) documents, such as GML, HTML, and XML.

Hotlink

→
A hotlink — also known as a remote or inline link — is a hyperlink that directs to a resource, such as an image or a file, without the owner’s explicit permission.

CSS Filters

→
CSS filters apply visual effects to HTML elements and their children using Cascading Style Sheets (CSS).

Element

→
Elements are individual HTML components that constitute a webpage’s structure and contents.

Front-end

→
The front-end refers to the visual and interactive part of a website or application that users directly interact with.

Empty state

→
The state displayed when a Collection contains no Collection items, which gives you the opportunity to design for a situation where there's no content to display. Check out the Empty States blog for inspiration.

Minimum height

→
The min-height CSS property defines an element’s minimum height.

Resource Description Framework (RDF)

→
The Resource Description Framework (RDF) is a standardized framework developed by the World Wide Web Consortium (W3C) that facilitates automated data exchange between web systems and applications.

Below the fold

→
Below the fold refers to the lower portion of a webpage that isn’t immediately visible to visitors upon loading until they scroll down.

Maximum width (max width)

→
The “max-width” CSS property defines an element’s maximum width.

Delete element

→
To remove an element and its content from your website.

Dashboard

→
From your Dashboard, you can access, create, and manage all your sites.

Right-side panel

→
The far right vertical panel in the Designer that houses the Style panel, Element settings panel, Style manager panel, and Interactions panel.

Read-only link

→
A way for others to test/debug your Webflow site in the Designer without being able to save or make permanent changes.

Display: inline

→
The display: inline CSS property is a display option that controls an element’s appearance and layout.

Product attribute

→
Product attributes are specific characteristics that define purchasable merchandise or commodities and provide relevant information to customers.

Registrar

→
A registrar, or domain name server (DNS) registrar, manages the sale of domain names and the assignment of IP addresses, ensuring each domain’s uniqueness.

Parent folder

→
A parent folder is a directory containing one or more folders and files.

Div block

→
A generic element that is typically used to group content, or when no other element is specifically or semantically suitable. Div blocks have no effect on the content or layout until styled with CSS.

Display property

→
Display properties operate like instructions that tell a browser how to display the different parts of a webpage. You can customize an element’s appearance and layout by changing its display property.

Folder settings

→
Allows you to manage your folders.

Mutable

→
In JavaScript, mutable objects are dynamic elements that allow post-creation changes and modifications.

Authorized applications

→
Third-party applications that you've given access to your personal and/or website data.

Automagically

→
Automagically describes complex content or functions that appear as if they were created using magic.

Dynamic element

→
Dynamic elements are web page components that can change based on user interaction or external factors, like a live news feed, without requiring a page refresh or reload.

Folder

→
A folder is a subdirectory of your site's root folder for organizing web pages.

Client-side

→
Client-side is the user-facing part of a web app, responsible for processing user input and displaying content.

Footer

→
A footer is a section at the bottom of a web page.

Save

→
The Designer automatically saves your progress every 10 seconds.

Element breadcrumbs

→
A navigational aid that displays in the bottom of the Webflow Designer to help you keep track of the element you're interacting with in relation to its hierarchy.

Common targets

→
All elements on the page targeted by the same selector.

PNG (Portable Network Graphics)

→
PNG, short for Portable Network Graphics, is a lossless image format known for its transparency support and quality retention after compression.
Hmm…we couldn’t find any results. Try a different search term or reset the filter.
Reset the filter
Load more