Website robots.txt file

TL;DR / Overview

The robots.txt file controls how search engines crawl your website. In Nilead, you can update this file easily through the admin panel. Just be cautious with the auto-generated sitemap line — it ensures search engines can find your site map and index your pages properly.

Prerequisites

• Admin access to your Nilead website

• Basic understanding of search engine indexing

• Optional: Familiarity with robots.txt syntax

Walkthrough

What Is the Robots.txt File?

robots.txt is a plain text file stored at the root of your website that tells web crawlers (like Googlebot) which parts of your site they can or cannot access.

Common use cases:

• Block access to certain folders or pages

• Prevent duplicate content from being indexed

• Point search engines to your sitemap

How to Edit Robots.txt in Nilead

To access and update your robots file:

1. Open the Quick Menu in the Nilead admin.

2. Search for “Robots” and select the Robots.txt management tool.

3. Edit the file contents as needed.

4. Click Save to apply your changes.

Example:

User-agent: * Disallow: /private/

Important Note on the Sitemap Line

Nilead automatically appends a Sitemap: line at the bottom of your robots.txt file. This line tells search engines where to find your sitemap.

Example:

Sitemap: https://yourdomain.com/sitemap.xml

🚫 Do not delete or modify this line unless you are absolutely sure. Incorrect edits can disrupt indexing of your entire site.

FAQs

1. Can I block certain pages from being indexed?
Yes. Use Disallow: followed by the path. For example: Disallow: /thank-you.

2. What happens if I remove the Sitemap line?
Search engines may not find your sitemap automatically, which can negatively affect your site’s indexing and SEO performance.

3. Can I edit robots.txt for specific bots like Bingbot?
Yes. You can define rules per bot using User-agent: sections.

4. Will changes to robots.txt affect existing indexed pages?
Yes. If you block pages that were previously indexed, search engines will eventually remove them.

5. Can I preview the file before saving?
Currently, there’s no live preview, but changes are instant and can be reverted manually if needed.