# Search Engine Robots

The Magento 2 configuration includes settings to generate and manage instructions for web crawlers and bots that index your site. The instructions are saved in a file called “robots.txt” that resides at the root of your Magento installation.

With the ScandiPWA theme, feel free to use the default Magento functionality to upload and configure the robots.txt.

### WHERE TO CONFIGURE?

1. On the Admin sidebar, go to **Content** → Design → **Configuration** 
2. Open **Global** configuration in **edit** mode 
3. Expand **Search Engine Robots**. 
4. Insert the instructions in **Edit Custom instruction of robots.txt** file and set the Default Robots. 
5. When complete, tap **Save Configuration**.

### Robots.tx template  <a href="#searchenginerobots-robots.txtemplate" id="searchenginerobots-robots.txtemplate"></a>

Even in the regular Magento projects, there are cases when we upload robots.txt directly to the server root directory and ignore the field in BE. The default robots.txt is usually taken and adjusted based on the project. As a template you are welcome to use the following generic robots.txt which can be used for your store that you can insert in the **Edit Custom instruction of robots.txt file** field in the Search Engine Robots configurations:

{% file src="<https://1702077504-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MJEKoY_RLJk3C3Fcnu6%2F-MKKzkmUeE6BaaCM9zIq%2F-MKL-qzXVOX8GFkvrSMu%2Frobots.txt?alt=media&token=b9af7d9a-0003-43d2-b567-aadcf43d4cf5>" %}

This file contains:&#x20;

* Standard M2 directories
* URLs that appear in FE but should be excluded from the index, e.g., My Account
* SID
* /catalog/view/


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://manual.scandipwa.com/v5.0.3/seo-and-search/seo-best-practices/search-engine-robots.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.