User Manual
DocsCreate ScandiPWA AppCreate Magento AppGitHub
v5.0.0
v5.0.0
  • ScandiPWA User Manual
  • Basic Configuration
    • Storefront Branding
      • Uploading Your Logo
      • Adding a Favicon
      • Copyright Notice
      • Store Demo Notice
      • Welcome Message
    • Store Details
      • Locale Options
      • Country Options
      • Currency
      • Contact Us
      • State Options
    • Websites, Stores & View
      • Scope
      • Single Store Mode
    • Industry Compliance
      • Privacy Policy
      • Cookie Law Compliance
  • Catalog
    • CMS Styling
    • Managing Price
      • Group Price
      • Tier Price
      • Special Price
    • Managing Inventory
      • Stock Options
      • Product Stock Options
      • Stock Management Methods
      • Product Alerts
    • Product Settings
      • Advanced Product Settings
      • Search Engine Optimization
      • Customizable Options
    • Related Products, Up-Sells, and Cross-Sells
      • Related Products
      • Up-sells
      • Cross-sells
    • Catalog Images and Video
      • Uploading Product Images
      • Adding Product Video
      • Placeholders
      • Swatches
      • Image zoom
      • Watermarks
    • Categories
      • Display Settings
      • Content Settings
      • SEO for Categories
      • Design Settings
      • Products in Category
    • Creating Products
      • Configurable Product
      • Simple Product
      • Grouped Product
      • Virtual Product
      • Bundle Product
      • Downloadable Product
        • Configuring Download Options
    • Product Attributes
      • Attribute Settings
      • Attribute Input Types
    • Catalog URLs
  • MSI
    • Multi Source Inventory
  • Marketing
    • Marketing Configurations
    • Product Reviews
      • Ratings
    • Recently Viewed Products
    • Compare Products
    • Wish Lists
      • Configuring the Wish List
  • Promotions
    • Promotions Configurations
    • Cart Price Rules
    • Catalog Price Rules
  • SEO & Search
    • Catalog Navigation
      • Layered navigation
      • Price Navigation
      • Filterable Attributes
      • Pagination Control
      • Breadcrumb Trail
      • Product Listings
    • Catalog Search
    • URL Rewrites
    • SEO Best Practices
      • Metadata
      • Search Engine Robots
      • Using a Site Map
    • Search Terms
  • Content
    • Adding a New Page
    • Content Blocks
    • Widgets
    • HTML Head
    • Core Content
    • Default Pages
  • Customers
    • Account Dashboard
      • Address Book
      • My Orders
      • My Wishlist
      • My downloadable
      • Newsletter Subscription
    • Configuring Account Options
      • New Account Options
      • Password Options
    • Creating a Customer Account
    • Customer Sign In
    • Newsletter
  • Communication
    • Dotdigital Chat
    • RSS Feeds
  • Sales
    • Shopping Cart
      • Cart Configuration
    • Checkout Steps
    • Order Confirmation
    • Checkout Configuration
      • Terms and Conditions
    • Basic Shipping Methods
      • Free Shipping
      • Flat Rate
      • Table Rates
      • In-Store Delivery
    • Carriers
    • Shipping Settings
    • Payment Methods
  • Taxes
    • Tax Configuration
  • System
    • Google reCAPTCHA
    • CAPTCHA
  • ScandiPWA Customization
    • Slider Management
    • Color and Content Customization
    • Menu Manager
    • Web manifest customization
    • Layout direction
  • PWA
    • Introduction to ScandiPWA
    • Cross-browser support
    • Magento version mapping
    • Offline Notice
    • Installation to Home Screen
    • The "New Version Available" Notification
  • FAQ
    • Frequently Asked Questions
Powered by GitBook
On this page
  • WHERE TO CONFIGURE?
  • Robots.tx template

Was this helpful?

  1. SEO & Search
  2. SEO Best Practices

Search Engine Robots

PreviousMetadataNextUsing a Site Map

Last updated 3 years ago

Was this helpful?

The Magento 2 configuration includes settings to generate and manage instructions for web crawlers and bots that index your site. The instructions are saved in a file called “robots.txt” that resides at the root of your Magento installation.

With the ScandiPWA theme, feel free to use the default Magento functionality to upload and configure the robots.txt.

WHERE TO CONFIGURE?

  1. On the Admin sidebar, go to Content → Design → Configuration

  2. Open Global configuration in edit mode

  3. Expand Search Engine Robots.

  4. Insert the instructions in Edit Custom instruction of robots.txt file and set the Default Robots.

  5. When complete, tap Save Configuration.

Robots.tx template

Even in the regular Magento projects, there are cases when we upload robots.txt directly to the server root directory and ignore the field in BE. The default robots.txt is usually taken and adjusted based on the project. As a template you are welcome to use the following generic robots.txt which can be used for your store that you can insert in the Edit Custom instruction of robots.txt file field in the Search Engine Robots configurations:

This file contains:

  • Standard M2 directories

  • URLs that appear in FE but should be excluded from the index, e.g., My Account

  • SID

  • /catalog/view/

400B
robots.txt