The robots.txt text file is a key element of SEO for eCommerce sites. Ecommerce websites are typically larger than other sites on the Web. They also have features like faceted navigation, which can exponentially increase their size. These sites must be able to control the way Google crawls their site. This allows these sites to manage their crawl budget and avoid low-quality pages being crawled by Googlebot.
The robots.txt issue with Shopify has been a long-standing problem for the SEO community. Shopify SEO’s biggest problem has been the inability to control the robots.txt for many years. This made it more difficult for the platform to use than other platforms such as SEO For Magento. Users have always been able edit the robots.txt. Although the default robots.txt blocks crawlers well, there are some sites that require modifications to this file. Shopify sites are becoming more popular and robust as we see more people using it. This means that robots.txt is required to perform more crawl intervention.
Shopify has done a tremendous job improving the user experience on their platform. Shopify has announced that from June 2021 you can now customize the robots.txt files for your site.
This is great news for SEOs as well as Shopify store owners, who have been waiting for years for the file to be adjusted. This also shows Shopify is listening to SEOs and taking steps to improve their platform search-wise.
Now that you know how to edit the file, let us talk about making those adjustments in situations where it might be worth doing so.
What is Shopify Robots.txt?
Shopify robots.txt tells search engines which URLs they should crawl on your site. The robots.txt can be used to block search engines from crawling low-quality pages. A file called robots.txt.liquid is used to generate the Shopify robots.txt.
What does Shopify’s default robots.txt block?
You might see a robots.txt already configured on your Shopify website. This file can be found by going to:
domain.com/robots.txt
This robots.txt file will show you that there are already many preconfigured rules.
These rules can be used to prevent search engines from crawling unneeded pages. Here are the top Shopify rules.txt files:
- Block: /search – Blocks site search
- Disallow : /cartBlocks Shopping Cart Page
- Blocks: /checkout
- Blocks: /account
- Blocks duplicate category pages created by the faceted navigation
- Sitemap – [Sitemap Links] – Refers to the sitemap.xml Link
Shopify’s default rules block the crawling of low quality pages on most websites. It’s probable that most Shopify store owners won’t have to modify their robots.txt files. Most cases should be handled by the default configuration. Shopify sites are generally smaller and crawl control shouldn’t be a problem for them.
As more websites adopt Shopify, the number of websites is growing. We are also seeing more websites with custom configurations, where the default robots.txt rules don’t suffice.
Shopify’s accounting rules are good enough for most cases. However, store owners may need to add rules to customize robots.txt to fit their site. You can do this by editing and creating a robots.txt.liquid.
How do you create the Shopify Robots.txt.liquid
The following steps can be used to create Shopify robots.txt.liquid files:
- Navigate to Online Store > Themes in the left sidebar on your Shopify admin page.
- Select Actions > Edit code
- Click the link “Add a New Template” under “Templates”.
- Click the dropdown to the left and select “robots.txt”.
- Select “Create template”
The Shopify robots.txt.liquid should now be visible in the editor.
How do you edit the Shopify Robots.txt file?
Adding a Rule
You can add additional code blocks to the robots.txt.liquid file to add a Shopify rule.
%- if group.user_agent.value == ‘*’ -%
‘Disallow: [URLPath]‘
% – endif –%
If your Shopify site uses the /search-results/ internal search function, and you want it to be blocked with robots.txt you can add the following command.
%- if group.user_agent.value == ‘*’ -%
{{ ‘Disallow: /search-results/. *’ }}
% -endif -%
You would add these two blocks to your file to block multiple directories (/search–results/ and /private/).
%- if group.user_agent.value == ‘*’ -%
{{ ‘Disallow: /search-results/. *’ }}
% -%
%- if group.user_agent.value == ‘*’ -%
{{ ‘Disallow: /private/. *’ }}
% – Endif -%
These lines should now populate in the Shopify robots.txt files:
Potential Use Cases
Despite the fact that robots.txt is sufficient for most websites, what situations could your site benefit from having Shopify’s robots.txt.liquid files edited? Here are some common scenarios where you might consider changing yours.
Internal Site Search
Blocking a site’s internal searches via robots.txt is a good SEO practice. Because there are so many queries users can type into a search engine, this is a good SEO practice. These pages could be crawled by Google, which could result in a lot of low-quality search results pages appearing in the index.
Shopify’s default bots.txt blocks standard internal searches with the following command
Disallow: /search
However, many Shopify sites don’t use Shopify’s default internal search. Many Shopify sites use apps or other internal search technology, as we have found. This often changes the URL for the internal search. This means that Shopify’s default rules no longer protect your site.
This site, for example, shows URLs that include /pages/search as the path.
These internal search URLs can be crawled and indexed by Google.
This website may want to edit Shopify’s robots.txt rules in order to add custom commands to stop Google from crawling your /pages/search directory.
Faceted Navigations
You might consider changing your Shopify robots.txt file if your site uses a faceted navigation. Faceted navigation is a filtering option that you can use on category pages. These are usually located on the left-hand corner of the page. Shopify allows users to filter products by color and size.
If we choose the “Yellow”, and “Black” color filters, it will show us that the URL with the “color” parameter has been loaded.
Shopify’s default robots.txt is able to block any page paths that a multi-faceted navigation may create. However, it cannot account for all possible uses. Google will be able to crawl this page because “color” has not been blocked.
Another reason to block Shopify pages using robots.txt is this. We might consider blocking many of these URLs to decrease the crawl of similar pages. This site could identify all parameters of the faceted navigation they wish to block (size, colour), and then create rules in robots.txt that will block their crawl.
Navigation Sorting
Many eCommerce websites offer sorting functionality on their category pages, similar to faceted navigation. These pages allow users to see products on their category pages in a different order.
This creates the problem of duplicate/similar content. These pages are variations on the original category page, but the products are in a different order. You can see below how a parameterized URL that sorts products alphabetically is created when you select “Alphabetically A-Z”. This URL uses the “.q” parameter added to the end.
This URL is not unique and should be crawled. It simply contains the same products as the original category pages, but sorted in a different way. The Shopify site may want to add a robots.txt policy that blocks crawling of all URLs ending in “?q”.
Conclusion
Shopify’s robots.txt.liquid file gives SEOs much more control over crawling their sites than they did previously. Shopify’s default robots.txt is sufficient to keep search engines away from undesirable areas on most sites. However, you may want to make adjustments if an edge case applies to your site. The likelihood of making adjustments to the robots.txt files will increase the bigger your store and the more customization you make to it. You can reach us if you have any questions regarding the robots.txt and Shopify SEO agency service.
