Index problem with my top of page

hafuuu
New Member
11 0 0

I just have launched my pages a week ago(https://interior-and-furniture.myshopify.com)

and I tried to send xml file to Search console to be fetched several times ,but it shows: couldn`t fetched.

I assume this is because robots.txt status is disallowed.(https://interior-and-furniture.myshopify.com/robots.txt)

However, shopify support says you cannot edit robots.txt status.

this topic says need new primary domain to be crawled by google:

https://community.shopify.com/c/Ecommerce-Marketing/Google-Search-Console-Sitemap-Error/td-p/552894

Is there any other way to crawled by google?

or Is there any other cause to prevent my pages from google crawling?

 

 

0 Likes
Edward-Newgate
Navigator
505 21 60

Hi @hafuuu 

As you said, the reason is your robot.txt file disallows Google bots so that bots can not visit your URLs and as a result, your pages don't get indexed.

To solve this issue you should change your robot.txt to allow status. Also, this guide: How to get your online shop on Google would help your pages get indexed more quickly.

Hope it helps!

Was my reply helpful? Click Like or Mark it as an Accepted Solution to let me know!
0 Likes

Hi @hafuuu

I checked your robots.txt file and there is nothing wrong with it. The XML file is not disallowed in your robots.txt file. All those disallowed URLs in your robots.txt are correct as it should be (since Google would not index pages like Orders, Cart, Account, or Collection search pages like /*/collections/*%2B*

Screenshot (48).png

Plus, how did you send your XML file to Google Search Console? There are two options when it comes to submitting XML files: 

 

  • Domain - Registering your store’s entire domain means that Google will be tracking data from all URLs on your website. This includes all subdomains (m. // www. // etc.) and all protocols (http // https // etc.)
  • URL Prefix - Registering a URL prefix means that Google will be tracking data on just this one specific URL. No different subdomains or protocols will be tracked, meaning that you can track, for instance, your blog separately from the rest of your site.

-> Usually, you’ll want to go with domain to ensure that every single one of your pages is indexed by Google.

 

image19_c9a8cdff-a2a0-4dc2-a152-9a7134cd25b3_2048x2048.png

Check out this video tutorial to see if there are any steps you're missing. And if possible, I'd advise you buy your own domain instead of using the default myshopify.com since your domain won't have any authority and it'll be difficult when you need your pages to get ranked. 

Anyhoo, let me know if my answer if of any help with a thumb up, or like  

Cheers

PageFly - Advanced Shopify Page Builder - Empowering 90.000+ active merchants.

Check our Evergreen Guide on Moving Your Brick-n-Mortar Store Online.

Want To Strive The Coolest E-commerce Knowledge? Explore Our FREE Video Resources.

Trevor
Community Moderator
Community Moderator
2557 334 522

Hello!

Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes
Steve759
New Member
3 0 0

Hi @Trevor ,

I need your help!

How I can remove some URLs from the sitemap.xml?
Documentation here (https://shopify.dev/themes/seo/robots-txt) allows us to disallow some URLs for indexing and we can add extra URL to the sitemap.
But don't know how to remove some URLs from sitemap.

Thank you.

0 Likes
Trevor
Community Moderator
Community Moderator
2557 334 522

@Steve759 

I'm afraid that would be outside my scope of expertise and outside the cope of Shopify Support. If you would like assistance customizing your robots.txt file you will need to hire a Shopify Expert as Shopify Support can not help with the edits you would like to make, nor provide feedback before implementing changes.

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes
Steve759
New Member
3 0 0

Hi @Trevor ,

Should I check with Shopify technical support team?

Thank you.

0 Likes
Trevor
Community Moderator
Community Moderator
2557 334 522

@Steve759 

Shopify does not support making changes to your robot.txt file. I would recommend creating a post in the Technical Q&A thread to see if other community members can provide an answer, or hire a Shopify Expert. 

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes