Shopify Discussion

Dan_Lawrence · ‎02-21-2019

Hi,

I want to run my dev zone site through Screaming Frog SEO web crawler to check its seo but following the web-forms authentication method brings up a warning notice that this could even lead to data loss etc

So to avoid potential problems i want to temporarily go live with it so i can crawl it, avoiding the web authentication method

Obviously its absolutly crucial the dev zone doesnt get indexed by Google whilst i do so, so i know i need to apply the no-index via the robots.txt file or globally via the header first.

Then i can go live knowing it wont get crawled by google and i can crawl it with Screaming Frog, obtain the report and then take it offline again after

So pls advise either how to do this (edit robots.txt or apply no-index globally via the header) ?

Or of course if you have knwoeledge of crawling shopify dev sites with Screaming Frog how best to do that, if not via the above ?

All Best

Dan

Dan Lawrence https://DigitalNomad.Global

Torkel_Öhman · ‎02-23-2019

Hi,

Is it a new site or an existing site? If it is a new site, I wouldn’t worry about Google indexing your site if you have it up temporary – unless you claim your site in Search Console, set up GA and so on it will take some time.

If it is a site that is up and indexed, just set nofollow & noindex tags in the header

You can read more here

https://developers.google.com/search/reference/robots_meta_tag

Best regards & good luck 🙂

Torkel

Dan_Lawrence · ‎02-25-2019

HI Torkel

Thanks for your reply

Thats why i trying to find out, how to add nofollow & noindex tags in the header via shopify ?

All Best

Dan

Dan Lawrence https://DigitalNomad.Global

Torkel_Öhman · ‎02-25-2019

Hey,

Here is a guide to how to do it, hope that helps.

https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines

And when you are ready, if you need help with setting up Google Shopping & Google Ads we have an app that is 100% automatic - 30-days free trial so you can see the results before you pay for it.

Best of luck 🙂

Dan_Lawrence · ‎02-26-2019

Thanks Torkel

ok i've read that page and says options for excluding an individual page, as well as the search template (whats that ??) BUT no instructions as far as i can see for excluding the entire site temporarily

So i would greatly appreciate it if you could let me know what to put in for excluding the entire site ?

Yes of course ill check it out your app, whats the link ?

All Best

Dan

Dan Lawrence https://DigitalNomad.Global

Torkel_Öhman · ‎02-26-2019

If you want the whole site excluded paste this line

<meta name="robots" content="noindex, nofollow">

into theme.liquid

You can paste it anywhere beteween <head> and </head>

And remember to remove it when you go live 🙂

I would also advise you to connect search console, sitemaps and Google Analytics when you go live.

Here is the app, appriciate if you check it out

The app is https://apps.shopify.com/amanda-automatic-google-ads

Dan_Lawrence · ‎02-26-2019

That's great thanks so much Torkel

Yup i've been doing SEO everyday since 1998 believe it or not (just not on the technical side and never with shopify), so yes know to add GSC, GA etc etc

Thanks for the app link ill check it out once im live

All Best

Dan

Dan Lawrence https://DigitalNomad.Global

Josh_Uebergang · ‎03-05-2019

Dan, add the nofollow and noindex directive then flick the website live. Just don't set your domain configuration to redirect to a primary domain. Then you can use the myshopify.com link of the store to crawl as you wish. This lets you test redirects and broken links within Screaming Frog. All URLs should be relative so that when you go live on the primary domain, everything is correctly canonicalized to the final URL source without redirect chains. Being an SEO vet, you're probably familiar with Screaming Frog, but in my Shopify SEO guide, I share some other ways I use the tool for Shopify stores, some of which is different to how you typically approach other platforms.

Run Google Shopping ads? Get the free definitive guide to Google Shopping for Shopify (no optin required): https://www.digitaldarts.com.au/google-shopping

Dan_Lawrence · ‎03-05-2019

That's brilliant thanks so much Josh

Dan Lawrence https://DigitalNomad.Global

russhuq · ‎03-12-2019

Hey Dan,

I actually had kind of a similar situation. The staging store was both password-protected and no-follow, no-index meta tagged. Here is what I did to get the store crawled

1. Ticked 'Follow Internal nofollow' under 'Configuration > Spider'

2. Login to the site using Forms-based authentication under 'Configuration > Authentication' which you already know

Thank you!

jackyle · ‎03-13-2019

Hi
This is a good question and I also want to find the same answer as you

But I wonder if we add nofollow & noindex code in this meta tag which will cause search engines like google though Not indexed but will we suspect spam?

Dan_Lawrence · ‎05-16-2019

Great info thanks so much 🙂

Dan Lawrence https://DigitalNomad.Global

Trevor · ‎06-21-2021

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.

Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify (Out of Office until Sept 7th)
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog

Shopify Discussion

Screaming Frog - how best to access a password protected site ??

Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Re: Screaming Frog - how best to access a password protected site ??

Community Blog Articles

How you can use PayPal to accept payments and pay your Shopify bill

Improving your online store speed

Optimizing Storefront for Mobile Shopping

Shopify Community

Support

Shopify

Quick Links