Screaming Frog - how best to access a password protected site ??

Excursionist
33 0 1

Hi,

 
I want to run my dev zone site through Screaming Frog SEO web crawler to check its seo but following the web-forms authentication method brings up a warning notice that this could even lead to data loss etc
 
So to avoid potential problems i want to temporarily go live with it so i can crawl it, avoiding the web authentication method
 
Obviously its absolutly crucial the dev zone doesnt get indexed by Google whilst i do so, so i know i need to apply the no-index via the robots.txt file or globally via the header first. 
 
Then i can go live knowing it wont get crawled by google and i can crawl it with Screaming Frog, obtain the report and then take it offline again after
 
So pls advise either how to do this (edit robots.txt or apply no-index globally via the header) ?
 
Or of course if you have knwoeledge of crawling shopify dev sites with Screaming Frog how best to do that, if not via the above ?
 
All Best
Dan
Dan Lawrence https://DigitalNomad.Global
0 Likes
Excursionist
17 1 1

Hi,

 

Is it a new site or an existing site? If it is a new site, I wouldn’t worry about Google indexing your site if you have it up temporary – unless you claim your site in Search Console, set up GA and so on it will take some time.

 

If it is a site that is up and indexed, just set nofollow & noindex tags in the header

 

<META NAME="robots" CONTENT="noindex,nofollow">

 

You can read more here

https://developers.google.com/search/reference/robots_meta_tag

 

Best regards & good luck :)

Torkel

 

0 Likes
Excursionist
33 0 1

HI Torkel

 

Thanks for your reply

 

Thats why i trying to find out, how to add nofollow & noindex tags in the header via shopify ?

 

All Best

Dan

 

 

Dan Lawrence https://DigitalNomad.Global
0 Likes
Highlighted
Excursionist
17 1 1

Hey,

 

Here is a guide to how to do it, hope that helps.

 

https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines

 

And when you are ready, if you need help with setting up Google Shopping & Google Ads we have an app that is 100% automatic - 30-days free trial so you can see the results before you pay for it. 

 

Best of luck :)

0 Likes
Excursionist
33 0 1

Thanks Torkel

 

ok i've read that page and says options for excluding an individual page, as well as the search template (whats that ??) BUT no instructions as far as i can see for excluding the entire site temporarily 

 

So i would greatly appreciate it if you could let me know what to put in for excluding the entire site ?

 

Yes of course ill check it out your app, whats the link ?

 

All Best

Dan

 

Dan Lawrence https://DigitalNomad.Global
0 Likes
Excursionist
17 1 1

If you want the whole site excluded paste this line

<meta name="robots" content="noindex, nofollow">

into theme.liquid

 

You can paste it anywhere beteween <head> and </head>

 

And remember to remove it when you go live :)

I would also advise you to connect search console, sitemaps and Google Analytics when you go live.

 

Here is the app, appriciate if you check it out

The app is https://apps.shopify.com/amanda-automatic-google-ads

 

0 Likes
Excursionist
33 0 1

That's great thanks so much Torkel

 

Yup i've been doing SEO everyday since 1998 believe it or not (just not on the technical side and never with shopify), so yes know to add GSC, GA etc etc

 

Thanks for the app link ill check it out once im live

 

All Best

Dan

 

Dan Lawrence https://DigitalNomad.Global
0 Likes

Dan, add the nofollow and noindex directive then flick the website live. Just don't set your domain configuration to redirect to a primary domain. Then you can use the myshopify.com link of the store to crawl as you wish. This lets you test redirects and broken links within Screaming Frog. All URLs should be relative so that when you go live on the primary domain, everything is correctly canonicalized to the final URL source without redirect chains. Being an SEO vet, you're probably familiar with Screaming Frog, but in my Shopify SEO guide, I share some other ways I use the tool for Shopify stores, some of which is different to how you typically approach other platforms.

Run Google Shopping ads? Get the free definitive guide to Google Shopping for Shopify (no optin required): https://www.digitaldarts.com.au/google-shopping
2 Likes
Excursionist
33 0 1

That's brilliant thanks so much Josh

Dan Lawrence https://DigitalNomad.Global
0 Likes
New Member
1 0 1

Hey Dan,

 

           I actually had kind of a similar situation. The staging store was both password-protected and no-follow, no-index meta tagged. Here is what I did to get the store crawled 

 

          1.  Ticked 'Follow Internal nofollow' under 'Configuration > Spider'

 

 

              screaming-frog.jpg

 

           

        2. Login to the site using Forms-based authentication under 'Configuration > Authentication' which you already know

 

             screaming-frog-authentication.jpg

 

             Thank you!

 

           

 

1 Like