How can I remove a spam page from my website index?

[MVUILL]

Hi! 

I received an alert from google search console saying they excluded a page from indexation, and when I look at the page it's spam publicity: 

URL : https://www.french-address.com/collections/vendors?q=Buy%20FUT%2023%20coins%2C%20Cheap%20FIFA%2023%2...

How can I delete this page, I can't find it anywhere? 

Thx for your help!!

[Shay · Accepted solution]

Hi @MVUILL 

I can definitely understand your concern! Based on the URL you shared it looks like this might be a collection created within your admin or possible from the new Shopify Collabs service. 

The "spammy" content of that page you shared is what appears to be the collection page title. If you still cannot find this page within your store admin please reach out to our live support team for additional help with this. 

Please know that our theme and technical support team may be limited in what they can do depending on where/how this page was created. We generally can only support theme edits on our own in-house themes or technical issues that resolve around the Shopify platform itself. That doesn't mean they won't do everything they can to help you get this resolved! 

To contact live support please follow this link: Contact Support - Shopify Help Center.

[Participant]

@jcop24 Your observation matches mine. That is, I believe you are correct in saying, "If you search (for phrases like) "euromillion shopify" in google, numerous sites pull up but you can tell they are all malware on normal sites." However, I would suggest that use of the word "malware" might be an overstatement at this time. "Spam" yes, but "malware"? Let's hope it hasn't gotten to that point, yet.

Another search that you can try is this:

If there is a unique/unusual phrase which you have observed which is repeatedly showing up in the spam pages, Google that phrase along with "powered by Shopify" (which is the text that often appears in the footer of Shopify stores). I have been using those combined search terms for about a month, and have watched the number of spam-related search results grow exponentially from about 350,000 to almost 3 million today. Please note that I am not posting the unique/unusual search phrase that I used, because I can see that there are legitimate stores that have been hit with this spam, and I do not want their businesses to get caught up in this mess any more than they may have been caught up in it already.

Shopify: Time is running out. This system-wide problem can be fixed by you, and only you. This is not simply a "backlink" issue, because if the spam pages could not be created by exploiting the Shopify system, then the pages would not be indexed by Google (or any other search engine), and therefore there would be no risk of customers seeing the spam pages and being scared away from our store (or potentially millions of other Shopify stores).

@Shay @Greg-Bernhardt

[Italia-Straps]

Was there a deeper investigation by the technical team and if so, what was the result?

Thanks!

[NEI-Arlene]

I have no idea, unfortunately I just received the above message from Shay. I did however input the code I posted above to prevent Google Bots from crawling and indexing that page. That didn't work due to Google policy that says if another website has a link to a page on your site then google will still index that page regardless if your robot.txt is asking not index. So I went ahead and requested the link to be removed temporarily from google console under removals then temporary removals. I also requested it to be removed under 'Outdated Content'. I'm hesitant to use disavow option because I've read bad things about using that tool in terms of it affecting ranking. 
Still following his thread in hopes that someone comes up with a real solution. 

Best regards!

[Italia-Straps]

RE: Google Disavow Tool

 It is my understanding that this tool is for Spammy backlinks from OTHER sites pointing to your site as opposed to Spammy links ON YOUR SITE which is the case here. In other words, you are disavowing a URL on your own site not from an outside site. Therefore, the SEO implications could be unexpected.

Since this is such a widespread problem, it seems reasonable for Shopify to get in touch directly with Google and coordinate on the issue.   The spam content that store owners are seeing is pretty consistent. Therefore, it seems feasible that Google could just ignore this content being injected into Shopify stores going forward.

[Shay]

Hi @Italia-Straps 

Our team is aware of the concerns merchants have around these spam backlinks being flagged for their store. I wanted to clarify that these links are not being injected into your store, but are being created using the website's search function. You can easily replicate how these spam backlinks work yourself by using the same URL query on your own store:

https://www.yourstoreurlhere.com/collections/vendors?q=fuzzypinkslippers  

The page for the spam query only exists so long as the query URL for the page exists. The only way to stop this from happening is to remove the URL from the website that is creating it. 

[DaveSweetCures]

Hi Shay, 

The query string can be change to anything and shopify is currently set to print whatever the query is like for example 

https://www.french-address.com/collections/vendors?q=hi-shay

So all anyone wanting to take advantage of this needs to do it is know that a site is on shopify. I believe an answer lies in 404 these queries, rather than printing the query on the page - as to Google this just makes it a "unique page, suitable for indexing". 

This needs solving ASAP, we're up to 17k indexed pages for a site that should have 200. 

[eheurlin]

Hi, we also have a ticket open for this -  34752336 - i believe we have 3.65mln fake pages. We are working on fixing this, I hope that Shopify is taking this seriously. 

[CD]

Yours looks like mine. blah blah blah /search?q=ddos攻击安卓手机... Did you find any solution? 

Please advise! 

Thanks

[Alexis_Théry]

i have a ticket number 35498846 on my website https://ursul.fr

Working alone on my website and dont find any expert to help me ...

I dont understand why everybody says to disavow a link cause its not a link from another website. what should we do???

[Mont]

Yes same issue. Google is not impressed ranking has dropped.  

[Greg-Bernhardt]

It's not a hack or malware, but a limitation of whatever theme/search is being used. Category and search pages without results should be 404ing or set to noindex (can be done using liquid) and definitely not print out the query. That is the solution. See how SKIMS does it. This tactic does not work on their store: https://skims.com/collections/vendors?q=Buy%20FUT%2023%20coins%2C%20Cheap%20FIFA%2023%20coins%20for%...

[DaveSweetCures]

Thanks Greg, this would seem to be the best solution - I just wish you had more details of how to implement - but assume this will be different theme by theme?

We're currently working at updating our site to be similar to skims - so thanks for the starting point. But as an interim fix we've applied a catch all in the following files to at least prevent indexing in Google.

robots.txt.liquid:

{{ 'Disallow: /*?q=*' }}

theme.liquid:

 {% if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}<meta name="robots" content="noindex">
 {% endif %}

[NEI-Arlene]

Remember this won't work due to Google policy that says if another website has a link to a page on your site then google will still index that page regardless if your robot.txt is asking not index. 

There are so many people that enter the noindex code but are still experiencing the same issue. 

Did Skims use the disavow tool?

[DaveSweetCures]

Ok, so as temp fix V2 - I wonder about <link rel="canonical" href="https://www.yoursite.co.uk/" />, making this display when the path is equal to path == '/collections/vendors' and collection.all_products_count == 0 as that rectified an issue we were having months ago regarding tags and tags appearing in google as /tag1+tag2+tag3 and all the many variations? 

BTW, Shopify solution as of 11am this morning is still, "it's a link problem" you need to disavow all the links with Google - not our problem. 

Skims appear to be displaying a 404 when the product=0, which would in theory stop google displaying the link in Google as the end result would be a 404. I've requested our dev team to look into this and will update if this indeed is a fix - not sure if theme dependant.

[NEI-Arlene]

It's definitely not theme based as over 200,000 sites are being affected.  Honestly, this is a big issue and seems to be a vulnerability on Shopifys end. I have yet to find a proper solution but doing all I can. Any update would be greatly appreciated.  

 

[DaveSweetCures]

Apologies, when I say theme based I mean that many themes are impacted by this - and that it maybe a different fix per theme. It is interesting that Skims don't have this issue. I do believe the key is in presenting collection pages with 0 product results with a 404 - but just not sure how to do this. 

[Mels1]

Hi there, @DaveSweetCures 

Wondering if you could elaborate on your temp fix v2. Do you mean an alteration to: 

 {% if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}

<meta name="robots" content="noindex">
 {% endif %}

I have placed the code above in our theme yesterday, but if you see better results with an alterion. Please do share.

Cheers,

Melissa

[DaveSweetCures]

Hi @Mels1 

After a bit of sleepless night I'm at this point, placing the following code in theme.liquid 

{%- if canonical_url != blank -%}
{% request.path == '/collections/vendors' and collection.all_products_count == 0 -%}<meta name="robots" content="noindex"> <link rel="canonical" href="{{ shop.url }}" /> {% else %} <link rel="canonical" href="{{ canonical_url }}" />
{% endif %}
{%- endif -%}

The above will do two things, 

Add <meta name="robots" content="noindex"> to each ?q= page where the products for that vendor =0 

and it also places 

<link rel="canonical" href="www.yoursite.com" />

To give Google a second indication that you don't want these pages to be ranked in Google. 

Please note: 

1. This is a temp fix for us, we want to prevent the site printing the query on a page full stop. And are working on a fix to send these type of requests to a 404. 
2. That said we will probably leave this code in as a back up.
3. You can easily remove thousands of pages from Google by using there remove tool an selecting the prefix option  www.yoursite.com/collections/vendors?q=
4. Please check current canonical setup before deploying if not wanting to do that additional canonical setup use the code below (that you've used already).

{% if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}

<meta name="robots" content="noindex">
 {% endif %}

5. I will update once the team have sorted a fix that 404's these queries.
6. If anyone can write this better - please feel free to update - thank you. 

[Greg-Bernhardt]

I would remove the canonical part of your liquid, it's unnecessary. Pages that aren't indexed, are not ranked and if you en masse canonicalize pages without a canonical to the homepage you may be shooting yourself in the foot unless you know all the pages on your site that it may affect, but really again, it's not neccessary and only adds complexity that is a risk.

[fabmol1]

Hi, I just saw these links in my google search console. FS. Is there a non-techny solution, has @Shopify done anything about this? Does it affect our rankings??

[Greg-Bernhardt]

@fabmol1 have you read through this thread?

[Participant]

Hello, Greg. Until Shopify patches this system-wide vulnerability, can we please have an official piece of code that has been vetted by Shopify Staff which we can implement into our stores — along with an official, easy-to-find Help Center article that clearly explains what the code is for, where to install it, and what its limitations might be?

Because as it stands, Shopify support agents are simply telling us (store owners) that this is essentially not a Shopify problem, even though it most certainly is. The agents are telling us to read articles on third-party sites, or "report the problem to Google", or refer to the community forum, for answers. These types of directions from the support agents are nothing less than acts of passing-the-buck, in my opinion. Shopify needs to own this spam issue. And it is irrelevant that similar problems may, or may not, be happening on other platforms. Shopify needs to do better; that is why we chose to host our store here — for the time being.

It is inexcusable that Shopify has not implemented a system-wide measure to prevent these spam pages from being generated by now. Based on the accounts that other users have posted, and based on my own experience and research, this spam problem has been going on for several weeks, if not months. Ultimately, it is Shopify's system that is allowing these spam pages to be generated (and yes, they are pages, each with their own individual URLs), so it is Shopify, and only Shopify, that should be responsible for fixing the problem. It is not the store owner's responsibility. With all of the resources that Shopify has, why has a fix not been implemented by now?

In the meantime, I await your reply with official Shopify-vetted code which we can use as a temporary bandage.

[Jesper_Skaane_B]

Been with Shopify chat support for 2 hours now, and they still seem like they have no idea what i am talking about. This is embarrassing since the problem have existed since September.

My solution so far is this:

Disallow: /collections/vendors*
which apparently doesn't work, since Google will allow a disallow when it includes a link

making a "temporary removals --> New request" on  /collections/vendors prefix.
Which will help, but not fix the issue.

What other measurements could be taken? (i see two solutions in the thread, one by @Allan-EP and one by @DaveSweetCure

@greg - you mention that you are not directly involved in this issue, but simply trying to help. This is appreciated, but why are no representatives that are involved in this issue commenting in this thread?

We are all affected and in the dark as to how to proceed.

[DaveSweetCures]

If it helps you any the solution I posted seems to working in the sense the number of these pages indexed by Google is dropping in webmasters / Google console. And the total number of these pages being found and being crawled hasn't increased (so fix preventing the print of query seems to have helped prevent anymore of these links being created as its pointless for the attacking site to do so).

The only downside is that the links google found and crawled before our fixes, google will still assume they are 'live' as it's not possible to serve a 404.

Interestingly since applying the fix and the total indexed page count dropping, our organic hits have increased - maybe a coincidence - but concerning if potentially if the two are linked. (NOTE: Our organic hits aren't anything to write home about and this is just for our UK site).

Have attached a screenshot which shows the indexed page count dropping with the fixes we implemented and interestingly the organic hits increasing. 

For reference the site is around 250 unique pages, but I am aware of additional pages / dupes we need to sort that inflates our page count. That said we still have some way to go from the currently indexed 52k of pages. 



NOTE: Do take note of Greg's comment regarding not including the /vendors/ directory in robots.txt until after Google has removed the pages from it's index as you do want Google to crawl them to read the no index command in the page. 

As mentioned in previous posts this could be such as simple fix, vendors functionality could be a simple tick box Y / N if you want to use or not. If not, Shopify could serve a 404 page to all hits on the vendors directory. 

[Jesper_Skaane_B]

@DaveSweetCurescan you not use the "removals --> temporary removals --> New reques" feature in Google Search Console to remove the already crawled sites? 

[DaveSweetCures]

@Jesper_Skaane_B - Yes you can and we did that as well, only issue being is without the other fixes, then the external links that create the pages will still continue to be created.

The temporary removal is also just a bandaid to allow sites time to fix issues. So technically speaking the page still exists - and is "potentially indexable", you've just requested it to be temporarily hidden. Google console also lags behind an changes considerably. 

Ultimately, the not indexed pages count won't drop until one of the following things happens,
1. The attacking sites realises there's no point anymore having links to our site - doubt they'll clean them up though. 
2. Google removes the spammy site / network on its own accord  - but even so it's still your responsibility as the content is created on your site - even tho you didn't create it. And, and another site/network will just appear in it's place.
3. Shopify do the right thing and publish a fix which serves a 404 on these types of queries*

*BTW, I can see Shopify's point of view this being a theme issue, as had some of these fixes being in place in the theme, arguably we would NOT be in the position we are now. As there would have been no point of the attacking site linking to us in the first place - had the query not printed in the page content. 

[DaveSweetCures]

*BTW, I can see Shopify's point of view this being a theme issue, as had some of these fixes being in place in the theme, arguably we would NOT be in the position we are now. As there would have been no point of the attacking site linking to us in the first place - had the query not printed in the page content. 

[Jesper_Skaane_B]

@DaveSweetCures  thanks for clearing that up. one would hope that shopify steps up. one of the reasons i am on shopify and not wordspress, is that one would think that there would be people handling these kinds of things.

in regards to it being a theme issue. Sure. however i am on a shopify native theme. so i am going to kick that ball right back to Shopify.

[DaveSweetCures]

wow ok, so it's even an issue with native themes. OK, be worth keeping an eye on updates for native themes - as any fixes may herald a different solution for those with themes created elsewhere. 

[Jesper_Skaane_B]

@DaveSweetCuresShopifys goto is always "third party app" or "theme issue" -__-'

I am on a theme called Supply, which is native shopify theme. i should be eligible for some dev. work, which i am waiting for answer on. but knowing shopify, it probably falls out of the scope of what they offer.

[AnthonyH1]

Hello, 

your code definitely helped to stock the hack from the /collections directory but now the /search seems to be the new entry door. 

What would you recommend to prevent this ?

Many thanks

Anthony

[Greg-Bernhardt]

That is why you noindex. If a page is marked noindex, it doesn't matter how many sites link to you. Google will ignore them while dropping these 0 result pages from their index. 

[shadi1]

Hi Greg

Two weeks ago i tried the no index proposed by Jizo_Inagaki :

{%- if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}
<meta name="robots" content="noindex">
{%- endif -%}

This solution seemed to work at first. But i just checked my google search counsel and here is a screen shot:

These pages are still being indexed!

Greg Please let us know if you have a better no index solution?

[Greg-Bernhardt]

Hi Shadi1, noindex is the way and is nearly always respected by Google. Reports in GSC can lag and are not at all current. It can take a few weeks or more for Google to drop a page depending on how well-trafficked it is and then more time for Google to update GSC. The best way to see the current status is to use the "Inspect URL" and that will show the current status. Also if you have a small set of pages you set to noindex, you can use the "Inspect URL" tool to request reindexing and that will speed things up. Note that if you block those pages with noindex before Google can recrawl and see that change, it will not drop the pages from the index, because it can't see the change.

[JenDeg]

I can verify this solution works. We added it to our site then forced Google to reindex and it seems to be working. 

[NEI-Arlene]

What is the solution? 

The no index code? This has been proven to not work. Did Skims use the Disavow tool? 

[MJ5]

Can Shopify advise on the code to use to redirect to a 404 page using the liquid. That would be a step in the right direction. This is a widespread problem for store owners and there seems to be a focus by Shopify on the backlinks and not the actual issue that we just want to prevent these pages from being generated by a search string. SKIMS has a good way of dealing with this which would hopefully assist the wider community if Shopify could communicate how to achieve the same result.

[z285chen]

Thanks, @DaveSweetCures  for the detailed steps. I think it's a temporary workout for us now until Shopify realizes it is a widespread serious bug. Anyways, we could not 404 those pages as I've also checked other websites like Allbirds or Fashion Nova which just redirect them to their homepage or collection/vendor instead.

[tc77]

@DaveSweetCures I think that @z285chen  might be onto something. Could we do a 301 redirect from:www.yoursite.com/collections/vendors?q=  just to the home page? or would that still leave it in the search console? 

[fabmol1]

I'm following this thread but a bit lost!

What should I do? I don;t want to risk doing anymore harm by trying to fix this and making things worse long term in Google.

[z285chen]

Hi guys, so I found the same issue today. I think the ideal solution so far is first to ask Google to remove those links with the same prefix /collections/vendors?q= temporarily and then add the code provided above by @DaveSweetCures to prevent others pages from being created in the same way. And robots.txt could be just a backup.

[DaveSweetCures]

Hi @z285chen 

That's two of several steps we've implemented. The issue with just requesting Google removes the URLs is that it's a temporary fix, meanwhile millions more links are being built and linked to your site - this isn't to say you shouldn't do this, but just bear in mind the problem is still there. Likewise with the robots.txt. as a back - Google is more than happy to ignore robots.txt if an indexable page is found through a link - so just using those two solutions won't work. We've setup the following which you are welcome to check on our site https://www.sweetcures.co.uk/collections/vendors?q=tryit 

1. Our web team prevented the query being printed in the body of the page using the below text. Please note this fix may vary on theme used.

Update sections/collection-content.liquid 

Line one above all script:  
{%- unless collection.handle == 'vendors' -%}
After all script above schema:  
{%- else -%}
<div id="page-404" class="container">
<div class="text-center margin-top-xl">
<h2 class="text-lg margin-bottom-lg">{{ 'general.404.title' | t }}</h2>
<p class="text-component margin-bottom-lg">{{ 'general.404.text' | t }}</p>
<p class="text-component">{{ 'general.404.subtext_html' | t }}</p>
</div>
</div>
{%- endunless -%}

2. Prevent the query being printed in the title tag

Update theme.liquid title tag to be as below

{%- if request.path == '/collections/vendors' and collection.all_products_count == 0 %}
<title>404 Not Found</title>
{%- else %}
<title>{{ seo_title | strip }}</title>
{% endif %}

3. Print a no index on these pages

{% if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}

<meta name="robots" content="noindex">
 {% endif %}

4. Add to robots.txt file - Note this is merely a back up Google uses this as nothing more than a guide and potentially will still list URLs with inbound links.

{{ 'Disallow: */vendors?q=*' }}

5. You can easily remove thousands of pages from Google by using there remove tool an selecting the prefix option  www.yoursite.com/collections/vendors?q=

6. Use Ahrefs free tool / webmaster tools to explore and download link profile - use this as a basis to upload a list to Google Disavow tool.

Were we are as of 06/12/2022 - Google Webmasters show 2,000,000 pages, 330,000 which have been indexed - bear in mind this is a 300 page site at most. Google webmasters is slow so I'm hopeful over time the above numbers will improve. Will update as we go. 

I have to say that I still don't feel this issue is fixed as even with no index and the above in place, the ability of a rouge party to create unique URLs makes this a very difficult issue to resolve when it's not possible to publish a 404 response for these queries. Really Shopify should make the vendor setup a YES / NO tick box option - you going to use it or not?. And then simply close this door with a 404 for those who aren't going to use it. Shopify are responsible for not millions, but hundreds of billions of pages of web spam. Passing it off a theme issue simply isn't true. Site owners really should be lobbing Shopify as this does have potential to ruin link profiles and even take sites down. 

 

[DaveSweetCures]

@Greg-Bernhardt Please could you request Shopify update this vendor setup to be an Y/N tick box option. This would instantly solve this problem and remove billions of page of spam. Thank you. 

[Greg-Bernhardt]

@DaveSweetCures nice job removing the printout. That is half the battle. Once the incentive is removed, the spammers will move on. If they can't print out the query then there is nothing for them to do. You may want to reconsider the 404 language however, as the page does exist, it's just that no venders or searches are found. That is a different communication to users. Also your robots block is preventing Google from seeing the noindex tag. The pages will then remain indexed. Only add the block once the pages are deindexed.

[DaveSweetCures]

Cheers @Greg-Bernhardt - that's actually a very good point re the robot.txt - I'll update the post. Thank you. 

[DaveSweetCures]

So if we said this was potentially a solution for this? 

1. Our web team prevented the query being printed in the body of the page using the below text. Please note this fix may vary on theme used. Please change the message to what's suitable - we've used a 404 message but this may not be suitable or overly relevant for all - step 1 & 2. 

Update sections/collection-content.liquid 

Line one above all script:  
{%- unless collection.handle == 'vendors' -%}
After all script above schema:  
{%- else -%}
<div id="page-404" class="container">
<div class="text-center margin-top-xl">
<h2 class="text-lg margin-bottom-lg">{{ 'general.404.title' | t }}</h2>
<p class="text-component margin-bottom-lg">{{ 'general.404.text' | t }}</p>
<p class="text-component">{{ 'general.404.subtext_html' | t }}</p>
</div>
</div>
{%- endunless -%}

2. Prevent the query being printed in the title tag

Update theme.liquid title tag to be as below

{%- if request.path == '/collections/vendors' and collection.all_products_count == 0 %}
<title>404 Not Found</title>
{%- else %}
<title>{{ seo_title | strip }}</title>
{% endif %}

3. Print a no index on these pages

{% if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}

<meta name="robots" content="noindex">
 {% endif %}

4. You can easily remove thousands of pages from Google by using there remove tool an selecting the prefix option  www.yoursite.com/collections/vendors?q=

5. Use Ahrefs free tool / webmaster tools to explore and download link profile - use this as a basis to upload a list to Google Disavow tool.

6. Add to robots.txt file - but only after the pages are no longer indexed by Google - Note this is merely a back up Google uses this as nothing more than a guide and potentially will still list URLs with inbound links.

{{ 'Disallow: */vendors?q=*' }}

[gmeneses]

Hello Dave, I need clarification on steps 4 and 5. Can I use the Google Disavow Tool for my URL? I thought this tool only worked to remove backlinks from other sites pointing to mine instead of links with my own URL, which is the case for this specific issue.

[gmeneses]

Hello @DaveSweetCures and @Greg-Bernhardt, I need clarification on steps 4 and 5. It is my understanding that the Google Disavow Tool is for Spammy backlinks from OTHER sites pointing to your site as opposed to Spammy links ON YOUR SITE, which is the case here. In other words, you are disavowing a URL on your own site, not from an outside site. Could you please clarify how to use this tool for this specific issue? Thanks for your help!

[Greg-Bernhardt]

Correct Disavow tool is for external backlinks and I would not recommend using it unless you have a manual penalty from Google. The Temporary Removal tool is for internal links.

The solve is simply two steps. 

• Remove in spammer incentive by ending the querystring printout on the page
• Noindex 0 result vendor and search pages.

[gmeneses]

Hi @Greg-Bernhardt quick clarification in regards to the post from @DaveSweetCures:

• When you say "ending the query string printout on the page," it would be steps 1 and 2.
• When you say "Noindex 0 result vendor and search pages," it would be step 3.

Correct? 

Thanks again for your help.