How can I remove a spam page from my website index?

[MVUILL]

Hi! 

I received an alert from google search console saying they excluded a page from indexation, and when I look at the page it's spam publicity: 

URL : https://www.french-address.com/collections/vendors?q=Buy%20FUT%2023%20coins%2C%20Cheap%20FIFA%2023%2...

How can I delete this page, I can't find it anywhere? 

Thx for your help!!

[Shay · Accepted solution]

Hi @MVUILL 

I can definitely understand your concern! Based on the URL you shared it looks like this might be a collection created within your admin or possible from the new Shopify Collabs service. 

The "spammy" content of that page you shared is what appears to be the collection page title. If you still cannot find this page within your store admin please reach out to our live support team for additional help with this. 

Please know that our theme and technical support team may be limited in what they can do depending on where/how this page was created. We generally can only support theme edits on our own in-house themes or technical issues that resolve around the Shopify platform itself. That doesn't mean they won't do everything they can to help you get this resolved! 

To contact live support please follow this link: Contact Support - Shopify Help Center.

[gmeneses]

Hi @Greg-Bernhardt quick clarification in regards to the post from @DaveSweetCures:

• When you say "ending the query string printout on the page," it would be steps 1 and 2.
• When you say "Noindex 0 result vendor and search pages," it would be step 3.

Correct? 

Thanks again for your help.

[CD]

I  just don't understand why we need to waste time trying to figure this out.

Shouldn't be included in the fee we pay every month? Or access to a manual? a training? anything?

I've read the whole thread, and I am lost. I am not sure what to do and have hundreds of pages INDEXED in Chinese.

Please please what should I do?

[DaveSweetCures]

@gmeneses - @Greg-Bernhardt Greg and others are correct in saying Google Disavow tool is a powerful tool that should be used with caution. However, this isn't just an internal issue, the reason these links are being indexed by Google is because they are links on another site. Google crawls the SPAM site, follows the query link and because it looks like unique content indexes it.

Sign up to HREFs free webmasters account and you can dump a backlink profile for your site and you'll see all the links for yourself. Image attached is a screenshot of our backlink profile, disavow will only bite you if you are foolish enough to disavow large chunks of links with no thought or care, but it's clear with the Chinese characters which are the spam sites and I see no issue using disavow. BUT please, this is IMO. 

 

[DaveSweetCures]

@Greg-Bernhardt 

We have 1.5million pages of spam created with this @eheurlin 3.5million, two sites responsible for 4 million pages of spam. 

And I believe this could be solved by making the vendor query setup something you can opt in or our of using. Literally a Yes / No option in the shopify backend. If No, would simply serve a real 404 on any traffic on the vendor URL - where a site doesn't need the vendor setup. 

Instantly this would remove billions of pages of spam from Google. 

Question is @Greg-Bernhardt how do we get this implemented please? As current solutions provided by Shopify simply aren't good enough IMO. 

[Greg-Bernhardt]

@DaveSweetCures I'm on the internal SEO team, so I can't comment on anything regarding Shopify product. Please keep this thread updated as you implement your solutions.

[tc77]

I'm getting "Title" for my title on all my collections here. Am I putting it in the wrong place?

[tc77]

@DaveSweetCures I'm not "technical" but was able to figure out that I needed to remove existing title code (after taking a backup!) and replacing with your title code. That clarification might help others that get stuck on it.

[CD]

Hi Dave,

Thanks for your reply. To me, this is the best solution and the only one that makes any sense.

However, I am not sure how to do this (or too dumb!)

Do you have the contact of your dev team? I would like someone to do it for me.

Thanks,

Claudia

[Maelb22]

Hello Dave,

Thank you for your precious sharing.

I just would like have some precision on the step 4 :

"4. You can easily remove thousands of pages from Google by using there remove tool an selecting the prefix option  www.yoursite.com/collections/vendors?q=" => I made that yesterday and it seems it desindexed all collections page (I'm not sure, but this is that I'm seeing now). 

So if it is the case, it is not the right url to send to "noindex" temporary all vendors url spam pages. Do you have the same behavior  on your side ?

Best regards

[Italia-Straps]

Has anyone raised this issue this issue directly in the Google Search Central Help Community?

https://support.google.com/webmasters/community?hl=en

Generally, there’s lots of good support on various issues there and Google Experts will often chime in with solutions.

This is a far-reaching problem and all things being equal, Google doesn’t want all the SPAM pages in their index either.

[gmeneses]

I used the solution provided by @DaveSweetCures, but I'm getting this "translation missing" text attached to my 404 Not Found text. Does someone know how to erase the "translation missing" portion? I think it has something to do with the en.default.schema.json Locales.

Here is my main-collection-banner.liquid line above all script:

Here are my main-collection-banner.liquid lines with some tweaks above the schema:

Here are my theme.liquid lines to prevent the query from being printed in the title tag:

Here is the en.default.schema.json section that I might need to update:

Thanks for the help!

[DaveSweetCures]

I think it's your h2 tag in the second image... 

change

{{ '404 Not Found' | t }}

to

404 Not Found

so it looks something like,

<h2 class ="text-lg margin-bottom-lg">404 Not Found</h2>

[gmeneses]

Thanks @DaveSweetCures. Your suggestion regarding the "translation missing" message worked.

@DaveSweetCures @Greg-Bernhardt What happens if I disable the search bar feature on my store? I know that even when you type anything after the /vendors?q= URL, it will show as the page's title, but deactivating the search bar might help with something. What do you think?

[tc77]

I emailed Tobias (CEO) on this. A "Senior Lead" responded saying that they are taking it seriously and working to get it resolve, but could not give me a timeframe. They said it should be done ASAP.

[Italia-Straps]

Did another review of Google Search Console today and discovered a new wave of SPAM on the “…collections/vendors?q=” pathway that so many shop owners are experiencing.

However, the difference now is the SPAM pages are in the category of “Blocked by robots.txt” or “Crawled - currently not indexed” in Google Search Console.

In other words, the new SPAM pages are not showing up as indexed which would seen to be a positive thing.

We have not made any coding changes on the site.

Does this indicate that Shopify has maybe made a change on their end?

Is anybody else seeing this in Google Search Console?

Thanks!

[Italia-Straps]

On the same subject this article came up doing some more research on this topic:

https://gofishdigital.com/blog/shopify-robots-txt/

Good info here on the default robots.txt file from Shopify. Apparently, most shop owners don’t change this. Interesting to note that the default robots.txt file blocks internal site search.

Questions:

• Can Shopify institute a change to the default robots.txt file to handle the “…collections/vendors?q=” SPAM?
• Is it a possible solution for store owners to just “Disallow: collections/vendors?q=” in their robots.txt file or would there be a downside?

Looking forward to input from others and hopefully Shopify staff.

Thanks again.

[Participant]

I also emailed Tobi, in mid-December (to the email address which he publicly lists: tobi@shopify.com). However, nobody from his office got back to me on this very serious issue.

I have also corresponded with at least three Shopify support agents. They have been a) either unaware of the problem, or b) they continually try to pass it off as a "backlink" issue.

This is not simply a "backlink" issue, because if the spam pages could not be created by exploiting the Shopify system, then the pages would not be indexed by Google (or any other search engine), and therefore there would be no risk of customers seeing the spam pages and being scared away from our store (or potentially millions of other Shopify stores).

[Egle]

Having same issue. Had 300+ pages indexed (as original content). Within a week, 3.5million pages were indexed and 3.7 million pages not indexed. In Total: 7.2 million pages ... Not sure what to do, added "noindex" meta tag, it helped me to not get all 7.2 mill indexed, but how to remove indexed ones? Because when you using disavow tool and add link /collection/vendors it removes whole /collections/ from indexing, not only collections/vendors/ .

[Maelb22]

Hi Eagle,

It seems ok, you can use "suppression" tool in search consol, it will only remove the "collections/vendors/" pages. I tested, and for the moment, it is ok from my side.

Use the prefix option  www.yoursite.com/collections/vendors?q=

To shopify team : 

Do you think it is normal to take months to add a line of "404 not found" code that would solve this problem? I don't think it's normal on my side, it's time to react.

It would also be interesting to advise your customers when we contact you by chat. I have contacted you 3 times about this, each time you told me that the problem was only from my side, and that no other shopify store had experienced this problem. It's quite bold, when you see the number of topics dedicated to this problem...

[TopWhiskies]

This is impacting my store too. Trying to work through a solution. Shopify support have referred me to the theme developer.

[Dan_Bennett]

Why is Shopify returning a 200 error on any invalid page going to /collections/vendors?q=...

Shouldn't this return a 404?

[Allan-EP]

This is exactly the issue - instead of 404-ing as expected, the page is rendered as 200 with whatever the value of "q" is as the collection object's title so it's outputted to the page, and that's the exploit the spammers are taking advantage of. 

[Maxime_Breton_V]

As mentioned by Allan-EP above, the issue is on Shopify's end is being exploited by spammers.

Shopify should not be referring people with this issue to third parties. The problem touches Shopify stores with Shopify-provided themes. Shopify should fix the exploit on their end.

It is not normal that websites are getting de-indexed by search engines and losing rankings for something that can easily be fixed by Shopify. Hopefully this thread will bring more attention to the issue, although it has been ongoing for months without any solution.

[Greg-Bernhardt]

@Allan-EP noindex is more appropriate. Whether or not a result is returned, the page exists. 0 is just as valid a number as any. 0 doesn't = page doesn't exist. 0 = 0 vendors. The danger of 404ing is that you may have a store that fluctuates between having a result for the query and not having a result. If you flip back and forth telling Google the page exists one week and not the next, it reduces confidence and could result in lost page equity/history from Google's side. By using noindex you remove the page from the index and keep its history in Google.

[Allan-EP]

You're not understanding the issue. "Vendors" is not a collection on our store (and I would expect this is the case for many of the other stores in this thread), so /collections/vendors/ shouldn't exist. A 404 is the expected response. On top of that, "q" is getting passed into the collection title ( {{collection.title}} in the template outputs the query parameter) for some reason. So, the spammers simply pass anything they want into "q"  at /collection/vendors/ for their exploit. 

[Greg-Bernhardt]

Hi Allan, I do understand the issue. If you want to talk priority, then I agree, the ending of the query printout is really all that matters here. That is the incentive. Then we can talk about noindex vs 404 and I agree in your case if you're manufacturing all your products, you don't need a vendors page, but to remain flexible for all merchants, noindex achieves the same solution.

[Allan-EP]

@Greg-Bernhardt So Shopify's position is, "every single store needs to fix this at an account level instead of implementing a system-wide fix" for this exploit? 

[Greg-Bernhardt]

Hi Allan, I'm not speaking for Shopify's position nor have visibility on the teams addressing this issue. I'm on the internal SEO team just popping in to help where I can.

[Allan-EP]

@Greg-Bernhardt How can this be escalated to a team that can help then? Suggesting your entire customer base to implement a noindex is unacceptable, and this exploit will continue to spread as the vulnerability becomes known.  

[Allan-EP]

Looking at this further, the whole issue (probably) stems from the url_for_vendor filter

Shopify was trying to be useful by creating an automatic collection-like page for vendors, and haphazardly allowed the q parameter to be used for the collection title. 

So, the platform fix should be to validate that the vendor name passed to q actually exists before outputting it to the page. Additionally, /vendors/ should 404 if no vendors are in the product catalog. 

[Allan-EP]

Here's the fix I ended up using in the collection template: 

{% if collection.id or collection.products.size > 0 %}
	<h1>{{ collection.title }}</h1>
{% endif %}

By checking for collection.id, we continue to support real collections created by the store owner. Otherwise, if this a product query by vendor, we make sure we have at least 1 product result. That way, spammers garbage won't get outputted to the page (assuming the product count is always 0, which it should be). 

[Maxime_Breton_V]

Hello Allan-EP,

For the less tech-savvy amongst us, could you provide additional information as to where exactly to implement your fix in the collection template code?

Would it be at the location with the red arrow in the attached screenshot?

Thank you in advance!

[Allan-EP]

@Maxime_Breton_V It will vary based on theme, but yes, your red arrow would be correct. 

<h1>{{ collection.title }}</h1>

 will become

{% if collection.id or collection.products.size > 0 %}
<h1>{{ collection.title }}</h1>
{% endif %}

[Vicky6]

Hi @Allan-EP,

     Thanks for this code. @Greg-Bernhardt thanks for all your help too. I feel you are the only Shopify Staff member trying to help up. The other no index code solution messed with my sites menu and breadcrumbs. Normal for me on 11/28 was 9K Non-Indexed and 4K Indexed. Today I'm at 2.1 Million NI and 1.7M indexed). The attack started 11/29. I contacted Shopify right away. Shopify is ignoring me as well and telling me to go to a theme developer. @Shay my ticket numbers are 34543316 & 34784932.

     I need some help finding where to put this code in my theme if there is any one that uses snippets... Here are a couple screen shots. I'm not sure which snippet the code needs to be in. I think mine is in the collection-item.liquid? It's the only place I find {{ collection_title }}.

     If someone has actually used a Shopify Expert who knows exactly what to do to fix this problem, I'd LOVE to know their contact info. I have a feeling they are trying to figure it out too!

     FYI, I've read all the posts in both community threads and have read all the suggested articles, so I'm up to speed. This is something that Shopify can prevent/fix and are staying silent about it. I hope they are working on a solution on their end. This is maddening! OK, rant over.

[Allan-EP]

@Vicky6 Can you attach a screenshot of collection-template.liquid under Sections? You’ll either add the code there or in the snippet, but I suspect it’s the section. 

[Vicky6]

@Allan-EP I checked. No {{ collection_title }} in that section. Thanks for your speedy help! 

[Allan-EP]

It may not be exactly collection_title like in your snippet. What you’re really looking for is collection.title

[Vicky6]

@Allan-EPcollection.title wasn't in collection-template.liquid but I did find it in collection-header.liquid. It was too large to screenshot. Here is the code for that section. Thanks for taking a look if you have time. 

<header class="page-header">

  {% if section.settings.display_image_cat %}

    <div class="img_cart">

      {% if section.settings.img_cat != blank %}

      <img src="{{ section.settings.img_cat | img_url :"master"}}" alt="{{ collection.title | split: '|' | first }}" />

     {% else %}

        {% capture current %}{% cycle 2%}{% endcapture %}

        {{ 'lifestyle-' | append: current | placeholder_svg_tag: 'placeholder-svg' }}

       {% endif %}

    </div>

  {% endif %}

  {% if settings.enable_multilang %}

  <h2>

    <span class="lang1">{{ collection.title | split: '|' | first }}</span>

    <span class="lang2">{{ collection.title | split: '|' | last }}</span>

  </h2>

  {% else %}

   <h2>{{ collection.title | split: '|' | first }}</h2>

  {% endif %}

  {% if collection.description != ''  %}

  <div class="collection-des">

    {% if settings.enable_multilang %}      

      <div class="rte lang1">{{ collection.description | split: '[lang2]' | first }}</div>

      <div class="rte lang2">{{ collection.description | split: '[lang2]' | last }}</div>

    {% else %}

     <div class="rte">{{ collection.description | split: '[lang2]' | first }}</div>

    {% endif %}

  </div>

  {% endif %}      

</header>

{% schema %}

  {

    "name": "Collection Header",

    "settings": [

  {

        "type": "checkbox",

        "id": "display_image_cat",

        "label": "Display Image Category?",

        "default": true

      },

      {

        "type": "image_picker",

        "id": "img_cat",

        "label": "Image Category"

      }

]

  }

{% endschema %}

[Allan-EP]

It’s probably this line: 

<h2>{{ collection.title | split: '|' | first }}</h2>

The “if” line goes above that and the “endif” line goes after it. I’d write the code fully out but am replying on my phone at an airport. 🙂

[Vicky6]

Thanks @Allan-EP,    I'm still stumped and hesitant to do this w/o guidance since the <h2>{{ collection.title | split: '|' | first }}</h2> is already wrapped in an if/then for {% if settings.enable_multilang %} as seen below:

{% if settings.enable_multilang %}
  <h2>
    <span class="lang1">{{ collection.title | split: '|' | first }}</span>
    <span class="lang2">{{ collection.title | split: '|' | last }}</span>
  </h2>
  {% else %}
  <h2>{{ collection.title | split: '|' | first }}</h2>
  {% endif %}

If you have time to type the whole thing out for me, I'd really appreciate it. I've seen my indexed vs. index drop a bit since I added the "no index" code below on 12/7 but it's been a month and I didn't have the results that @Egle  has had where the indexed pages have dropped back to normal. Mine are still at 1.5M indexed. They should be 4K.  {%- if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}
<meta name="robots" content="noindex">
{%- endif -%}

[Allan-EP]

@Vicky6 You can wrap that existing if/else statement with the one for the vendors check. So: 

{% if collection.id or collection.products.size > 0 %}
  {% if settings.enable_multilang %}
  <h2>
    <span class="lang1">{{ collection.title | split: '|' | first }}</span>
    <span class="lang2">{{ collection.title | split: '|' | last }}</span>
  </h2>
  {% else %}
  <h2>{{ collection.title | split: '|' | first }}</h2>
  {% endif %}
{% endif %}

and then be sure to add this in your theme.liquid between the <head> and </head> tags: 

{% if request.path == '/collections/vendors' and collection.all_products_count == 0 %}
<meta name="robots" content="noindex">
{% endif %}

[Vicky6]

Thanks so much @Allan-EP! I added it and previewed a spam URL. The collection title no longer shows but it shows up in my breadcrumbs. Is that good enough or do I need to find some code to get it not to show there as well?

[Allan-EP]

@Vicky6 Yes, you should put the same if/endif tags around the breadcrumb code as well to prevent the spam content from appearing. 

[Participant]

@Allan-EP A hundred thank-you's for this!

This code is working for our store, as it prevents the spam text from appearing as the page title.

Thank you for the very clear illustration of how to implement the code.

[shadi1]

I can't believe a Shopify wide fix has not been implanted yet. This problem is getting much worse and effects our google ranking and sales. Which in turn is effecting Shopify profits !

Or am i wrong this is not effecting sales and profits!

[bradm24]

Implement the code found here and then you should see it recover. Once it's implemented remove those bad links from google search console. 

https://community.shopify.com/c/shopify-discussions/website-hacked-help/m-p/1893630/highlight/true#M...

[Egle]

Hello, the only ONE and WORKING SOLUTION is:

{%- if request.path == '/collections/vendors' and collection.all_products_count == 0 -%}
<meta name="robots" content="noindex">
{%- endif -%}

All other solutions, codes, apps, google disavow and removal, etc., are temporary fixes, not permanent ones. Don't waste your time and energy focusing on anything other than this line of code.

We had 13 million not indexed and 2.5 million indexed pages. As you can see in the attached screenshot, it took just a little over a month to have almost all 2.5 million dropped.

Have patience; it will take some time. We worked on our keyword rankings for three years and were in TOP15; after this attack, our rankings for each keyword were 100+. Nonetheless, we were using the WEGLOT translate app for five different languages, and those attacks drained the "translations request" counter in a day, which meant that none of our pages were solved for a month cause visits from bots exceeded our request quota.

But as of right now, our keyword ranking is already coming back rapidly, it took just a couple of days, and some keywords from 94th place right now are ranked in 20-30th place.

[Jesper_Skaane_B]

@Egle Thanks! Where should it be implemented? I theme.liquid ? If so, any particular place?

[Allan-EP]

@Jesper_Skaane_B the meta tag would go in your theme.liquid, anywhere between the <head> and </head> tags but I tend to put something like that as close to the top as possible to find it later if needed.

[shadi1]

Egle did you try Allan-EP  solution?