Discuss and resolve questions on Liquid, JavaScript, themes, sales channels, and site speed enhancements.
Shopify has a new bug - they are indexing tons of useless pages in Google again. Is there any way to stop this from happening? The code seems to only be editable by shopify since it's in the content_for_header.
Do a google search for:
inurl:/web-pixels-manager@
(https://www.google.com/search?q=inurl%3A%2Fweb-pixels-manager%40)
Each site has multiple versions of this page indexed. Peets.com for example has 20.
The pages look blank, but include something like this in the code:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Web Pixels Manager Sandbox</title>
<script id="web-pixels-manager-setup">(function e(e,n,a,o,t){e&&(window.Shopify=window.Shopify||{},window.Shopify.analytics=window.Shopify.analytics||{},window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=function(e,n,a){window.Shopify.analytics.replayQueue.push([e,n,a])});var r,i,s,l,d,c,p,u,f=a+"/"+o+"."+function(){var e="legacy",n="unknown",a=null,o=navigator.userAgent.match(/(Firefox|Chrome)\/(\d+)/i),t=navigator.userAgent.match(/(Edg)\/(\d+)/i),r=navigator.userAgent.match(/(Version)\/(\d+)(.+)(Safari)\/(\d+)/i);r?(n="safari",a=parseInt(r[2],10)):t?(n="edge",a=parseInt(t[2],10)):o&&(n=o[1].toLocaleLowerCase(),a=parseInt(o[2],10));var i={chrome:60,firefox:55,safari:11,edge:80}[n];return void 0!==i&&null!==a&&i<=a&&(e="modern"),e}()+".js";r={src:f,async:!0,onload:function(){if(e){var a=window.webPixelsManager.init(e);n(a),window.Shopify.analytics.replayQueue.forEach((function(e){a.publishCustomEvent(e[0],e[1],e[2])})),window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=a.publishCustomEvent}},onerror:function(){var n=(e.storefrontBaseUrl?e.storefrontBaseUrl.replace(/\/$/,""):self.location.origin)+"/.well-known/shopify/monorail/unstable/produce_batch",a=JSON.stringify({metadata:{event_sent_at_ms:(new Date).getTime()},events:[{schema_id:"web_pixels_manager_load/2.0",payload:{version:t||"latest",page_url:self.location.href,status:"failed",error_msg:f+" has failed to load"},metadata:{event_created_at_ms:(new Date).getTime()}}]});try{if(self.navigator.sendBeacon.bind(self.navigator)(n,a))return!0}catch(e){}const o=new XMLHttpRequest;try{return o.open("POST",n,!0),o.setRequestHeader("Content-Type","text/plain"),o.send(a),!0}catch(e){console&&console.warn&&console.warn("[Web Pixels Manager] Got an unhandled error while logging a load error.")}return!1}},i=document.createElement("script"),s=r.src,l=r.async||!0,d=r.onload,c=r.onerror,p=document.head,u=document.body,i.async=l,i.src=s,d&&i.addEventListener("load",d),c&&i.addEventListener("error",c),p?p.appendChild(i):u?u.appendChild(i):console.error("Did not find a head or body element to append the script")})(null,null,"https://cdn.shopify.com/shopifycloud/web-pixels-manager/0.0.186","sandbox","0.0.186");</script>
</head>
<body></body>
</html>
Solved! Go to the solution
This is an accepted solution.
Hello everyone.
Thank you for your patience while we worked towards providing a fix to this issue and building out an FAQ that addresses the primary concerns and questions from this thread. You can find our FAQ on the Community Blog.
We welcome you to continue the conversation in the blog section of the blog post with any valid questions that aren't already answered by the FAQ. We will monitor the comments section for any valid feedback on how this change may impact Shopify stores and will actively remove or edit any comments that spread misinformation or speculation on the issue.
On that, this thread contains some misinformation and speculation on the issue and how it may impact ones online store. For this reason, we will no longer be monitoring this thread, but want to keep it open and available for historical purposes.
We greatly appreciate all of you for bringing this to our awareness and collaborating with us while we worked towards a solution. @Greg-Bernhardt deserves a special shoutout as they have championed this thread and this issue internally.
I'll be marking this as the solution as this has been resolved and helps surface this reply for anyone who may be new to the thread.
Trevor | Community Moderator @ Shopify
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog
This is an accepted solution.
Hello,
Thank you for your continued feedback on the web pixels issue. More recent replies have been sharing misinformation or misattribute the issue. For this reason, we've chosen to close this thread.
If you believe you have a new issue that isn't answered by our FAQ, then we'd encourage you to create a new thread in our Technical Q&A board with as much detail as possible.
Thank you.
Trevor | Community Moderator @ Shopify
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog
@MaBa that is not a problem and the expected result
@jackzhu what is the latest date?
To learn more visit the Shopify Help Center or the Community Blog.
Hi there! We get this on all our websites. I see this was marked as solved in January but we still have these 404's on all our websites?
@PixelsPhotoArt what report is that screenshot from?
To learn more visit the Shopify Help Center or the Community Blog.
It is from google search console.
@PixelsPhotoArt which report in GSC?
To learn more visit the Shopify Help Center or the Community Blog.
Greg, the 404 report. Should I forward them to you in a DM?
@PixelsPhotoArt great, that is the expected result, we want to see the old URLs in the 404 report. There is no harm there.
To learn more visit the Shopify Help Center or the Community Blog.
How can a 404 with no way to redirect can be an expected result that brings no harm when search engines will penalize websites with a high number of 404 errors, as it can indicate that the website is not well-maintained or has broken links. This can lead to lower search engine rankings and reduced visibility for the website.
@PixelsPhotoArt wrote:
How can a 404 with no way to redirect can be an expected result
Why would you want to redirect these "no content" pixel pages? A redirect means that the page now exists elsewhere. The pages are gone, so communicate that.
@PixelsPhotoArt wrote:
search engines will penalize websites with a high number of 404 errors
The context of those 404s is important. There is no "404 penalty" for Google. Those pixel pages do not exist and are not internally linked from anywhere. Google does not care about them or want to crawl them. 404 is the expected result.
If you suddenly have a high amount of high value and highly linked pages that go 404, that will harm performance, yes.
@PixelsPhotoArt wrote:
it can indicate that the website is not well-maintained or has broken links. This can lead to lower search engine rankings and reduced visibility for the website.
There are no broken links. 404ing pages that don't exist, by definition, is the sign of a well-maintained website instead of trying to trick Google with ineffective redirects.
To learn more visit the Shopify Help Center or the Community Blog.
why not fix these issues instead of semi-fixing them? Surely you can set the header to remove or redirect them?
I also do not want a bunch of 401's that will stay in the Google system forever for a coding mistake I did not make
Hey Greg-Bernhardt,
the problem of this semi-solution is the export-maximum of 404 errors in Google Search Console. You can export up to 1.000 404 errors, but if you have more errors than that, you can't export them. That's a big problem, especially for larger Online-Shops because you won't be able to see real problems in your shop then, if the whole list is full with these kind of URLs that don't have any benefit for Google or for Shopify customers.
In my opinion this issue should really be fixed immediately
"The pages are gone, so communicate that."
-Greg Bernhardt, Shopify Staff, 3/30/2023
Conveniently enough, there's an HTTP code for that that. It's called 410 - Gone! What's super cool is it tells google that the page is "gone", not "missing" so google stops looking for it, they publicly acknowledged that 410's get pulled faster. I know per our DM's that you are aware of every single piece of information above so your response defending the 404 decision is concerning.
Maybe you acknowledged what I said but just thought I was wrong, I'm just a forum member. Why don't we ask the most respected source out there, the Internet Engineering Task Force.
The 410 (Gone) status code indicates that access to the target resource is no longer available at the origin server and that this condition is likely to be permanent. If the origin server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) ought to be used instead.
The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed.
The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists. A 404 status code does not indicate whether this lack of representation is temporary or permanent; the 410 (Gone) status code is preferred over 404 if the origin server knows, presumably through some configurable means, that the condition is likely to be permanent.
Source Document:
https://www.rfc-editor.org/rfc/rfc9110.html#name-404-not-found
Here's what John Mueller from Google said about 404 and 410.
“From our point of view, in the mid term/long term, a 404 is the same as a 410 for us. So in both of these cases, we drop those URLs from our index.
We generally reduce crawling a little bit of those URLs so that we don’t spend too much time crawling things that we know don’t exist.
The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so.
So if you’re just removing content naturally, then that’s perfectly fine to use either one. If you’ve already removed this content long ago, then it’s already not indexed so it doesn’t matter for us if you use a 404 or 410.”
To learn more visit the Shopify Help Center or the Community Blog.
Hey Greg-Bernhardt,
the 410 solution would help to avoid the problem with too many 404 errors to export. Is it so hard to fix that, or why didn't you say anything to my comment? Here it is again:
Hey Greg-Bernhardt,
the problem of this semi-solution is the export-maximum of 404 errors in Google Search Console. You can export up to 1.000 404 errors, but if you have more errors than that, you can't export them. That's a big problem, especially for larger Online-Shops because you won't be able to see real problems in your shop then, if the whole list is full with these kind of URLs that don't have any benefit for Google or for Shopify customers.
In my opinion this issue should really be fixed immediately
Hi Greg-Bernhardt,
Thank you for working on this issue, and for having the attention of Shopify. It seems these attacks are some of the most important issues for Shopify - so it needs Shopify CEO attention and resources to get it fixed, and for Shopify to get ahead of it. This isn't where we are it seems.
However, it is confusing as to what I should or shouldn't do .... I have been noticing this issue for months on my Google Search Console and have been periodically reading this blog post.
The "Accepted Solution" is not valid as it is old. It must be updated please to tell people what they should do ... EXACTLY and clearly. Throwing additional comments midway in this blog isn't working!
For instance ..... I just inserted
Disallow: /web-pixels-manager
Disallow: /wpm
into my robots.txt.liquid file.
Should I have done that? I can't tell by reading the 9 pages of back and forth.
I still have lots of "non-indexed" files showing:
I have also created a whole bunch of redirects so that the 404's are presented on an actual webpage. Should I have done that?
I have trying work out what works and what doesn't now for months .... too long.
I await your reply please.
Thank you,
I-LIVE-ON
This is still a big problem, my indexed wpm pages are almost gone but not indexed pages count has gone from 20k to almost 60k in one month. Really tanked my ranking . spring time is my busy season and this is slowest i ever had in 10 years.
I have no clue what shopify is doing to fix this but they need to speed it up.
Also stop saying not indexed doesn't matter , i m sure a lot of business on shopify are feeling this. 20k to 60k count rise in not indexed pages is effecting our ranking.
If any one has any solution to speed this process up and get us back to normal please let us know?
God!! You are damn right, and god pls punish them!! what they are doing to us, our years' effort, just ruined by their tiny code!!!!!!!
They need to pay for this!!!
@modabilé 410s are also listed in the 404 GSC report. Google has documentation that the difference between 410 and 404 is a couple days at most. https://youtu.be/kQIyk-2-wRg?t=1721
@Andy-Envision I'm seeing that page return 404
@TIMECAPSULES nothing is required on your end
@ahgaos there is no evidence or communication from Google that number of noindex URLs results in a ranking loss. There have been a couple of Google Core Algorithm updates in the past few months. I would encourage you to investigate those.
To learn more visit the Shopify Help Center or the Community Blog.
That's because I removed my client's domain name and collection name, as you can see it's using website.com. I was only showing what the URL structure is appearing in GSC.
Thank you for Youtube link to 404 or 410 discussion. However, key here is speakers reference to "removing content naturally". Our issue, is with the thousands upon thousand of other pages that are somehow looking to be indexed. This so far is only happening with our Shopify site. I really do believe more attention needs to be paid here.
Of course Google Algorithm updates factor in, so does the economy. There are thousands of variables. Shopify can certainly argue to dismiss...as not their problem. It's certainly a problem for our store and I don't have the resources to ultimately rule this out as THE reason our SEO tanked. So....not sure what to do. Nov 2022 was our best month (as a start-up) Dec was good. Then ZIP
This gross stuff has been happening for 3 months, have you ever really thought about the solution to solve the problem?? and the fault you made?? sorry you don't think that's your fault, it's our fault, it's shopify users' fault to choose your! If what you did that you claim can solve our problem, why's here people keeping pooping out to complain?? We are your customers, but what you treat us like? keep thinking about this in your mind, we are hereby for real solutions, but not perfunctory words.
Again this has been 3 months, let's see how long does your company will keep running.
I'm not sure if this is helpful, so pls let me know if I've completely missed the mark here.
I responded to this thread months ago when I first noticed google had indexed some "non pages" on our website. I simply added a "no index" to any pages which had "vendors?q=" in the url and then added the offending url to my GSC removals.
I subsequently had another random "vendors?q=" type url index, I followed the same procedure.
I've had no issues.
What am I missing?
Greg I used to agree with you . not indexed pages shouldn't be effecting our seo. But in reality now i suspect it is effecting our stores . As for google algorithm updates even the best seo experts wont fully know whats going on with that update for a while as they see results come in. Who knows maybe not indexed pages would of been no issue last year but this year its an issue.
What i want to know is why "not indexed" pages count keeps going up ??? are those wpm pages still being generated? Take a look at the screen shot i just took of my GSC.
Just in the past few days it went up by 3k.
Anyone else experiencing this ?
@shadi1 the report will continue for as many pages you have on your website as there may be a corresponding WPM page. The noindex is informative, not errors.
To learn more visit the Shopify Help Center or the Community Blog.
@Greg-Bernhardt I understand , but i want to know is there still lots of wpm pages being generated by shopify?
@shadi1 as far as I'm aware, these WPM pages are used for various analytics tracking
To learn more visit the Shopify Help Center or the Community Blog.
Analytics tracking? whose? Why? If after all this was deliberately added… have no words.
@Greg-Bernhardt So when all the WPM are out of everyone's indexed pages count, are there any plans to go back and block those WPM pages in robot.txt and remove the no index that was placed in the header? Or would that bring us back to square one as more WPM pages are being generated and those new ones would be indexed thou blocked by robot.txt?
@shadi1 Correct, you'd just see those URL move from the noindex report to the robots.txt and those URL may become indexed
@kbarcant I'd imagine you're still getting 404s added as Google returns to those original URLs. It can take many weeks as we originally stated.
The 404s occur from the original path. The noindexes occur from the new URL. The 404s won't likely be removed from the report. What is most important is that they are removed from the Google index.
Google has stated that Crawl Budget is not a consideration until you're into the millions of pages. Google will also reduce crawling of pages that are 404 and noindex.
Your concern about the 404 report pushing out non-wpm URLs is valid.
Can you DM me some URLs that are returning 404 and still remain in the index?
The 404s are intended and not a bug. These were old pixel URLs. They no longer exist and thus are 404.
To learn more visit the Shopify Help Center or the Community Blog.
Good morning Greg,
Without meaning to sound rude, as I don't know where you sit in the hierarchy of things.
Has this been escalated or are you the only one firefighting this?
Kind Regards
Matt
I just want to say, Google tracking code also is used for various analytics tracking, but it could not generate so many pages. So do you think whether Shopify needs optimizate your tracking code to stop generating such shit pages !!!
This uptick in wpm pages being detected by Google Search Console started in early February for our store, around the time there was an announcement of the Server Pixels limited release forthcoming in the 04-2023 API updates.
February 6, we had 154 noindex pages detected in GSC. This has gone up to 4,443 since then.
Even if this doesn't impact ranking, it creates a lot of clutter in GSC. We can only view 1,000 noindex entries in the GSC interface (including export options), so we can't evaluate our noindex status for pages other than the wpm pages if they are not within the first 1,000 results.
We don't have any customer events configured that would use the web pixel manager system.
I have a bigger question - we should be given the option to "opt out" of Shopify "analytics".
It is a very poor tool and provides me with less than 10% of what i can see in Google analytics.
No shopify analytics = no need for WPM pages = problem solved
Our store in its “natural” state is only about 40 indexed pages. Every week we keep seeing an increase in not indexed pages especially “excluded by no index tag” wpm@ pages in GSC. The “excluded by no index tag” wpm@ pages just keep growing. We used to have none.
Question: Will these pages just keep growing in number forever?
As I mentioned, our store is small. We haven’t seen a ranking issue yet. It seems that some larger stores seem to have an exponential issue where they perhaps trigger a threshold with Google impacting ranking.
In other words, it makes sense Google would ignore some level no index pages but tens of thousands could be a negative ranking signal.
See how your site is doing and when/if trouble started https://feinternational.com/website-penalty-indicator/
Hi @Greg-Bernhardt,
For our site, the Excluded by ‘noindex’ tag (wpm and web-pixels-manager) URLs are steadily rising now at 4k. There are about 778 wpm/web-pixels-manager URLs that are indexed. We have not deployed any fixes on GSC. As a note, I checked the robots.txt.liquid file and it doesn't contain either Disallow: /web-pixels-manager or Disallow: /wpm.
I've been following this thread and so far, there isn't any clear answer. From the standpoint of NOT having done anything to "fix" the issue yet, what should we do at this point? Nothing? Or something? A clear step-by-step answer would be very much appreciated.
Thanks!
We're ready to try anything. The "do-nothing" approach has gotten <10 of our 3,000+ 404's removed in six weeks, and due to google search console being overwhelmed by the numbers I can't even see if that's 10 wpm pages or 10 other 404's unrelated to this entire ordeal.
At this rate we'll be dealing with wpm 404s until Q1 2028.
Our go-to "Shopify Only" guy (kbarcant, who I've seen in this thread a few times, thanks for translating this all to english) is at a loss on what we, as a company, can do here. I needed an answer so I spoke to a few other Shopify store developers about this.
The first one was basically said exactly what I'd heard already. The second one said something I hadn't heard yet. They informed us that the 404s are here to stay for a very long time, and that this negative SEO is applied to the domain, not the shopify site.
They recommended WE BUY A NEW DOMAIN AND MOVE OUR ENTIRE WEBSITE TO IT.
We need clarity on what exactly shopify is doing to clarify this. "Wait, it'll fix itself" isn't an answer. My employees paychecks aren't waiting, they're still getting sent out. Shouldn't a company with 10% of the market share of ecommerce should be able to make a call to google and figure out how to fix this??? The only answer I've gotten so far that makes a lick of sense is changing the domain, and I'll be honest, it's not something I'm taking very lightly, we've been using this same domain for fifteen years.
Can you please give us an update? Sitting on the fence isn't very comfortable, we have to make a jump soon.
The Support team and I are working on a communication doc to wrap this up. Absolutely DO NOT migrate to a new domain. That is patently absurd. There is no negative SEO applied from the 404s. Please wait for our document.
To learn more visit the Shopify Help Center or the Community Blog.
How soon - this has now been more than 3 months since this craziness started?
Dear Greg if you just say 10 time: "There is no negative SEO applied" it not will became true. Something is going on with shopify for a while. Please deep investigate problems and solve it. Everything not going to fix it up magically. Here is another relative problem: https://community.shopify.com/c/ecommerce-marketing/weird-url-adding-in-google-search-console/m-p/19...
Hi Greg,
I am a little concerned that you mention a communication doc as opposed to a support doc.
Does this mean that we are going to have our concerns dismissed?
I would like to see a doc that explains how we remove this noise from our data so we can see the wood for the trees.
Kind Regard
Matt
Hi Greg,
Thanks for the communication that a document from Shopify is being produced on this vexxing issue for Shopify's customers.. As you know, there is a lot of angst re this issue - per comments on this blog.
Regards
Document should help our SEO
free-fall down another 25% since last week.
@MontI'm surprised shopify hasn't told us to turn our site off and then turn it back on again, the level of troubleshooting they've provided here is dismal. I'm expecting to see a document that says "wait some more it'll fix itself, oh and 404s aren't bad for seo". Probably a good time to short some stock.
Moving to a new domain gets rid of the 404s, something that Shopify has spent an entire quarter failing at. Every single solution that's been applied in the last 3 months failed. Can you really blame us for looking at other solutions? We've had to cut our employees hours because of this and Shopify has been completely opaque, and every "fix" has generated another issue. Saying that the 404s aren't to blame makes this more like a cover-up than a solution. We need a date on when to expect that document, even if it's "within a month" or something vague.
Our goal is for the document to answer all the common questions we're seeing here. The 404s are not hurting your site although I understand the inconvenience of them clogging up the GSC report. Moving domains however is a massive risk and has the potential for legitimate disruption in your performance. We will NOT recommend that.
To learn more visit the Shopify Help Center or the Community Blog.
Adding thousands of 404s via bad links on high-value product pages has a profound undeniable negative effect on SEO.
I don't want anyone leaving the thread with the wrong idea.
Starting a B2B store is a big undertaking that requires careful planning and execution. W...
By JasonH Sep 23, 2024By investing 30 minutes of your time, you can unlock the potential for increased sales,...
By Jacqui Sep 11, 2024We appreciate the diverse ways you participate in and engage with the Shopify Communi...
By JasonH Sep 9, 2024