Re: Shopify Bug - Web Pixels Manager Sandbox

ChummyTees · ‎01-17-2023

Shopify has a new bug - they are indexing tons of useless pages in Google again. Is there any way to stop this from happening? The code seems to only be editable by shopify since it's in the content_for_header.

Do a google search for:

inurl:/web-pixels-manager@

(https://www.google.com/search?q=inurl%3A%2Fweb-pixels-manager%40)

Each site has multiple versions of this page indexed. Peets.com for example has 20.

The pages look blank, but include something like this in the code:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Web Pixels Manager Sandbox</title>
    <script id="web-pixels-manager-setup">(function e(e,n,a,o,t){e&&(window.Shopify=window.Shopify||{},window.Shopify.analytics=window.Shopify.analytics||{},window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=function(e,n,a){window.Shopify.analytics.replayQueue.push([e,n,a])});var r,i,s,l,d,c,p,u,f=a+"/"+o+"."+function(){var e="legacy",n="unknown",a=null,o=navigator.userAgent.match(/(Firefox|Chrome)\/(\d+)/i),t=navigator.userAgent.match(/(Edg)\/(\d+)/i),r=navigator.userAgent.match(/(Version)\/(\d+)(.+)(Safari)\/(\d+)/i);r?(n="safari",a=parseInt(r[2],10)):t?(n="edge",a=parseInt(t[2],10)):o&&(n=o[1].toLocaleLowerCase(),a=parseInt(o[2],10));var i={chrome:60,firefox:55,safari:11,edge:80}[n];return void 0!==i&&null!==a&&i<=a&&(e="modern"),e}()+".js";r={src:f,async:!0,onload:function(){if(e){var a=window.webPixelsManager.init(e);n(a),window.Shopify.analytics.replayQueue.forEach((function(e){a.publishCustomEvent(e[0],e[1],e[2])})),window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=a.publishCustomEvent}},onerror:function(){var n=(e.storefrontBaseUrl?e.storefrontBaseUrl.replace(/\/$/,""):self.location.origin)+"/.well-known/shopify/monorail/unstable/produce_batch",a=JSON.stringify({metadata:{event_sent_at_ms:(new Date).getTime()},events:[{schema_id:"web_pixels_manager_load/2.0",payload:{version:t||"latest",page_url:self.location.href,status:"failed",error_msg:f+" has failed to load"},metadata:{event_created_at_ms:(new Date).getTime()}}]});try{if(self.navigator.sendBeacon.bind(self.navigator)(n,a))return!0}catch(e){}const o=new XMLHttpRequest;try{return o.open("POST",n,!0),o.setRequestHeader("Content-Type","text/plain"),o.send(a),!0}catch(e){console&&console.warn&&console.warn("[Web Pixels Manager] Got an unhandled error while logging a load error.")}return!1}},i=document.createElement("script"),s=r.src,l=r.async||!0,d=r.onload,c=r.onerror,p=document.head,u=document.body,i.async=l,i.src=s,d&&i.addEventListener("load",d),c&&i.addEventListener("error",c),p?p.appendChild(i):u?u.appendChild(i):console.error("Did not find a head or body element to append the script")})(null,null,"https://cdn.shopify.com/shopifycloud/web-pixels-manager/0.0.186","sandbox","0.0.186");</script>
  </head>
  <body></body>
</html>

Chummy Tees

Trevor · ‎05-15-2023

Hello everyone.

Thank you for your patience while we worked towards providing a fix to this issue and building out an FAQ that addresses the primary concerns and questions from this thread. You can find our FAQ on the Community Blog.

We welcome you to continue the conversation in the blog section of the blog post with any valid questions that aren't already answered by the FAQ. We will monitor the comments section for any valid feedback on how this change may impact Shopify stores and will actively remove or edit any comments that spread misinformation or speculation on the issue.

On that, this thread contains some misinformation and speculation on the issue and how it may impact ones online store. For this reason, we will no longer be monitoring this thread, but want to keep it open and available for historical purposes.

We greatly appreciate all of you for bringing this to our awareness and collaborating with us while we worked towards a solution. @Greg-Bernhardt deserves a special shoutout as they have championed this thread and this issue internally.

I'll be marking this as the solution as this has been resolved and helps surface this reply for anyone who may be new to the thread.

Trevor | Community Moderator @ Shopify
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Trevor · ‎07-06-2023

Hello,

Thank you for your continued feedback on the web pixels issue. More recent replies have been sharing misinformation or misattribute the issue. For this reason, we've chosen to close this thread.

If you believe you have a new issue that isn't answered by our FAQ, then we'd encourage you to create a new thread in our Technical Q&A board with as much detail as possible.

Thank you.

Trevor | Community Moderator @ Shopify
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Greg-Bernhardt · ‎03-29-2023

@MaBa that is not a problem and the expected result

@jackzhu what is the latest date?

To learn more visit the Shopify Help Center or the Community Blog.

PixelsPhotoArt · ‎03-29-2023

Hi there! We get this on all our websites. I see this was marked as solved in January but we still have these 404's on all our websites?

Greg-Bernhardt · ‎03-29-2023

@PixelsPhotoArt what report is that screenshot from?

To learn more visit the Shopify Help Center or the Community Blog.

PixelsPhotoArt · ‎03-30-2023

It is from google search console.

Greg-Bernhardt · ‎03-30-2023

@PixelsPhotoArt which report in GSC?

To learn more visit the Shopify Help Center or the Community Blog.

PixelsPhotoArt · ‎03-30-2023

Greg, the 404 report. Should I forward them to you in a DM?

Greg-Bernhardt · ‎03-30-2023

@PixelsPhotoArt great, that is the expected result, we want to see the old URLs in the 404 report. There is no harm there.

To learn more visit the Shopify Help Center or the Community Blog.

PixelsPhotoArt · ‎03-30-2023

How can a 404 with no way to redirect can be an expected result that brings no harm when search engines will penalize websites with a high number of 404 errors, as it can indicate that the website is not well-maintained or has broken links. This can lead to lower search engine rankings and reduced visibility for the website.

Greg-Bernhardt · ‎03-30-2023

@PixelsPhotoArt wrote:

How can a 404 with no way to redirect can be an expected result

Why would you want to redirect these "no content" pixel pages? A redirect means that the page now exists elsewhere. The pages are gone, so communicate that.

@PixelsPhotoArt wrote:

search engines will penalize websites with a high number of 404 errors

The context of those 404s is important. There is no "404 penalty" for Google. Those pixel pages do not exist and are not internally linked from anywhere. Google does not care about them or want to crawl them. 404 is the expected result.

If you suddenly have a high amount of high value and highly linked pages that go 404, that will harm performance, yes.

@PixelsPhotoArt wrote:

it can indicate that the website is not well-maintained or has broken links. This can lead to lower search engine rankings and reduced visibility for the website.

There are no broken links. 404ing pages that don't exist, by definition, is the sign of a well-maintained website instead of trying to trick Google with ineffective redirects.

To learn more visit the Shopify Help Center or the Community Blog.

Denis_B · ‎03-30-2023

why not fix these issues instead of semi-fixing them? Surely you can set the header to remove or redirect them?

I also do not want a bunch of 401's that will stay in the Google system forever for a coding mistake I did not make

modabilé · ‎04-11-2023

Hey Greg-Bernhardt,

the problem of this semi-solution is the export-maximum of 404 errors in Google Search Console. You can export up to 1.000 404 errors, but if you have more errors than that, you can't export them. That's a big problem, especially for larger Online-Shops because you won't be able to see real problems in your shop then, if the whole list is full with these kind of URLs that don't have any benefit for Google or for Shopify customers.

In my opinion this issue should really be fixed immediately

purpose driven jewellery
FOR A WILD AND SUSTAINABLE FUTURE
inspired by the harz

kbarcant · ‎04-14-2023

"The pages are gone, so communicate that."
-Greg Bernhardt, Shopify Staff, 3/30/2023

Conveniently enough, there's an HTTP code for that that. It's called 410 - Gone! What's super cool is it tells google that the page is "gone", not "missing" so google stops looking for it, they publicly acknowledged that 410's get pulled faster. I know per our DM's that you are aware of every single piece of information above so your response defending the 404 decision is concerning.

Maybe you acknowledged what I said but just thought I was wrong, I'm just a forum member. Why don't we ask the most respected source out there, the Internet Engineering Task Force.

15.5.11. 410 Gone

The 410 (Gone) status code indicates that access to the target resource is no longer available at the origin server and that this condition is likely to be permanent. If the origin server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) ought to be used instead.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed.

15.5.5. 404 Not Found

The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists. A 404 status code does not indicate whether this lack of representation is temporary or permanent; the 410 (Gone) status code is preferred over 404 if the origin server knows, presumably through some configurable means, that the condition is likely to be permanent.

Source Document:

https://www.rfc-editor.org/rfc/rfc9110.html#name-404-not-found

Greg-Bernhardt · ‎04-14-2023

Here's what John Mueller from Google said about 404 and 410.

“From our point of view, in the mid term/long term, a 404 is the same as a 410 for us. So in both of these cases, we drop those URLs from our index.

We generally reduce crawling a little bit of those URLs so that we don’t spend too much time crawling things that we know don’t exist.

The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so.

So if you’re just removing content naturally, then that’s perfectly fine to use either one. If you’ve already removed this content long ago, then it’s already not indexed so it doesn’t matter for us if you use a 404 or 410.”

To learn more visit the Shopify Help Center or the Community Blog.

modabilé · ‎04-15-2023

Hey Greg-Bernhardt,

the 410 solution would help to avoid the problem with too many 404 errors to export. Is it so hard to fix that, or why didn't you say anything to my comment? Here it is again:

Hey Greg-Bernhardt,

the problem of this semi-solution is the export-maximum of 404 errors in Google Search Console. You can export up to 1.000 404 errors, but if you have more errors than that, you can't export them. That's a big problem, especially for larger Online-Shops because you won't be able to see real problems in your shop then, if the whole list is full with these kind of URLs that don't have any benefit for Google or for Shopify customers.

In my opinion this issue should really be fixed immediately

purpose driven jewellery
FOR A WILD AND SUSTAINABLE FUTURE
inspired by the harz

TIMECAPSULES · ‎04-16-2023

Hi Greg-Bernhardt,

Thank you for working on this issue, and for having the attention of Shopify. It seems these attacks are some of the most important issues for Shopify - so it needs Shopify CEO attention and resources to get it fixed, and for Shopify to get ahead of it. This isn't where we are it seems.

However, it is confusing as to what I should or shouldn't do .... I have been noticing this issue for months on my Google Search Console and have been periodically reading this blog post.

The "Accepted Solution" is not valid as it is old. It must be updated please to tell people what they should do ... EXACTLY and clearly. Throwing additional comments midway in this blog isn't working!

For instance ..... I just inserted

Disallow: /web-pixels-manager
Disallow: /wpm

into my robots.txt.liquid file.

Should I have done that? I can't tell by reading the 9 pages of back and forth.

I still have lots of "non-indexed" files showing:

Screenshot 2023-04-16 125656.png

I have also created a whole bunch of redirects so that the 404's are presented on an actual webpage. Should I have done that?

I have trying work out what works and what doesn't now for months .... too long.

I await your reply please.

Thank you,

I-LIVE-ON

shadi1 · ‎04-16-2023

This is still a big problem, my indexed wpm pages are almost gone but not indexed pages count has gone from 20k to almost 60k in one month. Really tanked my ranking . spring time is my busy season and this is slowest i ever had in 10 years.

I have no clue what shopify is doing to fix this but they need to speed it up.

Also stop saying not indexed doesn't matter , i m sure a lot of business on shopify are feeling this. 20k to 60k count rise in not indexed pages is effecting our ranking.

If any one has any solution to speed this process up and get us back to normal please let us know?

AngryLuisatwpb · ‎04-16-2023

God!! You are damn right, and god pls punish them!! what they are doing to us, our years' effort, just ruined by their tiny code!!!!!!!

AsyncLJSoon · ‎04-17-2023

They need to pay for this!!!

Greg-Bernhardt · ‎04-17-2023

@modabilé 410s are also listed in the 404 GSC report. Google has documentation that the difference between 410 and 404 is a couple days at most. https://youtu.be/kQIyk-2-wRg?t=1721

@Andy-Envision I'm seeing that page return 404

@TIMECAPSULES nothing is required on your end

@ahgaos there is no evidence or communication from Google that number of noindex URLs results in a ranking loss. There have been a couple of Google Core Algorithm updates in the past few months. I would encourage you to investigate those.

To learn more visit the Shopify Help Center or the Community Blog.

Andy-Envision · ‎04-17-2023

That's because I removed my client's domain name and collection name, as you can see it's using website.com. I was only showing what the URL structure is appearing in GSC.

Mont · ‎04-17-2023

Thank you for Youtube link to 404 or 410 discussion. However, key here is speakers reference to "removing content naturally". Our issue, is with the thousands upon thousand of other pages that are somehow looking to be indexed. This so far is only happening with our Shopify site. I really do believe more attention needs to be paid here.

Of course Google Algorithm updates factor in, so does the economy. There are thousands of variables. Shopify can certainly argue to dismiss...as not their problem. It's certainly a problem for our store and I don't have the resources to ultimately rule this out as THE reason our SEO tanked. So....not sure what to do. Nov 2022 was our best month (as a start-up) Dec was good. Then ZIP

Nandykool · ‎04-17-2023

This gross stuff has been happening for 3 months, have you ever really thought about the solution to solve the problem?? and the fault you made?? sorry you don't think that's your fault, it's our fault, it's shopify users' fault to choose your! If what you did that you claim can solve our problem, why's here people keeping pooping out to complain?? We are your customers, but what you treat us like? keep thinking about this in your mind, we are hereby for real solutions, but not perfunctory words.

Again this has been 3 months, let's see how long does your company will keep running.

Pip999 · ‎04-18-2023

I'm not sure if this is helpful, so pls let me know if I've completely missed the mark here.

I responded to this thread months ago when I first noticed google had indexed some "non pages" on our website. I simply added a "no index" to any pages which had "vendors?q=" in the url and then added the offending url to my GSC removals.

I subsequently had another random "vendors?q=" type url index, I followed the same procedure.

I've had no issues.

What am I missing?

shadi1 · ‎04-18-2023

Greg I used to agree with you . not indexed pages shouldn't be effecting our seo. But in reality now i suspect it is effecting our stores . As for google algorithm updates even the best seo experts wont fully know whats going on with that update for a while as they see results come in. Who knows maybe not indexed pages would of been no issue last year but this year its an issue.

What i want to know is why "not indexed" pages count keeps going up ??? are those wpm pages still being generated? Take a look at the screen shot i just took of my GSC.

Just in the past few days it went up by 3k.

Anyone else experiencing this ?

Greg-Bernhardt · ‎04-18-2023

@shadi1 the report will continue for as many pages you have on your website as there may be a corresponding WPM page. The noindex is informative, not errors.

To learn more visit the Shopify Help Center or the Community Blog.

shadi1 · ‎04-18-2023

@Greg-Bernhardt I understand , but i want to know is there still lots of wpm pages being generated by shopify?

Greg-Bernhardt · ‎04-18-2023

@shadi1 as far as I'm aware, these WPM pages are used for various analytics tracking

To learn more visit the Shopify Help Center or the Community Blog.

dededus · ‎04-18-2023

Mont · ‎04-18-2023

Analytics tracking? whose? Why? If after all this was deliberately added… have no words.

shadi1 · ‎04-18-2023

@Greg-Bernhardt So when all the WPM are out of everyone's indexed pages count, are there any plans to go back and block those WPM pages in robot.txt and remove the no index that was placed in the header? Or would that bring us back to square one as more WPM pages are being generated and those new ones would be indexed thou blocked by robot.txt?

Greg-Bernhardt · ‎04-18-2023

@shadi1 Correct, you'd just see those URL move from the noindex report to the robots.txt and those URL may become indexed

@kbarcant I'd imagine you're still getting 404s added as Google returns to those original URLs. It can take many weeks as we originally stated.

The 404s occur from the original path. The noindexes occur from the new URL. The 404s won't likely be removed from the report. What is most important is that they are removed from the Google index.

Google has stated that Crawl Budget is not a consideration until you're into the millions of pages. Google will also reduce crawling of pages that are 404 and noindex.

Your concern about the 404 report pushing out non-wpm URLs is valid.

Can you DM me some URLs that are returning 404 and still remain in the index?

The 404s are intended and not a bug. These were old pixel URLs. They no longer exist and thus are 404.

To learn more visit the Shopify Help Center or the Community Blog.

FSFMatt · ‎04-19-2023

Good morning Greg,

Without meaning to sound rude, as I don't know where you sit in the hierarchy of things.
Has this been escalated or are you the only one firefighting this?

Kind Regards
Matt

jackzhu · ‎04-19-2023

I just want to say, Google tracking code also is used for various analytics tracking, but it could not generate so many pages. So do you think whether Shopify needs optimizate your tracking code to stop generating such shit pages !!!

Andy__D · ‎04-18-2023

This uptick in wpm pages being detected by Google Search Console started in early February for our store, around the time there was an announcement of the Server Pixels limited release forthcoming in the 04-2023 API updates.

February 6, we had 154 noindex pages detected in GSC. This has gone up to 4,443 since then.

Even if this doesn't impact ranking, it creates a lot of clutter in GSC. We can only view 1,000 noindex entries in the GSC interface (including export options), so we can't evaluate our noindex status for pages other than the wpm pages if they are not within the first 1,000 results.

We don't have any customer events configured that would use the web pixel manager system.

Denis_B · ‎04-18-2023

I have a bigger question - we should be given the option to "opt out" of Shopify "analytics".

It is a very poor tool and provides me with less than 10% of what i can see in Google analytics.

No shopify analytics = no need for WPM pages = problem solved

Italia-Straps · ‎04-18-2023

Our store in its “natural” state is only about 40 indexed pages. Every week we keep seeing an increase in not indexed pages especially “excluded by no index tag” wpm@ pages in GSC. The “excluded by no index tag” wpm@ pages just keep growing. We used to have none.

Question: Will these pages just keep growing in number forever?

As I mentioned, our store is small. We haven’t seen a ranking issue yet. It seems that some larger stores seem to have an exponential issue where they perhaps trigger a threshold with Google impacting ranking.

In other words, it makes sense Google would ignore some level no index pages but tens of thousands could be a negative ranking signal.

Mont · ‎04-18-2023

See how your site is doing and when/if trouble started https://feinternational.com/website-penalty-indicator/

Gtalan · ‎04-19-2023

Hi @Greg-Bernhardt,

For our site, the Excluded by ‘noindex’ tag (wpm and web-pixels-manager) URLs are steadily rising now at 4k. There are about 778 wpm/web-pixels-manager URLs that are indexed. We have not deployed any fixes on GSC. As a note, I checked the robots.txt.liquid file and it doesn't contain either Disallow: /web-pixels-manager or Disallow: /wpm.

I've been following this thread and so far, there isn't any clear answer. From the standpoint of NOT having done anything to "fix" the issue yet, what should we do at this point? Nothing? Or something? A clear step-by-step answer would be very much appreciated.

Thanks!

oemseats · ‎04-24-2023

We're ready to try anything. The "do-nothing" approach has gotten <10 of our 3,000+ 404's removed in six weeks, and due to google search console being overwhelmed by the numbers I can't even see if that's 10 wpm pages or 10 other 404's unrelated to this entire ordeal.

At this rate we'll be dealing with wpm 404s until Q1 2028.

Our go-to "Shopify Only" guy (kbarcant, who I've seen in this thread a few times, thanks for translating this all to english) is at a loss on what we, as a company, can do here. I needed an answer so I spoke to a few other Shopify store developers about this.

The first one was basically said exactly what I'd heard already. The second one said something I hadn't heard yet. They informed us that the 404s are here to stay for a very long time, and that this negative SEO is applied to the domain, not the shopify site.

They recommended WE BUY A NEW DOMAIN AND MOVE OUR ENTIRE WEBSITE TO IT.

We need clarity on what exactly shopify is doing to clarify this. "Wait, it'll fix itself" isn't an answer. My employees paychecks aren't waiting, they're still getting sent out. Shouldn't a company with 10% of the market share of ecommerce should be able to make a call to google and figure out how to fix this??? The only answer I've gotten so far that makes a lick of sense is changing the domain, and I'll be honest, it's not something I'm taking very lightly, we've been using this same domain for fifteen years.

Can you please give us an update? Sitting on the fence isn't very comfortable, we have to make a jump soon.

Greg-Bernhardt · ‎04-24-2023

The Support team and I are working on a communication doc to wrap this up. Absolutely DO NOT migrate to a new domain. That is patently absurd. There is no negative SEO applied from the 404s. Please wait for our document.

To learn more visit the Shopify Help Center or the Community Blog.

Denis_B · ‎04-24-2023

How soon - this has now been more than 3 months since this craziness started?

DariusWS · ‎04-24-2023

Dear Greg if you just say 10 time: "There is no negative SEO applied" it not will became true. Something is going on with shopify for a while. Please deep investigate problems and solve it. Everything not going to fix it up magically. Here is another relative problem: https://community.shopify.com/c/ecommerce-marketing/weird-url-adding-in-google-search-console/m-p/19...

FSFMatt · ‎04-24-2023

Hi Greg,

I am a little concerned that you mention a communication doc as opposed to a support doc.

Does this mean that we are going to have our concerns dismissed?

I would like to see a doc that explains how we remove this noise from our data so we can see the wood for the trees.

Kind Regard
Matt

TIMECAPSULES · ‎04-24-2023

Hi Greg,

Thanks for the communication that a document from Shopify is being produced on this vexxing issue for Shopify's customers.. As you know, there is a lot of angst re this issue - per comments on this blog.

Regards

Mont · ‎04-24-2023

Document should help our SEO

free-fall down another 25% since last week.

Mont · ‎04-24-2023

https://support.google.com/webmasters/thread/207283469/our-shopify-store-google-search-performance-h...

oemseats · ‎04-24-2023

@MontI'm surprised shopify hasn't told us to turn our site off and then turn it back on again, the level of troubleshooting they've provided here is dismal. I'm expecting to see a document that says "wait some more it'll fix itself, oh and 404s aren't bad for seo". Probably a good time to short some stock.

oemseats · ‎04-24-2023

Moving to a new domain gets rid of the 404s, something that Shopify has spent an entire quarter failing at. Every single solution that's been applied in the last 3 months failed. Can you really blame us for looking at other solutions? We've had to cut our employees hours because of this and Shopify has been completely opaque, and every "fix" has generated another issue. Saying that the 404s aren't to blame makes this more like a cover-up than a solution. We need a date on when to expect that document, even if it's "within a month" or something vague.

Greg-Bernhardt · ‎04-24-2023

Our goal is for the document to answer all the common questions we're seeing here. The 404s are not hurting your site although I understand the inconvenience of them clogging up the GSC report. Moving domains however is a massive risk and has the potential for legitimate disruption in your performance. We will NOT recommend that.

To learn more visit the Shopify Help Center or the Community Blog.

kbarcant · ‎04-24-2023

Adding thousands of 404s via bad links on high-value product pages has a profound undeniable negative effect on SEO.

I don't want anyone leaving the thread with the wrong idea.

Re: Shopify Bug - Web Pixels Manager Sandbox

How can I prevent the new indexing bug from creating useless pages on Google?

Community Blog Articles