Have your say in Community Polls: What was/is your greatest motivation to start your own business?

Re: Shopify Bug - Web Pixels Manager Sandbox

Solved

How can I prevent the new indexing bug from creating useless pages on Google?

ChummyTees
Tourist
9 0 33

Shopify has a new bug - they are indexing tons of useless pages in Google again. Is there any way to stop this from happening? The code seems to only be editable by shopify since it's in the content_for_header.

 

Do a google search for:

inurl:/web-pixels-manager@

(https://www.google.com/search?q=inurl%3A%2Fweb-pixels-manager%40)

 

Each site has multiple versions of this page indexed. Peets.com for example has 20. 

bug1.jpg

 

The pages look blank, but include something like this in the code:

 

 

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Web Pixels Manager Sandbox</title>
    <script id="web-pixels-manager-setup">(function e(e,n,a,o,t){e&&(window.Shopify=window.Shopify||{},window.Shopify.analytics=window.Shopify.analytics||{},window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=function(e,n,a){window.Shopify.analytics.replayQueue.push([e,n,a])});var r,i,s,l,d,c,p,u,f=a+"/"+o+"."+function(){var e="legacy",n="unknown",a=null,o=navigator.userAgent.match(/(Firefox|Chrome)\/(\d+)/i),t=navigator.userAgent.match(/(Edg)\/(\d+)/i),r=navigator.userAgent.match(/(Version)\/(\d+)(.+)(Safari)\/(\d+)/i);r?(n="safari",a=parseInt(r[2],10)):t?(n="edge",a=parseInt(t[2],10)):o&&(n=o[1].toLocaleLowerCase(),a=parseInt(o[2],10));var i={chrome:60,firefox:55,safari:11,edge:80}[n];return void 0!==i&&null!==a&&i<=a&&(e="modern"),e}()+".js";r={src:f,async:!0,onload:function(){if(e){var a=window.webPixelsManager.init(e);n(a),window.Shopify.analytics.replayQueue.forEach((function(e){a.publishCustomEvent(e[0],e[1],e[2])})),window.Shopify.analytics.replayQueue=[],window.Shopify.analytics.publish=a.publishCustomEvent}},onerror:function(){var n=(e.storefrontBaseUrl?e.storefrontBaseUrl.replace(/\/$/,""):self.location.origin)+"/.well-known/shopify/monorail/unstable/produce_batch",a=JSON.stringify({metadata:{event_sent_at_ms:(new Date).getTime()},events:[{schema_id:"web_pixels_manager_load/2.0",payload:{version:t||"latest",page_url:self.location.href,status:"failed",error_msg:f+" has failed to load"},metadata:{event_created_at_ms:(new Date).getTime()}}]});try{if(self.navigator.sendBeacon.bind(self.navigator)(n,a))return!0}catch(e){}const o=new XMLHttpRequest;try{return o.open("POST",n,!0),o.setRequestHeader("Content-Type","text/plain"),o.send(a),!0}catch(e){console&&console.warn&&console.warn("[Web Pixels Manager] Got an unhandled error while logging a load error.")}return!1}},i=document.createElement("script"),s=r.src,l=r.async||!0,d=r.onload,c=r.onerror,p=document.head,u=document.body,i.async=l,i.src=s,d&&i.addEventListener("load",d),c&&i.addEventListener("error",c),p?p.appendChild(i):u?u.appendChild(i):console.error("Did not find a head or body element to append the script")})(null,null,"https://cdn.shopify.com/shopifycloud/web-pixels-manager/0.0.186","sandbox","0.0.186");</script>
  </head>
  <body></body>
</html>

 

 

 

 

Accepted Solutions (2)

Trevor
Shopify Staff (Retired)
3408 463 1005

This is an accepted solution.

Hello everyone. 

 

Thank you for your patience while we worked towards providing a fix to this issue and building out an FAQ that addresses the primary concerns and questions from this thread. You can find our FAQ on the Community Blog. 

 

We welcome you to continue the conversation in the blog section of the blog post with any valid questions that aren't already answered by the FAQ. We will monitor the comments section for any valid feedback on how this change may impact Shopify stores and will actively remove or edit any comments that spread misinformation or speculation on the issue. 

 

On that, this thread contains some misinformation and speculation on the issue and how it may impact ones online store. For this reason, we will no longer be monitoring this thread, but want to keep it open and available for historical purposes. 

 

We greatly appreciate all of you for bringing this to our awareness and collaborating with us while we worked towards a solution. @Greg-Bernhardt deserves a special shoutout as they have championed this thread and this issue internally. 

 

I'll be marking this as the solution as this has been resolved and helps surface this reply for anyone who may be new to the thread. 

Trevor | Community Moderator @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Trevor
Shopify Staff (Retired)
3408 463 1005

This is an accepted solution.

Hello, 

 

Thank you for your continued feedback on the web pixels issue. More recent replies have been sharing misinformation or misattribute the issue. For this reason, we've chosen to close this thread. 

 

If you believe you have a new issue that isn't answered by our FAQ, then we'd encourage you to create a new thread in our Technical Q&A board with as much detail as possible.  


Thank you. 

Trevor | Community Moderator @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Replies 385 (385)
Greg-Bernhardt
Community Manager
95 1 34

@ahsonmkhan the first linked URL is noindex and the second is 404

To learn more visit the Shopify Help Center or the Community Blog.

ahsonmkhan
Excursionist
31 0 3

@Greg-Bernhardt  Thank you for all the support you have extended. We hope these will be sorted out soon.

jackzhu
Excursionist
11 0 7

affected pages.pngdetail links.png

 

Hi Greg, there is no noindex in wpm@ pages and our site produces such pages everyday.  Google still indexed them even we have add  Disallow:/wpm@* in the robots.txt.  

 

Our site had indexed about 3 hundreds pages before. Now google shows more than 6000 pages and our traffic dropped by 2/3. Can you guys stop the code to produce such no content, meaningless pages?

 

jackzhu
Excursionist
11 0 7

@Greg-Bernhardt Can you guys stop the code to produce such no content, meaningless pages?

kbarcant
Shopify Partner
50 0 81

@Greg-Bernhardt 

indexrep1.pngIn one night one of my sites went from under 100 freshly fixed 404's to over 3,000. The next day (3/15) I'm down to 75% of the traffic 3/16 I'm at 50%. It's stayed steady since.

 

We need to get this resolved ASAP, this is not going to be easy to explain to the business owner why half of his traffic vanished. Please, any update would be appreciated. I'll be tapping F5. 

jonpemp
Tourist
8 0 3

its really bad that shopify is completely not addressing this issue when they are the CMS, majority of stores on teh platform are being affected.

I went to do a check on the sites in my country, a ton of them even big names are affected 

shadi1
Explorer
74 1 49

Hi

 

Where did the /web-pixel-manager@ come from, i have the /wpm@ issue going on right now. I had the web pixel manager issue and it was supposedly fixed few month ago.

 

The /wpm@ is getting fixed right now, my indexed pages dropped by 1000 last night, i tested to make sure @WPM is not blocked by robot.txt any more and its also not being indexed .

 

So The /wpm@ pages are dropping . Your screen shots are scaring me thou , I m really hoping the /web-pixels manager@ is not back now.

 

My Sales traffic and sales have also had a big drop the past few weeks. I haven't heard anything about compensation yet . This is not our fault we should be compensated some how!!!

jackzhu
Excursionist
11 0 7

Yeah, we need compensation !!!!!

ahsonmkhan
Excursionist
31 0 3

@Greg-Bernhardtplease have a look 

Indexable.jpg

 

 

Mont
Explorer
58 1 28

Please someone tell me if this makes sense , because it appears they're playing whack a mole my little store can't take being slammed by Google, they won't run my ads even. Am I correct the issue is that our auto generated sitemap.xml includes these bad urls. What if built my own tidy sitemap and sent that to Google instead. 

Mont
Explorer
58 1 28

OK Interesting...just went to collect my site map and "Products"  was changed by shopify from https://vr.supplies/sitemap_products_1.xml.

to

https://vr.supplies/sitemap_products_1.xml?from=1680358867017&to=6815972884560

 

In essence removing bad urls (404) that were there yesterday from reaching Google. I just sent a new index request to Google Console. Fingers crossed it worked.

Mont
Explorer
58 1 28

Just went to collect my site map and "Products"  was changed by shopify from https://vr.supplies/sitemap_products_1.xml.

to

https://vr.supplies/sitemap_products_1.xml?from=1680358867017&to=6815972884560

 

In essence removing bad urls (404) that were there friday from reaching Google. I just sent a new index request to Google Console. Fingers crossed it worked.

kbarcant
Shopify Partner
50 0 81

Are they seriously not going to resolve this? The silence is deafening.

Greg-Bernhardt
Community Manager
95 1 34
  • web-pixel-manager URLs will 404 and drop from the index
  • wpm URLs will be noindex and drop from the index

Both of these things will take weeks to complete. You will see an increase in your "pages with noindex" report and that is a nonissue in terms of SEO.

 

To learn more visit the Shopify Help Center or the Community Blog.

jackzhu
Excursionist
11 0 7

@Greg-Bernhardt 

  • wpm URLs will be noindex and drop from the index

What is the shopify's method?  Add noindex tag in wpm pages? I did not find such tag in these pages at present

 

You said nonissue in terms of SEO. Now Google has indexed these pages. If your guys do not know what is spam from Google, here is the definition from Google https://developers.google.com/search/docs/essentials/spam-policies .

 

Spammy automatically-generated content

Spammy automatically generated (or "auto-generated") content is content that's been generated programmatically without producing anything original or adding sufficient value; instead, it's been generated for the primary purpose of manipulating search rankings and not helping users. Examples of spammy auto-generated content include:

  • Text that makes no sense to the reader but contains search keywords
  • Text translated by an automated tool without human review or curation before publishing
  • Text generated through automated processes without regard for quality or user experience
  • Text generated using automated synonymizing, paraphrasing, or obfuscation techniques
  • Text generated from scraping feeds or search results
  • Stitching or combining content from different web pages without adding sufficient value

If you're hosting such content on your site, you can use these methods  to exclude them from Search.

 

 

Greg-Bernhardt
Community Manager
95 1 34

@jackzhu the noindex is sent via http header, not meta tag

To learn more visit the Shopify Help Center or the Community Blog.

jackzhu
Excursionist
11 0 7

@Greg-Bernhardt i see it. But our site also still get such pages indexed. Why shopify could not stop the code to produce such pages?

Denny10
Excursionist
20 0 17

"pages with noindex" are not problematic, if they are out of the index. The only thing that might need to be monitored is the crawl budget after everything is fixed.

The problem we had/have is "indexed, although blocked by robots.txt".

 

The GSC is usually also lagging a few days behind. That's why it might look like pages are still going to be indexed, which is not the case.

 

As long as the robots.txt doesn't block those pages (you can test the affected URLs here) & noindex-Tag is added to those sites (X-Robots or Meta robots, you can test the affected URLs here here), no more pages are going to be indexed.

 

The only thing you can do to clear the index faster is to go to "indexed, although blocked by robots.txt" in Search Console and start a review. You'll probably need several runs to get them all out of the index.

kbarcant
Shopify Partner
50 0 81

@Greg-Bernhardt 

  "web-pixel-manager URLs will 404 and drop from the index"

 

How? Is something changing in the future? Right now /web-pixels-manager pages are not blocked in the sitemap, not blocked with no-index tags, and not blocked with x-robots tags. What has changed and how can we verify that it's actually changed?

 

I can't tell google the problem is fixed and have them spend an exorbitant amount of our crawl budget over the next month to see that the pages are still indexable and being served by shopify. Stuff happens. Just be transparent with us and tell us what the roadmap is because saying indexable pages "will 404 and drop from the index" is a promise that seems unrealistic.

 

indexable.png

 

Greg-Bernhardt
Community Manager
95 1 34

@kbarcant can you please DM that link, thanks!

To learn more visit the Shopify Help Center or the Community Blog.

kbarcant
Shopify Partner
50 0 81

Dm'd.

Denis_B
Excursionist
28 0 44

@Greg-Bernhardt -This issue has draggged on for  nearly 3 months. For clarification, can you confirm:

 

1 - You state that the sandbox url's are "no-index" in the http header - but when I check one of my product skus url with the sandbox url

 

https://www.bunkbedking.com/web-pixels-manager@0.0.186/sandbox/products/bunk-beds-tall-stairway-gray...

 

(see screenshot) it shows that google can index:

https://www.dropbox.com/s/v1gjkjr6wrdunrw/Screenshot%202023-03-26%2010.49.19.png?dl=0

 

I think part of the problem is that shopify has created many url options of this sandbox pixel and now I have them scattered around in Google Page Indexing - I have versions in:

Blocked by Robots.txt

Crawled - currently not indexed

Excluded by ‘noindex’ tag

Not found (404)

 

So can you confirm exactly what steps are needed to clean up this mess for each of these 4 categories?

 

Reading thru your comments, it seems the solution is to do nothing and let the shopify created url fix the issue. If so, you need to go back to all the versions of this sandbox url shopify created and make them not not indexable ie as the wpm@0.0.245@ variant

 

https://www.dropbox.com/s/83dlit8nj3h96wi/Screenshot%202023-03-26%2011.08.55.png?dl=0

From our side;

We should not add a no-index, no-follow tag in Robots.txt for the sandbox url's?

We should not disavow the sandbox url's?

Thanks

Denis

 

as an aside why was this code not tested before it went live to ensure it functioned correctly?

 

Pascal_Keane
Shopify Partner
8 0 4

I'm astounded at the patience of this community. Clearly a lot of knowledgeable people in here who understand the impact of this issue. I don't have anything to contribute except to add that I have the same issue and have regularly been frustrated with Shopify's inability to address obvious issues in a timely manner (setting accurate times for shipping zones being one - still unresolved for the UK). It is clear that smaller retailers are all but ignored on this platform.  

jonpemp
Tourist
8 0 3

Hi @Greg-Bernhardt when can we expect shopify to come out with a fix for this...?

 

Just an estimated date of when the devs can fix this issue that is plaguing many merchants

Greg-Bernhardt
Community Manager
95 1 34

@Jon can you please articulate the exact issue you are facing?

To learn more visit the Shopify Help Center or the Community Blog.

Greg-Bernhardt
Community Manager
95 1 34

@Denis_B the pixel URL you provided redirects to https://www.bunkbedking.com/collections/bunk-beds-stairway. You'll want to investigate why.

 

@tom511 what report is that screenshot from? The new URL is from search, not web pixel.

To learn more visit the Shopify Help Center or the Community Blog.

ahsonmkhan
Excursionist
31 0 3

Hi @Greg-Bernhardt 

 

The rest of the indexed pages are getting deindexed slowly however I have seen a new indexed page today. Please guide me if there is any index tag that needs to be added like vendors/collections.

 

https://www.urbanbeauty.pk/search?q=%E5%9C%A8%E7%BA%BF%E9%BE%99%E8%99%8E%E6%96%97%2C%E3%80%90%E8%AE%...

 

 

Greg-Bernhardt
Community Manager
95 1 34

Thanks for the link @ahsonmkhan, I'll take this back to the devs

To learn more visit the Shopify Help Center or the Community Blog.

ahsonmkhan
Excursionist
31 0 3

Also please check if we can use this solution for the time being (Read this on another community post)

 

{% if request.path == '/search' and search.results_count == 0 %}

  <meta name="robots" content="noindex">

{% endif %}

Greg-Bernhardt
Community Manager
95 1 34

@ahsonmkhan yes you can use that. Once implemented, please confirm it's working.

To learn more visit the Shopify Help Center or the Community Blog.

Greg-Bernhardt
Community Manager
95 1 34

@ahsonmkhan that page is sending noindex via header but your theme is injecting a robots meta which says to index.

To learn more visit the Shopify Help Center or the Community Blog.

ahsonmkhan
Excursionist
31 0 3

@Greg-Bernhardt This is a bit complicated for me. please suggest should I go ahead with putting that noindex tag in the head section only

 

or is there anything else that should be done along with it

jackzhu
Excursionist
11 0 7

We all have lots of wpm@ urls in GSC indexed page. @Greg-Bernhardt Now what is the solution for these pages?

Denis_B
Excursionist
28 0 44

@Greg-Bernhardt

 

Hi Greg

 

Response please to my other questions:

 

I think part of the problem is that shopify has created many url options of this sandbox pixel and now I have them scattered around in Google Page Indexing - I have versions in:

Blocked by Robots.txt

Crawled - currently not indexed

Excluded by ‘noindex’ tag

Not found (404)

 

So can you confirm exactly what steps are needed to clean up this mess for each of these 4 categories?

 

Reading thru your comments, it seems the solution is to do nothing and let the shopify created url fix the issue. If so, you need to go back to all the versions of this sandbox url shopify created and make them not not indexable ie as the wpm@0.0.245@ variant

 

https://www.dropbox.com/s/83dlit8nj3h96wi/Screenshot%202023-03-26%2011.08.55.png?dl=0

From our side;

We should not add a no-index, no-follow tag in Robots.txt for the sandbox url's?

We should not disavow the sandbox url's?

Thanks

Denis

 

as an aside why was this code not tested before it went live to ensure it functioned correctly?

Greg-Bernhardt
Community Manager
95 1 34

@Denis_B 

 

Blocked by Robots.txt: These will reduce over time as we removed the block

Crawled - currently not indexed: this report is not an issue in this instance

Excluded by ‘noindex’ tag: this report is not an issue in this instance

Not found (404): this report is not an issue in this instance

To learn more visit the Shopify Help Center or the Community Blog.

tom511
Visitor
1 0 0

Capture d’écran 2023-03-27 à 13.38.31.png

The problem is still not solved knowing that it is more than 2 weeks old for the number 1 host of ecommerce site it becomes shameful. Each day more is less credibility for all your users. Moreover, even today I still have the Wpm and Web pixel Manager pages that appear but I also have a Chinese URL that was not created by me, which has just been detected. So I'm going to ask users if this is related to the same problem as WPM and if they too have detected this because shopify doesn't seem to want to help us this PROBLEM, or doesn't communicate enough which makes everything unclear and very unpleasant.

AngryLuisatwpb
Tourist
4 0 10

Why are you doing this on Earth? Do you know how hard we effort on our webs? While you can just leave with a tiny code bug bomb for us to deal with the shitty results? We don't accept it! YOU SOLVE THIS PROBLEM AT ONCE, AND YOU OWE US AN APOLOGY!

Andy__D
Visitor
2 0 2

@Greg-Bernhardt We are also experiencing this issue on our store. Should I DM you with details?

Mont
Explorer
58 1 28

Organic search result from Analytics for March 2023 if down by 36% from March 2022. 

ahgaos
Visitor
1 0 0

@Greg-Bernhardt 

 

Can you provide an update on this? I agree with @TIMECAPSULES , it is not clear from these 9 pages what we have to do to fix this issue. You keep asking for individual store pages with issues but what is the generic fix every store owner can use to resolve the indexing problem? 

 

Can you please provide an update? Or at least inform us on the status of fixing this? My SEO ranking is dropping severely, similar to other's. Please communicate a clear path forward or at least provide some insights from Shopify's side into the timeline of fixing this. Doing so also removes the urge for other's to keep contacting you about the same matter. 

 

Thanks in advance.

Andy-Envision
Shopify Partner
18 0 23

This is definitely not fixed. We have clients sending us their GSC error notification emails on a weekly basis. Now the error is showing up as a Server error (5xx) in GSC. 

 

https://website.com/wpm@0.0.280@5c46ad45w03c6b27bp74bf52d0m9e637fdc/web-pixel-shopify-custom-pixel@0530/sandbox/collections/collection-name

 

Mont
Explorer
58 1 28

I have a question for all - just curious. Of the few - throughout this community with similar issues going back for some as early as last summer - I noticed sites from EU, Canada etc... also NOT dot.com's

 

SO Questions are: What country is your affected site in? What is affected site domain name end suffix?

 

Just wondering if there's a pattern re outside the USA or Not a dot com

edubnix
Tourist
3 0 2

Great question. My issue is strictly on our Japan subdomain right now

LuluBeau
Tourist
5 0 5

Hi, 

I've been following this issue. Our site's ranking has dropped by almost 40% since March. According to the latest Google report, although there are not generating any new indexed pages (web or wpm), there are still constantly generating new web 404 pages and noindexed wpm. Theoretically, these pages shouldn't impact our rankings. However, the true problem is that our rankings aren't showing any signs of improvement and are continuing to worsen. I'm unsure about what steps to take next, and I'm feeling very anxious about the situation.

Pip999
Excursionist
41 0 7

Is there a list of all text affecting urls? This is what I've flagged and no indexed so far. Be good to add in any other text no so google doesn't index my pages if I'm affected by them.

 

collections/vendors?q=
web-pixels-manager@0.0.186/sandbox/
wpm@0.0.234@1543f512w2542bf6cpa9acea22m3c2685ed/sandbox/blogs/news
wpm@0.0.245@a1fcb853wff0d51fapfe890cd8ma6042ba1/sandbox/

LuluBeau
Tourist
5 0 5

Hello,

 

Given the current large number of noindex wpm pages, would disallowing them be more beneficial to reduce Google's crawling costs?

 

Thanks!

jasonwill21
Excursionist
18 0 14

Omg, happy to see people are having same issues as me. My traffic is turning BIG downwards due to this issue since Middle of March core update. Tons of pages are not indexed currently. Can anyone suggest me solution?Screenshot 2023-04-19 at 1.57.29 PM.pngScreenshot 2023-04-19 at 4.13.05 PM.png

shadi1
Explorer
74 1 49

So far shopify solution has been to no index, they placed a no index tag in the headers of all stores, they can't use a disallow tag in robot.txt as that would create a lot of indexed thou blocked by robot.txt pages.

I thought this would of been a temp solution and i m trying to find out if there a permanent one in the works!

Any news on that @Greg-Bernhardt 

 

 

Denis_B
Excursionist
28 0 44

The problem is that this issue has clogged up my no-index list in GSC - so i cannot see if any of my real skus are not being indexed - frustrating

 

I would really like to see a response from Shopify  of their plan to fix the current issue and ensure it is not an ongoing problem (not just comments from Greg who I thik is trying to do his best, but has no complete answers that solve our concerns)

 

Mont
Explorer
58 1 28

Agree, would appreciate knowing what Shopify intends to do beyond temp fixes, if anything.... or maybe they can't or don't know. Greg thank you for doing all you can.

 

I suspect Google's A.I. and algorithms will fix their problem with us. Who knows where that will lead. If we go one more month with SEO pointing in the wrong direction...whatever the cause. It just isn't sustainable for our Shopify store.