App Proxy removes custom headers

Solved
yadiguzel
Tourist
4 0 4

Hi,

I have a service worker for sending web push notifications which is hosted on my server and configured with header 'service-worker-allowed: /' (you can see below). 

curl -D - https://myapplicationserver/sw.js
HTTP/2 200 
date: Sat, 26 Dec 2020 08:03:34 GMT
content-type: application/javascript
content-length: 2213
service-worker-allowed: /
apigw-requestid: YJotkjunvHcEJmw=

When I am trying to host this script with App Proxy 'service-worker-allowed: /' header is removed from response (you can see below).

curl -D - https://myshop/apps/myapplication/sw.js
HTTP/2 200
apigw-requestid: YQUUTgckPHcES0A=
cf-cache-status: DYNAMIC
cf-ray: 6089fd490d5d3688-LAX
cf-request-id: 074a1ca1a300003688c987d000000001
content-encoding: br
content-type: application/javascript
date: Mon, 28 Dec 2020 08:42:42 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
nel: {"report_to":"network-errors","max_age":2592000,"success_fraction":0.0001}
report-to: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
server: cloudflare
set-cookie: secure_customer_sig=; path=/; expires=Tue, 28 Dec 2021 08:42:41 GMT; secure; HttpOnly
set-cookie: cart_currency=USD; path=/; expires=Mon, 11 Jan 2021 08:42:41 GMT; SameSite=Lax
set-cookie: cart_sig=160343e4d7c8c38e333a06286465b09f; path=/; expires=Mon, 11 Jan 2021 08:42:41 GMT; HttpOnly; SameSite=Lax
vary: Accept-Encoding
x-dc: gcp-us-central1,gcp-us-central1

 

Last week I didn't have the issue because i was able to send web push notification with same configuration.  Do you have any suggestions regarding how to solve this issue. 

Thanks

 

 

Accepted Solution (1)

Accepted Solutions
Shayne
Shopify Staff
Shopify Staff
208 9 47

This is an accepted solution.

Thanks for chiming in everyone, and for the detailed explanation from yadiguzel, Edgemesh, and PushOwl. While service workers on the root path (and the headers that they require) are no longer supported, you're still able to serve up service workers from your app proxy sub-path. There are a few apps on the app store who are already using this approach to send push notifications. Hopefully it's a pattern that will still offer the utility that your merchants need. If the change leaves you with missing features, either DM me or drop a message in the thread, and we can work through potential solutions together. We all want to provide as much merchant utility as we possibly can, but as the web evolves, we also need to make changes to keep merchant security ahead of the curve.

Shayne | Developer Advocate @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

View solution in original post

Replies 22 (22)
Greg_Kujawa
Shopify Partner
1020 83 262

I ran into something similar, in that my app proxy calls were missing header values that previously were honored. Started back on 18-Dec-2020. Not sure if there was some sort of change on Shopify's end, but it sure sounds like it. Maybe someone from Shopify can weigh in on this? 

HunkyBill
Shopify Expert
4762 54 548

Not sure it is appreciated, but I have to ask anyway. Why use custom headers in the first place? You know Shopify strips cookies and other extraneous values, trying to ensure the Proxy is not a vector for anything nasty. So why not just send data to the Proxy endpoint, work on it, and return results, all without jamming something into the header.

Is there something you can do with a header you can't do with the data payload? What is the purpose where you can achieve something special?

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
yadiguzel
Tourist
4 0 4

Hi,

for sending web push notifications you have to use service workers. Application it self does not require anything from headers. But if you want to use service worker in root scope you have to use this header.

for more details about service workers please refer

https://developers.google.com/web/fundamentals/primers/service-workers

Thanks

 

 

HunkyBill
Shopify Expert
4762 54 548

I am surprised you'd try and jam that via the Proxy in the first place. Walking a tightrope there.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
Edgemesh
Tourist
9 0 3

Shopify has sent us (and others who use the Service Worker framework) a note indicating that they will not support the Service Worker Header in the App Proxy. We (along with numerous other application developers in the community) are reaching out to the team there to see what the plan/status is. However - in the meantime I believe newly submitted App-Proxy apps the require the Service Worker will fail. I'd suggest opening a support ticket with the Shopify team as well. 

HunkyBill
Shopify Expert
4762 54 548

Totally not surprised that end run is getting blocked. 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
yadiguzel
Tourist
4 0 4

HunkyBill why you are thinking such? What is problem working with service workers from your perspective?  Some applications still has this custom header. Here is push owl service worker response headers. 

apigw-requestid: YS0HehHnoAMEV3g=
cache-control: max-age=604800
cf-cache-status: DYNAMIC
cf-ray: 60903b48eac23610-LAX
cf-request-id: 074e03619300003610be151000000001
content-encoding: br
content-type: application/javascript
date: Tue, 29 Dec 2020 02:53:35 GMT
etag: W/"c0e26ab6ad4b93d4f3cc5e183294f413"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
last-modified: Wed, 23 Dec 2020 07:53:37 GMT
nel: {"report_to":"network-errors","max_age":2592000,"success_fraction":0.0001}
report-to: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
server: cloudflare
service-worker-allowed: /
set-cookie: secure_customer_sig=; path=/; expires=Wed, 29 Dec 2021 02:53:35 GMT; secure; HttpOnly
set-cookie: cart_currency=USD; path=/; expires=Tue, 12 Jan 2021 02:53:35 GMT; SameSite=Lax
set-cookie: cart_sig=d445c8fcce2facbfd5b0add8f934de82; path=/; expires=Tue, 12 Jan 2021 02:53:35 GMT; HttpOnly; SameSite=Lax
vary: Accept-Encoding
x-amz-id-2: yhOAjtDPPVBsK4Rr+N6GTWv0sd5smxmbRhUmZUfOzKLNrbCX1cgWS92Rjj8RBMsBkUWUPRgMn6c=
x-amz-request-id: 313A0BD56BD914E8
x-dc: gcp-us-central1,gcp-us-central1

 

Greg_Kujawa
Shopify Partner
1020 83 262

Sounds like you will need to implement a "middleman" to receive the app proxy requests, massage the data into a format that your service worker will accept, then forward request that along, and finally take the service worker response and pass it back to the app proxy caller. 

My scenario where I found the app proxy mechanism had changed was actually a fortunate one. I had mistakenly referenced the app proxy URL with the partner development shop's DNS name. This had worked for about a year and then failed. Due to CORS. I corrected my goof-up by correctly referencing the app proxy URL with the shop's DNS name that it was installed on. Which was the way I should've implemented it all along. 

HunkyBill
Shopify Expert
4762 54 548

@yadiguzelthere is nothing about what push owl does that requires service workers. Everything that App offers is available through a regular call to a Proxy endpoint. So I see no point in arguing for these headers to be passed. I can see why Shopify would restrict that vector. So it is not like I have a problem with service workers, I am just commenting that I am NOT surprised Shopify does not allow them in the context of Proxy calls.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
Edgemesh
Tourist
9 0 3

As Push OWL uses the Push API  (like other push notification providers) it requires a Service Worker in order to access things like ServiceWorkerGlobalScope.onpush and ServiceWorkerRegistration.pushManager.  This is also true for similar services such as Aimtell, Re:Amaze, etc. 

Since the Service Worker is client-side (e.g. on the browser ) but NOT being served from the root (e.g. https://store.com/sw.js) apps need to set this header to define the Scope.
This is a critical feature here on Shopify, as there is not a way to serve a file on the root directly (as files uploaded reside in the /assets path).
As such, removing this header from the app-proxy disables all service-worker functionality on Shopify.

Google docs is probably a good example of this header and it's scope requirements being set in the wild, where users work on the path https://docs.google.com/document/u/0/ but the Service Worker lives at https://docs.google.com/document/offline/serviceworker.js. In order to allow the Service Worker (which is orthogonal to the page itself) to operate at a deeper scope of https://docs.google.com/document , the Service-Worker-Allowed header is set to "/document" (see image below).

googleDocsSW.png

HunkyBill
Shopify Expert
4762 54 548

I appreciate all the attention to detail here, in the marvelous world of service workers. As per the OP problem, I am simply throwing in my 2 cents that it seems that if Shopify was actively removing the service worker header entry from App Proxy calls, it was due to them not wanting to introduce this as a vector. If they find the concept of service workers to be benign and fine, and that that headers is fine in App Proxy requests, then they will allow it.

I am making no claim to the use case being valid, but I am simply stating, App Proxy calls are usually stripped of anything other than the bare minimum needed to pass data to the correct endpoint, and to render the resulting data, whether it be HTML/Liquid or JSON. How this ever became mired in service workers is clearly a difficult construct to explain away.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
Shashank_Kumar
Shopify Partner
5 0 6

Shashank here from PushOwl. Hope I can add some context.

Service workers are an essential part of the modern web and adopted by modern websites to provide functionalities like push notifications, PWA, client-side caching, and much more. Service workers have some pre-requisites in order to run properly on a store:

  • The javascript file should be loaded from the same domain (hence the use of App Proxy to load service workers)
  • Service workers have scopes. The scope defines where the service workers will work. The scope needs to be '/' in most cases so that service worker can be loaded across all sub-paths like `/collections/..` or `/products/...` or just landing page of a store itself `/`

Shopify till now allowed both the options ie. loading service workers via proxy and also setting the `service-worker-allowed` header. There was a recent change from Shopify which started removing any custom header and also throwing an error message on the client-side. This broke apps esp. push notifications, PWA, and client-side caching apps. Shopify deployed an immediate quick fix which allowed existing apps to still load the header. 

The reason cited for the removal was a security issue. We are still waiting for details about that. 

I really hope Shopify either allows already vetted apps to continue as usual or have secure ways of loading the service workers. Service workers make the web better. And hence I am waiting for an email with a positive note for Shopify in 2021.

Shashank_Kumar
Shopify Partner
5 0 6

In case of folks more curious about the tech, do have a read here: https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers

HunkyBill
Shopify Expert
4762 54 548

No question they serve a useful purpose. Thing is, they can also be abused, and hence are a vector for abuse. So that is probably why the move to strip the headers. No one is debating the utility of the service workers. You say they are essential, and to you they are, but in the context of playing in someone else's sandbox, they might not see your Tonk truck as essential to their fun in the sandbox.

So it remains an issue, till Shopify either capitulates and lets ALL service worker headers through the Proxy, or none. Clearly, everyone is waiting with bated breath.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
yadiguzel
Tourist
4 0 4

Hi  Shashank_Kumar,

Did Shopify team make any update about topic. I checked change logs, but i did not find anything related.

Thanks for your replies 

Greg_Kujawa
Shopify Partner
1020 83 262

Seeing this was a conscious decision made by Shopify and implemented not too long ago, I'm not so sure how quickly they would reverse it out of the blue. Not to mention that the change was likely implemented due to some possible abuse or misuse at the very least.

The solution isn't terribly complex. Rather than point an app proxy at a third party service worker that has custom header requirements, implement a small lightweight middleware service to handle the app proxy calls. Pass along whatever data is important in the request body. Take all of that in the middleman and pass along whatever you want to the third party service worker. The entire middleman might consume maybe a screen or two's worth of code.

Shayne
Shopify Staff
Shopify Staff
208 9 47

This is an accepted solution.

Thanks for chiming in everyone, and for the detailed explanation from yadiguzel, Edgemesh, and PushOwl. While service workers on the root path (and the headers that they require) are no longer supported, you're still able to serve up service workers from your app proxy sub-path. There are a few apps on the app store who are already using this approach to send push notifications. Hopefully it's a pattern that will still offer the utility that your merchants need. If the change leaves you with missing features, either DM me or drop a message in the thread, and we can work through potential solutions together. We all want to provide as much merchant utility as we possibly can, but as the web evolves, we also need to make changes to keep merchant security ahead of the curve.

Shayne | Developer Advocate @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

gamenow
New Member
2 0 0

@Shayne 
I am serving service-worker using app-proxy and still, I am getting scope error - "The path of the provided scope ('/') is not under the max scope allowed ('/apps/notifier/'). Adjust the scope, move the Service Worker script, or use the Service-Worker-Allowed HTTP header to allow the scope."

jvasek
New Member
1 0 0

Hello Shayne,

 

We are in the partner program and would like to have a conversation as to what the best approach would be to work within the guidelines of this new restriction.  I have tried contacting you via DM but this community account is new and I therefore do not have the privilege to send/receive such messages.  Would you mind enabling that feature for this account so that I can reach out to you privately?

 

Thank you for your time in this matter.