Application Proxy - There was an error in the third-party application.

Thomas_Kwan
Shopify Partner
25 0 0

Hi there

 

Our app suddenly started getting this error since Friday (21 Jun, 2019). And the error can be seen from here - https://etechfocus-demo.myshopify.com/apps/simple-testimonials

 

That URL will hit our proxy URL ( https://myshopifyapps.com/simple-testimonials/proxy?shop=etechfocus-demo.myshopify.com) which is returning valid liquid codes. So we are not sure why our clients and our test app are getting this error.

Thanks for your help in advance.

Replies 34 (34)

Thomas_Kwan
Shopify Partner
25 0 0

Actually, we notice that our proxy was not even called. We setup our proxy as

 

Subpath prefix: apps

Subpath: simple-testimonials

Proxy URL: https://myshopifyapps.com/simple-testimonials/proxy

 

When we accessed our test store's https://etechfocus-demo.myshopify.com/apps/simple-testimonials, our proxy was not called. 

Saw the error code 500 from cloudflare

 

* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 500
< date: Mon, 24 Jun 2019 14:50:10 GMT
< content-type: text/html; charset=utf-8
< set-cookie: __cfduid=df665a7c97b070af8aed7c0543ae7f76f1561387809; expires=Tue, 23-Jun-20 14:50:09 GMT; path=/; domain=.myshopify.com; HttpOnly
< set-cookie: _shopify_y=62873d60-a98e-4222-9d6a-0f59242c3fa1; path=/; expires=Thu, 24 Jun 2021 02:28:34 -0000
< set-cookie: _orig_referrer=; Expires=Mon, 08-Jul-19 14:50:10 GMT; Path=/; HttpOnly
< set-cookie: _landing_page=%2Fapps%2Fsimple-testimonials; Expires=Mon, 08-Jul-19 14:50:10 GMT; Path=/; HttpOnly
< set-cookie: secure_customer_sig=; path=/; expires=Fri, 24 Jun 2039 14:50:10 -0000; secure; HttpOnly
< set-cookie: cart_sig=; path=/; expires=Mon, 08 Jul 2019 14:50:10 -0000; HttpOnly
< content-security-policy: block-all-mixed-content; frame-ancestors *; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=app_liquid&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fapp_proxy&source%5Bsection%5D=storefront&source%5Buuid%5D=7d0fe70a-cdd0-4cd2-8d06-6afae0b17d26
< vary: Accept-Encoding
< x-shopid: 21465161
< x-shopify-stage: production
< x-sorting-hat-podid: 52
< strict-transport-security: max-age=7889238
< x-dc: gcp-us-east1,gcp-us-east1,gcp-us-east1
< x-sorting-hat-shopid: 21465161
< x-shardid: 52
< x-content-type-options: nosniff
< x-request-id: 7d0fe70a-cdd0-4cd2-8d06-6afae0b17d26
< x-download-options: noopen
< x-xss-protection: 1; mode=block; report=/xss-report?source%5Baction%5D=app_liquid&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fapp_proxy&source%5Bsection%5D=storefront&source%5Buuid%5D=7d0fe70a-cdd0-4cd2-8d06-6afae0b17d26
< x-permitted-cross-domain-policies: none
< content-language: en
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 4ebf8333caa6a366-HKG

jem_punsalan
New Member
5 0 0

We are also getting this issue, when we tried to use proxy tunnel (ngrok) pointing to our local app, it is working fine. But when using a domain for the app proxy it is the same as what you are getting. Anything else we need to look into? Thanks

Thomas_Kwan
Shopify Partner
25 0 0

Hi jem_punsalan

 

Are you just recently getting this error? I started getting this error since around last Friday 06/21/2019.

 

Thomas

jem_punsalan
New Member
5 0 0

Hi @Thomas_Kwan we only get this issue yesterday, am not sure if theres new implementations related to App Proxy for Shopify.

pixeledluke
Tourist
8 0 1

Hi Thomas,

 

We have been experiencing exactly the same issue at: https://unionsubscriptions.myshopify.com/apps/fulfilment which is supposed to be proxying a subdomain, but doesn't seem to be getting hit at our side. Getting the same 500 error you have reported.

 

If we visit the proxying url directly, it works, just as you are seeing. We first started noticing this on the 20/06/2019.

 

I've seen your thread with CloudFlare, and seen they mentioned that Shopify have just started working with them. Have you have any update from them, or have you got any further with the issue your side?

 

Cheers,

Luke.

Thomas_Kwan
Shopify Partner
25 0 0

Hi Luke

 

I did not get any further update from Shopify yet. Currently as a short-term fix, I have setup a ngrok channel and I have changed the proxy url to point to ngrok instead of my server directly.That seems to be working fine.

 

We have 2 apps and I have left one app pointing to our server so that I can tell if the problem has been resolved or not.


https://etechfocus-demo.myshopify.com/apps/simple-testimonials (With short-term fix, running thru ngrok)

https://etechfocus-demo.myshopify.com/apps/proxy (Pointing to our server, still broken)

 

I will update this thread when i have additional info.

 

thanks

thomas

Thomas_Kwan
Shopify Partner
25 0 0

Here is timeline with the Shopify Support so far

 

Jun 24, 03:55 (13350334)

- Reported the error and I was told the problem will be reported to the Partners team

"regarding to that I can ask the Partners team, I have your email and i will give it to the said team so that they can contact you"

 

Jun 25, 04:41 (13363587)

- Enquiring status and I was told that the ticket is in the system but the ticket was NOT assigned
"The partner specialists have the ticket in the system so you have done everything right, this is a technical query so a Guru like me would be of no help to you, but they will reach out as soon as the ticket arrives on their desk"
" Then I can tell you that at this time there is no partner manager assigned to your account"

 

Jun 26 17:35 (13383873)

- Asking about status and got help from support to raise the priority up
"I will set the ticket as more urgent priority so they review it sooner but typically it does take our escalated teams 24-72hrs to review tickets"
"And it was escalated on Monday, I will also include your 2nd support ticket with it so they can review both at once"

 

Jun 27 16:55 (13395784)
- Asking for status

"just reaching out to our partner team to what's going on with the ticket since I can see it was opened on Monday"
"It appears that we can escalate this to our technical team as well so I will do that, I'm just going to ask you a few more questions so that I can get all of the information the first time and save some time"

jem_punsalan
New Member
5 0 0

Thanks @Thomas_Kwan let us know for any updates. For now we use ngrok to tunnel our app as short term solution.

Thomas_Kwan
Shopify Partner
25 0 0

Jun 28 18;11 Got email from Partner Support

 

"Sorry to see the issue has been open for so long, I'm going to escalate to our Developer support team for further assistance. They should be in touch in a couple business days!"

Thomas_Kwan
Shopify Partner
25 0 0

July 1 4:15pm (13430540)

- sent 2 emails in last 2 days and got no response

- asking for update via support chat

"Okay, since this has already been sent to our Developer Support, technical has no control on when our Developer Support will reply. I see it is marked as a high priority, but I can put this one up to urgent for you so they can get onto it straight away. How does that sound?"

Alex
Shopify Staff
1561 81 341

Hey guys.

 

Are you absolutely certain you are not receiving these web requests? Going over our app proxy logic, it's pretty explicit in that there are two code paths leading to that error in third party application state.

 

1. Your response is too large. If you cannot replicate the error by sending a tiny response as a test, then that would be your issue.

2. If the response from your server is in the 500 range. The expression to check this is quite literally if response.status/100.floor == 5 (this is the most common case).

 

Let me know what you guys think. I do know cloudlfare has been having some issues lately, where do you guys host your apps?

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

Thanks for getting back. I am pretty sure I am not getting the web request. I have been monitoring the access log on the server when I tried to access the proxy link on the Shopify side. This is the normal flow (correct me if I am wrong)

 

  Shopify's proxy URL (for example  https://etechfocus-demo.myshopify.com/apps/proxy) -> Our server (https://myshopifyapps.com/simple-meta/proxy)

 

When I hit https://etechfocus-demo.myshopify.com/apps/proxy I don't see any new entry in our access log on our server. Yet, the browser returns a 500 error from Cloudflare. I believe Shopify calls Cloudflare which calls our server. You can try it yourself.

 

If I change the flow like the following, it works (without any server side changes on our end)

 

  Shopify's proxy URL -> ngrok (a free proxy service) -> Our server

 

The issue could be on Cloudflare' side so even the logic you showed works, it does not mean much. The Cloudflare integration is on Shopify side, so we have no way to check why Cloudflare is returning 500.

 

Alex
Shopify Staff
1561 81 341

Thanks for that information @Thomas_Kwan.

 

The 500 you're seeing in the browser from cloudflare is the status that we're resolving to after determining that the proxied app response is in the 500 range, so a 500 leading to a 500 from us. That said, I'm working on making certain that what I'm seeing in the logic is in fact not a 500 range response from some proxy or middleware which that code is hitting rather than your app host itself. Stay tuned on that one.

 

Cheers.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

pixeledluke
Tourist
8 0 1

Hi Alex,

 

To confirm from our side, we are also checking our access logs on our server, and are not getting anything coming through when using the proxy:

https://unionsubscriptions.myshopify.com/apps/fulfilment => https://checkout.unionroasted.com/

 

As per Thomas's issue, we are also seeing the 500 error from Cloudflare, but via ngrok, our server gets the request, however this cannot be used as a permanent fix in our case.

 

As for the size of the request, we have adjusted the response we are sending back to one line of text, which results in the same error.

Alex
Shopify Staff
1561 81 341

Thanks @pixeledluke. That helps.

 

I'll update this thread when I've got something.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Alex
Shopify Staff
1561 81 341

Hey guys.

 

I spoke to the teams responsible for the app proxy logic and I can confirm that there is only two cases where the error you're seeing is surfaced:

 

1. Your response is too long, try shortening it first to make sure that's not the issue (you guys have ruled this out already I believe).

2. Your server is returning a status in the 500 range.

 

When specifically referring to 2, the code path is effectively if res.status/100.floor == 5 (then throw the error). res.status I have confirmed is absolutely the response from the server we're hitting, we don't hit any middleware of our own, the code calling this is the proxying mechanism. Is it possible you guys are behind a reverse proxy or load balancer which is where the 500 response is being sent back, before it hits your app server maybe?

 

Cheers.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

We are not 1 or 2. Things work when ngrok is being used in front of our service. We do not have any other proxy service running.

 

We have these configured on the proxy setting page

 

Subpath prefix: apps

Subpath: proxy

Proxy URL: https://myshopifyapps.com/simple-meta/proxy

 

Can you confirm with your team the following

 

a) When Shopify see 500 as error, can you capture the entire response and send it to us?

b) Or,  Is there a debug mode that we can turn on to get more info in the http response?

 

thanks

pixeledluke
Tourist
8 0 1

Hi @Alex 

 

As confirmed previously, we can also access the endpoint directly, which functions correctly with a non-500 error, and have confirmed that this also works via ngrok, as does @Thomas_Kwan 's proxy.

 

The response limit has been disregarded previously in this thread, as we've tested a response that was less than 1kb, which results in the same issue.

 

I will await your response to Thomas's questions, as this seems to be an identical error.

 

Thanks.

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

We have these configured on the proxy setting page

 

Subpath prefix: apps

Subpath: proxy

Proxy URL: https://myshopifyapps.com/simple-meta/proxy

 

If you hit "https://myshopifyapps.com/simple-meta/proxy" directly, you will see a response of "<div>Hello</div>", and it will return no 500 internal error.

But when accessing "https://etechfocus-demo.myshopify.com/apps/proxy", we see 500. So the 500 is not from our server.

Alex
Shopify Staff
1561 81 341

@Thomas_Kwan as I mentioned before, the 500 you're seeing in the browser is raised by us when we supposedly see a 500 from your server. The 500 our proxy mechanism sees leads to a 500 raised for the client on the storefront. To answer your prior questions:

 

a) We don't currently capture the response, but I'm going to look into getting this added in order to give everyone a clearer picture as to what's happening (body, headers, etc). It would be a good idea to do so.

b) There is not presently a debug mode, but hopefully adding logging for the above details would be a strong first step into making it easier to work through issues like this one.

 

For what it's worth, I'm not seeing other reports on this issue elsewhere, to give you the support perspective. That's not me saying it's a non issue, I'm just adding observations from my end.

 

I'll let you guys know when we've got some logging added to capture more details of the responses which allegedly have a 500 status. I don't have an explanation for why it works when accessing the page directly vs approaching it through an app proxy without some more details, which is why my guess was that it was possible that the issue was occurring between ourselves and your app server. I've confirmed we don't hit cloudflare before hitting your app server (or whatever might be in front of it, which would make some sense as to why it's not working in your production environment).

 

For good measure, I didn't see if you guys ended up confirming with me where you host your production app. That might help a bit on my end.

 

Cheers.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

pixeledluke
Tourist
8 0 1

Thanks @Alex ,

 

We are currently hosting our app on Rackspace.

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

Our server is with Linode and the location of the server is Fremont, California.

 

And I think how Cloudflare involved is as follows:

 

  Cloudflare -> https://etechfocus-demo.myshopify.com/apps/proxy (Shopify's Link) -> https://myshopifyapps.com/simple-meta/proxy (Our Server)

 

Note that Shopify's proxy link is hosted behind Cloudflare.

 

Can u try and confirm if you see the followng?

 

1) When you hit our proxy link (https://myshopifyapps.com/simple-meta/proxy, this is what we have configured on the proxy setting page) directly in the browser, it returns VALID liquid.

2) But when you hit https://etechfocus-demo.myshopify.com/apps/proxy (which behind the scene hit the link in 1 as well ) it returns "error in third party application "

 

Do you have other possible explanations for the above behavior?

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

Another data point, I changed the proxy settings from

 

  Subpath prefix: apps

  Subpath: proxy

  Proxy URL:https://myshopifyapps.com/simple-meta/proxy

 

to

 

  Subpath prefix: apps

  Subpath: proxy

  Proxy URL:https://myshopifyapps.com/this_return_404_file_not_found.html

 

If you hit https://myshopifyapps.com/this_return_404_file_not_found.html directly, you will see 404. But when using the proxy link (https://etechfocus-demo.myshopify.com/apps/proxy), I am still seeing "There was an error in the third-party application". And I do not see new entry in the access log when hitting the service via Shopify's proxy link.

 

Let's pretend the request did hit our server. Doesn't this show that Shopify shows the third-party application even if our server returns non-500 errors? What I am trying to demonstrate is that there is more going on behind the scene than the logic that you described.

 

thanks again for your help and support

pixeledluke
Tourist
8 0 1

Hi @Alex 

 

Do you have any update on this following @Thomas_Kwan 's info?

 

Thanks.

Thomas_Kwan
Shopify Partner
25 0 0

Hi @Alex 

 

I want to follow up and see if you have found additional info for the issue.

 

thanks

thomas

Alex
Shopify Staff
1561 81 341

Hey guys, I unfortunately don't have anything to share just yet, but I'll update this thread as soon as I've got more info.

 

Cheers.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

Do u have any update on the situation?

 

thanks

thomas

Thomas_Kwan
Shopify Partner
25 0 0

Hi Alex

 

Any update?

 

thanks

thomas

pixeledluke
Tourist
8 0 1

@Thomas_Kwan / @Alex ,

 

An update from our side:

We have renewed our client's SSL certificate today, which has resolved the error that was originally reported.

 

Our previous SSL supplier was GoDaddy, and is now TSO Host. Both certificates are standard SSL certificates for a subdomain.

 

Not sure if this will help resolve the issue, or will help in any way for anybody else.

 

Thanks.

Alex_Stanhope1
Shopify Partner
4 0 2

For anyone following this thread in the future, I had a similar problem caused not by an expired cert, but a dodgy Apache vhost config.

 

The Shopify app proxy will verify the certificate of the target system, which means you need to set in your vhost:

SSLCertificateFile

SSLCertificateKeyFile

SSLCACertificateFile

 

Sometimes SSLCACertificateFile is only set if SSLVerifyClient is also set.

 

If you're debugging this sort of thing in the future, don't just rely on Chrome Developer tools.  I had two URLs (one working with app proxy, one not).  Both showed a valid certificate, both responded with identical content and near identical headers.  In the end I had to use cURL to expose the problem:

 

curl -XGET -v <url>

 

which highlighted the error:

 

* NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.
* Closing connection 0
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

 

Shopify couldn't validate my cert because it didn't know where to find the CA bundle.

mjmabato
New Member
9 0 0

are you using development store?

Thomas_Kwan
Shopify Partner
25 0 0

Hi jem_punsalan

 

We just tried ngrok, and yeah, it worked for us. This will be a good short-term work around for us. Almost feel like there is cloudflare is calling some IPs (ngrok's ) but not all (our server for example).

So with ngrok.io domain in the proxy URL works, but not with our domain. Using ngrok introduces an extra layer of networking routing which is not desirable.

 

thomas

jem_punsalan
New Member
5 0 0

Hi @Thomas_Kwan we only use ngrok for local testing our app. As for the domain app proxy issue, still not clear of the root cause. We are still trying to find the solution for this. Let us know if you also find something related to the app proxy issue.

adelespinasse
Shopify Partner
9 0 3

I got this error when the proxy URL pointed to a Localtunnel URL (similar to ngrok). It works fine when pointed to our production service.

 

Pretty annoying since it makes it hard to test in a development environment. I don't want to have to switch to ngrok.