Changed the storefront App Proxy URL (in app listing) and getting a 500 error on the new address

Shopify Partner
11 0 3

I changed the address for an App Proxy inside my application configuration.

Now when I load the address I get a 500 with the message `There was an error in the third-party application`.

The request is not proxied to the URL and my server never receives a request.

Is there some sort of delay when changing the app proxy URL of an app?