Experiencing API outage

Solved
Highlighted
Shopify Partner
13 4 4

We are currently experiencing an issue with all of our customer's Admin APIs. When calling we get this response consistently:

 

{
    "errors": "Exceeded 0 calls per second for api client. Reduce request rates to resume uninterrupted service."
}

We are testing this with simple Postman requests against ALL our shopify customers and the experience is the same. 

0 Likes
Highlighted
Shopify Partner
13 4 4

We are passing 2 hours of downtime with our APIs and level 1 support is trying desperately to find someone on the products team to help. Is there anyone monitoring the Forums that can provide assistance? Thanks

0 Likes
Highlighted
Shopify Partner
1840 177 707

Are you actually receiving 429 status in your response? The "exceeded 0 calls" is confusing me a little, but if you are actually getting 429's then it's clearly a throttling issue. Without more context it's really impossible to tell.

Liked this post? You might also like our fantastic upsell apps Candy Rack and Candy Cart or offer free gifts with Gift Box. All made with ❤️  and care by Digismoothie
0 Likes
Highlighted
Shopify Partner
13 4 4

We were approved for Payment Processing today and after that it seems like all our API limits are set to 0 for all of our existing client installs. So the message is more or less saying that I am exceeding my throttle limit of 0 allowable calls and gives the standard 429 status code. Very strange and seems isolated to our application. We are desperately trying to reach out to every support channel in hopes of getting the issue escalated with the Shopify application teams.

0 Likes
Highlighted
Shopify Partner
1840 177 707

So you are saying your X-Shopify-Shop-Api-Call-Limit is always 40/40 and doesn't leak? I am just trying to understand how come you're getting a 429. Either you're constantly exceeding your burst limit and not allowing it to go down e.g. by sending retries, or it's something else neither of us are aware of yet.

 

p.s. Ah the joy of Friday releases :-p

Liked this post? You might also like our fantastic upsell apps Candy Rack and Candy Cart or offer free gifts with Gift Box. All made with ❤️  and care by Digismoothie
0 Likes
Highlighted
Shopify Partner
13 4 4

I certainly have never seen this before. I am not seeing the header X-Shopify-Shop-Api-Call-Limit at all in any of the responses.

 

{ HTTPError: Response code 429 (Too Many Requests)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
  name: 'HTTPError',
  method: 'GET',
  path: '/admin/orders.json?status=any&limit=250&page=9',
  protocol: 'https:',
  url: undefined,
  statusCode: 429,
  statusMessage: 'Too Many Requests',
  headers: 
   { server: 'nginx',
     date: 'Sat, 16 Feb 2019 02:03:25 GMT',
     'content-type': 'application/json; charset=utf-8',
     'transfer-encoding': 'chunked',
     connection: 'close',
     'x-sorting-hat-podid': '82',
     'x-sorting-hat-podid-cached': '1',
     'x-sorting-hat-shopid': '7047393',
     'x-sorting-hat-privacylevel': 'default',
     'x-sorting-hat-featureset': 'default',
     'x-sorting-hat-section': 'pod',
     'x-sorting-hat-shopid-cached': '1',
     'referrer-policy': 'origin-when-cross-origin',
     'x-frame-options': 'DENY',
     'x-shopid': '7047393',
     'x-shardid': '82',
     'x-stats-userid': '0',
     'x-stats-apiclientid': '2417604',
     'x-stats-apipermissionid': '104610365523',
     'retry-after': '0.0',
     'strict-transport-security': 'max-age=7889238',
     'x-request-id': 'bf761e0a-2053-497d-904f-6df84fffac07',
     'x-shopify-stage': 'production',
     'x-content-type-options': 'nosniff',
     'x-download-options': 'noopen',
     'x-permitted-cross-domain-policies': 'none',
     'x-xss-protection': '1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Forders&source%5Bsection%5D=admin_api&source%5Buuid%5D=bf761e0a-2053-497d-904f-6df84fffac07',
     'x-dc': 'chi2,gcp-us-central1' } }
0 Likes
Highlighted
Tourist
6 1 0

We were also approved for payment processing today and now our API calls are returning: "Exceeded 0 calls per second for api client. Reduce request rates to resume uninterrupted service." ... weird.

0 Likes
Highlighted
Shopify Partner
13 4 4
We desperately been trying to escalate the issue over the weekend. Thanks for reporting this. I advise to contact support and reference this Forum post and feel free to reference our ticket number #12061062 and state you have the same symptoms. We will do the same. They already have our notes documented in the ticket.
1 Like
Highlighted
Tourist
6 1 0

Okay will do -- also trying to get this fixed as it's affecting our clients :(.

0 Likes
Highlighted
Shopify Partner
13 4 4

This is an accepted solution.

Apps team responded and said that there was an issue:

 

After some digging, there was an issue that occurred when the scoping for payment processing was activated that set the API calls by client to 0, instead of the normal 100 per second amount. This has now been fixed and will not happen again with the integration.

1 Like