Experiencing API outage

[emrecelik]

We are currently experiencing an issue with all of our customer's Admin APIs. When calling we get this response consistently:

{
    "errors": "Exceeded 0 calls per second for api client. Reduce request rates to resume uninterrupted service."
}

We are testing this with simple Postman requests against ALL our shopify customers and the experience is the same. 

[emrecelik]

We are passing 2 hours of downtime with our APIs and level 1 support is trying desperately to find someone on the products team to help. Is there anyone monitoring the Forums that can provide assistance? Thanks

[KarlOffenberger]

Are you actually receiving 429 status in your response? The "exceeded 0 calls" is confusing me a little, but if you are actually getting 429's then it's clearly a throttling issue. Without more context it's really impossible to tell.

[emrecelik]

We were approved for Payment Processing today and after that it seems like all our API limits are set to 0 for all of our existing client installs. So the message is more or less saying that I am exceeding my throttle limit of 0 allowable calls and gives the standard 429 status code. Very strange and seems isolated to our application. We are desperately trying to reach out to every support channel in hopes of getting the issue escalated with the Shopify application teams.

[KarlOffenberger]

So you are saying your X-Shopify-Shop-Api-Call-Limit is always 40/40 and doesn't leak? I am just trying to understand how come you're getting a 429. Either you're constantly exceeding your burst limit and not allowing it to go down e.g. by sending retries, or it's something else neither of us are aware of yet.

p.s. Ah the joy of Friday releases :-p

[emrecelik]

I certainly have never seen this before. I am not seeing the header X-Shopify-Shop-Api-Call-Limit at all in any of the responses.

{ HTTPError: Response code 429 (Too Many Requests)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
  name: 'HTTPError',
  method: 'GET',
  path: '/admin/orders.json?status=any&limit=250&page=9',
  protocol: 'https:',
  url: undefined,
  statusCode: 429,
  statusMessage: 'Too Many Requests',
  headers: 
   { server: 'nginx',
     date: 'Sat, 16 Feb 2019 02:03:25 GMT',
     'content-type': 'application/json; charset=utf-8',
     'transfer-encoding': 'chunked',
     connection: 'close',
     'x-sorting-hat-podid': '82',
     'x-sorting-hat-podid-cached': '1',
     'x-sorting-hat-shopid': '7047393',
     'x-sorting-hat-privacylevel': 'default',
     'x-sorting-hat-featureset': 'default',
     'x-sorting-hat-section': 'pod',
     'x-sorting-hat-shopid-cached': '1',
     'referrer-policy': 'origin-when-cross-origin',
     'x-frame-options': 'DENY',
     'x-shopid': '7047393',
     'x-shardid': '82',
     'x-stats-userid': '0',
     'x-stats-apiclientid': '2417604',
     'x-stats-apipermissionid': '104610365523',
     'retry-after': '0.0',
     'strict-transport-security': 'max-age=7889238',
     'x-request-id': 'bf761e0a-2053-497d-904f-6df84fffac07',
     'x-shopify-stage': 'production',
     'x-content-type-options': 'nosniff',
     'x-download-options': 'noopen',
     'x-permitted-cross-domain-policies': 'none',
     'x-xss-protection': '1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Forders&source%5Bsection%5D=admin_api&source%5Buuid%5D=bf761e0a-2053-497d-904f-6df84fffac07',
     'x-dc': 'chi2,gcp-us-central1' } }

[lewisf]

We were also approved for payment processing today and now our API calls are returning: "Exceeded 0 calls per second for api client. Reduce request rates to resume uninterrupted service." ... weird.

[emrecelik]

We desperately been trying to escalate the issue over the weekend. Thanks for reporting this. I advise to contact support and reference this Forum post and feel free to reference our ticket number #12061062 and state you have the same symptoms. We will do the same. They already have our notes documented in the ticket.

[lewisf]

Okay will do -- also trying to get this fixed as it's affecting our clients :(.

[emrecelik · Accepted solution]

Apps team responded and said that there was an issue:

After some digging, there was an issue that occurred when the scoping for payment processing was activated that set the API calls by client to 0, instead of the normal 100 per second amount. This has now been fixed and will not happen again with the integration.