Re: Consistent delay (20~ second average) of webhooks delivered via EventBridge

Consistent delay (20~ second average) of webhooks delivered via EventBridge

Shopify Partner
43 0 9

I recently switched from HTTP to EventBridge for receiving webhooks. I've noticed consistent delays, averaging ~20 seconds for `orders/create` and ~ 12 seconds for `orders/paid`, for example.

I send EventBridge events to a lambda function, where I then calculate how long it took the function to receive the payload by taking the difference of the `event['time'` from the EventBridge event and the `payload['created_at']` field. Code for that below. Then I generate a metric in DataDog to track this over time. 


According to the `event['time'] attribute is likely sent by Shopify, so by using this time (rather than ``), this should take any lambda issues out of the equation (e.g. function cold starts, concurrency, function code, etc).


How I calculate the delay:


def calculate_event_delay(event_time, topic, payload)
  action = topic.split('/').last
  timestamp = case action
    when 'create'
    when 'update', 'updated', 'paid', 'delete', 'edited'
    when 'cancelled'
    when 'uninstalled'
  if timestamp
    (event_time - Time.parse(timestamp)) * 1000



Where `event_time` is the timestamp provided directly by the event notification.

Here are my metrics in DataDog:


Screen Shot 2021-02-09 at 2.52.51 PM.png

 These two webhook topics should be delivered in near real time, as opposed to a customers/update or products/update webhook, which I understand can be throttled so the app doesn't get spammed.


Would really appreciate this gets looked into because my app relies on a timely `orders/create` webhook as I have a widget on the Order Status Page (confirmation page). 


Replies 5 (5)

Shopify Staff (Retired)
61 7 13

Hi @Jeff-Blake ,


When did you notice the increase in times? Both HTTP and EventBridge delivery latency should be roughly the same. We're rolling out some improvements to try and reduce the latency while not increasing any risk of duplicates. You should start seeing a decrease across both HTTP and EventBridge delivery times starting today.



To learn more visit the Shopify Help Center or the Community Blog.

Shopify Partner
43 0 9

Hey Mike. Thanks for responding.

Unfortunately, I only started tracking this recently... as after switching to EventBridge I noticed that my order status page widget was simply not loading fast enough (even though my app was doing all the processing it needed to within 300-500ms), i.e. I have the widget set to poll every 2 seconds, and with HTTP for the past ~600 days has always been mostly great.. my widget would load in 1-3 seconds. After 10 polls (20 seconds) my app kicks off a job to look for that order manually, and I saw a huge spike in those jobs recently, which led me to tracking this metric. I never noticed any serious latency with HTTP, and I would have heard complaints if that was the case.

I just checked my metrics and it looks like you guys made improvements around 7:30 AM PST! This is great, but still, this seems like way higher latency than it ought to be (or was, with HTTP). 


Past day:

Screen Shot 2021-02-10 at 9.34.22 AM.png


Past 1 hour (average around 13s):

Screen Shot 2021-02-10 at 9.35.58 AM.png


Hope this helps and if you need any more information let me know. 

Shopify Partner
43 0 9

One observation is the orders/paid and orders/update eventbridge notifications average around 5 seconds, whereas orders/create averages 13 seconds.


This graph shows the past 4 hours

Screen Shot 2021-02-10 at 11.35.14 AM.png

5 0 1

Did you find a solution? I’m subscribing to orders/paid in order to print a receipt after POS checkouts but it can take more than 10 seconds to trigger making the process too slow to be useful 😞

Shopify Staff (Retired)
61 7 13


We strive for low-latency on the delivery but there's a lot of factors that can cause some delays along the way. The size of the order being one of them. Larger orders (and therefore, larger payloads) may contribute to a slight delay in delivery.

You should see most webhooks arrive within 10s from the time the event gets triggered. If there's a failure on the destination (ex: a destination API endpoint is failing/timing out) it will get retried several times with each attempt adding more delays.



To learn more visit the Shopify Help Center or the Community Blog.