Inconsitency found in Shopify Webhooks

[Tom13]

Hi,

I have recently started development of an app that accept requests from Shopify Webhooks(Settings > Notifications > Webhooks).

For the Shopify webhooks that I received on my server, I realized that the date/time found under the payload property "updated_at" is inconsistent with the sequence of the webhooks sent.

For example:

I received "refunds/created" event before "orders/canceled" event, but after a closer look at the "updated_at" property in each payload, I discovered that the date/time is "2018-12-07 12:30:11" and "2018-12-07 12:30:10" respectively, indicating that "orders/canceled" had occurred first on the Shopify's server before the "refunds/created" event.

Considering the inconsistency above, could anyone advise on an approach? Whether to process the webhooks according to the sequence received OR the date/time found in the "updated_at" property?

Looking forward to hear from you.

[KarlOffenberger]

Considering webhooks are fire & forget, if fired and if received at all, I would not rely on the sequence of receiving said webhooks.

Best regards

[Tom13]

Hi Karl,

Thanks for your input. Yes, I agree with you and am not depending on the webhooks for accomplishing "mission critical" tasks.

However, the order of clearing the webhooks queue is important to record the historical events that occurred to an order.

For example:

The sequence of webhooks received for a sales order:
1. Order/Paid
2. Order/Created
3. Order/Canceled
4. Refunds/Created

If we clear the webhooks sequentially, our current queue clearing logic will face a few issues.

#1 - Logic will discover that there is still no order created in order to mark the order as paid.
#2 - Logic will create an order but not mark as paid(which is inaccurate, the order has been paid in #1).
#3 - Logic will cancel the order. No problem.
#4 - Logic will discover that order has been canceled, thus no refunds applicable(which is inaccurate, a refund is required in #3).

Of course, we could redesign our logic to work differently, however, with a clearer understanding of the webhooks sequence and "updated_at" property in the payload data, it will help lessen the time for trial and error, or maybe our approach is wrong. There may be a better way out there.

Thus, looking forward to more developers familiar with the above to help shed some light on the matter. Many thanks!

[ClementG]

I've been wrestling with that issue for a very long time and never found a satisfactory solution, besides calling back into the rest api.

IMO each webhook should come with an incrementing sequence id or version id to disambiguate to help resolve requests out of order.

[KarlOffenberger]

Hmm... we must all realize that ordered message processing at internet scale is a massively difficult problem. Tak any message queue broker, be it RabbitMQ or even Kafka and try to achieve above. Message processing with multiple parallel consumers cannot be guaranteed by most of these - even Kafka cannot make such claims across multiple partitions.

Shopify is an outfit that operates at considerable scale - I am pretty sure they don't have one sole queue chugging along somewhere in the cloud, happily and oblivious of anything else, taking in and orderly sending out. That's easy when you have it running on your local dev box or a small outfit where one queue instance gladly abides.

IMO, it would be a safer bet to not rely on order or even delivery of webhook events. There are several techniques how to accomodate for this depending on how "timely" your service needs to be.

Just my 2c - hope you guys find a way to sort (no pun intended) this out 😉

[Tom13]

Hi, thank you all for the replies and comments. Really appreciate it. 

Well, looks like we may need to rethink the approach than using the current webhooks method. 

Yes, it has to be as timely as possible, and assuming we call the rest API, how do we know exactly “when” to do it? I don’t think polling it consistency with close frequency is a good idea. Has anyone implement something that is more “event-driven”?

[KarlOffenberger]

Haven't done anything in code, but first thing that comes to mind is using the /admin/orders/#{order_id}/events.json endpoint and corelating the verbs found in there with whatever webhook you received. If you receive refunds/created before orders/cancelled you should be able confirm by requesting the order's events (without actually having to poll or load full order which you basically get within the webhook payload).

It's still chatty and doesn't solve the issue of "dropped" webhooks, but off the top of my head I cannot think of a less chatty solution and am pretty sure a more resilient solution with webhooks alone isn't possible - Shopify could implement backoff retry's for webhook delivery, but even that wouldn't save us in case of catastrophic failure.

What would be "nicer" is if we could use the GraphQL API to query events and filter by subjectType=order (or product and so on) as well as having field parity with the REST API which also includes verbs etc. As is, the GraphQL BasicEvent is rather useless for applications other than displaying message strings.

[Ryan]

Karl's right on the money, webhooks can't be guaranteed to arrive in the exact correct order, or even arrive at all.  It's a best effort exercise, and while mostly reliable, are not 100%. 

Shopify could implement backoff retry's for webhook delivery

Failed webhook delivery does have a back off.

What would be "nicer" is if we could use the GraphQL API to query events and filter by subjectType=order (or product and so on) as well as having field parity with the REST API which also includes verbs etc. As is, the GraphQL BasicEvent is rather useless for applications other than displaying message strings.

Not sure if this exactly what you are looking for but you can do something like this:

# simple query to get top 10 messages from order timeline
query {
  order(id: "gid://shopify/Order/597885419542") {
    events(first: 10) {
      edges {
        node {
          __typename message
          createdAt
          id
        }
      }
    }
  }
}

Return from my test shop:

[
  {
    "order": {
      "events": {
        "edges": [
          {
            "node": {
              "__typename": "BasicEvent",
              "message": "Kiarra Little placed this order on Developer Tools.",
              "createdAt": "2018-09-25T17:20:28Z",
              "id": "gid://shopify/BasicEvent/9217353023510"
            }
          },
          {
            "node": {
              "__typename": "BasicEvent",
              "message": "Developer Tools fulfilled 4 items from Shopify2.",
              "createdAt": "2018-09-25T17:20:29Z",
              "id": "gid://shopify/BasicEvent/9217353482262"
            }
          },
          {
            "node": {
              "__typename": "BasicEvent",
              "message": "This order was archived.",
              "createdAt": "2018-09-25T17:20:29Z",
              "id": "gid://shopify/BasicEvent/9217353809942"
            }
          },
          {
            "node": {
              "__typename": "BasicEvent",
              "message": "A $140.00 CAD payment was processed on the card.",
              "createdAt": "2018-09-25T17:20:29Z",
              "id": "gid://shopify/BasicEvent/9217353842710"
            }
          },
          {
            "node": {
              "__typename": "BasicEvent",
              "message": "Received new order <a href=\"https://ryanink.myshopify.com/admin/orders/597885419542\">#1135</a>.",
              "createdAt": "2018-09-25T17:20:32Z",
              "id": "gid://shopify/BasicEvent/9217356595222"
            }
          }
        ]
      }
    }
  },
  {
    "cost": {
      "requestedQueryCost": 13,
      "actualQueryCost": 8,
      "throttleStatus": {
        "maximumAvailable": 1000,
        "currentlyAvailable": 992,
        "restoreRate": 50
      }
    }
  }
]

[KarlOffenberger]

Hi Ryan, thanks for clarifying!

Would you be able to elaborate on the backoff strategy? Is it doumented somewhere?

As for GraphQL, yes, I am aware of being able to do that, but the BasicEvent object has little value if you want to determin the actual event type.

For instance with REST Admin API, I could

// Request

GET /admin/orders/879868051571/events.json



// Response (removed a few fields for brevity)

{
  "events": [
    {
      "id": 13117662789747,
      "subject_id": 879868051571,
      "subject_type": "Order",
      "verb": "cancelled",
      "arguments": [
        "api_client_id",
        2632381
      ],
      "message": "test canceled this order. Reason: Other.",
    },
    {
      "id": 13117662724211,
      "subject_id": 879868051571,
      "subject_type": "Order",
      "verb": "mail_sent",
      "arguments": [
        "Order Cancelled",
        "____@foo.no",
        "api_client_id",
        2632381
      ],
      "message": "test sent an order cancelled email to Karl Offenberger (___@foo.no).",
    },
    {
      "id": 13117662330995,
      "subject_id": 879868051571,
      "subject_type": "Order",
      "verb": "refund_success",
      "arguments": [
        905898786931,
        "380.00",
        "CZK",
        "api_client_id",
        2632381
      ],
      "message": "test refunded 380,00 Kč on the Bogus ending in •• 1.",
    }
  ]
}

Now apart for message, we have subject_type, subject_id, verb and arguments. In GraphQL we have... a message string 😛

In GraphQL it would be nice to have more implementations of Event interface or add same fields available in REST Event to BasicEvent. Also, would be nice to have an event connection on the query root so we can

query {
  events(first: 250, after: $lastKnownCursor, query: "subjectType:'order'") {
    edges {
      node {
        id
        message
      }
      cursor
    }
    pageInfo {
      hasNextPage
      hasPreviousPage
    }
  }
}

Notice the query filter on subjectType? That would be a massive help!

Best regards!

[Tom13]

I went poking around events.json today and in the documentation I discovered the following. 

The events returned by the Event resource should not be considered to be realtime. Events might not be created until a few seconds after the action occurs. In rare cases it can take up to a few minutes for some events to appear.

It’s really a downer and signal yet another round of caffeinated creative  thinking. Haha.

Despite the above, unlike webhooks, events.json seems like a more certain way to determine if an event has occured. 

But in order to have an event driven approach, I don’t think one can escape usage of webhooks at least in the Shopify arena. 

Thus, I’m thinking of an hybrid approach.

1) To have a custom logic to “moderate” the webhooks received, and create our own sequence(ie. Created > Paid > Refund > Cancel) of the webhooks -since they are not sent in a strict order anyway. This way, we can still retain historical events occured to the sales order in our app.

2) However, diff events for the same sales order may occur at diff time, not necessary in the same batch of the webhooks sent. Thus, modify our original logic to act according to the payload data, and less dependent on the sequence we received it. This way, it’s more accommodative to the “arbitrary” way that webhooks are sent.

3) To address the possible delivery failure of the webhooks, I am thinking to introduce this new component for a “timely reconciliation”. It could be every end of day/week/month etc. depending on the volume. This component will correlate webhooks received to events.json(creates/paid/refund/cancel) to check for any discrepancy and determine if any reconcile work is required. 

Whew.. what a mouthful. Thanks for all your sharing, keep it coming if you see anything to improve. I believe this could help a lot of “real-time / events-driven” Shopify apps to be more efficient.