Recently updated customers results are inconsistent (replication lag or cache issue)

SplioConnectors
New Member
2 0 0

Hello,

We are using the Shopify API to synchronize data in our warehouses in real-time, so data consistency is important. We had issue with using Shopify webhooks (mainly lost requests), so we fallbacked to doing frequent polling to Shopify endpoints using since_id or updated_at_min or created_at_min query parameters to ensure we incrementally fetch all newly created or updated resources.

We found out an issue with the customers endpoint when used in combination with the updated_at_min parameter (ie, we try to fetch every 5 minutes all the customers updated in the last 5 minutes), which looks related to some caching or replication lag issue on the Shopify server side

The steps to reproduce:

  1. Hit the /admin/customers.json?updated_at_min=2021-05-06T17:21:15Z endpoint with a timestamp which is a few seconds before you current local time. It should return an empty result.
  2. Update a test customer (on my video I added a metafield on one)
  3. Hit again the /admin/customers.json?updated_at_min=2021-05-06T17:21:15Z endpoint several times in a row. You will notice that you will still get empty result for the next 30 to 40 seconds, and you might even see you test customer appears, then disappears and appears again (which leads me to think I am being load balanced on different shopify servers with different cache data, or shopify is hitting different database slave servers with replication lag)

Here is a video replicating the problem: 4pDQoTl.mp4

For your information, other findings:

  • it also applies to created customers (ie, creating a user and then hitting the customers.json?created_at_min=... endpoint)
  • I replicated the issue on 3 different shopify stores
  • I cannot reproduce the issue on the products endpoint (ie, product.json?updated_at_min immediately returned the updated product)

 

Let me know if you need more information

0 Likes
SplioConnectors
New Member
2 0 0

Up! any update on this issue? If confirmed this is a real problem, it makes any system trying to sync customers data out of Shopify being out of sync at one point.

0 Likes