Happening now | Office Hours: Customizing Your Theme With Moeed | Ask your questions now!
Our Partner & Developer boards on the community are moving to a brand new home: the .dev community forums! While you can still access past discussions here, for all your future app and storefront building questions, head over to the new forums.

Re: Poor performance when syncing changed variants out of Shopify

Solved

Poor performance when syncing changed variants out of Shopify

HarryAtGadget
Shopify Partner
6 0 12

With the newly formatted webhook body that only includes the IDs of the extended variants, it requires a bunch more expensive API calls to get the details of those variants if you want to save their data to some external system, like an ERP.

 

For apps doing this kind of syncing, they *have* to fetch and re-fetch all those variants by ID each time a webhook arrives because there's no way to tell what might have changed with them. If you want to update titles or published status or whatever you can't tell from the webhook payload if that may have changed or not, so you must refetch every omitted variant every time. This makes it remarkably easy to consume an app's entire GraphQL rate limit just fetching and re-fetching these variants, leaving no headroom at all for doing other useful work in the app (like syncing changes back the other way).

 

I get why Shopify needs to send condensed payloads -- the full payload of 2k variants is too big to send all at once. But, it seems unsustainable, and like it will generate a bunch of unnecessary load on Shopify, if every app that syncs variants has to re-load all those variants each time a webhook comes in.

 

One way around this would be including an `updated_at` in the condensed payload. Instead of sending just the GID, send both the id and the `updated_at` that Shopify already stores. Syncing apps can then compare each variant's new updated_at to the previously witnessed one, and avoid re-fetching/re-syncing if it hasn't changed. All the variants would still need to be fetched up front during an initial sync, but after that things get snappy.

 

Webhooks are so powerful because they don't consume any rate limit at all, but now, we're forced to use up precious precious rate limit just to re-assemble the webhook payload we actually want before even doing the actual business logic of the app. Please throw us a bone and let us know if anything has changed!

Accepted Solution (1)
Ashish_Shah
Shopify Staff
22 7 11

This is an accepted solution.

Hey @HarryAtGadget 

Thanks for bringing this up for discussion. We are actively discussing solutions we can offer here. Adding an `updated_at` field alongside the variant ids is also under consideration.

ALIAS | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

View solution in original post

Replies 7 (7)

jineshshah36
Shopify Partner
7 0 3

I think a separate webhook for variant changes would work as well, and require no extra fetches.

HunkyBill
Shopify Partner
4853 60 571

So you get a product webhook. You're saying that if you now want all the variants (up to the limit of 2000), you don't want to make API calls? Really? I mean, it is standard looping where even before the 2000 bump we would ask for say 20 in a cursor, then the next 20, etc... so 5 API calls to get all 100 variants. Now if we have 2000, it is true, that would be 100 calls, getting 20 at a time. Your suggestion of only asking for the ones that matter is interesting but how would you do that? If someone changed all 2000 and you got a webhook, you'd have to do 2000 product variant calls based on seeing 2000 updated_at dates newer than what you saved from your last operations? Wow. Whole lotta rosy there!

Neat issue anyway. Based on the FAQ, seems like Shopify has a ton of surfaces in flux here for both them and us. Probably gonna let the dust settle before expecting them to move any deck chairs around on the boat.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
HarryAtGadget
Shopify Partner
6 0 12

@HunkyBill wrote:

So you get a product webhook. You're saying that if you now want all the variants (up to the limit of 2000), you don't want to make API calls? Really? I mean, it is standard looping where even before the 2000 bump we would ask for say 20 in a cursor, then the next 20, etc... so 5 API calls to get all 100 variants.

I don't think that's true -- previously, when you got a webhook with a product and up to 100 variants, all the variant data was included right there with the webhook payload, you didn't need to make any extra calls to get a full picture of the state of the product. For syncing off to external systems all the data is right there already, no API calls necessary.  With the bump up to 2000, this has changed, and now you only have a partial picture.

 

And, I'd be fine making API calls to fetch the data if the rate limits weren't so low! The combination of forced API calls to fetch up-to-date data whenever a webhook comes in with the low rate limit creates this issue. If we need to make API calls to fetch data after every webhook, there will always be a threshold where products are changing fast enough that you need to use your *entire* rate limit just to fetch the changed details about the omitted data, with no rate limit left over for actually doing anything useful. 

 

 


@HunkyBill wrote:

If someone changed all 2000 and you got a webhook, you'd have to do 2000 product variant calls based on seeing 2000 updated_at dates newer than what you saved from your last operations? Wow. Whole lotta rosy there!

That's the worst case scenario indeed, but I suspect that would be rare. Without the `updated_at` information in the webhook payload though, we're forced to assume the worst every time and re-fetch up to 1900 every time.

HunkyBill
Shopify Partner
4853 60 571

I was not speaking of the webhook payload. Clearly, you'll never get 2000 variants in that. I was strictly speaking of the fact you get a complete API call picture (after you get the webhook, this is what you're gonna do). But you understand that 🙂

As for limits, since code handles limits, and limits are a real concern, we code in limit handling code. It is pretty hard to make this particular aspect go away, and this scenario you point out is not unique! Clearly Shopify cannot give carte blanche to this particular situation. So trade offs! We're good at that. 

 

As you said, with dates you could do calls, but that is actually worst case processing. You still have to examine your 1900 variants and ask the dumb question "Are you OK Willis?". 

So I suspect we'll be able to best deal with this by Shopify perhaps alerting us to "This set of variants are interesting, they changed", and we then use the API to get that set of variants, but not with brute force, instead, it is something "defined ahead of time" or I dunno. 

Fact is, playing in any store now, with any products, at any serious volume of use both in sales, customization and management, with 2000 variants and perhaps N options, means we are still without a clear picture of what is going to happen. 

So asking for more definition now is good! I am sure they're well aware and cognizant of the Pandora's box they opened to appease their masses!

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
Ashish_Shah
Shopify Staff
22 7 11

This is an accepted solution.

Hey @HarryAtGadget 

Thanks for bringing this up for discussion. We are actively discussing solutions we can offer here. Adding an `updated_at` field alongside the variant ids is also under consideration.

ALIAS | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

AsafGitai
Shopify Staff
109 16 45

Hi Folks, wanted to close the loop on this.
We have now started sending updated_at fields along side the id, and will be sorting the list desc.
https://shopify.dev/docs/api/admin-rest/unstable/resources/webhook#event-topics-products-create

HunkyBill
Shopify Partner
4853 60 571

Your link is products_create but I think you meant products_update? Not sure there is any need for updated_at flags on a create.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com