Fetching delivery location group zones via graphql

stickfigure
Shopify Partner
37 2 5

I'm trying to synchronize (fetch/create) fairly complicated delivery profile. ~60 zones with ~50 weight conditions each.

I can't fetch this structure in a single graphql query, much less create it. So it seems obvious to break this down and do it one zone at a time.

How can I fetch a single zone (by id) with graphql? There doesn't seem to be anything like this:

 

	deliveryProfile(id: $profileId) {
		profileLocationGroups {
			locationGroupZone (id: $theZoneId) {

 

There's no top-level locationGroupZone query, and deliveryProfile->profileLocationGroups->locationGroupZones only seems to take first/last/before/after. Near as I can tell, the only way to fetch a zone is to query for a list of zones in the group, then use the cursor of the previous zone to fetch one forward. Which seems totally insane. What am I missing?

Thanks in advance.

Replies 11 (11)

HunkyBill
Shopify Expert
4845 60 547

So you know you have 60 zones. And each one is meaty. So you ask for them one at a time. Whatever right? It is just a loop. So you get a Zone, and all the meat and potatoes from said zone. You know you have a cursor for the next one, so you process the next one once your first one is done and in the books.

Are you saying it is impossible for you to get ONE zone and the interesting info contained in that zone, as the expense is > 1000? Why is it insane to simply iterate over zones?

It is probably not something they thought would be needed, to ask for a single zone by ID, but maybe that is something coming. Point is, even then, you still have to persist the zone ID's in your persistence layer to do a sync on a zone, so for now, just sync all zones, and you're golden. I sync 5 or 6 zones and rates and services of each of the providers in one nice but yes, using cursors in there is not without some pondering.

 

 

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
stickfigure
Shopify Partner
37 2 5

Thank you for the response.

I started by iterating and syncing each zone but I quickly got throttled.

The cost to fetch each zone is a couple hundred points. I don't know the cost to update a zone, but it's probably more. With 60 zones, fetching + updating is going to take a long time with all the carefully placed sleeps to let the counter catch up. It will be brittle because it runs for extended periods at the edge of the rate limit; the system might try to do other work on that shop. A failure means restarting (mostly) from scratch.

Also, I have a hard infrastructure limit of 10m before these job queue requests are cut off. It would really be much better to process each zone separately.

I didn't quite understand the point about my persistence layer. I don't need to save the zone ids; I just need to fetch the state of a delivery profile, compare it against what it should be, and make appropriate changes. It doesn't seem to be viable to do this all at once, so I'm looking for strategies to break the work down into smaller chunks (ie, per-zone). Unless there is some better approach?

HunkyBill
Shopify Expert
4845 60 547

When you say "compare it to what it should be", that is data coming for your persistence layer. I am sure you'll experience a break-through at some point. Downloading an entire set of zones cannot take that long. 10m is an eternity in terms of available resources for just that. On the other hand, updating seems like it could drag due to the fact you have that extra task of figuring out what to actually update, and then organizing the actual update calls. I guess you just keep watching the schema evolve and soon enough a version will be released that aids in your task.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
stickfigure
Shopify Partner
37 2 5

You'd think 10m is an eternity, but I've found that I need a 5s sleep to stay under the rate limit (4s gets throttled, so each zone fetch costs somewhere between 200 and 250 points). That's 5 minutes just to fetch the 60 zones. Half my time budget and we haven't gotten to the updates yet. When we do get to the updates, it's not uncommon for a single zone create or delete call to timeout and trash the whole process.

I've optimized the zone fetch to the minimum data, so I don't believe the 200-250 points can be reduced.

Is there a way of sending feature requests to the dev team?

FWIW the raw country/weight/price data comes from an elaborate set of CSV files that are loaded and merged at app startup, not a database. And the comparison logic is practically instantaneous.

Thank you.

HunkyBill
Shopify Expert
4845 60 547

I do not understand your crazy 10m window. Can you shed some light on why you cannot do your entire zone download prior to that window? If for example you started downloading your zones, and it took say 15 minutes to do just that task, and so the second your 10m window opens to AFFECT things at Shopify, you're already loaded up and ready for bear? I know you could still blow 10m updating, but surely you'd also have an occasion where your updates would be minimal and done in a minute.

 

Just curious...

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
stickfigure
Shopify Partner
37 2 5

My application runs on Google App Engine. There are many wonderful things about GAE, but also some limitations. One of those limitations is that all requests are cut off after 10m.

Generally speaking, the GAE-idiomatic solution to this sort of problem is to break it down into chunks and put them on task queues. As a bonus, I can schedule them for future execution in a cascade, spreading them out so they are less likely to get throttled. But Shopify's API design makes it quite challenging to fetch "one zone" in a standalone task!

Now, 5 minutes is still less than 10m so I could have an initial task that iterates through the zones (with 5s sleeps), does the comparisons, and spawns subtasks to perform the activity. But it's pretty hacky, and if I'm going to do hacky, I might as well make a fake "fetch zone" method that gets an overview of the whole list and fetches one out by using the cursor of the previous zone. At least that could be replaced by a more direct method if it appears in the API.

HunkyBill
Shopify Expert
4845 60 547

I am not understanding a 10m request window on GAE. What does that even mean? If you want to use GAE to do some computing that takes 10 hours, I am pretty sure they will take your money, and run your task for 10 hours. As an engineer, I won't be stupid and guess at something I know nothing about, but I am pretty sure, GAE is capable of doing some task for more than 10m before dying. It seems incomprehensible to me, that tasks are limited like that. Why would anyone use such a thing? Google may be good at Search, but it is pretty much the only thing they excel at, so you could be right I guess, and GAE is just some junk that cannot hold a candle to other services that can run 24 hours or more without a hiccup.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
stickfigure
Shopify Partner
37 2 5

@HunkyBill wrote:

If you want to use GAE to do some computing that takes 10 hours, I am pretty sure they will take your money, and run your task for 10 hours.

They will not.

They will point you at different services in the Google Cloud Platform[1] but they have different programming models with different tradeoffs. If you can live within Standard GAE's programming constraints, you can scale up without hiring ops or devops. My last company hit $100M ARR selling tshirts, peak loads in the hundreds of hits per second, with 3 engineers total. And nobody wore a pager.

This is really straying from the topic. I could just as easy complain "Why can't Shopify update 3000 data points in less than 10 minutes? Why would anyone use such a thing?" It doesn't matter, that is the constraint we live with. I need to make it work and I'm not going to build out a whole new technology platform just to solve this issue.

[1]: There actually are different instance types within the "Google App Engine" umbrella that can run longer workloads, but the same caveats apply - it's a different programming model and requires separate deployments.

HunkyBill
Shopify Expert
4845 60 547

Chill dude. You come at me with one of the strangest constraints I have heard in years, namely you get 10 minutes to do something or else you turn into a pumpkin.

Forgive me for being a little surprised at that. And note, flashing bizarre credentials around like $100M ARR does nothing to inform on that subject. I get it, you wear big boy pants and know how to do basic Internet Computing. No questions there.

Thanks for enlightening me though. I do understand now. If you choose to use this "10 minutes or you're a pumpkin" aspect of GAE is for all intents a great way to string together a bunch of routines that get poop done. Bravo! I will keep that in mind as a "good to know" thing in case it ever comes up for me.

I hope you publish how you solve this issue here, it truly is interesting as it rises well above a lot of the mundane ones I see. Good effort!

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
stickfigure
Shopify Partner
37 2 5

Sorry, I was just trying to illustrate the positive side of this particular platform. I don't think it would have been possible to scale to that size on just 3 engineers without GAE (or a similar PaaS like Heroku). If it means living with a 10m request deadline (which yes I do find occasionally annoying), it has proven a worthwhile tradeoff.

At any rate it sounds like there's no straightforward solution and I'm on my own. I'll figure something out. I appreciate your time.

HunkyBill
Shopify Expert
4845 60 547

There are Shopify engineers out there that you can engage, and since they work on exactly your domain here (Delivery Profiles) I am sure they would LOVE to hear about your issue. I have been helped over the years many times by these "angels", in the sense that here in the forums, I get crickets chirping often, as no one can help me. Annoying as hell yes! But c'est la vie.

I was also stumped by Delivery Profiles and was gifted some nice code that explained it much much better than I could've gleaned from the documentation. So as a stop-gap measure sometimes they do reach out.

Often I hit the wall, and realize, there are zero others out there that have a solution for me because there just is not a solution at this time. So I have learned to be somewhat patient. That being said, if years pass, I get frustrated, as I see it as money left on the table.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com