Syncing large matrix of products + variants + locations + inventory levels

[Lynn-srs]

I'm working on creating an intermediate app to sync all relevant information from an ERP to Shopify, where the ERP is leading in terms of most product information, variants, inventory levels, and locations.

While I'm constantly running into issues because of limitations in Shopify, I'm currently kind of stuck with inventoryBulkToggleActivation or inventoryActivate, and the inventory levels themselves. The current tenant I'm working with has ~100k variants, with ~20 locations, this is up to 2M Inventory Items.

Right now I'm running a script to call inventoryBulkToggleActivation to activate all locations for all inventory items that have not been activated yet, because otherwise I can't send inventory levels. This is honestly a weird setup and I'd expect inventory tracking to be automatically enabled when I tell the system there's inventory at this locations, but it is what it is. Unfortunately for a smaller subset (probably 60k variants and up to 6 locations as a test) and this has been taking way too long already. For each variants I do a call with up to 6 locations, and at the time of writing this process has been running for over 40 minutes, how can I speed this up?

The second part that I'm worried about is the actual inventory level sync. When we want to sync all inventory item levels this will take ~5minutes for 120k records (limited locations for testing), which when scaled up will be way too slow. This system uses a locations bulk operation query, and executes this for each locations, this already is fairly slow.

The main problem is that we have to retrieve everything and calculate the new levels. I've implemented a partial update that can run during the day, which is much faster, but will fall back to the above if it detects a larger set of modified levels. Generally speaking the partial updates will be in the range of 0 to 100 modified variants every update, which we want to schedule every 2~5 minutes (shorter is better).

How should I approach these syncs? The calls I found existing right now seem to be too slow overall to use a "pull from system X and pull from system Y and send the new state back to X". Currently we plan on syncing the full inventory levels every night, and every few minutes we send the deltas for variants with inventory level changes in the past ~10 minutes, which will only send the deltas using inventoryAdjustQuantities.

[Dotsquares]

hi @Lynn-srs 

Thanks for the thorough breakdown — syncing large-scale ERP data to Shopify definitely comes with some unique challenges, especially at the scale you're working with (~2M inventory items across multiple locations).

You're absolutely right about the current process being inefficient — Shopify's inventory activation flow (via inventoryBulkToggleActivation or inventoryActivate) isn't ideal when scaling. Unfortunately, inventory tracking doesn't auto-enable, and you must activate tracking per location/variant before updating levels, which adds significant overhead.

Here are a few suggestions to optimize your process:

1. Parallelization & Rate Limits
Make sure you're parallelizing your activation and inventory sync operations as much as Shopify's API rate limits allow. Shopify allows 2 calls per second and a bucket of 40, but if you're using the GraphQL Admin API, it uses a leaky bucket algorithm which can offer more throughput — especially if your queries are well-batched.

Use GraphQL bulk operations where possible, particularly for writes. They let you enqueue a large mutation job and get notified when it's done, without worrying about rate limits.

If you're currently activating items per-variant/location using regular calls, consider batching via GraphQL mutations if available, or using Shopify Plus endpoints if your merchant has access.

2. Use InventoryItem-Level Caching or Indexing
Maintain an internal index (in your app) of which inventory items are already activated at which locations. You can sync this index nightly and then only activate newly created items or location-variant combos as needed. This avoids unnecessary checks and cuts down bulk activation runs.

3. Smarter Full Syncs
For the nightly full syncs:

Consider breaking it into smaller location-specific jobs.

You might also pre-process the changes in your ERP to pre-aggregate deltas — for example, export a file with only changed SKU+location+quantity instead of calculating diffs in Shopify’s data model.

4. Delta Updates Strategy
You’re on the right track using inventoryAdjustQuantities for real-time partial syncs. A few tips here:

Make sure you're not sending the same quantity again — Shopify will silently fail if there's no change.

If you're doing updates every 2–5 minutes, try switching to an event-driven approach, where the ERP pushes changes into a queue (like Kafka or even a lightweight pub/sub) and your sync system consumes and applies deltas continuously.

5. Alternative Design Options
Depending on your ERP and business needs:

Headless inventory: Use an external inventory system and let Shopify query it via a storefront function or an app proxy (though this sacrifices native inventory tracking features).

Shopify Plus: If the tenant is on Plus, you could potentially work with Shopify's support to explore more performant internal tools or beta features for high-scale operations.

[Lynn-srs]

1. Parallelization & Rate Limits
Make sure you're parallelizing your activation and inventory sync operations as much as Shopify's API rate limits allow. Shopify allows 2 calls per second and a bucket of 40, but if you're using the GraphQL Admin API, it uses a leaky bucket algorithm which can offer more throughput — especially if your queries are well-batched.

Use GraphQL bulk operations where possible, particularly for writes. They let you enqueue a large mutation job and get notified when it's done, without worrying about rate limits.

If you're currently activating items per-variant/location using regular calls, consider batching via GraphQL mutations if available, or using Shopify Plus endpoints if your merchant has access.

2. Use InventoryItem-Level Caching or Indexing
Maintain an internal index (in your app) of which inventory items are already activated at which locations. You can sync this index nightly and then only activate newly created items or location-variant combos as needed. This avoids unnecessary checks and cuts down bulk activation runs.

I reach the rate limit in a single thread. I put stuff in bulk operations where possible, but a lot of calls simply don't support this. Currently our flow is to fetch all Shopify products, match them against our products, then match all variants vs our variants, create where missing (this internally seems to create an InventoryItem), and update where it exists. The update variant bulk payload contains the locations which I compare to our known locations and send in the missing inventory items through that bulk call (not the bulk operation). Unfortunately sending in all those single calls hits a rate limit and I can't solve this any other way from what I can tell, it's just going take an unreasonable time.

I have no idea how the Shopify Plus endpoints work, how do I check this? I believe this tenant has Shopify Plus.

3. Smarter Full Syncs
For the nightly full syncs:

Consider breaking it into smaller location-specific jobs.

You might also pre-process the changes in your ERP to pre-aggregate deltas — for example, export a file with only changed SKU+location+quantity instead of calculating diffs in Shopify’s data model.

The choice is either a massive full update, or a delta from the last x minutes, there's no in-between. The full update is required because otherwise we have constantly keep track of which are updated where. So during the day we'll have smaller frequent updates (this works fine), but during the nightly update we still have a process that takes ~ 5 to 15 minutes. I don't think this can be made more efficient because we rely on having to call a bulk operation query to fetch all locations. We can't do this async because we're allowed a single bulk operation per type, which means our stock fetching and product sync at night are already at risk of conflicting.

4. Delta Updates Strategy
You’re on the right track using inventoryAdjustQuantities for real-time partial syncs. A few tips here:

Make sure you're not sending the same quantity again — Shopify will silently fail if there's no change.

If you're doing updates every 2–5 minutes, try switching to an event-driven approach, where the ERP pushes changes into a queue (like Kafka or even a lightweight pub/sub) and your sync system consumes and applies deltas continuously.

We have a queue right now. A cronjob runs every few minutes and checks an SFTP server for files that are being pushed by the ERP. Each mutation file is aggregated into a single changeset and then processed. Up to 250 variants are processed using direct API calls, because that's the limit of a single query. Unfortunately there's no location filter, so it's a lot of overhead... So I'm not worried about this as everything is already running async and we are again limited by either the API rate limit from Shopify, or the Bulk Operation limit. I can do a second query to raise this limit to 500 variants, but it's far from ideal.

5. Alternative Design Options
Depending on your ERP and business needs:

Headless inventory: Use an external inventory system and let Shopify query it via a storefront function or an app proxy (though this sacrifices native inventory tracking features).

Where would I find more information on this?