Still trying to get my custom developed product and customer integration apps to play nicely. Long story short is I have three Shopify stores. Each of them has roughly 1,700 products with at least one product variant associated with it. Each store has around 15,000 customers. I am looking to refresh product counts, prices, etc. and update customer records on a regular, automated basis.
So from what I've read on here I am limited to an maximum of two API calls per second. I assume this is per store. Is this also per API key as well? My product integration has unique API keys for each store. Same with my customer integration. Unique API keys apart from the product integration on a per store basis.
Does this mean that my product integration can run independently from the customer integration and neither one will step on each other's toes? As long as each program honors the two API calls per second limit?
Even if I handle the 429 exceptions and retry them this solution isn't scaling very well for refreshing my back-end data to keep it somewhat current. If I am looking at my customer base, checking/updating customer metafields, inserting new customer records, etc. the process takes hours and hours to complete. I will keep refining my code to better optimize and limit my API calls. But the process of doing so is onerous at best.
As long as I know the ground rules I can hopefully keep cleaning up my code to meet the requirements. Any help is appreciated. Thanks!
Thanks for your reply. I did read the document but unless I'm missing something it doesn't say if "your app" having a limit means per API key/password or the store has the limit across all "apps." That's my scalability issue where I have to look at each customer metafield set for tens of thousands of customers. And this apparently doesn't pass along with the basic customer record.
I did find this quote in that doc amusing, since in my case it's my users noticing the automated difference between 12 and 24 hours!
Automated tasks are great because you don't need to wait around while things get done. If an automated task takes 30 minutes to complete rather than 15, you won't feel the difference if you're doing something unrelated, fun, productive etc.
The bane of using Metafields. I recently had to rewrite an entire App when it became apparent that using them was a very poor choice algorithmically. I found the best option was to store the records externally in my App, and keep the data that would've been metafields as a key:value assignment on a column. It sped things up O(N) and I have not regretted dumping the use of metafields for anything but the most trivial stuff.
Thanks for the clarification. To me some of the "gotchas" I've encountered using the Shopify API include the following:
If I am pulling a product then its variants are also passed along as an array. If I am pulling a customer then its addresses are also passed along as an array. But metafields unfortunately do not follow this same convention and logic. And the fact that there is the lack of batch operations when adding/updating only compounds my scalability issues. When I was simply pulling a customer list from one Shopify store, looking at the customer metafield values, and then looking at their wishlist items this process started on 9/20 at 16:04. Completing on 9/22 at 01:16.
I will refactor my wishlist pulls to help some. But otherwise this is pretty frustrating...
Interestingly there are PUT options in the API where you can indeed update metafields in the same API call. See the Product update for an example. Not ideal but I think it exhibits this behaviour. It re-inforces why using Metafield resources is tricky at best and how there remain inconsistencies in the API. Be prepared to support Metafields with long-running background tasks, long-running being the operative term :)
For example, a shop can have 10,000 products. Each product can have 100 variants. Therefore to deal with all the variant metafields you need to touch 10,000 * 100 resources. Wow. How fast is that ever going to be :0
I have looked at refactoring my code, and the challenges still exist. In short, I need to pull the customer record, pull their metafields, and then pull their wishlists. Perhaps I can combine functions into a single method call, but the three separate pulls still would require the API call throttling. So now that across all three Shopify stores I am totalling close to 50K customers, I am looking at upwards of a full 24 hour time period to pull this data. Not to mention any adds/updates that might come into play.
Are there any plans to add any sort of batch methodology into the Shopify API? I can certainly pull customer records in batches. But then again it doesn't pull their metafield data. If I want to add/update metafield data that's another area that lacks this batch methodology. I know that wishlists are a separate Shopify App add-on, so I can't expect that logic to inherit some batch methodology.
My company's website design development resource exclusively works with Shopify. So I went into this project being forced into working with this API. But checking with my peer resources I have found that Magento supports batch functionality to a much larger degree. Not saying I know the technical ins and outs of that assertion being the case, but for larger-scale clients trying to automate working with thousands of products and customers I have to say that Shopify doesn't seem to suit our purposes.
50K records is minor. Small potatoes. In this day and age, you can rent production quality DB's with GB of storage for peanuts. Store all your data in your App, and skip using Shopify as your underlying transactional DB. You will thank yourself later. Using metafields will almost certainly never be performant. If they have not made it so after 5 years, why do you think one day soon that will change? If you want performance, do the data processing in your App's DB and use Shopify as the originating source of data, when speed is not important.
Not sure about the "data in your App" aspect. Here is the nuts and bolts of it. We have brick and mortar stores with a separate POS/Inventory/CRM system. We want any new B&M customers to be ported into Shopify to get an "account activation" e-mail for our web store. Then any wishlist items either created in our web store or in our POS/Inventory/CRM system will synch across the two systems. We link Shopify/internal customer ID's through a Shopify customer ID metafield.
|an hour ago|
|an hour ago|
|3 hours ago|
|3 hours ago|