I have 6 million products to import. Will it really take 34 days?

Highlighted
Tourist
8 0 12

I'm a book seller. My supplier has provided me with exports of their catalog so that I can import them into my store. This initial import is 6M products. Since the the data is about 10GB, manual uploads of 15MB CSV files won't really work. Therefore, I intend to utilize the REST API to import products (which I'd prefer to use for updates, anyway). However, I quickly realized that it would take 34 days to import given the 2 calls per second limit. I contacted support to see if the rate could be raised for my initial import, but they told me that API requests have to be handled in the forum. So is this something that we can do?

0 Likes
Highlighted
Tourist
8 0 12

Support said that questions about the API are outside of their scope of work, but that devs are active and responsive in the forums. This does not appear to be true.

0 Likes
Highlighted
Trailblazer
139 13 24

I don't know what the deal is. I've asked multiple questions in the past couple of weeks (including pointing out a potential documentation error) and haven't gotten replies. I see other individuals asking straightforward API questions as well and getting left hanging. 

 

I see the potential of the Shopify app store and becoming an app developer here, but my experience with App Bridge and its documentation thus far has been incredibly frustrating.

 

For your use case, have you researched using GraphQL instead of REST API? https://shopify.dev/tutorials/perform-bulk-operations-with-admin-api

0 Likes
Highlighted
Tourist
8 0 12

Thanks for the reply. I was leaning towards REST initially because of my familiarity with it, but I'm going to do some tests with GraphQL bulk queries to see what the point cost is for creating many products at once. Dealing with asynchronous reporting of bulk query status is really going to complicate what would otherwise be a straightforward process.

I'll report back on my findings.

0 Likes
Highlighted
Shopify Expert
4188 29 412

There is no bulk import to massively create products. If you look at the BULK API at this time, it is meant for consuming data.

Sure importing 6M products will take time. What did you expect? Products are quite complex if you study them, so there is no simple way of building them super super quick. So just suck it up, and tell you client if they are convinced they can manage 6 million products, they might have to wait a bit to get all of them going, but in the meantime, they have plenty of other fish to fry, like marketing, organizing how to SEO properly, heck, even figuring out the organization.

You see, one critical aspect of this you left out, which speaks volumes, is the fact that importing products is one thing, but it is silly to go about doing that if you don't even have your basic collections (your catalog hierarchy) figured out. If you don't have sci-fi, fantasy, drama, true crime or other collections first, it makes no sense to import 6 million products. What are you gonna do? Ask your merchant client to sort them and place them into collections? No. You have to do this job right. First you study a sample of say 2000 products (books). Then you write a script that nails the import of those 2000 products. You can tear them down, and build them up with a button press. Once you have that, THEN AND ONLY then do you drag your 6 million into play.

I have built 50,000 SKUs using script without too much trouble, from a single button press and crappy CSV data from ancient Windows or AS-400 crap systems that broke years ago, and from that exprience I can tell you, do NOT go straight to thinking 6 million. Your chances of success hover in single-digits IMO if you go at it that way.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Highlighted
Tourist
8 0 12

The client is me. The business is a profitable three year old B&M, so the fish have been fried. ETL of the data from our supplier to a local database has been performed. A test import of 1000 products via CSV was performed with good results. I do expect importing 6 million SKUs to take time, but 34 days seems a bit much.

My original question about raising the API limit was directed to Shopify's developers since support directed me here. I guess I didn't make that clear enough. But thanks for your input, Bill.

1 Like
Highlighted
Shopify Expert
4188 29 412

You are missing one key thing. Even though you are rate limited to 2 API per second, you are not limited in the same fashion with GraphQL. In GraphQL, you get a complexity score and credits. So if your API call to make a product has a score out of 1000... you get 1000 minus that amount to make another call. And so on. And the credit pile keeps replentishing. So in fact, if you just do the math of 2 per second and go to 6 million, and count 34 days, you're not really going to see that.

Just some advice to pile on since you seem keen and all that. Shopify is not perfect, and long running jobs tend to experience what are called Gateway errors. So PLEASE for the love of all things computer, ensure you can restart your import from where you left off if you get killed that way. As far as any other problems go, you see ready to let it rip. All the best! Have fun with it, it is small pain for long term gain.

And if you did not know, now you do, if you pay the Plus subscription to Shopify, they can boost you to 10 API calls per second, as a nice gesture. That is a 5-fold increase for you that you might consider valuable.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes