Fastest way to import a lot of products

Shopify Partner
61 0 13

<redacted>

0 Likes
Shopify Staff
Shopify Staff
591 0 44

If you wanted to import via the API it can be done, though with your number of products it will still take a while, but you will have more insight into where you are and can probably pause the job without any repercussions.

If you were to make Product creation API requests that includes the variant information in them (assuming 3 variants per product) and using a default API limit (refresh of 2 requests per second, or 2 concurrent requests) you could probably finish creating all those products in about 67 minutes.

0 Likes
Shopify Expert
3980 13 309

If 67 minutes is composed of 60 seconds for each minute, you get 4020 API calls at 1 per second. Double that to 2/second and you can make 8040 API calls. Since creating a product is an API call, you make 8040 products in the best possible case in 67 minutes. I think the OP has 24,000 so in fact you might want to suggest triple those minutes or 201 minutes. Reading a CSV and composing the correct JSON or other high level objects in Ruby or Python etc. to add/update the products also consumes clock cycles, so he would have to prepare all 24,000 objects representing the products before entering this loop to have any chance of achieving this kind of rate. 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Shopify Partner
61 0 13

I'm using the following Ruby code to import all products:

https://gist.github.com/narzero/8d2fa41c7f52b392e4ea

It currently takes around 3 seconds to add a single product. At this rate it would take 20 hours to add 24000 products. That's unacceptable. Everything under 2 hours is acceptable.

How could I speed things up?

1 Like
Shopify Staff
Shopify Staff
591 0 44

If you sign up for a Plus account you can get increased API limits, but at the moment that's about it. You could perhaps try splitting your single CSV out as well, don't know how much of a performance improvement you will get from it.

0 Likes
Shopify Expert
3980 13 309

I think the takeaway is simply this.

If you think the native built-in Shopify is slow (i.e.) the product CSV import, you are not going to achieve any kind of speedup using the API. 

The API is not a work-around or replacement for things considered to be slow in Shopify. It is for adding value, not substituting equivalence.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Shopify Partner
61 0 13

It seems to be slow because it takes a long time to download the images from the URL's.

It takes around 3 seconds to create a single product with one image. It takes around 0.65 seconds to create a product without an image.

When downloading the images myself, it's much faster. I figure if I download the images on the local machine and then add them as an attachment it should speed things up. But I can't seem to add them as an attachment to the product. I keep getting a `Encoding::InvalidByteSequenceError: “\xFF” on US-ASCII` error when I try to save the new product.

I've created a post on StachExchange, please take a look if you have some time: http://stackoverflow.com/questions/25979950/encodinginvalidbytesequenceerror-xff-on-us-ascii

0 Likes
Highlighted
Shopify Staff
Shopify Staff
591 0 44

Are you Base64 encoding your images?

0 Likes
Shopify Partner
61 0 13

I wasn't. Thanks!

0 Likes
Shopify Staff
Shopify Staff
83 0 6

In addition to everything already mentioned, consider running your task in parallel chunks. If you cannot hit 2 requests per second because a Product takes longer to create, you don't need to create one product at a time. You could have more than 1 request firing to create a product at a time.

Adam Doeler, API Team, Developer at Shopify
0 Likes