If you wanted to import via the API it can be done, though with your number of products it will still take a while, but you will have more insight into where you are and can probably pause the job without any repercussions.
If you were to make Product creation API requests that includes the variant information in them (assuming 3 variants per product) and using a default API limit (refresh of 2 requests per second, or 2 concurrent requests) you could probably finish creating all those products in about 67 minutes.
If 67 minutes is composed of 60 seconds for each minute, you get 4020 API calls at 1 per second. Double that to 2/second and you can make 8040 API calls. Since creating a product is an API call, you make 8040 products in the best possible case in 67 minutes. I think the OP has 24,000 so in fact you might want to suggest triple those minutes or 201 minutes. Reading a CSV and composing the correct JSON or other high level objects in Ruby or Python etc. to add/update the products also consumes clock cycles, so he would have to prepare all 24,000 objects representing the products before entering this loop to have any chance of achieving this kind of rate.
I'm using the following Ruby code to import all products:
It currently takes around 3 seconds to add a single product. At this rate it would take 20 hours to add 24000 products. That's unacceptable. Everything under 2 hours is acceptable.
How could I speed things up?
I think the takeaway is simply this.
If you think the native built-in Shopify is slow (i.e.) the product CSV import, you are not going to achieve any kind of speedup using the API.
The API is not a work-around or replacement for things considered to be slow in Shopify. It is for adding value, not substituting equivalence.
It seems to be slow because it takes a long time to download the images from the URL's.
It takes around 3 seconds to create a single product with one image. It takes around 0.65 seconds to create a product without an image.
When downloading the images myself, it's much faster. I figure if I download the images on the local machine and then add them as an attachment it should speed things up. But I can't seem to add them as an attachment to the product. I keep getting a `Encoding::InvalidByteSequenceError: “\xFF” on US-ASCII` error when I try to save the new product.
I've created a post on StachExchange, please take a look if you have some time: http://stackoverflow.com/questions/25979950/encodinginvalidbytesequenceerror-xff-on-us-ascii
In addition to everything already mentioned, consider running your task in parallel chunks. If you cannot hit 2 requests per second because a Product takes longer to create, you don't need to create one product at a time. You could have more than 1 request firing to create a product at a time.
|3 hours ago|
|5 hours ago|
|5 hours ago|
|10 hours ago|
|10 hours ago|