Paginate API results

Shopify Partner
33 0 5

By the way, thanks @CSaunders for the answer. I did sort that out from @HunkyBill's second response, but I appreciate breaking it down for me as a clear snippet.

0 Likes
Tourist
69 0 2

The following snippet is cleaner and more idiomatic for Ruby, and also comes with these special bonuses relative to the snippet above:

  • It handles errors, which are unfortunately fairly common with the Shopify API.
  • It doesn't copy all of the values into memory, but lets you operate directly on a single batch (which is useful for, say, Order, where you might have hundreds of thousands of them).
  • It's prettier.

This should probably go into the Shopify gem, but for now, enjoy:

module ShopifyAPI
  class Base
    RETRY_AFTER = 60

    def self.find_all(params = {}, &block)
      params[:limit] ||= 50
      params[:page] = 1
      retried = false
      
      begin
        until find(:all, :params => params).each { |value| block.call(value) }.empty?
          params[:page] += 1
          retried = false
        end
      rescue ActiveResource::ServerError => ex
        unless retried
          sleep((ex.response['Retry-After'] || RETRY_AFTER).to_i)
          retried = true
          retry
        else
          raise ex
        end
      end
    end
  end
end

If you include that somewhere in your project (either in initializers or lib) then you can then just do, e.g.

ShopifyAPI::Customer.find_all do |customer|
  # do something with the customer
end

Or:

ShopifyAPI::Order.find_all(:status => :any) do |order|
  # do something with the order
endOr:
(And now, after reading more carefully, I see that you're looking for something in PHP, but leaving the comment for posterity / others looking for a solution to said problem in Ruby.)
Directed Edge recommendations app for Shopify: http://bit.ly/de-app
0 Likes
Shopify Partner
1171 0 55

Nice one Scott. I wasn't aware of 'Retry-After'.

Also for prosperity, the status code returned when too many calls are made is now a 429, so you will need to change it to rescue ActiveResource::ClientError (instead of ServerError).

Take a look at our Apps: www.bookthatapp.com ♥ www.searchifyapp.com ♥ www.productsassistant.com
0 Likes
New Member
11 0 0

@Scott Wheeler - Thanks so much for posting that! Very cool.

I think I further enhanced it by changing your until condition to be like this:

until find(:all, :params => params).each { |value| block.call(value) }.length != params[:limit]

This will prevent the loop from making an extra round trip to the server at the end simply to get a blank return. (Unless of course you get results equally divisible by your limit :-) )

Here is the code, as modified by BBG and me:

https://gist.github.com/4243115

0 Likes
Tourist
69 0 2

Here's an updated version with the considerations above, plus some updates we stuck in at some point in our version:

module ShopifyAPI
  class Base
    RETRY_AFTER = 60

    def self.find_all(params = {}, &block)
      params[:limit] ||= 50
      params[:page] = 1
      retried = false

      begin
        until find(:all, :params => params).each { |value| block.call(value) }.length < params[:limit]
          params[:page] += 1
          retried = false
        end
      rescue ActiveResource::ConnectionError, ActiveResource::ServerError,
        ActiveResource::ClientError => ex
        unless retried
          sleep(((ex.respond_to?(:response) && ex.response['Retry-After']) || RETRY_AFTER).to_i)
          retried = true
          retry
        else
          raise ex
        end
      end
    end
  end
end
Directed Edge recommendations app for Shopify: http://bit.ly/de-app
0 Likes
Shopify Partner
8 0 0

Hunk Bill, the problem seems to be the docs.   Why is limit listed as a parameter but page is not?  https://help.shopify.com/en/api/reference/products/product#index

 

Screen Shot 2019-03-14 at 1.35.47 PM.png

 

 

0 Likes
Highlighted
Shopify Partner
1841 169 469

C'mon @Bryan30 what's up with necro-posting? This thread is from 2012!!!

 

Besides, paging is documented more broadly as it applies to several resource endpoints. Possibly elsewhere too.

 

Either way, why bother resurrect an ancient thread unless you have a question and if you do, why not post a new one so this one can be retired? It just messes up Google search results or prompts more users to chime in on old threads by mistake thinking they're current.

I turn coffee in to code - since 1998
1 Like