414 Request-URI Too Large error

Michael_King3
Tourist
24 0 2

Hello all. I'm trying to manually sort a collection of approx. 2000 products via the API. I can do smaller collections without an issue, but this one causes the "414 Request-URI Too Large" error. I realize it's doing it because the querystring is too long, but I'm wondering if anyone has a workaround for what I'm trying to do?

Here is the API call:

ServerXmlHttp.open "PUT", "https://(my-storename).myshopify.com/admin/smart_collections/(CollectionID-value)/order.json?sort_or...;... (repeat "&products[]=(value)" 2000 more times.)

ServerXmlHttp.setRequestHeader "Content-type","application/json"
ServerXmlHttp.send

I would do this in chunks (say 100 items at a time), but then my sort order would get overwritten with each chunk I submit.

Any help in achieving this would be greatly appreciated, even if it's using a different API call.

Thank you all so much in advance!

0 Likes
Jeremy_Mercer
New Member
4 0 0

Hi Adam,

I'm having the exact same issue - did you find a work around for this?

This could be simply fixed if they allowed us to send the array of ordered products as a JSON string in the request body - but it dosn't work.

 

Thanks

0 Likes
Michael_King3
Tourist
24 0 2

I went many different ways (and several days of coding) to sort products in the order I wanted, which is by the release date of an item. I first tried the manual sort but ran into the issue above. I then used the API to update the "published_at" date with the item release date and that works - but NOT if you use pagination. So page 1 results were good, page 2 results were good, etc. but going from page 1 to page 2 gave a random set of records.

I finally settled on adding the release date before the Title of every item and sort by Title descending by default. This actually works, but is quite a hack. I read from other posts people did it this way and I wanted to avoid it as much as possible but it works.

Here is how I write the date before every title

<!--(release date of each item - formatted YYYY-mm-dd)-->(Title)

Example:
<!--20150630-->Eminem - 'The Best Of Eminem'
<!--20010215-->Eminem - 'The Slim Shady LP'

By doing it this way, I can easily split out the date on pages that display Title, and it won't show on Facebook shopping and other apps due to the comment out line.

If you need further help or explanation, just holla!

0 Likes
Jeremy_Mercer
New Member
4 0 0

Hi Adam,

Thanks for your reply, and being so kind to share your solution!

We are trying to sort first by in-stock / not in-stock, then by price in ascending order. The only way I could do this is by pre-pending an incrementing integer to our product titles (which I suppose I could hide using our theme).

I'm currently experementing with another possible solution - if it works, I will post all details here for you.

Likewise, if you have any queries, please feel free to ask away - we have heavily implemented the Shopify API with a catalog of over 30,000 products - and we're more than happy to share.

Thanks.

0 Likes
Michael_King3
Tourist
24 0 2

Jeremy - sounds like we have similar data - 50,000+ items, need to sort by "instock desc, release date desc, title asc". This was easy to do on our old system since I built the entire site/database and just added a column called "instock" and it was checked off to a value of 1 for any item that was in stock. With Shopify... not so easy.

For your database, I would use my method above and code it like this:

<!--(InStock or OutOfStock - use a 0 or 1. This is because you want to sort by price least to most)(Price) --->(Title)

Example of in stock items with a price of $20.00, $25.00, $29.99 and a title of "Sony Camera"):
<--02000-->Sony Camera
<--02500-->Sony Camera
<--02999-->Sony Camera

Example of out of stock items with a price of $20.00, $25.00, $29.99 and a title of "Sony Camera"):
<--11999-->Sony Camera
<--12500-->Sony Camera
<--12999-->Sony Camera

Sorting the 6 items above by Title A>Z will look like this which is what you want:
<--02000-->Sony Camera
<--02500-->Sony Camera
<--02999-->Sony Camera
<--11999-->Sony Camera
<--12500-->Sony Camera
<--12999-->Sony Camera

 

By doing this, if you sort by Title Oldest to Newest, all items beginning with "0" (aka In Stock) will display first in order by price.

Here is the Liquid code to split out the beginning sort/price comment part

{% assign product_title_alt = product.title | split: "-->" %}
{% if product_title_alt.size > 1 %}
	{% assign product_title_alt = product_title_alt[1] %}
{% endif %}

{% comment %}
Display product title below:
The title of this item without HTML comment is
{% endcomment %}
{{product_title_alt}}

 

Let me know your thoughts and your alternative solution you are thinking of.

0 Likes
Jeremy_Mercer
New Member
4 0 0

Hi Adam,

You beat me to it! That is exactly what we have done...

We've appended a numerical value to the front of each title, separated with a pipe symbol “|” to split this in our theme. We are migrating from our own eCommerce platform, so it automatically generates a "display_order" field conveniently for us (which is in the order we require).

As a matter of interest, do you experience any difficulties with the 2 requests per second limit?

Our system has to synchronise with our retail sales platform, and we have had to develop an elaborate queuing system to cope with this restriction.

0 Likes
Michael_King3
Tourist
24 0 2

I would recommend putting whatever you want in front of the Title to be surrounded by the HTML comment quotes like I showed you. If you don't, your title will appear with everything before and including the pipe with any app (e.g. Facebook Buy, 3rd party search apps, etc.).

As for the 2 requests per second - yes, quite annoying. I read in some documentation that there is a container of 400 requests it will hold up to, that is why your first couple of hundred probably have no issues but then... the que runs out of space and you need to pause.

Two ideas on how to fix this:

1) Include a pause script in your loop. Here is my code for when the Shopify API gives that error:
 

If ServerXmlHttp.status <> 200 AND ServerXmlHttp.status <> 201 Then
        ErrorText = ErrorText & "GET : " & ShopifyAPIURL & "/metafields/" & VariantIDVideos & ".json"
End If

If ErrorText <> "" Then
    Response.Write ErrorText & "<br>"
    Response.Write "Error updating ShopifyID: " & ShopifyID & " (UPC: " & UPC & ") on Shopify! (" & ServerXmlHttp.status & ") (" & ServerXmlHttp.responsetext & ")<br>"
    
    ' Too many API calls per second. Take a breather.
    If ServerXmlHttp.status = "429" Then
        ' Pause for 30 seconds to ensure data is populated in Access
        dtCurrent = Now
        Do Until Abs(DateDiff("s", dtCurrent, Now)) >= 30
             dummy = dummy & "a"
        Loop
    End If
    
    Response.Flush
Else
    sql2 = "DELETE FROM UPCTemp WHERE UPC = '" & UPC & "'"
    Set rs2 = Conn.Execute(sql2)
End If

Here's what that code is doing:

- I add every item I want to update into a table used strictly for updating the Shopify API. We'll call it "UPCTemp".

- My script uses this table to loop through my items to update via an INNER JOIN to my Items table.

- Within the loop, I'm calling the Shopify API and checking for errors via the response given by the Shopify API. Anything other than code 200 or 201 from the server is an error. I log those errors into a variable to display later.

- If the error code is 429, then I have hit the API limit. I tell my code to take a 5 second pause.

- If no error, I delete the UPC from my temp table.

- Once the loop is done, it checks for any UPCs in the temp table and lists them out as "Error - the following UPCs did not get updated (list of UPCs here)." I know those need to be rerun or fix the errors in the data causing the Shopify errors.

- An alternative way to do this is instead of using a temp table, just add a column to your Items table called "ShopifyUpdated". Beginning of script, you update that column to have a value of 0 (i.e. False). As the script runs, if you have no errors just replace my DELETE SQL with an UPDATE SQL to update that checkbox to 1 (i.e. True). Then you can run a seperate loop once the loop is done to display all items that still have a value of 0.

If you have a different idea, let me know.

0 Likes
Jeremy_Mercer
New Member
4 0 0

Hi Adam,

Our soliution is quite different. Our internal systems are written in Ruby, and we have constructed a queing system to handle all requests to and from Shopify.

So basically we have overwritten the save, update, and delete methods in our corresponding classes to generate a request which is placed in our queue (SQL table). Our queue stores the path, JSON request, and a priority. The priority is used to determine which records require updating more urgently.

We have then written a daemon which processes this queue, and handles the inherent limit imposed on us by Shopify. This daemon spawns itself as necessary depending on how many items are waiting in the queue to be processed. Here is the snippet of code which we use to handle our upload rate calculations:

# Calculate how long until another upload can be made for a rate limited service
# This version waits no time (i.e. not rate limited)

class UploadRateCalculator

  # Calculate when the next upload time will be
  def next_upload_time()
    return Time.now
  end

  # and then call this when you want to sleep for the defined time
  def sleep_until_time(time)
    now = Time.now

    record_sleeping_time(now)

    duration = time - now
    if duration > 0
      sleep duration

    end

  end

  private

  # empty method to potentially be overridden
  def record_sleeping_time(time)
  end
end
class ShopifyLeakyBucketUploadRateCalculator < UploadRateCalculator

  MAX_UPLOAD_RATE_PER_SECOND = 2
  BUCKET_SIZE = 40

  def initialize()
    # record the time each request is made


    @call_history = Array.new(BUCKET_SIZE)

    earlier = Time.now - 2

    # these initial values will mean that it will start off with
    # an empty Leaky Bucket

    for index in 0... BUCKET_SIZE
      @call_history[index] = earlier
    end

    @upto = 0
  end

  # Calculate when the next upload time will be
  def next_upload_time()

    now = Time.now

    # was the 40th call ago less than one second ago?
    if (now - @call_history[(@upto - 1)/BUCKET_SIZE]) <= 1.0
      # 40 requests in a second, default back to the standard upload rate
      return now + 1.0 / MAX_UPLOAD_RATE_PER_SECOND

    else
      # otherwise full steam ahead!
      return now
    end
  end

  private

  # record the time each call is made so we can monitor the rate of sends
  def record_sleeping_time(time)
    @call_history[@upto] = time
    @upto = (@upto + 1) % BUCKET_SIZE
  end

end

It's quite a complicated solution, but it is necessary when we perform batch updates to our product cataloge which require urgent attention (i.e. price / stock level updates).

0 Likes
Michael_King3
Tourist
24 0 2

I just read your post and was thinking about my calculations more this morning. My "wait 30 seconds" idea is not efficient - yours is much more using the exact 2 calls/second, 40 max in bucket idea. Very good, I'm trying to implement this now myself for faster API processing.

If you ever want to talk code or have questions, I am always available since I love it! You def seem to know your stuff. My contact info is on our site:

http://www.undergroundhiphop.com/contact/

0 Likes
Michael_King3
Tourist
24 0 2

So I came up with a solution that might help you revise your script. You can get the bucket count from Shopify via the header. If the bucket size hits 40, then pause for 40 seconds in order to reset the bucket. I am pausing at 35 instead of 40 so I never hit the bucket max.

Here is the code I am using:

' Max bucket size = 40 API Calls. Pause it just short of 40.
APIBucketSizeRequest = ServerXmlHttp.getResponseHeader("HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT")
If APIBucketSizeRequest <> "" Then
	APIBucketSizeSplit = Split(APIBucketSizeRequest, "/")
	APIBucketSize = APIBucketSizeSplit(0)
End If

If APIBucketSize >= 35 Then
	response.write "<font color=""red"">pausing API for 35 seconds...</font><br>"
	response.flush
	dtCurrent = Now
	Do Until Abs(DateDiff("s", dtCurrent, Now)) >= 35
		dummy = dummy & "a"
	Loop
End If

Have fun!

0 Likes