How do I pull down xml content programatically from my shopify store?

Shopify Partner
21 0 0
Hi there, I look after a shopify site, and while I'm doing local development, it's very useful to pull down the xml for the pages and blogs we run on the site. I've worked out that I can access the api from my browser by appending .xml to various pages, like so :


http://domain.myshopify.com/admin/pages # request standard page

http://domain.myshopify.com/admin/pages.xml # request xml feed for pages, as per api documentation

This works when I'm using a browser, but I'm unable to do this using curl:

~$ curl -v -u chris@stemcel.co.uk:t0pSekritPassW0rd http://anonycorpstaging.myshopify.com/admin/blogs.xml
* About to connect() to anonycorpstaging.myshopify.com port 80 (#0)
*   Trying 174.143.61.89... connected
* Connected to anonycorpstaging.myshopify.com (174.143.61.89) port 80 (#0)
* Server auth using Basic with user 'chris@stemcel.co.uk'
> GET /admin/blogs.xml HTTP/1.1
> Authorization: Basic Y2hyaXNAc3RlbWNlbC5jby51azpmQjc5cFdaVg==
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: anonycorpstaging.myshopify.com
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Content-Type: application/xml; charset=utf-8
< Connection: close
< Status: 401
< X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.4
< Cache-Control: no-cache
< Content-Length: 33
< Set-Cookie: _shopify_session=BAh7BjoPc2Vzc2lvbl9pZCIlODRiZmUzNzRmNGU0OWJkZDIyNjAyYTg3MDUzYWM3OGM%3D--2bb33a6b5714b8ef450d8e1104c8f0bf4422396d; path=/; HttpOnly
< Server: nginx/0.6.37 + Phusion Passenger 2.2.4 (mod_rails/mod_rack)
< 
* Closing connection #0
Creating a full on shopify app to do a simple database dump seems more involved than it really needs to be for this. What is the recommended way of getting data out of a shopify site for more accurate localhost or staging development here? Is "the Ruby::Mechanize approach":http://d-jones.com/2007/5/7/how-to-export-all-your-products-from-shopify-in-xml really my best option here still?
0 Likes
Shopify Partner
21 0 0

Update—

It turns out that mechanize looks really handy for this, and I was being a total wimp about trying it out.

This altered version of the mechanize script linked above:

1) logs into your store with your credentials,

2) pulls down an xml dump of your pages,

3 ) tries to convert it to yaml,

4) writes this into a yaml file called pages.dump.yaml, for dumping into vision.app

Hope this is useful for someone else – it should be fairly trivial to iterate through the blogs and products to pull down all the content.

Once you have that, you can drop it into the vision yaml database, and (huzzah!) Design a site with your own products instead of those snowboards all the time.


require 'rubygems'
# you need to gem install crack and mechanize first for this
require 'mechanize' # Required to login and download the products feed
require 'crack' # required to parse the xml and spit it out as yaml

# set a few variables first - you need to edit these with your credentials

  URL = "http://domain.myshopify.com" 
  LOGIN = "you@yourdomain.com" 
  PASSWORD = "your_password" 

  # Download the XML feed from Shopify
  def download_xml
    # Request the login page
    agent = WWW::Mechanize.new
    page = agent.get("#{URL}/admin/auth/login")

    # Input the login creditionals into the form
    form = page.forms.first
    form.login = LOGIN
    form.password = PASSWORD
    page = agent.submit(form, form.buttons.first)

    # Request all the pages in the store.
    page = agent.get("#{URL}/admin/pages.xml")
    page.body
  end

# crack parse the xml simply for us
c = Crack::XML.parse(download_xml)

# create an emtpy file object
f = File.new('pages.dump.yaml', 'w+')

# then convert the xml into yaml, and write it into the file
f.write(c.to_yaml)

# clean up after ourselves
f.close


0 Likes
Shopify Staff
Shopify Staff
195 0 3

Chris,

I admire the mechanize/crack script :)

Next time, try using an private api app to do this sort of thing.

Then


curl http://[api-key]:[password]@http://anonycorpstaging.myshopify.com/admin/blogs.xml

Edward Ocampo-Gooding – Shopify Developer Advocate
0 Likes
Shopify Staff (Retired)
Shopify Staff (Retired)
5841 0 121

That’s pretty sweet Chris! How did I miss this? Thanks for sharing!

http://shopifyplus.com ::: http://twitter.com/bacchus
0 Likes
Shopify Expert
3994 14 321

@Chris.

Are you sure about your huzzah! there and dumping your inventory into Vision and having success?

Last time I checked, for trivial things, that approach could work, but there are many ways in which it won’t due to the underlying Ruby code of Vision. For example, some things are hard-coded, and no matter your database of yaml, you won’t get the full benefits of your own inventory.

I often hack product.type and product.vendor in conjunction with blog articles for support.. and this is not trivial to hack into Vision… among other things.

Anyway, I have not played with Vision code since V3 or something, so maybe it has changed enough so you can actually think about designing with your own inventory.

Please let me know if in fact you do have some success with this and what, if any limitations you run into.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Shopify Partner
21 0 0

Hi Edward,

Thanks for the tip about the private api key – that’s exactly what I wanted, and Il be able to refine accordingly now.

@hunky bill – you’re right – for checking content on custom design pages and blogs this gets me part of the way, but I’m not familiar enough with vision’s internals to hack away yet.

Will update once I have some time for this.

0 Likes
Shopify Partner
21 0 0

Hmm…

This kinda works, but Ruby doesn’t seem to like it when I try using @curb, instead of curl.

Here’s what it spits out: (please note, I’ve trashed this key since generating, so it should be basically useless to anyone else)



~$ irb
>> require 'curb'
=> true
>> c = Curl::Easy.perform('http://e5bb34c44cd81b8a21696ea4e8902879:245f458fb22673f96fb7228b2a3d8a6d@wornagainstaging.myshopify.com/admin/pages.xml')
#<Curl::Easy http://e5bb34c44cd81b8a21696ea4e8902879:245f458fb>
>> c.verbose = true
=> true
>> c.perform
* About to connect() to domain.myshopify.com port 80 (#0)
*   Trying 174.143.61.89... * Connected to domain.myshopify.com (174.143.61.89) port 80 (#0)
> GET /admin/pages.xml HTTP/1.1
Host: domain.myshopify.com
Accept: */*

< HTTP/1.1 401 Unauthorized
< Content-Type: application/xml; charset=utf-8
< Connection: close
< Status: 401
< X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.4
< Cache-Control: no-cache
< Content-Length: 33
< Set-Cookie: _shopify_session=BAh7BjoPc2Vzc2lvbl9pZCIlOTQ2ZmJjNjFmYjJjMmYxYTYyMGQ2NjAzMTc2N2EwZWQ%3D--a8a9ce0b4b1dad9df74b767e3833bd70053e621a; path=/; HttpOnly
< Server: nginx/0.6.37 + Phusion Passenger 2.2.4 (mod_rails/mod_rack)
< 
* Closing connection #0
=> true
>> 


Anyone have an idea why this is happening?

0 Likes
Shopify Staff
Shopify Staff
195 0 3

I’ve seen cases where there’s some authentication header that isn’t handled properly.

Chris: have you thought about just using the shopify_app Rails plugin for what you’re doing? It might be a little overkill, but It handles the auth. stuff correctly and lets you write code like:


products = ShopifyAPI::Product.find(:all, :params => {:limit => 3} )

puts products.to_xml

Edward Ocampo-Gooding – Shopify Developer Advocate
0 Likes