How can I effectively set up ETL for data analytics in an online store?

[ClaireBLondon]

Hello!

This is my first message here; I am a beginner data analyst (retraining from a biology background!) and I started an internship at a Shopify shop in which I am asked to do a job that is more of a data engineer one:

TASKS
- Set up data analytics warehouse --> data pipeline
- ETL (Extract-Transform-Load) from the Shopify data software tools (API) to xls/csv (Excel)
- Track KPIs (defined thereafter) monthly then weekly (and daily?) --> schedule jobs
- Create dataviz dashboards & reports using Google Data Studio
(+ CRM Customer Relation Management?)

KPIs
- # mailing list subscribers
- # website visits
- # orders
- # returns
- # customers placing 2+ orders
- # orders per brand --> trends
- profitability = order value €
- products on wishlists (desired/populars)

I suppose I need to connect to the Shopify Admin API then do the ETL process, but I never did that and I am self-taught so I feel a bit lost.
I already asked the shop to create a private app and got its credentials, but is it the right option or should I create a custom app instead?
I don't have direct access to the shop, and I have a partner account, not a staff one...
How do I proceed from there?  Where can I find the relevant information?
I found some documentation but this is a bit overwhelming, there is a lot, it is quite technical and I am not sure which option applies to my case:

https://github.com/Shopify/shopify_python_api/blob/master/README.md  
I am stuck at step 4, probably because I've not yet figured out what to put as 'request_params':

NameError Traceback (most recent call last)
<ipython-input-5-fba5ce29d6d1> in <module>
1 session = shopify.Session(shop_url, api_version)
----> 2 access_token = session.request_token(request_params) # request_token will validate hmac and timing attacks
3 # you should save the access token now for future use.

NameError: name 'request_params' is not defined

https://help.shopify.com/en/manual/apps/private-apps  
https://shopify.dev/tutorials/build-a-shopify-app-with-node-and-react/embed-your-app-in-shopify 
https://shopify.dev/tutorials  
https://community.shopify.com/c/Shopify-Apps/Shopify-Custom-app-to-get-store-data/m-p/1142391#M35486 

[Anto_S]

Hi @ClaireBLondon 

You are on the right track - a private app with the right read access rights is the right way to access the data, and a partner account should suffice.

We've worked with multiple clients to setup the ETL flow & transformations from Shopify to a sheet, you can email me at anto@opstack.co, happy to help.

thanks

Anto

[ClaireBLondon]

Thank you Anto_S, I really appreciate!

[gravitydata]

That looks good but there also many vendors who can sync data for you if you didn't get this working.  https://gravitydata.co also allows you sync sub minute and it takes a few min to setup. There is a free account too

[SuneelG]

Hi @ClaireBLondon did you figure out what to put as 'request_params' ? Im also stuck at this point

[ElijahStownton]

For ETL operations, you can use the simple and convenient Skyvia service. It's a no-code cloud solution