I'm creating an app which does some analytics on store orders. For these analytics to be any useful the app should be able to load all orders (currently I'm using only the simple read_orders permission, and last 2 months is enough). The current graphql query that I'm using is able to load 1K orders in about 50-60 seconds on an unloaded cheapest DO server.
I'm mostly concerned about the following -
1. What is the general average total number of orders (last 2 months) per store. How big can this number get ?
2. Once my server starts getting loaded, the time 50-60 seconds would only increase, and if the number of orders increases, then so would the load time
One of my solution is to either restrict the maximum amount of orders my app would load, like around 2 - 3K or let the user decide how many orders they want to load with the caveat that the more orders they load, more time the app would take.
I'm looking for suggestions from people here who've worked or stumbled upon this kind of a situation and how did they fix it, or how would they proceed in this kind of a situation ?