I am running a multi-threaded ingestion engine to process a store's product details. The ingestion engine queries tons of data about product, the variants, their media and the markets (contextual pricing) data.
I've managed to compress most of what I need in a single query that can end up costing between 600-700 points when the query includes contextual pricing, and <200 points when the pricing is queried separately. The ingestion engine also maintains a track of points left in order to ensure we do not hit the rate limits.
However, off late, I have started noticing that we are unable to fetch the details about a product when there are multiple threads running, which Shopify seems to be silently failing. One such example is:
Query:
fragment fieldsForMediaTypes on Media {
alt
mediaContentType
status
... on Video {
id
sources {
format
height
mimeType
url
width
}
originalSource {
format
height
mimeType
url
width
}
}
... on ExternalVideo {
id
host
embeddedUrl
}
... on MediaImage {
id
image {
altText
originalSrc
}
}
}
query ($productID: ID!, $maxVariants: Int!) {
product(id: $productID) {
description
status
productType
handle
hasOnlyDefaultVariant
hasOutOfStockVariants
isGiftCard
legacyResourceId
tags
title
totalInventory
tracksInventory
updatedAt
vendor
media(first: 100) {
edges {
node {
...fieldsForMediaTypes
}
}
}
metafields(first: 200) {
edges {
node {
namespace
key
description
value
}
}
}
options {
name
values
position
optionValues {
name
}
}
variantsCount {
count
}
variants(first: $maxVariants) {
edges {
node {
id
title
displayName
createdAt
legacyResourceId
availableForSale
inventoryPolicy
inventoryQuantity
image {
altText
url
}
selectedOptions {
name
value
optionValue {
name
hasVariants
swatch {
color
}
}
}
}
}
}
feedback {
details {
messages {
message
}
}
summary
}
}
} | Result: {'product': None}This query, when run in a multithreading environment, returns `None` for a product ID, whereas, when run separately after a cooldown period of 5 seconds, easily returns the expected product and only costs 23 points.
MainThread - DEBUG - GraphQL query 'get_product_data_segregated_pricing' costed: 23 points
In the above case, where the execution is amongst a multi-threaded environment, I can guarantee there are enough points to execute the query, but still I end up getting a None.
Is this some sort of a hidden rate limit? Is there a cap on concurrent requests sent to a store?
In order to replicate, execute the same query in a 5-10 thread environment and watch it return None vs executing it normally to get the result.
I'm seeing this happen not just with this query, but quite a few others. Any help in getting an insight into what may be happening here would be super helpful!
