Re: Full process for uploading files to the Files API

Full process for uploading files to the Files API

Celso_White
Shopify Partner
23 0 29

I spent hours trying to figure out the full flow of uploading files to the files api. Download file contents > Create a staged target > Upload the file to the staged target > Upload file to Shopify. This solution is node js specific but may be helpful for other languages. Although Shopify has the base documentation in place, its missing a comprehensive breakdown of the steps to take and parameters to set. Hope this helps someone looking to upload files to the api.

 

See here for the gist.

 

 

 

/*------------------------
Libraries
------------------------*/
const axios = require("axios");
const fs = require("fs");
const FormData = require("form-data");

/*------------------------
Download the file.
Good article on how to download a file and send with form data - https://maximorlov.com/send-a-file-with-axios-in-nodejs/
------------------------*/
const file = await fs.readFile("./your-image.jpg"); // This can be named whatever you'd like. You'll end up specifying the name when you upload the file to a staged target.
const fileSize = fs.statSync("./your-image.jpg").size; // Important to get the file size for future steps.

/*------------------------
Create staged upload.
---
Shopify sets up temporary file targets in aws s3 buckets so we can host file data (images, videos, etc).
If you already have a public url for your image file then you can skip this step and pass the url directly to the create file endpoint.
But in many cases you'll want to first stage the upload on s3. Cases include generating a specific name for your image, uploading the image from a private server, etc.
------------------------*/
// Query
const stagedUploadsQuery = `mutation stagedUploadsCreate($input: [StagedUploadInput!]!) {
  stagedUploadsCreate(input: $input) {
    stagedTargets {
      resourceUrl
      url
      parameters {
        name
        value
      }
    }
    userErrors {
      field
      message
    }
  }
}`;

// Variables
const stagedUploadsVariables = {
  input: {
    filename: "example.jpg",
    httpMethod: "POST",
    mimeType: "image/jpeg",
    resource: "FILE", // Important to set this as FILE and not IMAGE. Or else when you try and create the file via Shopify's api there will be an error.
  },
};

// Result
const stagedUploadsQueryResult = await axios.post(
  `${your_shopify_admin_url}/graphql.json`,
  {
    query: stagedUploadsQuery,
    variables: stagedUploadsVariables,
  },
  {
    headers: {
      "X-Shopify-Access-Token": `${your_shopify_admin_token}`,
    },
  }
);

// Save the target info.
const target =
  stagedUploadsQueryResult.data.data.stagedUploadsCreate.stagedTargets[0];
const params = target.parameters; // Parameters contain all the sensitive info we'll need to interact with the aws bucket.
const url = target.url; // This is the url you'll use to post data to aws. It's a generic s3 url that when combined with the params sends your data to the right place.
const resourceUrl = target.resourceUrl; // This is the specific url that will contain your image data after you've uploaded the file to the aws staged target.

/*------------------------
Post to temp target.
---
A temp target is a url hosted on Shopify's AWS servers.
------------------------*/
// Generate a form, add the necessary params and append the file.
// Must use the FormData library to create form data via the server.
const form = new FormData();

// Add each of the params we received from Shopify to the form. this will ensure our ajax request has the proper permissions and s3 location data.
params.forEach(({ name, value }) => {
  form.append(name, value);
});

// Add the file to the form.
form.append("file", file);

// Post the file data to shopify's aws s3 bucket. After posting, we'll be able to use the resource url to create the file in Shopify.
await axios.post(url, form, {
  headers: {
    ...form.getHeaders(), // Pass the headers generated by FormData library. It'll contain content-type: multipart/form-data. It's necessary to specify this when posting to aws.
    "Content-Length": fileSize + 5000, // AWS requires content length to be included in the headers. This may not be automatically passed so you'll need to specify. And ... add 5000 to ensure the upload works. Or else there will be an error saying the data isn't formatted properly.
  },
});

/*------------------------
Create the file.
Now that the file is prepared and accessible on the staged target, use the resource url from aws to create the file.
------------------------*/
// Query
const createFileQuery = `mutation fileCreate($files: [FileCreateInput!]!) {
  fileCreate(files: $files) {
    files {
      alt
    }
    userErrors {
      field
      message
    }
  }
}`;

// Variables
const createFileVariables = {
  files: {
    alt: "alt-tag",
    contentType: "IMAGE",
    originalSource: resourceUrl, // Pass the resource url we generated above as the original source. Shopify will do the work of parsing that url and adding it to files.
  },
};

// Finally post the file to shopify. It should appear in Settings > Files.
const createFileQueryResult = await axios.post(
  `${your_shopify_admin_url}/graphql.json`,
  {
    query: createFileQuery,
    variables: createFileVariables,
  },
  {
    headers: {
      "X-Shopify-Access-Token": `${your_shopify_admin_token}`,
    },
  }
);

 

 

 

Replies 29 (29)

PaulNewton
Shopify Partner
7201 637 1496

Good stuff @Celso_White  

If I get time I'll slap it in a glitch.me app and pop through it.

 

Though Shopify has the base documentation in place, its missing a comprehensive breakdown of the steps to take and parameters to set.

Each dev doc page should have a feedback box either in sidebar or at bottom of the page;  for the relevant docs your talking about I'd suggest shamelessly linking to this post with a complaint about the lack of specificity that could be enhanced based on your codes comments about the process.

 

Or if you go through the partner dashboard support file an issues/feature-request/bug-report with partner support.

Contact paull.newton+shopifyforum@gmail.com for the solutions you need


Save time & money ,Ask Questions The Smart Way


Problem Solved? ✔Accept and Like solutions to help future merchants

Answers powered by coffee Thank Paul with a Coffee for more answers or donate to eff.org


BrianDHogg
Visitor
2 0 0

Hey Celso, 

 

Thanks for this! I'm having what I guess is a slight issue, which is that I'm getting an error when trying to post the file contents to S3:

 

Invalid Argument: 

Only AWS4-HMAC-SHA256 is supported

 

I'm trying to sort out if this is a configuration or implementation issue on my end, so if you're able to shine any light on this, it would be greatly appreciated. 

 

Cheers!

BrianDHogg
Visitor
2 0 0

Actually, never mind! I was having an issue with file-saving and had set the parameters from the pre-signing to be JSON.stringified and forgot to flip it back.

 

Cheers!

Celso_White
Shopify Partner
23 0 29

Hi @BrianDHogg glad you were able to figure it out. Yes, all parameters need to be posted to AWS exactly as you receive them from stagedUploadsCreate. Shopify will provide the exact authentication parameters you need in the response.

For anyone else getting this error make sure to add the AWS parameters to the formData as indicated here.

MerchFaceMark
Visitor
1 0 0

Switching to Axios (from just fetch) was a big help to me (in addition to all the rest).  Thanks!

den232
Shopify Partner
213 8 55

This is brilliant!  Thanks so much.  jb

peepers-rick
Shopify Partner
10 0 5

Wow, I was so close to getting this but kept running into random little issues that were driving me crazy. You got me across the finish line. Amazing post, thanks a ton!

Viglucci
Shopify Partner
3 0 1

I had a similar issue in my app, which turned out to be the missing headers.

 

const headers = {
...formData.getHeaders()
};

 Thanks for sharing your gist, helped me solve my issue. 

Celso_White
Shopify Partner
23 0 29

Hey just a heads up it appears that Shopify is now using google for the staged targets. Before creating your headers, check to see which staged target url Shopify returns. If they return a google url then don't include the Content-Length header. If they return an aws url then include the Content-Length header. Check out the gist for updated code.

den232
Shopify Partner
213 8 55
Thanks so much! What would be an example of the staged target URLs in
both cases? Cheers jb
Celso_White
Shopify Partner
23 0 29

No problem! See below for the targets I'm getting in my app:

Celso_White
Shopify Partner
23 0 29

Hi all, just a note that the above code is no longer working if your file is above 2MB. That's specifically caused by the Staged Uploads part of the code. At this stage I'm going to create my own staged file upload system with Google Cloud Storage so I have more control over that process.

 

If you don't plan on uploading files over 2MB or the issue is randomly fixed then you should be good to use the code above.

peepers-rick
Shopify Partner
10 0 5

Hi @Celso_White ,

I've found your work here very helpful and wanted to let you know that I'm consistently having success with files > 2 mb if I change this line: 

form.append("file", file);

to this

form.append("file", file, "somefilename.someextension")

 

After scouring across the internet for a few frustrating hours, I stumbled upon this to help me find this solution: https://issuetracker.google.com/issues/225060183

Xen-dev
Shopify Partner
16 0 1

Hi,

thanks for an elaborate response.

 

But currently I am facing an issue while getting the request response headers from axios as it says : Property 'getheaders' does not exist on type 'FormData'. in the following code:  

Any help regarding this would be grateful.

    const formData = new FormData();

    for (var param of target.parameters) {
      formData.append(param.name, param.value);
    }

    const blob = new Blob([fileData]);

    formData.append("file", blob);
    await axios.post(target.url, formData, {
      headers: {
        ...formData.getheaders()
        "Content-Type": "multipart/form-data",
      },

 

ShopifyDevSup
Shopify Staff
1453 238 511

Hey @Xen-dev , I've followed up here for you https://community.shopify.com/c/graphql-basics-and/issues-when-obtaining-and-maintaining-urls-for-pr... 

Developer Support @ Shopify
- Was this reply helpful? Click Like to let us know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

Xen-dev
Shopify Partner
16 0 1

Hi , thanks this detailed and elaborate reply helps a lot !!

Is it possible to use the productCreateMedia mutation instead of fileCreate?

 

den232
Shopify Partner
213 8 55

I use my own Amazon s3 location for staging file uploads, so I do not use stagedUploadsCreate .  I only use productCreateMedia ...  Works just fine.  A lot simpler too.  Good luck, jb

Xen-dev
Shopify Partner
16 0 1

I used the productCreateMedia in the manner given below, but receiving this error

"message":"Invalid Model 3d

would be great if you could suggest a solution.

 

    const fileCreateResponse = await admin!.graphql(
      `#graphql
      mutation productCreateMedia($media: [CreateMediaInput!]!, $productId: ID!) {
        productCreateMedia(media: $media, productId: $productId) {
          media {
            alt
            mediaContentType
            status
          }
          mediaUserErrors {
            field
            message
          }
          product {
            id
            title
          }
        }
    
      }`,
      {
        variables: {
          media:[{
            alt: "3d models",
            mediaContentType: "MODEL_3D",
            originalSource: "https://d18bonzukonoby.cloudfront.net/testshopify/xyz.usdz",
          }] ,
          productId: "gid://shopify/Product/723xxxxxxxx",
        },
      },
    );

    console.log(fileCreateResponse, "fileCreateResponse");

    const productData = await fileCreateResponse.json();

    return json({
      data: productData.data,
    });
  } catch (error: any) {
    console.error("Error:", error);
    return json({ error: error.message }, { status: 500 });
  }
};

 

ShopifyDevSup
Shopify Staff
1453 238 511

Hi @Xen-dev 

Please see the responses to this error we've provided in the following threads, and if you have any further issues with this, please do reach out to our Support Team directly via our Shopify Help Center, and we can absolutely help take a closer look into this error you're experiencing with specific examples provided.
 

I hope this helps, and I hope you have a great day 🙂

 

Developer Support @ Shopify
- Was this reply helpful? Click Like to let us know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

ShopifyDevSup
Shopify Staff
1453 238 511

Hi @Xen-dev 

While we are unable to comment on the code shared above, I can confirm that the productCreateMedia mutation can be used with Staged Upload URLs created by the stagedUploadsCreate mutation after the files have been uploaded to the url. The productCreateMedia argument originalSource, can accept both staged URLs and external URLs as well.

More information on the using Staged Upload URLs with the productCreateMedia mutation can be found in our Shopify.Dev documentation. Here are some guides and resources from our documentation that maybe of further help.

 

 

I hope this helps, and I hope you have a great day 🙂

Developer Support @ Shopify
- Was this reply helpful? Click Like to let us know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

xao0isb
Shopify Partner
7 0 1

Thanks man! This is very helpful since shopify does not always provide us with documentation.

 

I was guided by your post but replaced node.js with ruby on rails and also wrote whole article with flow/logic , implementation and a little bit went into details: https://dev.to/xao0isb/shopify-send-file-to-shop-with-ruby-on-rails-44i7

 

So anyone who still struggles with understanding on how to upload file to shopify shop can also be guided by the article.

den232
Shopify Partner
213 8 55

Good work!  It does utterly amaze me how difficult and error-prone Shopify has made this simple matter!  Cheers jb

Ido1425
Shopify Partner
26 0 3

I am facing an issue uploading image buffers to Shopify files through my app.

the process is working but in the Shopify files dashboard, there is a processing error on all images.

The error:

Ido1425_0-1720599354834.png

 

This is the code for the process(this is gadget.dev platform app):

 

async function uploadImage(productId, imageId, compressedBuffer, imageFormatTemp, connections, logger) {
  try {
    // Step 1: Create the image asset in Shopify using stagedUploadsCreate mutation
    const uploadQuery = `
      mutation stagedUploadsCreate($input: [StagedUploadInput!]!) {
        stagedUploadsCreate(input: $input) {
          stagedTargets {
            url
            resourceUrl
            parameters {
              name
              value
            }
          }
        }
      }
    `;

    const uploadVariables = {
      input: [
        {
          filename: `${imageId}-compressed.${imageFormatTemp}`,
          mimeType: `image/${imageFormatTemp}`,
          httpMethod: "POST",
          resource: "FILE",
          fileSize: `${compressedBuffer.length}`
        },
      ],
    };

    const uploadResponse = await graphqlWithRetry(uploadQuery, uploadVariables, connections, logger);
    const stagedTarget = uploadResponse.stagedUploadsCreate.stagedTargets[0];
    const { url, parameters } = stagedTarget;

    // Step 2: Upload the compressed image to the pre-signed URL
    const formData = new FormData();
    parameters.forEach(param => formData.append(param.name, param.value));
    formData.append('file', compressedBuffer, `${imageId}-test.${imageFormatTemp}`);
    await fetch(url, {
      method: 'POST',
      body: formData,
      headers: {
        ...formData.getHeaders(), // Pass the headers generated by FormData library
      },
    });

    // Step 3: Create the image asset in Shopify using fileCreate mutation
    const createFileQuery = `
      mutation fileCreate($files: [FileCreateInput!]!) {
        fileCreate(files: $files) {
          files {
            id
            alt
            createdAt
          }
        }
      }
    `;

    const createFileVariables = {
      files: [
        {
          alt: "test upload Image",
          contentType: "IMAGE",
          originalSource: stagedTarget.resourceUrl,
        },
      ],
    };

    await graphqlWithRetry(createFileQuery, createFileVariables, connections, logger);
    logger.info('New compressed image created');
  } catch (error) {
    logger.error(error, `Failed to update image ${imageId} for product ${productId} in Shopify.`);
  }
}

all the process seems to work well on my side, I get the 'New compressed image created' log, but in Shopify, I keep getting the error with extra info with the error.

 

I’ve tried to implement your solution, but I still keep getting an error

xao0isb
Shopify Partner
7 0 1

Hi! I think problem with the resource argument in stagedUploadsCreate mutation and the contentType argument in fileCreate mutation. Since you are uploading an image try to set resource argument in stagedUploadsCreate to IMAGE, not FILE. Let me know if it works or not.

Ido1425
Shopify Partner
26 0 3

Hi,

I will try that.

I have changed it to FILE in the stagedUploadsCreate mutation because of the explanation above.

do you mean you think the issue is that the mutation resource and the content type do not match?

 

Ido1425
Shopify Partner
26 0 3

Its still doesn't work, do you have any other idea?

Ido1425
Shopify Partner
26 0 3
xao0isb
Shopify Partner
7 0 1

I don't have any other ideas. Are you still struggling?

michaelscottWP
Visitor
1 0 0

This was very helpful. 

I decided to write a PowerShell script that I could run that would recursively iterate through subdirectories and upload the files.  That way, I could keep things organized on my end.  I also used the CDN to check for existence to reduce API calls (not sure if that matters or was a good idea, and there is a delay of a few seconds, but I was worried about rate limiting).

This does a two-stage upload through Google, as some have talked about in other comments. 

 

.\Shopify-UploadFiles.ps1 "{TARGET_PARENT_DIRECTORY}"

 


Hope it helps.

edit: I updated the script to be simpler (non-parallel) because some errors were problematic.  Also, I discovered Shopify changes spaces to underscores so it was not finding existing files that had spaces in their names.  That was fixed.

 

param (
    [string]$folderPath
)

# Load necessary assemblies
Add-Type -AssemblyName "System.Net.Http"

# Define your Shopify store details and API credentials
$shopifyStore = "XXXXX.myshopify.com"
$apiAccessToken = "XXXXXX"
$cdnUrlPrefix = "https://cdn.shopify.com/s/files/XXXXXXX"

# Resolve the folder path to an absolute path
$folderPath = [System.IO.Path]::GetFullPath($folderPath)

# Ensure the folder exists
if (-Not (Test-Path $folderPath)) {
    Write-Output "Folder not found: $folderPath"
    exit
}

Write-Output "Processing folder: $folderPath"

# Function to check if a file exists on the CDN
function Test-FileExistsOnCdn {
    param (
        [string]$url
    )

    Add-Type -AssemblyName "System.Net.Http"
    $httpClient = [System.Net.Http.HttpClient]::new()
    try {
        $response = $httpClient.SendAsync([System.Net.Http.HttpRequestMessage]::new([System.Net.Http.HttpMethod]::Head, $url)).Result
        return $response.IsSuccessStatusCode
    } catch {
        return $false
    }
}

# Function to upload a file to Shopify
function Upload-FileToShopify {
    param (
        [string]$filePath,
        [string]$fileName,
        [byte[]]$fileContent,
        [int]$fileSize
    )

    try {
        # Stage the file upload
        $graphqlMutationStage = @'
mutation stagedUploadsCreate($input: [StagedUploadInput!]!) {
  stagedUploadsCreate(input: $input) {
    stagedTargets {
      resourceUrl
      url
      parameters {
        name
        value
      }
    }
    userErrors {
      field
      message
    }
  }
}
'@

        $stagedUploadsVariables = @{
            input = @(
                @{
                    filename = $fileName
                    httpMethod = "POST"
                    mimeType = "image/jpeg"
                    resource = "FILE"
                }
            )
        }

        $apiUrl = "https://$shopifyStore/admin/api/2024-07/graphql.json"

        $response = Invoke-RestMethod -Uri $apiUrl -Method Post -ContentType "application/json" -Body (@{ query = $graphqlMutationStage; variables = $stagedUploadsVariables } | ConvertTo-Json -Depth 10) -Headers @{ "X-Shopify-Access-Token" = $apiAccessToken }

        if ($response.data.stagedUploadsCreate.userErrors) {
            Write-Output "Failed to stage upload for $fileName. Errors: $($response.data.stagedUploadsCreate.userErrors | ForEach-Object { $_.message } -join ', ')"
            return $false
        }

        $stagedTarget = $response.data.stagedUploadsCreate.stagedTargets[0]
        $uploadUrl = $stagedTarget.url
        $parameters = $stagedTarget.parameters
        $resourceUrl = $stagedTarget.resourceUrl

        # Prepare multipart form data using HttpClient
        $boundary = [System.Guid]::NewGuid().ToString()
        $content = [System.Net.Http.MultipartFormDataContent]::new($boundary)

        foreach ($param in $parameters) {
            $stringContent = [System.Net.Http.StringContent]::new($param.value)
            $stringContent.Headers.ContentDisposition = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
            $stringContent.Headers.ContentDisposition.Name = "`"$($param.name)`""
            $content.Add($stringContent)
        }

        $fileContentStream = [System.IO.MemoryStream]::new()
        $fileContentStream.Write($fileContent, 0, $fileContent.Length)
        $fileContentStream.Position = 0
        $fileContentContent = [System.Net.Http.StreamContent]::new($fileContentStream)
        $fileContentContent.Headers.ContentDisposition = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
        $fileContentContent.Headers.ContentDisposition.Name = "`"file`""
        $fileContentContent.Headers.ContentDisposition.FileName = "`"$fileName`""
        $fileContentContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse("image/jpeg")
        $content.Add($fileContentContent)

        $httpClient = [System.Net.Http.HttpClient]::new()

        # Upload the file to the staged target
        try {
            $responseUpload = $httpClient.PostAsync($uploadUrl, $content).Result
            if ($responseUpload.IsSuccessStatusCode) {
                Write-Output "Uploaded $fileName successfully.`n"
            } else {
                Write-Output "Failed to upload to staged target for $fileName. StatusCode: $($responseUpload.StatusCode)"
                Write-Output "Response: $($responseUpload.Content.ReadAsStringAsync().Result)"
                return $false
            }
        } catch {
            Write-Output "Failed to upload to staged target for $fileName. Error: $_"
            return $false
        }

        # Prepare the GraphQL mutation for creating the file in Shopify
        $graphqlMutationCreate = @'
mutation fileCreate($files: [FileCreateInput!]!) {
  fileCreate(files: $files) {
    files {
      alt
    }
    userErrors {
      field
      message
    }
  }
}
'@

        $createFileVariables = @{
            files = @(
                @{
                    alt = "alt-tag"
                    contentType = "IMAGE"
                    originalSource = $resourceUrl
                }
            )
        }

        # Create the file in Shopify
        $responseCreate = Invoke-RestMethod -Uri $apiUrl -Method Post -ContentType "application/json" -Body (@{ query = $graphqlMutationCreate; variables = $createFileVariables } | ConvertTo-Json -Depth 10) -Headers @{ "X-Shopify-Access-Token" = $apiAccessToken }
        if ($responseCreate.data.fileCreate.userErrors) {
            Write-Output "Failed to create file for $fileName. Errors: $($responseCreate.data.fileCreate.userErrors | ForEach-Object { $_.message } -join ', ')"
        } else {
            Write-Output "Uploaded $fileName successfully.`n"
        }
    } catch {
        Write-Output "Failed to upload $fileName. Error: $_"
    }
}

# Get all files from the specified folder and child folders
$files = Get-ChildItem -Path $folderPath -Recurse -File

Write-Output "Found $($files.Count) files."

foreach ($file in $files) {
    $fileName = $file.Name
    $fileFullPath = $file.FullName
    
    # Shopify replaces spaces with underscores in file names, so we need to do the same in our check
    $escapedFileName = $fileName -replace ' ', '_'
    $fileUrl = "$cdnUrlPrefix$escapedFileName"

    # Check if the file exists on the CDN
    if (Test-FileExistsOnCdn -url $fileUrl) {
        Write-Output "File $fileName already exists. Skipping upload."
    } else {
        Write-Output "File $fileName does not exist. Preparing to upload."

        # Read the file content and get the file size
        $fileContent = [System.IO.File]::ReadAllBytes($fileFullPath)
        $fileSize = (Get-Item $fileFullPath).length

        # Upload the file to Shopify
        Upload-FileToShopify -filePath $fileFullPath -fileName $fileName -fileContent $fileContent -fileSize $fileSize
    }
}