Product API body_html fails with 400 on characters above u0080

Solved
Highlighted
Shopify Partner
11 1 0

This took me a day to figure out. I was trying to copy products from one store to another using the APIs and I kept getting 400 errors when POSTing certain products. It turns out that the POST /admin/api/2020-04/products.json API does not accept characters u0080 or greater. This includes non-english accented characters such as ü, symbols such as •, non-US currencies such as £, and non-breaking spaces.

I got around this by converting those characters to numerical HTML equivalents in the form of "&#xxx;".

For example, ü (U+00FC) becomes ü

Here is Javascript code to do that:

new_body_html = old_body_html.replace(/[\u0080-\uFFFF]/gimfunction (i) {
return'&#' + i.charCodeAt(0) + ';';
    });

It would be great if the API was updated to allow characters above \u0080, or at least indicate this limitation in the documentation.

0 Likes
Highlighted
Shopify Staff
Shopify Staff
586 71 128

Hey @David_Sturman,

 

Can you provide more details about the client you're using to make these calls? I just tested with Insomnia, but I was unable to replicate the 400 error when sending the characters you mentioned in the html_body:

 

20-04-o4258-vz4jv

 

If you can provide the X-Request-ID value from the headers of a failed call, I can also use that to check our logs and see how that call is coming through on our end.

JB | Developer Support @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Shopify Partner
11 1 0

Hi JB,

Thanks for the reply.

I'm using the following code in NodeJS (v12.9.1)

 

    console.log(urloptions);
    const req = https.request(urloptionsfunction (res) {
        console.log('statusCode:'res.statusCode);
        console.log('headers:'res.headers);
    });
    console.log(postdata);
    req.write(postdata);
    req.end();
}
The POST fails when there are u0080+ characters in the body.html, such as the ® after the second occurrence of "Cubebot". There are also non-breaking spaces in the string.
Here is the program output in that case

URL:
https://key:password@mystore.myshopify.com/admin/api/2020-04/products.json

 

OPTIONS:
{
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Content-Length': 1732 }
}

POSTDATA:
{"product":{"product_type":"Lifestyle","body_html":"<meta charset=\"utf-8\">\n<p><span>Have you met Cubebot?</span></p>\n<p>Introducing the Cubebot® Capsule Collection.  Five new multi-colored Cubebots in two sizes. Cubebot, a foldable wooden robot puzzle with hundreds of poses but only one solution.</p>\n<p>Recommended for Ages 3+</p>\n<meta charset=\"utf-8\">\n<p><span>Have you met Cubebot?</span></p>\n<p>By <a href=\"https://queen-of-hearts-and-modern-love.myshopify.com/pages/areaware\" target=\"_blank\" title=\"robot, childrens games, games, cubebot\" rel=\"noopener noreferrer\">Areaware </a></p>\n<meta charset=\"utf-8\">\n<p class=\"small\">Beech and Elastic<br>Micro <br>4.25\" tall<br>5.5\" arm span<br>1.5x1.5\" cube</p>\n<p class=\"small\">Small<span> </span><br>6.75\" tall<br>9.25\" arm span<br>2.5x2.5\" cube</p>\n<p> </p>","title":"Areaware Cubebot Capsule Collection Micro by David Weeks - Red Multi","tags":"games, lifestyle","vendor":"Areaware","image":{"position":1,"alt":null,"src":"https://cdn.shopify.com/s/files/1/1248/8951/products/Red_Multi.jpg?v=1570733904"},"images":[{"positi... Title"]}],"variants":[{"title":"Default Title","price":"10.00","option1":"Default Title","option2":null,"option3":null,"sku":"708389997156","inventory_quantity":4,"inventory_management":"shopify"}]}}

STATUS CODE:
statusCode: 400

RESPONSE HEADERS:
headers: {
date: 'Tue, 21 Apr 2020 13:26:55 GMT',
'content-type': 'application/json',
'transfer-encoding': 'chunked',
connection: 'close',
'set-cookie': [
'__cfduid=dd7895ac082f41e7c7d2b14025316481f1587475615; expires=Thu, 21-May-20 13:26:55 GMT; path=/; domain=.myshopify.com; HttpOnly; SameSite=Lax'
],
'x-sorting-hat-podid': '50',
'x-sorting-hat-shopid': '7425359923',
'x-shopify-stage': 'production',
'x-content-type-options': 'nosniff',
'x-download-options': 'noopen',
'x-permitted-cross-domain-policies': 'none',
'x-xss-protection': '1; mode=block',
'x-dc': 'gcp-us-east1,gcp-us-east1',
'x-request-id': '62739686-69d6-4e86-a3c2-fb209381d664',
'cf-cache-status': 'DYNAMIC',
'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
server: 'cloudflare',
'cf-ray': '58777086dfc06a41-LHR',
'alt-svc': 'h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400',
'cf-request-id': '023e84a84b00006a416a903200000001'
}

 

If I replace the characters in the body_html with &#xxx; format, the write succeeds.

 

Here is that result:

URL:
https://key:password@mystore.myshopify.com/admin/api/2020-04/products.json

 

OPTIONS:
{
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Content-Length': 1762 }
}

 

POSTDATA:
{"product":{"product_type":"Lifestyle","body_html":"<meta charset=\"utf-8\">\n<p><span>Have you met Cubebot?</span></p>\n<p>Introducing the Cubebot&#174; Capsule Collection.&#160; Five new multi-colored Cubebots in two sizes. Cubebot, a foldable wooden robot puzzle with hundreds of poses but only one solution.</p>\n<p>Recommended for Ages 3+</p>\n<meta charset=\"utf-8\">\n<p><span>Have you met Cubebot?</span></p>\n<p>By <a href=\"https://queen-of-hearts-and-modern-love.myshopify.com/pages/areaware\" target=\"_blank\" title=\"robot, childrens games, games, cubebot\" rel=\"noopener noreferrer\">Areaware&#160;</a></p>\n<meta charset=\"utf-8\">\n<p class=\"small\">Beech and Elastic<br>Micro&#160;<br>4.25\" tall<br>5.5\" arm span<br>1.5x1.5\" cube</p>\n<p class=\"small\">Small<span>&#160;</span><br>6.75\" tall<br>9.25\" arm span<br>2.5x2.5\" cube</p>\n<p>&#160;</p>","title":"Areaware Cubebot Capsule Collection Micro by David Weeks - Red Multi","tags":"games, lifestyle","vendor":"Areaware","image":{"position":1,"alt":null,"src":"https://cdn.shopify.com/s/files/1/1248/8951/products/Red_Multi.jpg?v=1570733904"},"images":[{"positi... Title"]}],"variants":[{"title":"Default Title","price":"10.00","option1":"Default Title","option2":null,"option3":null,"sku":"708389997156","inventory_quantity":4,"inventory_management":"shopify"}]}}

 

STATUS CODE:
statusCode: 201

 

RESPONSE HEADERS:
headers: {
date: 'Tue, 21 Apr 2020 13:31:24 GMT',
'content-type': 'application/json; charset=utf-8',
'transfer-encoding': 'chunked',
connection: 'close',
'set-cookie': [
'__cfduid=d53dd9b8481e47f5e56cce23600036e3c1587475879; expires=Thu, 21-May-20 13:31:19 GMT; path=/; domain=.myshopify.com; HttpOnly; SameSite=Lax'
],
'x-sorting-hat-podid': '50',
'x-sorting-hat-shopid': '7425359923',
'referrer-policy': 'origin-when-cross-origin',
'x-frame-options': 'DENY',
'x-shopid': '7425359923',
'x-shardid': '50',
'x-stats-userid': '',
'x-stats-apiclientid': '2917325',
'x-stats-apipermissionid': '91298889779',
'x-shopify-api-terms': 'By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms',
http_x_shopify_shop_api_call_limit: '1/40',
'x-shopify-shop-api-call-limit': '1/40',
'x-shopify-api-version': '2020-04',
location: 'https://stq-dev2.myshopify.com/admin/products/4506957250611',
'strict-transport-security': 'max-age=7889238',
'x-request-id': 'c8a47772-0a8f-457e-9fb8-f3c67fd77f01',
'x-shopify-stage': 'production',
'content-security-policy': "default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=c8a47772-0a8f-457e-9fb8-f3c67fd77f01",
'x-content-type-options': 'nosniff',
'x-download-options': 'noopen',
'x-permitted-cross-domain-policies': 'none',
'x-xss-protection': '1; mode=block; report=/xss-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=c8a47772-0a8f-457e-9fb8-f3c67fd77f01',
'x-dc': 'gcp-us-east1,gcp-us-east1',
nel: '{"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}',
'report-to': '{"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}',
'cf-cache-status': 'DYNAMIC',
'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
server: 'cloudflare',
'cf-ray': '587776f95a91dc37-LHR',
'alt-svc': 'h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400',
'cf-request-id': '023e88afd40000dc37ac18f200000001'
}

 

0 Likes
Highlighted
Shopify Partner
11 1 0

Hi JB,

Any progress on what I'm doing wrong? I'm still frustrated by not being able to write characters above u0080.

Let me know if you need me to test again and supply new X-Request-ID values from the headers of a failed call

Thank you,

 -djs

0 Likes
Highlighted
Shopify Partner
521 38 109

Silly question, but since you are apparently using NodeJS...have you tried using the escape() function to automatically escape these characters?

0 Likes
Highlighted
Shopify Partner
11 1 0

Thank you for the suggestion.
I didn't try it for the two following reasons:

1) escape() is to turn a string into an HTTP URL query-friendly string. That's not what I need for POST data (perhaps for GET it would work).

2) Use of escape() is not recommended: see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/escape

3) Near as I can tell, the Shopify API won't decode the "URL safe" hexadecimal sequence created by escape() back into the characters I want.

Even so, I gave it a shot:

escape() turned:
{
    "product": {
        "title": "Burton Custom Freestyle 151",
        "body_html": "<strong>Good snowboard!</strong>",
        "vendor": "Burton",
        "product_type": "Snowboard",
        "tags": [
            "Barnes & Noble",
           "John's Fav",
           "\\"Big Air\\""
           ]
      }
}

into

%7B%20%20%20%20%22product%22%3A%20%7B%20%20%20%20%20%20%20%20%22title%22%3A%20%22Burton%AE%20Custom%20Freestyle%20151%22%2C%20%20%20%20%20%20%20%20%22body_html%22%3A%20%22%3Cstrong%3EGood%20%u20AC%20%A3%20snowboard%21%3C/strong%3E%22%2C%20%20%20%20%20%20%20%20%22vendor%22%3A%20%22Burton%22%2C%20%20%20%20%20%20%20%20%22product_type%22%3A%20%22Snowboard%22%2C%20%20%20%20%20%20%20%20%22tags%22%3A%20%5B%20%20%20%20%20%20%20%20%20%20%20%20%22Barnes%20%26%20Noble%22%2C%20%20%20%20%20%20%20%20%20%20%20%20%22Johns%20Fav%22%2C%20%20%20%20%20%20%20%20%20%20%20%20%22%5C%22Big%20Air%5C%22%22%20%20%20%20%20%20%20%20%5D%20%20%20%20%7D%7D

and I got a 400 error.

0 Likes
Highlighted
Shopify Partner
521 38 109
Highlighted
Shopify Partner
11 1 0
I tried again using the Shopify API example. Same failure: Here's the result
 
URL:
https://key:password@mystore.myshopify.com/admin/api/2020-04/products.json
 
OPTIONS:
{
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Content-Length': 1732 }
}
 
and passed in as a string)
{
    "product": {
    "title": "Burton® Custom Freestyle 151",
    "body_html": "<strong>Good € £ snowboard!</strong>",
    "vendor": "Burton",
    "product_type": "Snowboard",
    "tags": [
        "Barnes & Noble",
        "Johns Fav",
        "\"Big Air\""
    ]
    }
}
 
JAVAscript&colon;
const https = require("https");
writeproduct(URL, POST_DATA, OPTIONS);  // see above for what was passed in
function writeproduct(url, postdata, options) {
    const req = https.request(url, options, function (res) {
        console.log('statusCode:', res.statusCode);
        console.log('headers:', res.headers);
    });
    req.write(postdata);
    req.end();
}
 
STATUS CODE:
statusCode: 400
 
RESPONSE HEADERS:
headers: {
  date: 'Tue, 16 Jun 2020 16:47:30 GMT',
  'content-type': 'application/json',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'set-cookie': [
    '__cfduid=dbd101f37ae2cb1bacd617660d946d2731592326049; expires=Thu, 16-Jul-20 16:47:29 GMT; path=/; domain=.myshopify.com; HttpOnly; SameSite=Lax'
  ],
  'x-sorting-hat-podid': '50',
  'x-sorting-hat-shopid': '7425359923',
  'x-shopify-stage': 'production',
  'x-content-type-options': 'nosniff',
  'x-download-options': 'noopen',
  'x-permitted-cross-domain-policies': 'none',
  'x-xss-protection': '1; mode=block',
  'x-dc': 'gcp-us-east1,gcp-us-central1,gcp-us-central1',
  'x-request-id': '964fa051-1379-47df-ad76-36569589843c',
  'cf-cache-status': 'DYNAMIC',
  'cf-request-id': '035fa068410000088ba1a9e200000001',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  server: 'cloudflare',
  'cf-ray': '5a4603539bf5088b-CDG',
  'alt-svc': 'h3-27=":443"; ma=86400'
}
 
0 Likes
Highlighted
Shopify Partner
521 38 109

I just copied and pasted your JSON request body into Postman and issued this against our shop. It worked fine, as-is. I'll include the API request/response pair below. Definitely strange :( 

 

POST https://{my_shop}.myshopify.com/admin/api/2020-04/products.json HTTP/1.1
Content-Type: application/json
Authorization: Basic {my_access_token}
User-Agent: PostmanRuntime/7.25.0
Accept: */*
Host: {my_shop}.myshopify.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 292
Cookie: __cfduid=d474d283195031f2c442a1f3fcd4abe0d1591975172

{
    "product": {
    "title": "Burton® Custom Freestyle 151",
    "body_html": "<strong>Good € £ snowboard!</strong>",
    "vendor": "Burton",
    "product_type": "Snowboard",
    "tags": [
        "Barnes & Noble",
        "Johns Fav",
        "\"Big Air\""
    ]
    }
}

HTTP/1.1 201 Created
Date: Tue, 16 Jun 2020 17:22:43 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
X-Sorting-Hat-PodId: 161
X-Sorting-Hat-ShopId: 3036253
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 3036253
X-ShardId: 161
X-Stats-UserId: 
X-Stats-ApiClientId: 309925
X-Stats-ApiPermissionId: 8304915
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2020-04
Location: https://{my_shop}.myshopify.com/admin/products/5336559583394
Strict-Transport-Security: max-age=7889238
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopifycdn.net https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=da3c4e82-2d6c-4975-8e18-905f27860f79
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=da3c4e82-2d6c-4975-8e18-905f27860f79
X-Dc: gcp-us-central1,gcp-us-east1,gcp-us-east1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
X-Request-ID: da3c4e82-2d6c-4975-8e18-905f27860f79
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
CF-Cache-Status: DYNAMIC
cf-request-id: 035fc0a54c000058a4abb86200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5a4636e87c1958a4-ORD
alt-svc: h3-27=":443"; ma=86400
Content-Length: 1280

{"product":{"id":5336559583394,"title":"Burton® Custom Freestyle 151","body_html":"\u003cstrong\u003eGood € £ snowboard!\u003c\/strong\u003e","vendor":"Burton","product_type":"Snowboard","created_at":"2020-06-16T17:22:42Z","handle":"burton-custom-freestyle-151","updated_at":"2020-06-16T17:22:43Z","published_at":"2020-06-16T17:22:42Z","template_suffix":null,"published_scope":"global","tags":"\"Big Air\", Barnes \u0026 Noble, Johns Fav","admin_graphql_api_id":"gid:\/\/shopify\/Product\/5336559583394","variants":[{"id":34709060419746,"product_id":5336559583394,"title":"Default Title","price":"0.00","sku":"","position":1,"inventory_policy":"deny","compare_at_price":null,"fulfillment_service":"manual","inventory_management":null,"option1":"Default Title","option2":null,"option3":null,"created_at":"2020-06-16T17:22:43Z","updated_at":"2020-06-16T17:22:43Z","taxable":true,"barcode":null,"grams":0,"image_id":null,"weight":0.0,"weight_unit":"lb","inventory_item_id":36639142314146,"inventory_quantity":0,"old_inventory_quantity":0,"requires_shipping":true,"admin_graphql_api_id":"gid:\/\/shopify\/ProductVariant\/34709060419746"}],"options":[{"id":6813484253346,"product_id":5336559583394,"name":"Title","position":1,"values":["Default Title"]}],"images":[],"image":null}}
0 Likes
Highlighted
Shopify Partner
11 1 0

I tried that a while ago and it didn't work.

For reference, the following code works for encoding a product's body_html because it is interpreted by Shopify as HTML.

// replace any characters with codes \u0080 or larger
//   with their HTML code equivalent
function replaceCodes(string) {
    return string.replace(/[\u0080-\uFFFF]/gimfunction (c) {
        return '&#' + c.charCodeAt(0) + ';';
    });
}
 
However, this doesn't product the right result for the title field or any other field that is interpreted as text.
In any case, the API is supposed to take the higher-order characters, so something I'm doing is wrong.
 
@_JB could my problem be with the "Content-Type" in the header options? I'm using 'application/json'.
0 Likes