Bing Webmaster Sitemap Failure

[Mount_Republic_]

Hi,

Has anyone successfully submited their sitemap to bing webmaster?

I have submitted it several times over the last few months and for a long time it was pending. Finally, this past month the status turned to "failed".

I fetched the sitemap as a bing bot and this is what the status comes up as: "Status: The DNS resolution for the host of the url (or the redirected url) could not be resolved."

Anyone else have had this issue? And if so, what have you done to fix it?

Thanks,

Matt

[Doug_Gorman]

Hi Matt,

I am having the same porblem as well, I have reached out to bing multiple times, Shopify, and Godaddy as well. I get the same error its been going on for months. 

[Jason]

What's the url you added for the sitemap?

[Mount_Republic_]

Bummer. Well hopefully someone will be able to hell us out.

[Mount_Republic_]

Hi Jason,

The url I used is: https://mountrepublic.com/sitemap.xml

This is the same url I've used for Google's search and haven't had any problems there.

[St3v3]

I am having the same issue with Bing reporting a DNS failure. Google has indexed the site witrh no issues reported.

I have spoken to Bing they say I need to speak with the Domain provider, so I have contacted them (Namesco) and as the domain is forwarded to Shopify there is nothing they can do, so I need to contact Shopify. So I have done that and Shopify say I need to ask Namesco.

Bit stuck on this now and its very frustrating.

The only thing to add to this is I cant see all these different domain providers having the same issue, so the only common thing with all these problems is Shopify.

Hopefully Shopify can assist (have emailed them again) with this or it does limit the exposure the shop will be getting and I may have to consider other options. Not what I want to do with the time I have put into this new shop.

[Doug_Gorman]

Status: The DNS resolution for the host of the url (or the redirected url) could not be resolved.

Still geetting this error, Had Godaddy work with me on the  CNAME, everything is directed properly but not geting any response from bing. Shopify has told me Im on my own and I should try the forums????

[Don_Martin]

Exact same problem on 2 stores. This appears to be a Bing issue. I am following and will try to help if I find anything (over 20 years IT and Web exp)

[Doug_Gorman]

FYI I finally heard something from Bing. Waiting on hold with godaddy to see if they can help.

Hi Douglas,

Hope you are doing good. As per the Engineering team, when Bingbot is trying to crawl the sitemap it is receiving DNS Failure error and the server is blocking the Bingbot from crawling the sitemap. Here is the error message what we are receiving while making an attempt to crawl the sitemap "DNSFailure 5004This status code indicates that the DNS resolution for the host of the URL (or the redirected URL) could not be resolved. This status code indicates that DNS could not be resolved for the URL when the URL timed out." I would request you to check this issue with the server or DNS service provider and ask them to fix the issue with DNS so that Bingbot will crawl the sitemap and it will get submitted to Bing webmaster tool successfully. Once the issue on the server side is resolved, I would request you to please resubmit the sitemap or please let me know as soon as possible so that  I will resubmit the sitemap from my end. Sorry for the delay response because the Engineering took time to find the root cause of the issue. 

I hope that the information provided above would be helpful. If you have any follow-up questions or concerns, please let me know as soon as possible so that I would be glad to assist you.

Thank you,

Vivekchandra Awalkar | TECHNICAL SUPPORT ENGINEER
Bing Ads Support | Supporting your needs 24 x 7

Bing Ads - Check Out What's New in the Latest Release!

[BrentA]

---

@Doug_Gorman wrote:

FYI I finally heard something from Bing. Waiting on hold with godaddy to see if they can help.

Hi Douglas,

Hope you are doing good. As per the Engineering team, when Bingbot is trying to crawl the sitemap it is receiving DNS Failure error and the server is blocking the Bingbot from crawling the sitemap. Here is the error message what we are receiving while making an attempt to crawl the sitemap "DNSFailure 5004This status code indicates that the DNS resolution for the host of the URL (or the redirected URL) could not be resolved. This status code indicates that DNS could not be resolved for the URL when the URL timed out." I would request you to check this issue with the server or DNS service provider and ask them to fix the issue with DNS so that Bingbot will crawl the sitemap and it will get submitted to Bing webmaster tool successfully. Once the issue on the server side is resolved, I would request you to please resubmit the sitemap or please let me know as soon as possible so that  I will resubmit the sitemap from my end. Sorry for the delay response because the Engineering took time to find the root cause of the issue. 

I hope that the information provided above would be helpful. If you have any follow-up questions or concerns, please let me know as soon as possible so that I would be glad to assist you.

Thank you,

Vivekchandra Awalkar | TECHNICAL SUPPORT ENGINEER
Bing Ads Support | Supporting your needs 24 x 7

Bing Ads - Check Out What's New in the Latest Release!

---

 I got the same response from Bing from my support request with them about it:

Hi Brent,

Hope you are doing good. After my detailed on this issue, I see that when Bingbot is trying to crawl the sitemap it is receiving DNS Failure error and the server is blocking the Bingbot from crawling the sitemap. Here is the error message what we are receiving while making an attempt to crawl the sitemap "DNSFailure 5004." This status code indicates that the DNS resolution for the host of the URL (or the redirected URL) could not be resolved. This status code indicates that DNS could not be resolved for the URL when the URL timed out." I would request you to check this issue with the server or DNS service provider and ask them to fix the issue with DNS so that Bingbot will crawl the sitemap and it will get submitted to Bing webmaster tool successfully. Once the issue on the server side is resolved, I would request you to please resubmit the sitemap or please let me know as soon as possible so that I will resubmit the sitemap from my end. 

I hope that the information provided above would be helpful. If you have any follow-up questions or concerns, please let me know as soon as possible so that I would be glad to assist you.

Have a nice day!

Thank you,

Vivekchandra Awalkar | TECHNICAL SUPPORT ENGINEER
Bing Ads Support | Supporting your needs 24 x 7

[Doug_Gorman]

Still now clue as to what is happeinging. Godaddy again was no help. I did figure out that when I submit my site map via Bing Bot using http://ultra45.com I get the Re-Direct to Https, and bing is working!!!  However it is once the re-direct poits to the ssl [https://ultra45.com] the problem occours.

[Mount_Republic_]

---

@Doug_Gorman wrote:

Still now clue as to what is happeinging. Godaddy again was no help. I did figure out that when I submit my site map via Bing Bot using http://ultra45.com I get the Re-Direct to Https, and bing is working!!!  However it is once the re-direct poits to the ssl [https://ultra45.com] the problem occours.

---

I've noticed if I fetch the "http://" link then I get the same status as you. However, when I do it with "https://" the status is "completed".

[Doug_Gorman]

One more time, I called shopify, they said there was not problem. Just realized that when bing reads the xml file there is no format on the site map.....

This XML file does not appear to have any style information associated with it. The document tree is shown below.
        

[Mount_Republic_]

---

@Doug_Gorman wrote:

One more time, I called shopify, they said there was not problem. Just realized that when bing reads the xml file there is no format on the site map.....

This XML file does not appear to have any style information associated with it. The document tree is shown below.
        

---

This is normal. There isn't have to be any styling here. XML is simply a file for to allow search engine crawlers to crawl and index your website. 

If I type my link to direct me to the sitemap.xml this is what I get:

This XML file does not appear to have any style information associated with it. The document tree is shown below.

"<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<!--

This is the parent sitemap linking to additional sitemaps for products, collections and pages as shown below. The sitemap can not be edited manually, but is kept up to date in real time.

-->

<sitemap>

<loc>

https://www.mountrepublic.com/sitemap_products_1.xml?from=1504400441403&to=1622977937467

</loc>

</sitemap>

<sitemap>

<loc>https://www.mountrepublic.com/sitemap_pages_1.xml</loc>

</sitemap>

<sitemap>

<loc>

https://www.mountrepublic.com/sitemap_collections_1.xml

</loc>

</sitemap>

<sitemap>

<loc>https://www.mountrepublic.com/sitemap_blogs_1.xml</loc>

</sitemap>

</sitemapindex>"

[St3v3]

I have not had a response from Shopify since my last email which is disapointing, so currently I am no further forward on this.

[Mount_Republic_]

Hi all,

Thanks for all the updates. 

As for me, I'm waiting to hear back from Shopify and Bing. But so far no luck with anything.

- Matt

[BrentA]

I'm having the same issue.  Bing says it is an issue with my DNS or website is blocking them.  My domain is registered with GoDaddy and I contacted them via chat but they were no help, the guy I was chatting with kept asking me how I wanted him to fix the issue.  I'm still talking to Shopify support, so far they have been trying to help with the issue, but they haven't been able to figure out a resolution to the issue yet.

[BrentA]

For the ones having the issue is your www   CNAME in your DNS set to shops.myshopify.com or something else like @ ?

[Don_Martin]

I have the same issue and I have worked for 20 + years in IT. I have a question I have 2 sites failing this way 1 is on Name Cheap and 1 is on GoDaddy. The question I have is are people submitting their sitemaps with the https://www.sitename.com/sitemap.xml or the https://sitename.com/sitemap.xml format I am going to attempt the www method as I have only submitted the other way so far.

---

[Mount_Republic_]

---

@Don_Martin wrote:

I have the same issue and I have worked for 20 + years in IT. I have a question I have 2 sites failing this way 1 is on Name Cheap and 1 is on GoDaddy. The question I have is are people submitting their sitemaps with the https://www.sitename.com/sitemap.xml or the https://sitename.com/sitemap.xml format I am going to attempt the www method as I have only submitted the other way so far.

---

---

I've submitted both. Both have been pending for a while and now since bing manually resubmitted recently I'll be waiting longer. Once something changes, I'll let you know.

Supposedly, you should submit the one which is your primary domain. Mine is www.sitename.com so I should submit https://www.sitename.com/sitemap.xml. Previously, I had sitename.com. 

[GinaYAAHealth]

Are you all still having trouble with this? I am having the same issue with Bing not indexing the site.

I have contacted bing and shopify. I have referred the shopify support tech to this forum case. I will be sharing Bing results with Shopify support to be refered to Tech at Shopify. Hopefully we can get this fixed.

[Jpatrickm]

I've been getting the 403 Forbidden error for awhile but I was able to finally get this working by submitting the non-secure version of my site http:// without the www. No idea why that works, but it did.

[codecreate]

It's the same for me and I am not a newbie. I already host custom ecom sites I made from scratch on Google Cloud. This bug with Bing and Shopify has been going on for what seems like years from the posts in these forums. Each does nothing and each blames each other. I am with 123-reg for my domains so not godaddy like many. 

I suspect the issue is DNS / Cloudflare related with SSL. When using the tools 'Fetch as Bingbot' it works fine, even when using cURL on the command line with 'Bingbot' as the user agent it still returns a 200 status but when submitting sitemaps and pages with https in Bing Webmaster tools it gives 403 errors. Obviously Google Search Console is perfectly fine so the error / bug is probably be more Bing related but don't expect anything other than pre-templated replies from Bing and silence from Shopify.

My store for my client is ranking in Google, all is fine, only the homepage is in Bing, yet I have over 50 other non-shopify sites all ranking and indexed as expected with Bing.

I would suspect many Shopify owners who don't do any web dev won't even know this or be monitoring but it really is beyond a joke and certainly will be meaning I never go past one shop on this platform. It's Bing and Yahoo remember, I know Google is the majority but you least expect to have Shopify sorting this out directly with Bing.

It is very disappointing indeed, I wasted over a day trying everything and anything to no avail. 

http:// submits OK but the pages in the sitemap are https so it's a waste of time. I am waiting to see how manually submitted http versions pan out. The fact we can't even edit our sitemaps and add in internationalised pages with hreflang is also rather weird to say the least. 

It is only a time issue that is preventing me rolling out my own for this particular client. I don't know how anyone can recommend Shopify while this issue is left unattended and ignored.

[EUGENEAR]

Bing it is showing 403 (Forbidden)

is shopify or cloudflare blocking bing bots by mistakes?

[altCookinghub]

Eugene, I don't know. How do one verify and confirm that?

Thank you for your input and guide.

[codecreate]

---

@EUGENEAR wrote:

Bing it is showing 403 (Forbidden)

is shopify or cloudflare blocking bing bots by mistakes?

---

It appears it is something related to the DNS so I suspect it is a Cloudfare/Shopify/Bing specific error but can't pinpoint which one.

[EUGENEAR]

403 is a client server-side message from shopify servers telling Bing that bingbot has no access to that sitemap url via SSL connection. As it was suggested by another poster, if you use non secure protocol, the message will go away and bingbot will be able to crawl your sitemap. instead of https://yourdomain/sitemap.xml change it to http://yourdomain/sitemap.xml

[altCookinghub]

All, I want to share the email exchanges that I'm in the middle with both Bing support team and Shopify team.

1st response from Bing:

Thank you for the patience during our investigation, This is to inform you that when Bingbot is trying to crawl the site it is receiving 403 error and the server is blocking the Bingbot from crawling the site ( https://altcookinghub.com/sitemap.xml ). Here is the error message what we are receiving while making an attempt to crawl the site "403 HTTP Status " this status code indicates that the server is refusing access. I would request you to check with your Web Host/Web Service Provider, .htaccess files, virus detection, and data center for any unintentional blocks placed on our User Agents. Once the issue on the server-side is resolved, Our Bingbot will crawl the site and it will get site verified in Bing webmaster tool successfully. Please see the below screenshots for your reference.

Response from Shopify:

Michael here with Shopify Technical Support; it's a pleasure to meet you.

Kelly brought me up to speed on the issues you're having with submitting your sitemap to Bing. This issue would be on Bing's end and the way they handle HTTPS Server Name Indication (SNI). The sitemap is readily available for them to crawl, however in some situations we have notice Bing reporting issues due to a 403 error. This comes down to Bing needing to whitelist HTTPS SNI so they can correctly crawl through the sitemap and index everything for the storefront. We wouldn't be able to make any adjustments on our end for this as the issue is entirely coming from their end and they should be able to go through further steps for getting that whitelisted.

It is still finger pointing at this time. I'm forwarding Shopify response to Bing and awaiting for Bing's response back. I will update you guys if I hear from them back.

Comments are welcome!

[codecreate]

Still unresolved after all this time, having successfully verified and submitted sitemaps with my google cloud server plus another US based host, the issue is obviously Shopify but they send only unhelpful copy and paste replies.

---

@altCookinghub wrote:

All, I want to share the email exchanges that I'm in the middle with both Bing support team and Shopify team.

1st response from Bing:

Thank you for the patience during our investigation, This is to inform you that when Bingbot is trying to crawl the site it is receiving 403 error and the server is blocking the Bingbot from crawling the site ( https://altcookinghub.com/sitemap.xml ). Here is the error message what we are receiving while making an attempt to crawl the site "403 HTTP Status " this status code indicates that the server is refusing access. I would request you to check with your Web Host/Web Service Provider, .htaccess files, virus detection, and data center for any unintentional blocks placed on our User Agents. Once the issue on the server-side is resolved, Our Bingbot will crawl the site and it will get site verified in Bing webmaster tool successfully. Please see the below screenshots for your reference.

Response from Shopify:

Michael here with Shopify Technical Support; it's a pleasure to meet you.

Kelly brought me up to speed on the issues you're having with submitting your sitemap to Bing. This issue would be on Bing's end and the way they handle HTTPS Server Name Indication (SNI). The sitemap is readily available for them to crawl, however in some situations we have notice Bing reporting issues due to a 403 error. This comes down to Bing needing to whitelist HTTPS SNI so they can correctly crawl through the sitemap and index everything for the storefront. We wouldn't be able to make any adjustments on our end for this as the issue is entirely coming from their end and they should be able to go through further steps for getting that whitelisted.

It is still finger pointing at this time. I'm forwarding Shopify response to Bing and awaiting for Bing's response back. I will update you guys if I hear from them back.

Comments are welcome!

---

[Mark_Harding]

I had this issue any attempt to submit my Bing sitemap resulted in verify failure.

https ver of sitemap url failed over and over

https://yoursite.com/sitemap.xml

http worked ok.

http://yoursite.com/sitemap.xml

Regards Mark H

[codecreate]

Yep http works, just a pity that many won't realise this issue or simply give up. It is surprising that this still has not been resolved.

---

@Mark_Harding wrote:

I had this issue any attempt to submit my Bing sitemap resulted in verify failure.

https ver of sitemap url failed over and over

https://yoursite.com/sitemap.xml

http worked ok.

http://yoursite.com/sitemap.xml

Regards Mark H

---

[codecreate]

When I get time I will try submitting https sitemaps using the API, which I suspect will work.

[altCookinghub]

Guys, I'd recommend you to re-submit the https sitemap as I received email that it is fixed and indeed mine is fixed now. If resubmit didn't work for you, then I think you may want to contact Bing. Hope this helps

[codecreate]

Still not working for my site on Shopify, I have spent many hours emailing both Shopify and Bing to no avail. I am pleased your site is now accepting https on Bing, hopefully others on the same server are too. It might be IP or location specific.

[Don_Martin]

Also to your question in Namecheap it is shops.myshopify.com. I

---

@BrentA wrote:

For the ones having the issue is your www   CNAME in your DNS set to shops.myshopify.com or something else like @ ?

---

Also to your question in Namecheap it is shops.myshopify.com. In GoDaddy it is "shopname".myshopify.com

[Mount_Republic_]

Hey all,

So I've heard from bing. Apparently they weren't able to reproduce my DNS failure. They took a look at it and from what they see is that on the back-end bing is able to get to the sitemap, but for whatever reason it isn't getting updated on the webmaster portal. They have manually resubmitted my sitemap and will follow up with me. So far, my website hasn't been indexed so we'll see...

Have you guys verified your domain through bing? 

Shopify got back to me as well. So far all they've recommended is (1) to make sure the website is verified on Bing (which it is already) as well as offered a link to purchase a shopify app that can help with verification (don't need it), and (2) to use this forum as guidance (no help there as I'm the one who opened this thread..............).

-Matt

[Mount_Republic_]

---

@BrentA wrote:

For the ones having the issue is your www   CNAME in your DNS set to shops.myshopify.com or something else like @ ?

---

I have my domain through shopify, and yes, CNAME RECORD is as follows (Name: www ) and (points to: shops.myshopify.com).

[vapecommerceuk]

Also having a problem with Bing Webmaster Tools, I cannot verify the domain... I will try again later in case of a temporary problem or timeout.

[codecreate]

That is usually a DNS CNAME issue, might just take some time to propagate. If you are still stuck let me know, I can send a screen shot of my DNS if it helps

[i0nkol]

send shot please

[DabDude]

Any updates on this I am having the same issue all was fine until i migrated to shopify

[codecreate]

After weeks and weeks I now have 7 urls for this site listed in Bing. The only way it seems to work is by manually submitting URLs using http:// instead of https:// even then 7 urls compared to 300 in Google is beyond a joke.

Once again, the issue only occurs with the 1 Shopify site I have. All others are perfectly fine, I have been a web dev for almost 20 years so it's not like I don't understand it. My other sites all rank very well and all crawl without any issues in Bing.

This is obviously a Bing + Shopify + Cloudflare issue or some combination of those three +- one. I wasted days on this and refuse to keep wasting time on bugs I can't fix. Sadly this is one of the downsides of using a hosted platform.

On my other sites I have built and watch over Bing traffic is very low anyway compared to Google, especially UK sites. 5% and this is a not a Shopify site.

[DabDude]

I was thinking about flipping DNS from Godaddy to CloudFlare and just use the free account to see if anything changes or using Linode DNS because i have a couple of boxes there and they offer DNS... Right now I am still having issues with verifying my site by TXT record with Google Webmaster Tools. It just constantly goes to Unsuccessful which i thought it was odd. Its been days and still same message - https://www.screencast.com/t/pOLsPTrtbkFP

Edit: Just Cloudflard DNS Only

[codecreate]

I tried Google DNS, no changes with Bing, Cloudflare might be an option actually if that is the issue, please let us know, thank you.

As for google verification here is my setup below for this shopify site, apologies for scrubbing parts out but you can't be too careful. Maybe this might help you with the Google TXT verification, notice I put it in * and www and @ . 

[DabDude]

alll good thanks again, I will keep you updated!

[codecreate]

Now done, sorry I missed this reply.

[codecreate]

Bing has crawled the site from going by the Index Explorer, it just seems to be very slow showing them up when using the site: operator. So submitting the pages with http:// does seem to have worked. 

I used cURL eg :

curl --head --user-agent "BingBot" https://www.spiceapparel.co.uk

and the response was correct, 200 returned (see below) so the https issue seems like it is Bing Webmaster Tools specific and not bing or bingbot

HTTP/2 200 
date: Mon, 08 Jul 2019 16:39:15 GMT
content-type: text/html; charset=utf-8
set-cookie: __cfduid=db95960f831ae9b739af927e81b80faf71562603955; expires=Tue, 07-Jul-20 16:39:15 GMT; path=/; domain=.www.spiceapparel.co.uk; HttpOnly
x-sorting-hat-podid: 83
x-sorting-hat-shopid: 7153680468
x-frame-options: DENY
x-shopid: 7153680468
x-shardid: 83
content-language: en
strict-transport-security: max-age=7889238
x-alternate-cache-key: cacheable:2629dfd6ec48aa4cf7e9c767b50606d6
x-cache: miss
set-cookie: _shopify_y=7d588732-ed75-4696-a040-cb22d7e7d368; path=/; expires=Thu, 08 Jul 2021 04:17:39 -0000
x-request-id: 5f4f688d-77a1-4bdb-9cb9-7ab6bc8fe294
x-shopify-stage: production
content-security-policy: block-all-mixed-content; frame-ancestors 'none'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fshop&source%5Bsection%5D=storefront&source%5Buuid%5D=5f4f688d-77a1-4bdb-9cb9-7ab6bc8fe294
x-content-type-options: nosniff
x-download-options: noopen
x-permitted-cross-domain-policies: none
x-xss-protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fshop&source%5Bsection%5D=storefront&source%5Buuid%5D=5f4f688d-77a1-4bdb-9cb9-7ab6bc8fe294
x-dc: gcp-us-central1,gcp-us-east1
set-cookie: _orig_referrer=; Expires=Mon, 22-Jul-19 16:39:15 GMT; Path=/; HttpOnly
set-cookie: secure_customer_sig=; path=/; expires=Fri, 08 Jul 2039 16:39:15 -0000; secure; HttpOnly
set-cookie: _landing_page=%2F; Expires=Mon, 22-Jul-19 16:39:15 GMT; Path=/; HttpOnly
set-cookie: cart_sig=; path=/; expires=Mon, 22 Jul 2019 16:39:15 -0000; HttpOnly
nel: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
report-to: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f337e3eec473516-LHR

[altCookinghub]

Hi, I'm a newbie particularly on this SEO and indexing subjects.

Do I understand your latest post correctly that it is equivalent and working to submit the sitemap using http:// instead of https:// because using https:// will actually result in 403 error? Please kindly share. Thank you in advance.

[codecreate]

altCookinghub - yes that is correct, submit http://  sitemaps and urls, it seems to be the only workaround.

[Astha]

Hello Matt,
Try following bing tools to get the exact solution of your query
    1- Fetch as Bingbot
    2-Markup Validator
    3-SEO Analyzer
The DNS resolution for the host of the URL (or the redirected url) could not be resolved." this status can be because of the wrong url, your server is not responding, your URL is expired, etc
Contact your server team to resolve this issue
rest refer this https://www.bing.com/webmaster/help/why-is-my-site-not-in-the-index-2141dfab

thanks!