Getting verified for web crawling

rylancole
New Member
1 0 0

I am working on a custom search engine that will need to crawl and index various product pages on Shopify sites. We are using Apache Nutch for our project, but these lines in robots.txt will block us.

User-agent: Nutch
Disallow: / 

I understand the need for this broad blocking, and we are months off from launch still, but I am just wondering who I should get in contact with to get our user-agent verified so Shopify could add us to their robots.txt file?

User-agent: <my-agent>
Disallow: 
0 Likes