Is it possible to disable password page for search engines to allow crawling of the site?
I’m wondering if i can do it with Javascript to detect the robot?
var botPattern = "(googlebot\/|bot|Googlebot-Mobile|Googlebot-Image|Google favicon|Mediapartners-Google|bingbot|slurp|java|wget|curl|Commons-HttpClient|Python-urllib|libwww|httpunit|nutch|phpcrawl|msnbot|jyxobot|FAST-WebCrawler|FAST Enterprise Crawler|biglotron|teoma|convera|seekbot|gigablast|exabot|ngbot|ia_archiver|GingerCrawler|webmon |httrack|webcrawler|grub.org|UsineNouvelleCrawler|antibot|netresearchserver|speedy|fluffy|bibnum.bnf|findlink|msrbot|panscient|yacybot|AISearchBot|IOI|ips-agent|tagoobot|MJ12bot|dotbot|woriobot|yanga|buzzbot|mlbot|yandexbot|purebot|Linguee Bot|Voyager|CyberPatrol|voilabot|baiduspider|citeseerxbot|spbot|twengabot|postrank|turnitinbot|scribdbot|page2rss|sitebot|linkdex|Adidxbot|blekkobot|ezooms|dotbot|Mail.RU_Bot|discobot|heritrix|findthatfile|europarchive.org|NerdByNature.Bot|sistrix crawler|ahrefsbot|Aboundex|domaincrawler|wbsearchbot|summify|ccbot|edisterbot|seznambot|ec2linkfinder|gslfbot|aihitbot|intelium_bot|facebookexternalhit|yeti|RetrevoPageAnalyzer|lb-spider|sogou|lssbot|careerbot|wotbox|wocbot|ichiro|DuckDuckBot|lssrocketcrawler|drupact|webcompanycrawler|acoonbot|openindexspider|gnam gnam spider|web-archive-net.com.bot|backlinkcrawler|coccoc|integromedb|content crawler spider|toplistbot|seokicks-robot|it2media-domain-crawler|ip-web-crawler.com|siteexplorer.info|elisabot|proximic|changedetection|blexbot|arabot|WeSEE:Search|niki-bot|CrystalSemanticsBot|rogerbot|360Spider|psbot|InterfaxScanBot|Lipperhey SEO Service|CC Metadata Scaper|g00g1e.net|GrapeshotCrawler|urlappendbot|brainobot|fr-crawler|binlar|SimpleCrawler|Livelapbot|Twitterbot|cXensebot|smtbot|bnf.fr_bot|A6-Indexer|ADmantX|Facebot|Twitterbot|OrangeBot|memorybot|AdvBot|MegaIndex|SemanticScholarBot|ltx71|nerdybot|xovibot|BUbiNG|Qwantify|archive.org_bot|Applebot|TweetmemeBot|crawler4j|findxbot|SemrushBot|yoozBot|lipperhey|y!j-asr|Domain Re-Animator Bot|AddThis)";
var re = new RegExp(botPattern, 'i');
var userAgent = navigator.userAgent;
if (re.test(userAgent)) {
console.log('DON'T REDIRECT TO PASSWORD PAGE');
}
Thoughts?
Don’t think it’d work with JS because the password template is loaded serverside, and there’s no content available no matter how much JS you throw at it.
The opposite could work depending on your use case, ie make site live to bots, hide with JS to non bots (ie real users) with an interstitial or blur+popup or similar.
Bummer, ok fair enough. I had a good search through the files yesterday trying to find where the redirect is triggered and o think you just confirmed the reason I didn’t find it. Thanks
Show More
Not a bad shout about the pop up though. Might see if I can use the code for the size chart or sorting and trigger it using the Javascript user agent test
Is it possible to add a check to see if the user is logged in as admin as well as the user agent check through JavaScript?
So here is the two tests I have come up with. Can anyone help me combine them so that the overlay does not apply for admin or robots?
{% capture CFH %}{{ content_for_header }}{% endcapture %}{{ CFH }}
{% if CFH contains 'adminBarInjector' %}
{% assign isAdmin = true %}
{% endif %}
{% unless isAdmin == true %}
{% endunless %}
any help much appreciated!
Looks like you’re using the Freak Design way to test for staff/admin login state?
I would just do something like this maybe:
- Setup both isadmin (wrap some JS in liquid to define the var) and isbot as a true/false variable in JS.
- Then test for if [isadmin OR isbot] then if true change the popup div to style=“display:none;”.
Gotta point out though, it’s bit uhh “cloaky” when serving different content to bots or not, there are SEO risks attached to going down this route. Also, a bit curious what the end goal is behind this whole change, is all this necessary?
While the site is sat with a password page the bots can’t begin to index the site..
”wrap some JS in liquid to define the var” that’s the bit I’m not sure on how to accomplish.
Thanks.
I will re-evaluate, I might just make a custom homepage to be my index page for everyone other than admin, at least this way I can build a homepage that can be indexed.