Possible suspicious Bot activity?

MagazineCanteen
Tourist
11 0 1

Hello.... Does anyone have any idea of what a recent visit to our site may be? 

We usually get 3 simultaneous visits from the USA as a google bot does a crawl on our site, daily. 

 

But a strange one this morning. can anyone shed some light on it? We've had about 100 visits over the space of a few hours. Each visit seems to be working its way alphabetically through our inventory, page by page totalling around 100 product pages each visit. With a break between each session of about 10 minutes. 

 

Usually I would have thought a BOT would just visit, collect and leave. 

Not work through page after page. 

 

We've an inventory of magazine data totally 40-50k products. Its usual that people visit daily to collect information from one year of Vogue for example but never on this scale?

 

The visits are from various IP addresses in Nuremberg, Bavaria and Frankfurt Germany. Their IP address usually states DWL BACKBONE NETWORK BLOCK INCLUDES WAN AND LOOP

https://www.google.com/#q=DWL-Back%20Bone%20Network%20Block%20includes%20WAN%20and%20Loop

 

Is this something to be worried about or could it be part of the google bot network?

 

Thank you in advance...

 

 

Replies 2 (2)
MagazineCanteen
Tourist
11 0 1

Dear Sonia. Wow. Thank you for taking the time to offer a very informative reply. 

You've helped...Its still going on, though I've made my site password protected while I look into this. The fact that the visits are still in their hundreds assures me its not an actual person, who would give up after 100 password pages but it must be a bot. 

The cynic in me, is suspecting that it might be as you say, scraping for data.... the reason I would imagine is that I'm 6 years into creating one of the most comprehensive data references for 1000s of magazines. Of course, this is what makes sales but the information is valuable... I think it could be someone has found a bot to crawl this information on their behalf. 

 

I don't quite understand the steps you've suggested to fix this but will spend the morning investigating and also seek help from those that might. 

 

Sincerely grateful. 

MagazineCanteen
Tourist
11 0 1

This has continued for around 8 hours now. The bot crawler from Germany is using multiple IP addresses and different server hosts. Usually in batches of 100 magazines at a time, every 5 minutes. 

Ive made the rubbish call of closing my store. They are now reaching 100s of pages of password-needed page only. 

 

I have researched tot he best of my knowledge the liquid code required as mentioned by Sonia for the Robot Text. I can't tinker with this myself for fear of ruining the site. 

 

Does anyone have a page of code that would be suitable enter, in a FIVERR/Buy Me a coffee kind of way? 

 

I obviously don't want Google to be blocked from searching my site, so I don't know how practical it would be to block bots scraping all info... Unless to only allow the SEO segment, 

 

Any ideas guys? Sincerely.