What's the recommended way of blocking bot spam (from a single source)?

[Matt_T2]

Hello,

Our website is getting hammered with bot spam. It all originates from Boardman, Oregon with the hostname "amazon" -- so it's clearly from Amazon's data center at that location. Thankfully, the bots aren't signing up or checking out, but their direct visits now comprise 50+% (!!!) of our daily traffic. This volume has destroyed the usefulness of our Shopify dashboard/metrics... thankfully, we can filter it out in Google Analytics.

What we're looking for is a way to block this bot spam. What is Shopify's official, recommended means of blocking bot spam, since we don't have access to server files (such as .htaccess) to do so ourselves? Are apps like TrafficGuard or Visitor Blocker the recommended method? We're hoping for a comprehensive answer that can help the community as well as ourselves.

I've seen a few threads asking similar questions, and haven't been impressed by the responses. Please note that we're not looking for advice on how to filter out this traffic in GA, nor are we gathering others' opinions on the harmfulness of bot spam on SEO or SERP rankings. "Just don't worry about it" is not a fix. We're looking for a real solution; if a teenager can fix it in five minutes with .htaccess, then a $16B-market-cap company and its intelligent community can also find a solution.

Thanks!

[ArtinCoins]

I'm afraid we are in the same boat here at Art in Coins and I'm rather disappointed that Shopify doesn't have any server side tools in place to assist.  That's what I would expect with what I pay to use this platform and I'm not at all pleased to find that my only recourse is a very expensive third party app.  (such as Traffic Guard / Visitor Blocker) 

C'mon Shopify, time to get on the ball (please) and get some tools in place to help users block unwanted traffic.

[Betty2]

Hmmm . . . no offical response.

From what I'm seeing elsewhere this started being a problem for a wider audience (not just shopify) as of June 19th.

Disappointing.

[LizSwiatlo]

I tried to do research on Shopify and found nothing. I also tried to find addresses in the Audience/Technology/Network field and all data was not set.  did find two poorly rated ip addresses from Amazon in Boardman, Oregon so I filtered them out in Google analytics. I'll let you know if it works. 

[RussW]

I've registered my site url with Cloudflare.com today and within the first 6 hours it has blocked all the bots - we had issues with john smith and the like abandoning the cart causing our stats to badly skew...it was easy enough to do, although Cloudflare are meant to have a partner deal with Shopify so we shouldn't need to do this (apparently late 2019 the set this up) and there may be some issues in using Cloudflare with a Shopify url, but I have not experienced any yet - added bonus, the site is loading 10 times faster!

When you register the url, only use the basic free service, nothing more needed - you will then need to go to your url provider (123-reg for me) and change the DNS setting to the Cloudflare settings ( they tell you what to use) - seemed simple enough 🙂

Russ

[Bikes-Angel]

I can't believe that this thread is out here since 2018 and shopify still has done nothing!   I have other sites on cloudflare so I guess I'll just put the shopify site on there too the only thing I don't like is that cloudflare then only reports their ip addresses and not the original visitors ip to your traffic logs etc.  So you'll see everything coming from them whether it's rogue (and they let it through) or not.  Have you found a way around that?

[Zoolander]

Many reputable companies use cloudflare not just bots and spammers. I did block cloudflare for a few days and found it problematic as some Shopify apps or the developers/ website use CloudFlare. So instead I now reverse search the IP address to determine which website owns it and if it's an ongoing problem, I block that IP. 

[JoeCuv]

Shopify, we've seen steady traffic from this bot every month. We have also experienced a steady decline in organic search traffic in that same period. WTH! Do something!!! This thread is 2 years old. If Amazon is stealing traffic from Shopify stores doesn't that bother you?

[grantlindburg]

This is a **bleep** joke.

[Robin_Burton]

Same problem!

[ArtinCoins]

During my investigation, I have come across the following bash script on Github that can be employed as a server side solution.  I sent the information to Shopify support to investigate and hopefully deploy for their users.  The script is for either Unix or Linux servers so hopefully compatible with Shopify.

AWS-BLOCKER
A simple bash script to block all AWS IP ranges using iptables.
https://github.com/corbanworks/aws-blocker/blob/master/aws-blocker

#!/bin/bash -e
#
# Amazon AWS blocker through iptables.
#
# First we use curl to grab the official list of ranges from Amazon. The -s
# prevents extraneous output from curl, and the -L makes it follow redirects.
#
# The ranges are passed to jq, a JSON parser. The -r makes jq output raw data
# without quotes. We only need the list of prefixes, so we discard everything
# else.

POSITION=1
FILTERS=""
JSON_URL="https://ip-ranges.amazonaws.com/ip-ranges.json"

# Get the line where the jump will be inserted at.
# Useful if you want e.g related / established rules for outgoing traffic.
if [[ -n $1 ]]; then
    POSITION=$1
    shift
fi

##
# Builds region filters based on CLI arguments
#
# Arguments: CLI arguments as passed by $*
#
function build_filters() {
    for arg in ${@:1}; do
        if [[ -n $filters ]]; then
            filters=$filters", "
        fi

        filters=$filters"select(.region | contains(\"$arg\"))"
    done

    if [[ -n $filters ]]; then
        filters=" | "$filters
    fi

    echo $filters
}

##
# Extracts IP ranges from an Amazon JSON file
#
# Arguments:
#     $1 AWS JSON content
#     $2 Prepared filter string
#     $3 Group to extract IP ranges from (e.g. prefixes)
#     $4 Object key for IP ranges (e.g ip_prefix)
#
function extract_ip_ranges() {
    local json=$1
    local filters=$2
    local array=$3
    local prefix=$4

    local group='group_by(.'$prefix')'
    local map='map({ "ip": .[0].'$prefix', "regions": map(.region) | unique, "services": map(.service) | unique })'

    local to_string='.ip + " \"" + (.regions | sort | join (", ")) + "\" \"" + (.services | sort | join (", ")) + "\""'
    local process='[ .'$array"[]$filters ] | $group | $map | .[] | $to_string"

    local ranges=$(echo "$json" | jq -r "$process" | sort -Vu)
    echo "$ranges"
}

##
# Creates the AWS iptables chain if it doesn't exist, then flushes it
#
# Arguments:
#     $1 Version to use. Omit for v4
#     $2 Position to insert chain statement at
#
function create_and_flush_chain() {
    local version=$1
    local position=$2
    local cmd=ip${version}tables

    $cmd -n --list AWS >/dev/null 2>&1 \
        || ($cmd -N AWS && $cmd -I INPUT $position -j AWS)

    $cmd -F AWS
}

##
# Adds an iptables rule for each line in ranges
#
# Arguments:
#     $1 Version to use. Omit for v4
#     $2 Prepared lines
#
function add_iptables_rules() {
    local version=$1
    local cmd=ip${version}tables
    local lines
    local data

    IFS=$'\n' lines=($2)
    unset IFS

    for line in "${lines[@]}"; do
        eval local data=($line)
        local ip=${data[0]}
        local regions=$(echo ${data[1]} | tr '[:upper:]' '[:lower:]')
        local services=$(echo ${data[2]} | tr '[:upper:]' '[:lower:]')

        $cmd -A AWS -s "$ip" -j REJECT -m comment --comment "$regions = $services"
    done
}

# Retrieve IP ranges definition
# Either from an URL or file input (e.g. "< ranges.json")
if [ ! -t 0 ]; then
    JSON=$(cat - <&0)
else
    JSON=$(curl -s -L $JSON_URL)
fi

FILTERS=$(build_filters "$*")

# IPv4
create_and_flush_chain "" $position
V4_RANGES=$(extract_ip_ranges "$JSON" "$FILTERS" "prefixes" "ip_prefix")
add_iptables_rules ""  "$V4_RANGES"

# IPv6
create_and_flush_chain 6 $position
V6_RANGES=$(extract_ip_ranges "$JSON" "$FILTERS" "ipv6_prefixes" "ipv6_prefix")
add_iptables_rules "6" "$V6_RANGES"

[Spirit_Wanderer]

That is awesome. My traffic over time was replaced by 98% bots.

[James96]

Has anyone made any progress on this?

[German_Toy_Stor]

Same problem here! Any progress on that topic?

[James96]

Doesnt appear so. Its creating malformed URL's on my so I now have a huge amount of 404's.

[German_Toy_Stor]

We actually have two shops which are affected. Several emails with Shopify, but so far they only referred to 3rd party apps. Absolutely not satisfying. 

[ArtinCoins]

Hello all,

I have been in recent email discussions with one of Shopify's network specialists and they are looking at some options to combat this issue.  One thing I am working on is compiling a list of IP blocks that this Boardman Bot is using to spam my wesbite.

What is see used most is ip address starting with:

18, 32, 34 & 54

If anyone has observed other IP block's please post them here and I'll be happy to communicate them with Shopify.  I'm happy they are taking note and looking to take action either on a case by case basis, or, if this is now widespread, a broad platform solution so we won't have to resort to expensive 3rd party apps.

Speaking of apps.  I use the following:  Back in Stock, Delerious Profit, Mailchimp, Order Printer, Product Reviews and Tawk.to Chat.  I'm taking a little poll to see if perhaps there's an app on the platform that's got a malicious backdoor to it that's directing this bot activity.  Not likely since the Boardman Bot is bombing tons of websites and not just Shopify's, but I am very curious about how this fake search traffic got turned on to my particular store.

Cheers all, light at the end of the tunnel.

[Olive_McKillop1]

Well it looks like loads of Shopify clients have been trying to get an answer to the problem of stopping spam bots from subscribing to their website with no response from Shopify.  I too have this irritation.  Shopify when are we going to get an answer from you that works.  Dont see why we should have to pay other organisations for an App that you can produce!  Angry here in Australia!

[Katlin]

Same issues here. 3/4 of my traffic is now some Amazon bot, and apparently there is no way to stop it. Thanks Shopify.

[moonspinners]

I have been having the same issue as well from about the middle of June.  Somebody needs to find an inexpensive, and preferably free, fix to this problem. Shopify help us out please!!

[shop36478]

@ArtinCoins So you spoke to them 2 years ago and they've still done nothing?

GUYS - we need to be LOUDER on this issue. make your voices heard. This is unacceptable and we need to make sure they know. SPEAK UP!!

[21stcustom]

The "boardman bot" has hit me almost 100 sessions in the last 2.5 hrs. I turned on my password protect which flushed it for now at least. Here is my GA screengrab.

[AmazonBots]

Replying because I am too facing this issue and have thought about using some of the competing platforms as one has this exact issue fixed in their pitch, because it still has real developers behind it. 

[gm412]

Same problem here from Boardman Oregon and Kansas. This should be an easy fix on server side but so far seems like Shopify is unwilling to do anything.

[Spirit_Wanderer]

I lost two online stores on Dec 26, 2019 due to BOT attacks and DDoS attacks. Since then, I have tried to make the shopify experts see the harm these things are doing to all of us. The replies I have been getting from the technical department staff are stupid ones like, it is your third party app's, these are Google bots (when in fact they are FAKE GOOGLE BOTS on the entirely wrong Ip address) or my favorite in the case of Oberlo imports that do not bring all the information of the product in the import or need 5 refreshes to get the pics in, apparently we are making mistakes when selecting data ..... and blah blah blah!

Today I received another answer that nothing is dangerous for my business, yet Cloudflare (yes we moved our stores out of Shopify's reach to cloudflare) tells me a very different story. WHAT THEB F... K, Like i'm supposed to believe in their "infaillble protection) ! NO WAY!!!!

So we opened another shop, kept it under wraps, did EVERYTHING to hide it from the bots and as soon as it was online the bots started up and then kept on coming.

So now i have 3 online stores

When nobody knows you are opening a  online Store, that you put your password protect every page, and supposedly not one human can enter your store,  Now please explain to me how you wind up with 500 visits to your site and your password gets changed for Botwaw on the page that protects your store that bots aren't harmful.

You should be able to ask questions to the "Experts" and get an intelligent answer.

I know I'm not the only one who has "bot" problems and I have proof and data that the server where is my store is located receives  AUTOMATIC bot attacks and that these attacks are called on by the Shopify server, I'm talking about the server that houses ALL of our collective stores under ONE SINGLE IP ADDRESS. Did. you know that this server is actually red flagged in IPWatson.com as containing at least 2 major malwares AND IS FLAGGED AS A VERY HIGH RISK OF HACKING OR BEING HACKED?

I ask all those who have the same problems and who are tired of trying to make Shopify understand the situation and who make are thought of themselves as being crazy. That's what happens to me and the answers I get imply that I don't know anything.

Can you also tell me how these bot attack the stores of Shopify using RUBY, LIQUID PYTHON NUMPY, PYTHON BOOLEEN.

When I ask, I get this answer "It is impossible our protection is impenetrable".

WTF !!!!!!! We have proven it isn't and nobody is listening.

Those who are really tired of this fight with SHOPIFY, email me at spiritwanderer2018@gmail.com or fashionlimonde@gmail.com

Because the police have my files and I work with them to supply them all the evidence I have.

Because according to the law I HAVE THE RIGHT TO SUE SHOPIFY BECAUSE THEY AREN'T  DOING ANYTHING AND AS FAR AS WE ARE CONCERNED THEY ARE IN BREACH OF CONTRACT. WE CAN ALSO ACCUSE THEM OF BEING RESPONSIBLE FOR THESE ATTACKS.

[bonhommie]

We've been on Shopify Advanced for almost 5 years now and never noticed any bot issues but since Feb 9, 2020, we've had hundreds of daily spy bot visits! I thought this would be an easy/quick fix and I'm shocked to see that Shopify has no way to combat/block malicious spam bots. Definitely following this thread.

[FlairTrade]

Just experienced this for the first time from Boardman, OR and Kansas. Can't believe they haven't implemented some authentication process for legit traffic. It's possible many of the APP providers use AWS so some of this maybe legit traffic but it does pollute the analytics data.

[Greenbobbin]

So we have a new website and new to shopify for about 4 weeks (great timing I know!) but over half our traffic is coming in from Boardman, Oregon. Clearly its not real traffic and we dont even sell in the US. So what are folks doing about it? Has Shopify done anything to help?

[Zoolander]

I asked Shopify about this two weeks ago and even requested the IP address which they could or would not provide. We are all spending a lot of time and / or money trying to resolve what is probably a very simple fundamental issue for Shopify to resolve or block.  If I find a solution I will post it on the Shopify For Beginners Facebook page

[sasamara]

Is an anybody familiar with this referral traffic source websitebottraffic.pw ?

[Spirit_Wanderer]

No sorry! But check for clouflare!

[Ravster]

got a shocker today when I saw something like that happen with me as well. No clute what harm it is causing and how to deal with it. Shopify, I see not much has been done about it even when shop owners have pointed this out as early as 2019. Time you really share some credible information about what this is all about.

[David_Foster]

We run into the same issues with Shopify. I actually had to send them a with out prejudice letter this morning. They appear to have a system Bias when it comes to working on there own issues. They also do not appear to understand that they have a feedback loop happening in their service work. Its their dime.

Here is a link that was forwarded to me by one of my great customers. https://developer.amazon.com/support/amazonbot

[grantlindburg]

Hey SHOPIFY,

You gonna **bleep** say something or what? Some of us have serious business we'd like to conduct.

As a developer, I'm happy to just take the money you would otherwise get and build a solution that actually **bleep** does something about this, even if I have to build it from scratch. This is trash, and you're supposed to be better than this. If you can't be? Well hey, kiss tens of thousands in revenue goodbye for all I care.

[shop36478]

Shopify - this has become completely out of control. This is super simple for you to fix on your end and you're just turning the other way and avoiding it. DO SOMETHING IMMEDIATELY or you're going to lose all of us to another platform that actually cares.

[kenlague]

Dear Shopify,  this thread underscores the need for a more comprehensive solution to block bot traffic from Shopify-hosted sites. 

Other majors have had traffic filtering capabilities in place for years.   For example, Google Analytics allows excludes of city/region/hostname.   Why aren't Shopify admins able to do the same?

A few weeks ago our store joined the ranks of others getting hit by the server farm in Boardman, Oregon among others.  Always the same bot behavior.  100 or so unique visitors every day, each for a few seconds, each about 20 seconds per user session, bouncing off the home page.  It's significant enough to throw all of our visitor reports into disarray.  

We've added the captcha challenge for Cart and account signups, so far we haven't suffered malicious attacks at this level.  I really feel for other Shopify store owners who have more serious bot attacks, I wish them every success getting this fixed.   

The visitor bots are not quite as bad as this, but still a serious problem.  The bot visits cause a loss of confidence among our staff using Shopify reports.   

Third-party apps don't seem to address the central problem of filtering bot traffic appearing in Shopify Live View and reports.

Further, we're not inclined to tinker with no-follow's and meta properties to prevent Amazon bot indexing.  This is not for the faint of heart, and would put our efforts to get indexed with legitimate crawlers at risk.

Any other suggestions to filter out bots so they don't hit the visitor traffic reports would be welcome. Meanwhile, we're hoping Shopify's army of devs can come around to this soon. 

[SarahA]

Hi there, 

I just installed Google Analytics a week ago and have discovered, to my dismay, that 30 PERCENT of my new user traffic is coming from Linux OS in Boardman, Oregon. After searching for more info, I've discovered your thread and others indicating this is AWS bot traffic. Have you gotten any response from Shopify yet or found a solution? I've set a filter in my google analytics blocking traffic from aws but I'm sure that will also filter out legitimate visitors coming through an AWS hosted platform as well.

And, that doesn't prevent them from actually trolling through my site or doing who knows what other stealing and damage, which is MORE concerning to me than just skewing my reporting. Would love to hear if you found any solutions. I want those bot F***ers blocked from even accessing anything on my website at all 😞

Thanks!

[Greenbobbin]

Hi - we didnt get any response from Shopify about this, which was
disappointing to say the least.
We are a UK based retail shop and realistically dont expect to ship outside
of our own country so was surprised to find traffic in the US. Even more
surprised to find it was coming from one town (Boardman). After researching
I found this was bot related traffic. It has died off in the last 90 days
/ month / week so I've not looked into it further.
Hope you get to find a way to block it! Good luck as you'll be doing this
yourself, without shopify help!

[dylanpierce]

I know you're looking for an in-platform solution, but unfortunately Shopify hasn't prioritized a solution for this problem yet. So I decided to step up and making something. There are other blocking apps on the marketplace but Blockade is different in that is does a reverse DNS lookup on the client's IP address.

That way if the visitor is actually a bot coming from the servers hosted by scraping sites such as pexgle.com, ecomlad.com, babaoadspy.com, or others you can block them.

In addition it has the range of popular hosting providers such as AWS, Google Cloud Services & Azure, so if these abusive scrapers host their bots away from their web app, you still have a decent chance at blocking them.

You can download it here: https://apps.shopify.com/blockade

Happy to answer any questions about blocking and strategies to do so without affecting your real traffic.

Dylan

[hanujewels]

@dylanpierce I see you reply and suggestion for blockage app, can you educate us how we can use this app to block boardman specifically.  It does not help if we are not aware of the solution, subscribing to your app requires some kind of education.  Its great that you replied, but with very little details.  Please educate us on this community

[dylanpierce]

Hi @hanujewels 

Great question and thanks for the callout, I'm happy to explain.

Blockade is different than other Shopify traffic blocking apps in that it uses reverse DNS lookups to detect the origin of the traffic.

For example, when you're visiting a website with your browser, the normal DNS system takes your website URL (example shopify.com) and converts it to an IP address to retrieve the webpage from.

In reverse DNS, the opposite is done. So Blockade uses the IP address of visitors on your store and translates them into web addresses. For your normal traffic, these reverse DNS queries resolve to their home ISP or cellular network. However if a script hosted by a website's server visits your store, then the app is able to detect that the origin is that website.

In the "Blocked Websites" portion of the app, you can define websites to block:

In addition you can block known datacenters like AWS, Google Cloud Services & Microsoft Azure. Traffic coming from these datacenters are usually scripts/bots or VPNs. I hope this answers your question. Is there anything else I can help you with?

Thanks,

Dylan

[TAGSAngel]

I believe the App suggester is unfamiliar with your problem so the reply is a little incorrect.   The bordman ORE (not com) traffic is coming from amazon aws's bot  which is generally 
some form of "compute-1.amazonaws.com"    Just open a cloudflare free account and block everything containing amazon  amazon.com  amazonaws  etc    You'll find there's many more than that one that's hitting you daily unfortunately.   fake bots that make themselves look like bingbot   and other creeps that just want to get all your stats, your keywords and keyword rich content so they can get above you on the search ranks (or want to cram up your bandwidth).   And, most of the cloud protection blocking sites are also hosting a multitude of those bots too  it's a catch 22 and a never ending whack a mole battle I'm afraid. 

[Zoolander]

Boardman, Oregon is usually Google they have a data centre there.  Took me a long time to discover this.  I have stopped worrying about it, pretty sure you can too 😊

[aainaas]

Gosh this thread seems so old and still no answers yet. My site is having same issue and I am bombarded with bots from Ashburn hitting my site and messing my Google analytics and bounce rate. I am not sure how to fix the bot problem. If anyone has any idea and were able to successfully install anything and were able to block or remove the bots please help me out. Would love to hear from those who got this fixed. 

Shopify Tech team hopefully you will see this thread and take this seriously and escalate it and help us out. 

[MarissaPL]

Did you ever get an update from @Shopify ?

[aainaas]

No @MarissaPL none. 

[GSBS-TechDev]

Just wanted to share that there is a solution that is provided by Shopify for this, though they don't support it fully (needs a bit of coding know-how)

https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt

https://developer.amazon.com/support/amazonbot

Add the code that Amazon gives to block their bots, and your good to go.  As to why this isn't done by default is a good question that Shopify should still answer, but you have the power to address it all the same.