Tax Options Don't Correspond With Shipping Options

11 1 1

So I got my 2 shipping zones here: Canada and the United States

Now before, I had another zone called "Rest of the World" in which I placed a bunch of countries inside. 
I then decided I'm not going to ship there cause I've been researching VAT taxing and I honestly just don't understand how it works. So I proceeded to delete the "Rest of the World" zone. 

So I go into taxes to proceed to input the tax values in. 
But then I see here that I still have the countries that were listed in "Rest of the World" here. 
So My question is, does this really matter? Since they're not set up then technically they won't work right? Another question is: Is there a way to get rid of all these listed countries that are not included in my shipping zones? 

It'll be great if someone could answer, thank you for taking the time to read.