Solved! Go to the solution
Can't say I've tried, I think it's there for accessibility reasons so I've left it alone. But this article looks like it might shed some light on how to remove it https://stackoverflow.com/questions/20340138/remove-blue-border-from-css-custom-styled-button-in-chr...
I know that's on about Chrome but looks like the same thing.