I completely understand a concern you might have with site speed, but I don't think it's something to worry about too much unless it is very low! My store site speed is only 35 too. In saying that though, it definitley doesn't hurt to improve it if you can. There are some features you can control when it comes to site speed and these are the things you can look into:
There are also some other factors which you can't control such as:
There is a guide which goes into all of these in more depth here. Other than store speed, how is the rest of your store going?