My website seems to scroll down every time a variant is selected
That’s probably because the variant is linked to one of your product images, so when you select the variant, it’s trying to show the associated product image.
https://help.shopify.com/en/manual/products/product-media/add-images-variants