I ended up biting the bullet and getting this app - https://apps.shopify.com/variant-swatch-king
Works exactly as intended and very easy to use. Setup didn’t integrate completely with theme but Devs were super fast about getting this fixed and working properly.