We were having this same issue. We found a fix by making sure that the second variant we had on each sku all matched. Below is the scenario:
We have some variant material hidden its used for an internal integration and customers don't need that to make the choice. They can always see the variant Size.
Material options are either JU, QT or PT.
Because we have hidden the material variant, if you are on the "GALLON" / "JU" variant, if you change GALLON to Quart, you will end up in the "QUART" / "JU" variant, which does not exist (you only have the combination "QUART / "QT".
Therefore we either needed to show the material visually so that people can change it to switch to a specific combination, or else you will need to make sure that the material has the same value for all variants.