I’ve just learned that there are apps that can configure this and it apparently works on any plan! The app I ended up using is this https://apps.shopify.com/payfy-conditional-payments and it’s working very well for me. For some reason it says “Shopify Plus features” on the app store page but we are on the Advanced plan and it works perfectly! As a bonus this app also has other features to set rules to rename payment methods and conditionally hide payment methods.
Of course I’d prefer this to be configurable without an app but this will have to do for now.