So Managed to find consistently where the address is showing up.
When you go to the final payment step you get the option to change your billing address. If you click change here then the addresses on file are offered.
Not really what we intended.
Small chance this is due to earlier testing having test addresses used and shopify is being to smart about remembering these. I’ll try another fresh user incognito and see what I get.