Nice work, yes we managed it the same way, “Variant” metafields was the key, was a bit confusing at first, to put variant metafields on products with no variants, but now our colour filter is working the same as yours which is great!
I’m also using an app called “Metafields Guru” which has an excel like editor, which helps if you have lots of products, to make sure they all have a metafield value, and to check there are no anomalies.