I've been running my head into this problem too. I don't have a perfect solution, but I've solved my UX headache by adding a "Reset Filter" button next to the filter button (for Drawer style). It only shows up when filters are selected.
I've gone ahead an added a similar button within the filter drawer, which you can see here just below the active filter tags (this will also show up in sidebar view).
My added code for each:
Full disclaimer here: I'm not a coder and am teaching myself basics through a lot of forums to figure this out. If you have a more elegant solution, please share!