Doing some investigation I've found that if you use

Filter.then(RoutingHttpHandler)

it will apply the filter only on the found routes (not the 404 resulting from a route not found). Is that how you're doing it?