Gradle plugins are applied to Projects, not Components, Sourcesets, Tasks, etc. It’s up to the plugin to configure itself for those, if needed, so you would probably need to dig into the plugin source code to see if there’s a way to configure it to run only for test sourcesets.
A bit of a hack would be to simply disable/skip the tasks for non-test sourcesets, though that might cause problems if it produces Outputs that other non-disabled tasks depend on.
As a last resort, you might need to move those particular tests into their own Gradle module and apply the plugin there.