I am struggling with the same in Gemini and Google Search built in tool, in the end, it feels like easiest way for me to solve it will be using my own agent that will use the original API as a tool.
eg. Function calling agent, that uses searchWebTool, which is a wrapper for using original Gemini or OpenAi API with this tool directly.