https://kotlinlang.org logo
Join Slack
Powered by
Hey everyone! Looking for recommendations on local...
# koog-agentic-framework
t
Hey everyone! Looking for recommendations on local models that actually have reliable tool-calling support when running through Ollama. I’ve been testing Deepseek R1 but keep running into issues https://github.com/ollama/ollama/issues/8517 Thanks in advance!
d
I personally use Qwen 3 Instruct (A3A 30B) with a lot of success for calling tools. (All the Qwen 3 models for that matter, VL models included.) Otherwise, if I need a smaller model, I use Tool Ace 2 (a Llama 3.1 8B fine-tune for tools). As with lots of Llama 3.1 models, you will however need some response post-processors at the executor level, because they often produce Python-formatted or lenient JSON tool calls.
🙌 1
t
Thanks for your suggestion.
Both Qwen3 Instruct and ToolAce 2 can be run using the Ollama client, right? Or do they require any special configuration?
👍 1
s
https://ollama.com/search?c=tools Here are all the Ollama models from their site filtered by tool support. Pick your preferred parameters, size, and quantization.
However, I’d actually recommend using LM Studio. It uses llama.cpp as the backend, which is much faster if you have an Nvidia GPU, and even CPU-only inference is noticeably quicker. It also has a much wider selection of models, since it pulls them directly from Hugging Face. LM Studio also shows which models support tool calling. I usually go with a Qwen3 Thinking or Instruct model, since their latest ones are all trained for tool use, but I like to keep a Mistral model around as well.
thank you color 1
d
@tung for ToolAce 2, you have to pull it from HuggingFace directly through Ollama. For example:
Copy code
ollama pull <http://hf.co/mradermacher/ToolACE-2.5-Llama-3.1-8B-i1-GGUF:Q4_K_M|hf.co/mradermacher/ToolACE-2.5-Llama-3.1-8B-i1-GGUF:Q4_K_M>
thank you color 1
s
The GPT-OSS 20b/120b model is also quite good, though it depends on the task, of course.
v
GPT OSS is compatibly good, also @Antonii Belyshev is preparing a PR with a mechanism for fixing weak model responses containing tool calls in text
👍 1
s
Oh, that’s great news! I wrote something like that myself, a tool call interceptor to extract tool calls from mixed LLM responses.
thank you color 1
t
Thanks for your suggestion.kodee greetings
Hello, I’ve been trying to use the 
gpt-oss-20b
model through OpenRouter, but I’m encountering a serialization issue when calling tools that have no parameters. I get the following exception:
Copy code
Exception in thread "main" io.ktor.serialization.JsonConvertException: Illegal input: Field 'arguments' is required for type with serial name 'ai.koog.prompt.executor.clients.openrouter.OpenRouterFunction', but it was missing at path: $.choices[0].message.toolCalls[0].function
I saw that https://github.com/JetBrains/koog/issues/489 was supposed to fix this problem, but the fix doesn’t seem to work in my case. Here is the code for my tools:
Copy code
class GetDashboardsTool(private val dashboardUseCase: IGetDashboard) :
    Tool<Unit, DashboardsResultConvert>() {
    override val argsSerializer: KSerializer<Unit>
        get() = Unit.serializer()
    override val description: String
        get() = "provide all dashboards belong to account"
    override val resultSerializer: KSerializer<DashboardsResultConvert>
        get() = DashboardsResultConvert.serializer()

    override suspend fun execute(args: Unit): DashboardsResultConvert {
        return kotlin.runCatching {
            dashboardUseCase.invoke().getResultOrNull()
        }.fold(onFailure = {
            DashboardsResultConvert(error = it.message.orEmpty(), listOf())
        }, onSuccess = {
            DashboardsResultConvert(dashboard = it?.dashboard ?: listOf(), error = "")
        })
    }
}
model config:
Copy code
object GptOss20BModel : LLModelDefinitions {
    public val GptOssModel: LLModel = LLModel(
        provider = LLMProvider.OpenRouter,
        id = "openai/gpt-oss-20b:free",
        capabilities = listOf(
            LLMCapability.Temperature,
            LLMCapability.Speculation,
            LLMCapability.Tools,
            LLMCapability.Completion,
        ) + listOf(
            LLMCapability.Schema.JSON.Standard,
            LLMCapability.ToolChoice
        ),
        contextLength = 131_072,
        maxOutputTokens = 32_768,
    )
}
Please correct me if i have any miss take. Koog version: 0.5.3
s
connecting to this model from OpenRouter might not work because GPT models use their own tool call format called Harmony. If the OpenRouter provider doesn’t map this format to a standardized JSON format, it won’t work with Koog(need to be verified). I tested this model with LM Studio, and LM Studio specifically maps tool calls for this model to JSON. I’m not sure if it works the same way for ollama, since I only use LM Studio.
Also, you need to verify whether your selected provider on OpenRouter supports tool calls for this model. And even if they do, I’m not sure if they indicate whether they convert Harmony tool calls to the regular JSON format. Perhaps Koog could be doing this at some point
If you already use OpenRouter, I recommend this model. It worked the same for me in Ollama and LM Studio. I don’t remember for sure, but I think I also tried it on OpenRouter, and tool calls were working.
thank you color 1
t
@Sergey Y. Thanks for your feedback. I've tried this model with other tools that have parameters, and it works well. I'm not sure, but I think the issue may be that the model doesn't parse an empty parameter into an empty JSON object
@Sergey Y. Can you share your hardware setup needed to run gptoss20 locally?
s
I’d say 24+ GB RAM and a GPU with 12+ GB VRAM, ideally 16–24 GB to handle a decent context window.
thank you color 1
My PC has lots of RAM but only 12 GB of GPU memory, so I keep the context cache on the GPU and the model weights on the CPU.
a
@tung, the error about 
arguments
field likely means that the model returned a json like
Copy code
{ tool: "tool_name" }
without "arguments" field at all. However, to call a tool with no arguments, the model is expected to return a json similar to
Copy code
{ tool: "tool_name", arguments: {} }
with an empty arguments parameter. I think, it could be possible to catch and fix such errors. I am working to add a general component for fixing json errors in tool calls, and I will consider including a specific fix for this particular case
thank you color 2
kodee loving 1
t
Hi @Antonii Belyshev I got a crash while parsing the error response from OpenRouter. I think can catch it. log:
Copy code
Exception during executing subgraph 'clarifyUserQuestion': Unexpected JSON token at offset 741: Expected beginning of the string, but got { at path: $.choices[0].error.metadata['raw'] (Ask Gemini)
JSON input: .....,"code":502,"metadata":{"raw":{"message":"Hyperfusion.Servic.....kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 741: Expected beginning of the string, but got { at path: $.choices[0].error.metadata['raw']
JSON input: .....,"code":502,"metadata":{"raw":{"message":"Hyperfusion.Servic.....
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:587)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:585)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenient(AbstractJsonLexer.kt:453)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenientNotNull(AbstractJsonLexer.kt:425)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeString(StreamingJsonDecoder.kt:338)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:131)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:127)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:168)
	at kotlinx.serialization.encoding.CompositeDecoder.decodeSerializableElement$default(Decoding.kt:539)
	at kotlinx.serialization.internal.MapLikeSerializer.readElement(CollectionSerializers.kt:111)
	at kotlinx.serialization.internal.MapLikeSerializer.readElement(CollectionSerializers.kt:84)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:78)
	at ai.koog.prompt.executor.clients.openrouter.models.ErrorResponse$$serializer.deserialize(OpenRouterChatCompletion.kt:145)
	at ai.koog.prompt.executor.clients.openrouter.models.ErrorResponse$$serializer.deserialize(OpenRouterChatCompletion.kt:145)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:78)
	at ai.koog.prompt.executor.clients.openrouter.models.OpenRouterChoice$$serializer.deserialize(OpenRouterChatCompletion.kt:95)
	at ai.koog.prompt.executor.clients.openrouter.models.OpenRouterChoice$$serializer.deserialize(OpenRouterChatCompletion.kt:95)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
2025-12-10 19:52:23.489  9253-9253  a.k.a.c.a....ntSubgraph              E  	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:168) (Ask Gemini)
	at kotlinx.serialization.encoding.CompositeDecoder.decodeSerializableElement$default(Decoding.kt:539)
	at kotlinx.serialization.internal.CollectionLikeSerializer.readElement(CollectionSerializers.kt:80)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:168)
	at ai.koog.prompt.executor.clients.openrouter.models.OpenRouterChatCompletionResponse$$serializer.deserialize(OpenRouterChatCompletion.kt:156)
	at ai.koog.prompt.executor.clients.openrouter.models.OpenRouterChatCompletionResponse$$serializer.deserialize(OpenRouterChatCompletion.kt:156)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.json.Json.decodeFromString(Json.kt:149)
	at ai.koog.prompt.executor.clients.openrouter.OpenRouterLLMClient.decodeResponse(OpenRouterLLMClient.kt:181)
	at ai.koog.prompt.executor.clients.openrouter.OpenRouterLLMClient.decodeResponse(OpenRouterLLMClient.kt:51)
	at ai.koog.prompt.executor.clients.openai.base.AbstractOpenAILLMClient.getResponse(AbstractOpenAILLMClient.kt:252)
	at ai.koog.prompt.executor.clients.openai.base.AbstractOpenAILLMClient.access$getResponse(AbstractOpenAILLMClient.kt:89)
	at ai.koog.prompt.executor.clients.openai.base.AbstractOpenAILLMClient$getResponse$1.invokeSuspend(Unknown Source:15)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at android.os.Handler.handleCallback(Handler.java:995)
	at android.os.Handler.dispatchMessage(Handler.java:103)
	at android.os.Looper.loopOnce(Looper.java:248)
	at android.os.Looper.loop(Looper.java:338)
	at android.app.ActivityThread.main(ActivityThread.java:9067)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:593)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:932)
response:
Copy code
{
  "id": "gen-1765372428-79CQvojxQA9JEwFK3HhT",
  "provider": "NCompass",
  "model": "qwen/qwen3-30b-a3b",
  "object": "chat.completion",
  "created": 1765372428,
  "choices": [
    {
      "error": {
        "message": "Upstream error from NCompass: Hyperfusion.ServiceUnavailableError: litellm.MidStreamFallbackError: Hyperfusion.APIConnectionError: APIConnectionError: OpenAIException - list index out of range. Received Model Group=qwen/qwen3-30b-a3b\nAvailable Model Group Fallbacks=None",
        "code": 502,
        "metadata": {
          "raw": {
            "message": "Hyperfusion.ServiceUnavailableError: litellm.MidStreamFallbackError: Hyperfusion.APIConnectionError: APIConnectionError: OpenAIException - list index out of range. Received Model Group=qwen/qwen3-30b-a3b\nAvailable Model Group Fallbacks=None",
            "type": null,
            "param": null,
            "code": "503"
          },
          "provider_name": "NCompass"
        }
      },
      "logprobs": null,
      "finish_reason": null,
      "native_finish_reason": null,
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "refusal": null,
        "reasoning": "\nOkay, the user said \"hi, give me all dashboards.\"....",
        "reasoning_details": [
          {
            "format": "unknown",
            "index": 0,
            "type": "reasoning.text",
            "text": "\nOkay, the user said \"hi, give me all dashboards.\" ...."
          }
        ]
      }
    }
  ],
  "usage": {
    "prompt_tokens": 582,
    "completion_tokens": 535,
    "total_tokens": 1117,
    "cost": 0.00019636,
    "is_byok": false,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0,
      "video_tokens": 0
    },
    "cost_details": {
      "upstream_inference_cost": null,
      "upstream_inference_prompt_cost": 0.00004656,
      "upstream_inference_completions_cost": 0.0001498
    },
    "completion_tokens_details": {
      "reasoning_tokens": 535,
      "image_tokens": 0
    }
  }
}
138 Views
Open in Slack
PreviousNext
Powered by