Maybe not a question for here yet it might be usef...

# koog-agentic-framework

[maarten ha]

Maybe not a question for here yet it might be useful for many. If I need to post it somewhere else please let me know ❤️.
For a product that I am building I want to create AI agents with Koog. The challenge I put up is that the model needs to be able to run locally on the device. Since these things are new to me I do not exactly know how to test out different models and see what fits the best. Are there some guides online which helps with choosing the right model (chatgpt, claude, gemini, qwen, deepseek, IBM Granite…) without needing to study for months how to setup my own LLM? The learning of the LLM will eventually get there, I just want to make a start with a few models to test and then improve. Thank you so much

[Kris]

You can brew install ollama and then ollama pull qwen2.5:0.5b to get the model locally and start using it. It will require a machine with some memory to work properly, a mac with 128gb+ unified memory or a desktop with a 5090 always helps.

[maarten ha]

I already have a few models running locally yet I am not sure how to check if the model really meets my needs. Have no clue how to test these things

[Vadim Briliantov]

We are planning to support model selection API based on requirements.

In addition to promptExecutor.execute(prompt, tools, model) there will be something like:

promptExecutor.execute(prompt, tools) { availableModels ->
    availableModels
      .withCapabilities(Tools, ToolChoice, Vision) // ex: extension
      .filter { it.contextLength >= 100500 }
      .filter { it.lattency <= 300ms }
      .cheapest()
}

or

promptExecutor.execute(prompt, tools) { availableModels ->
    availableModels
      .filter { Moderation in it.capabilities }
      .filter { it.costPerMTokens <= 2.USD }
      .maxBy { it.contextLength }
}

etc.

or just:

promptExecutor.execute(prompt, tools) { availableModels ->
    availableModels.withCapabilities(Moderation).cheapest()
}

or even ask LLM to decide on a fly:

promptExecutor.execute(prompt, tools) { availableModels ->
    promptExecutor.execute(
        prompt = prompt("choose-llm") {
            system("You are professional LLM selection helper")
            user { markdown {
               h1("Your assignment")
               text("Help me decide which LLM is the most capable in execution of my task."
               h2("Available models")
               +"I have the following models available to execute my task:"
               bulleted {
                   availableModels.forEach {
                       item("provider: ${it.provider}, model: ${it.name}")
                   }
               }
               
               h2("Initial task:")
               text("Here is the description of the initial task that I am struggling to select the model for:")
               h3("Prompt:")
               text(prompt.toString) // TODO: format prompt history in XML
               h3("Tools:")
               text(tools.joinToString { it.toString() }) // TODO: format in JSON
            }
        },
        model = OpenAIModels.Chat.GPT_5 // or whatever oracle :)
    )
}

And similar API for AIAgent abstractions

[Vadim Briliantov]

Hope it would make it easier to decide on the model

[Vadim Briliantov]

Though, still models differ in terms of quality and reliability, which is still essential. And it is always changing and depending on the specific task. So, unfortunately, some experimentation for your particular task is essential and unavoidable.
Simply said -- just try it and see how it works for you 🙂

[maarten ha]

Thank you for the detailed answer. I’ll give it a try when I have time

[darkmoon_uk]

You certainly don't need 128GB to run a 0.5B param model.