I'm trying to understand the intended pattern for ...

# koog-agentic-framework

[darkmoon_uk]

I'm trying to understand the intended pattern for sending structured messages with attachments through an Agent, as part of a chat application that allows attachments.
The agent.run(string) API is confusing me as both: a) prominent in the Koog API, but b) restricting access to Koog features 🤔

Use Case:
• User sends a message like "Here's my receipt" with a PDF attachment
• LLM needs to 'see' both the text and the attachment
• Koog's ChatMemory should still handle conversation history
What I've tried:

// Currently using agent.run(String) which only takes text
val response = agent.run("Here's my receipt", sessionId)

// This creates Message.User with ContentPart.Text only

Questions:
Is there a way to run the agent with a pre-constructed Message.User that includes ContentPart.Attachment (e.g., ContentPart.File or ContentPart.Image)?
1. If not via agent.run(), can we use createSession() + session-based APIs to build a prompt with structured user messages?
2. Alternatively, is it intentional that Koog handles only text content, and attachments should be managed separately at the application layer (hybrid approach)?
What I've observed:
• Message.User supports parts: List<ContentPart> which can include ContentPart.File, ContentPart.Image, etc.
• agent.run(String) converts the string to Message.User(parts=[ContentPart.Text(string)]) internally
• We'd need to either:
• Find a way to pass structured messages to the agent
• Use a hybrid approach where attachments stay at the app layer
Any guidance on the intended pattern would be much appreciated! 🙏

[Didier Villevalois]

Hey Chris! The agent input and output types are type parameters. You can instantiate an Agent<MyInput, MyOutput> (and for your strategy too). Or you can also do an Agent<Message.User, List<Message.Response>>, which I often do.

Does that answer your question?

[Didier Villevalois]

But I agree that having Agent<String, String> everywhere in the examples and docs can be misleading.

[El Anthony]

I had a similar problem. I don't have my computer nearby, but I remember creating a message using the DSL 'prompt' with text and an attached image, and then using an LLMExecutor (instead of an Agent) to send it to my Ollama model.

[Didier Villevalois]

@El Anthony well that's not necessary. Why did you not use a typed agent as I explained above? By bypassing directly to the executor, you lose all (well most of) the capabilities of Koog.

[El Anthony]

I'll try again. It didn't work that time, but perhaps it's for other reasons. I think "Agent.run" wasn't accepting a prompt.

[Didier Villevalois]

Agent.run is not supposed to take a full Prompt but only the information you need to add to the current prompt of the agent.

Your agent maintains a prompt. You call run at every user interaction round and pass it what you need to add to the current prompt.

[Didier Villevalois]

By popular demand, here is an example of an agent with custom input and output types. I did a very simple strategy with only one node just for the sake of demonstration, but I hope you get the gist of it.

import ai.koog.agents.core.agent.AIAgent
import ai.koog.agents.core.agent.config.AIAgentConfig
import ai.koog.agents.core.dsl.builder.node
import ai.koog.agents.core.dsl.builder.strategy
import ai.koog.prompt.dsl.prompt
import ai.koog.prompt.executor.llms.all.simpleOllamaAIExecutor
import ai.koog.prompt.executor.ollama.client.OllamaModels

data class MyCustomInput(
    val content: String,
    val imageUrl: String? = null,
)

data class MyCustomOutput(
    val content: String,
)

suspend fun main() {
    val promptExecutor = simpleOllamaAIExecutor()

    val strategy = strategy<MyCustomInput, MyCustomOutput>("my-custom-strategy") {
        val llmRequest by node<MyCustomInput, MyCustomOutput> { input ->
            llm.writeSession {
                appendPrompt {
                    user {
                        +input.content
                        input.imageUrl?.let { image(it) }
                    }
                }

                val response = requestLLM()

                MyCustomOutput(response.content)
            }
        }

        nodeStart then llmRequest then nodeFinish
    }

    val agent = AIAgent<MyCustomInput, MyCustomOutput>(
        promptExecutor = promptExecutor,
        agentConfig = AIAgentConfig(
            prompt = prompt("my-custom-agent") {
                system("You are a helpful assistant.")
            },
            model = OllamaModels.Meta.LLAMA_3_2_3B,
            maxAgentIterations = 50,
        ),
        strategy = strategy,
    )

    val output = agent.run(MyCustomInput("Hello, world!"))

    println(output.content)
}

@darkmoon_uk, @El Anthony please tell me if things are more clear with this example.

[Didier Villevalois]

@Vadim Briliantov I really think that the docs should be enhanced to cover typed inputs/outputs for agents and strategies. This thread and the next one are a sign that this is confusing people.

Absolutely all the examples in the "Agents" section of the documentation are examples of String to String agents, and there is not a single example with other types as either input or output. I think I already said that in the past, so sorry if this is tiring, but IMO String -> String is not the most common use case, even less when you consider the pervasiveness of multi-modal interactions.

I think that from the beginning of the "Basic agents" section there should be, if not a full example, at least a very explicit notice that "for simplicity, the examples in this section are examples of agents taking a simple primitive String as input and returning a simple primitive String as output" and that "this is not the general case, and Koog allows agent to receive whatever custom types they need as input and output (cf. this section of the documentation describing agents with custom input and output types)".

[Vadim Briliantov]

@Didier Villevalois thanks! That’s indeed a good point regarding docs.

Also, have you seen structuredOutputWithToolsStrategy ?

[Vadim Briliantov]

It is available out of the box:

/**
 * Defines a strategy for handling structured output with tools integration using specified configuration and execution logic.
 *
 * This strategy facilitates a structured pipeline for generating outputs using tools and large language models (LLMs),
 * enabling transformations between input, intermediate results, and structured output based on the provided configuration and execution behavior.
 *
 * @param Input The type of the input to be processed by the strategy.
 * @param Output The type of the structured output generated by the strategy.
 * @param config The configuration for structured output processing, specifying schema, providers, and optional error handling mechanisms.
 * @param transform A suspendable function that accepts the input of type `Input` and produces a string output
 *                that serves as the input for further processing in the structured output pipeline.
 */
@JvmOverloads
public inline fun <reified Input, reified Output> structuredOutputWithToolsStrategy(
    config: StructuredRequestConfig<Output>,
    fixingParser: StructureFixingParser? = null,
    parallelTools: Boolean = false,
    noinline transform: suspend AIAgentGraphContextBase.(input: Input) -> String
): AIAgentGraphStrategy<Input, Output> = strategy<Input, Output>("structured_output_with_tools_strategy") {
    val setStructuredOutput by nodeSetStructuredOutput<Input, Output>(config = config)
    val transformInput by node<Input, String> { transform(it) }
    val callLLM by nodeLLMRequestMultiple()
    val executeTools by nodeExecuteMultipleTools(parallelTools = parallelTools)
    val sendToolResult by nodeLLMSendMultipleToolResults()
    val transformToStructuredOutput by node<Message.Assistant, Output> { response ->
        llm.writeSession {
            parseResponseToStructuredResponse(response, config, fixingParser).data
        }
    }

    // Set the structured output, get the input and then call the llm
    nodeStart then setStructuredOutput then transformInput then callLLM

    // On tools
    edge(callLLM forwardTo executeTools onMultipleToolCalls { true })
    edge(executeTools forwardTo sendToolResult)

    // On assistant messages
    edge(
        callLLM forwardTo transformToStructuredOutput
            onMultipleAssistantMessages { true }
            transformed { it.single() }
    )

    // Post tool result
    edge(sendToolResult forwardTo executeTools onMultipleToolCalls { true })
    edge(
        sendToolResult forwardTo transformToStructuredOutput
            onMultipleAssistantMessages { true }
            transformed { it.first() }
    )

    // Finish
    transformToStructuredOutput then nodeFinish
}