Hi! I’m trying to set up evaluations in Langfuse ...

# koog-agentic-framework

[Jose]

Hi!
I’m trying to set up evaluations in Langfuse for my Koog agent, but it seems the spans are never populating the Input and Output fields. Could someone help me understand how I can get those fields to populate?

Wouldn't it make seen Koog populated them automatically?

[Anastasiia Zarechneva]

cc @Sergei Dubov 🙏🏻🙏🏻🙏🏻

[Jose]

FYI, I tried setting them on the agent event handlers but that didn't work, the accessed span is not valid:

PropagatedSpan{ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}}

install(EventHandler.Feature) {
    onAgentStarting { e ->
        val userInput = e.context.agentInput.toString()
        val span = Span.current()
        span.setAttribute("langfuse.trace.input", userInput)
    }
    onAgentCompleted { e ->
        val span = Span.current()
        span.setAttribute("langfuse.trace.output", e.result.toString())
        logger.info("React Strategy result: ${e.result}")
    }

[Hywel Bennett]

I believe you need to enable verbose outputs:

install(OpenTelemetry) {
                setVerbose(true)

[Jose]

i already have it enabled:

install(OpenTelemetry) {
    setVerbose(true)
    addLangfuseExporter(
        traceAttributes = listOf(
            CustomAttribute("langfuse.trace.tags", listOf("rag", "production")),
            CustomAttribute("langfuse.trace.name", "myagent"),
        )
    )
}

[Jose]

NOTE: traces do show up in Langfuse, the problem here is that input and output are never set up

[Jose]

image.png,image.png

[Sergei Dubov]

Hi @Jose. Thank you for raising the question.

The input and output from your initial image corresponds to the very specific system __finish__ node. Each node in Koog has input and output where the content of this fields is related to what is going on inside a node and inside a strategy that you have. If you transfer an empty string to the __finish__ node, than you will have an empty input for this node.

You can enable trace log level for the ai.koog.agents.core.agent.entity package to see what is the result of the node execution. This might help to understand the reason. You can also use EventHandler or Tracing features for that and intercept NodeExecutionStarting / NodeExecutionCompleted events that contains info about node input and output in the event context. Could you please check what do you receive here?

@Hywel Bennett, thank you for your suggestion about the verbose flag. It still should write you an input and the output for the messages. The different is in the content. With verbose set to false (default), you should not be able to see it, and get HIDDEN:non-empty string value instead on a real content.

[Jose]

Hi @Sergei Dubov Thank you for taking the time to answer. Here's an example of my latest node.nodeCallLLM invocation before node.finish. As you can see, input and output are null

[Jose]

the general tracing view also show those fields empty

[Jose]

image.png

[Jose]

However, during the execution of onNodeExecutionStarting the input information is present:

[Jose]

at the final onNodeExecutionCompleted execution we also get the output:

[Jose]

My guess here is that langfuse.trace.input and langfuse.trace.output are not being populated by Koog

[Jose]

and that's why they don't appear in Langfuse

https://langfuse.com/integrations/native/opentelemetry#input
https://langfuse.com/integrations/native/opentelemetry#output

[Sergei Dubov]

Hm. Thank you for details. It seems like a bug for me now. Would you mind sharing a strategy structure? I will try to reproduce this on my end.
Could you please also clarify if you have this issue for all or only particular nodes?

[Jose]

It happens for all nodes, I'm using the built-in reactStrategy

[Sam]

Hi all, this happens to me as well:

https://kotlinlang.slack.com/archives/C08SLB97W23/p1758632793546549

[Sergei Dubov]

Folks, thank you for reporting this. This is the issue on OpenTelemetry feature side in NodeSpan. I will push a PR with the fix shortly. Thank you for catching this!

[Jose]

Thank you @Sergei Dubov!!!

[Sam]

Ditto, thanks a bunch! @Sergei Dubov

[Sergei Dubov]

Created a PR with changes to add input and output into nodes.

[Sergei Dubov]

@Sam, I’ve also created a separate ticket to resolve your request as well (please see KG-502).

It will require to add an extra layer to spans for the agent. Please stay tuned. I will try to fix it shortly as well. Meanwhile, you can workaround missing langfuse.trace.input and langfuse.trace.output through the input/output for start and finish nodes (will be available from the PR above).

[Jose]

Hi @Sergei Dubov I’ve been giving a try to the new updates and your PR works like a charm! I can see the new properties added per each node. And the workaround is enough for setting up our evaluations until we have the langfuse properties. Thank you for your quick solution!!

[Sergei Dubov]

@Jose, thank you for a feedback! 👍