Hi all, just wondering if anybody has any tips/tri...

# koog-agentic-framework

[Sam]

Hi all, just wondering if anybody has any tips/tricks to provide a nice agent "frontend". For instance, agents currently only output text, however in my chatbot, I would like the agent to be able to output clickable buttons, images, videos, carousels, etc. Now these images/videos won't be generated/created by the agent, but rather they will pre-exist maybe as part of a knowledge base etc. And then based on the context and the response the agent wants to provide to the user, it will deem it necessary to include them in the response. A great example of something like this is:

https://youtu.be/ZPo8n66_6-w?t=40▾

Rather than just outputting text, I also outputs a carousel with images, item names, prices, clickable buttons, etc.

Do you reckon they are simply prompting the agent to output a structured response in this instance, then using this to construct their frontend view? So for the media it would just be the URL to said media, then the agent can include that in the structured response?

So a dumbed-down "graph" might look something like:

1. user question
2. main llm call (ask model to include any references to any media URLs or general URLs it wishes to use later
3. assistant response
4. take the assistant response and ask LLM to "categorise it" into a structured output. dynamically include media URLs, buttons/links, etc
5. use structured response to render a nice front-end view
Do you reckon this is the "best" approach, or would there been an alternate method I should be exploring instead? Perhaps something that utilise tool calling a bit more, etc

[Filip Zalewski]

I do a small amount of this. Agents return JSON with some metadata describing the output and the frontend is solely responsible for the rendering of it. We use React and have a few custom built components that get rendered based on the response.

It’s not super sophisticated, and admittedly does not parse out parts of a response and apply rendering to them, but I think I would probably lean towards having the frontend be responsible for all of what you described.

[Sam]

yeah, I guess I'm more asking about techniques to generate the metadata that the agent outputs, rather than turning said output into actual frontend code

[Sam]

do you have the metadata generation stuff included as nodes in your strategy graph?

[Filip Zalewski]

I misread the question initially :) I am interested in this topic as well. We do output per agent right now but would welcome a better approach

[micah]

I would suspect that Shopify is probably using tool calling to some extent. I would expect in this case, there is a call to some backend that the LLM can call in order to search for the relevant catalog search that then gets sent back as structured to the UI (or maybe goes through additional refinement steps)