Hello, is there a way to get a count of the tokens...

# koog-agentic-framework

[Max]

Hello, is there a way to get a count of the tokens used and cost?

[TanVD]

As far as I see, not right now, but we for sure will add it

[TanVD]

We have an internal prototype that do it @Vadim Briliantov

[Vadim Briliantov]

Hi! It is already planned and actually in progress 🙂
Thanks for your request, also there was an issue for that on GH: https://github.com/JetBrains/koog/issues/145

[Max]

Thanks guys, I really like the library, I will keep an eye out 👍

[Vadim Briliantov]

Hi! The PR is merged now: https://github.com/JetBrains/koog/pull/184

You can use latestTokenUsage inside the prompt (or metaInfo from the Message.Response) — that’s what you get from LLMs. Example:

edge(nodeExecuteToolMultiple forwardTo nodeCompressHistory
  onCondition { llm.readSession { prompt.latestTokenUsage > MAX_TKNS } })

Also there is now a MessageTokenizer plugin for estimating more fine-grained token counts (not only for LLM responses and whole prompts before them, but also for each single user message):

install(MessageTokenizer) {
  tokenizer = ... // ex: `TiktokenEncoder` or your own Tokenizer impl
  enableCaching = false // by default -- `true`
}

and then you can use tokenizer extension from inside nodes:

node { 
 llm.readSession {
    tokenizer.tokenCountFor(prompt)  // for whole prompt
    tokenizer.tokenCountFor(prompt.messages[12])   // for single message
 }
}

[Max]

That's excellent, great work guys 👏