Does KotlinDL offer a pathway to doing Vector Embe...

# kotlindl

[darkmoon_uk]

Does KotlinDL offer a pathway to doing Vector Embedding inversion within Kotlin ecosystem, as described by this python tool? https://github.com/vec2text/vec2text

[darkmoon_uk]

Context: Automatic extraction of tags from vector-cluster centroid embeddings. I know a typical approach is to take the closest n original text records from to the centroid then get an LLM to summarise that, but I'm interested in the atypical approach 😜

[Michal Harakal]

If I understood your question correctly, for inversion you need a Transformer decoder, and as far as I know, this is not directly supported by KotlinDL.
I didn’t evaluate the possibilities with the ONNX Runtime, which KotlinDL supports.
I would be interested to learn what your motivation is for running such a rather tricky setup in Kotlin.

[darkmoon_uk]

It just my simple brain saying - if I want to automatically categorise things, embed them, cluster the embeddings, then take the vector-centroid of each cluster and invert that embedding back to text.

[darkmoon_uk]

I know a typical approach may be to take n closest things to the centroid then summarise their original texts with an LLM; but I fancy the embedding inversion approach to be potentially more precise.

[darkmoon_uk]

I may also be simply wrong about this, not having much experience.

[Michal Harakal]

The best way to find out is to put it to the test. Did you make any progress on that?
Setting aside your idea to improve embedding inversion, I’m still trying to understand where the Kotlin use case fits in this story.

[Michal Harakal]

Hi Chris @darkmoon_uk , I just came across this thread again and was wondering: did you ever end up finding a good solution for this?
I’m currently experimenting with a KMP-capable vec2text-style prototype in SKaiNET, a framework I’m working on, so your use case came back to mind. I’d be curious whether this is still something you’re interested in exploring.

[darkmoon_uk]

Hi @Michal Harakal; I did not but my searching lead me to understand that this is a generally lacking area in the field of LLMs, not specific to Kotlin.

[darkmoon_uk]

I suppose because of the ambiguity of how you'd reverse; but I can't help thinking that some vector arithmetic to 'fine tune' the types of reversals you expect, then embeddings to specify within that context would be somehow generalisable.

[Michal Harakal]

Yes, that matches my understanding too — the inversion itself is inherently ambiguous, which is probably why there isn’t a clean general solution.
My admittedly naive approach is to follow the engineering path and see which building blocks are actually required to move towards something workable, in KMP context. Your idea of using vector arithmetic to constrain or steer the expected reconstruction also sounds like an interesting direction to explore.
Even at this early stage, the exercise has already been useful for SKaiNET: it pushed me to add support for more advanced embedding models such as T5, and it also exposed a gap in our KV-cache implementation that we’ll address soon.
So regardless of whether embedding inversion itself turns out to be practical, exploring the problem is already helping improve the framework. I’m curious to see how far the idea can be pushed.

[darkmoon_uk]

I'd be keenly interested in any developments in this space too; my use-case was to auto-categorise financial transactions; in a way that's highly customised to each user: Cluster, centroid, then - yes - attempt to invert the centroid embedding to a description of its nearest concept.

[darkmoon_uk]

I thought perhaps by multiplying all the inputs by the embedding for 'category' first, then the eventual centroid may end up in the neighbourhood of a viable, overall categorical embedding.