https://kotlinlang.org logo
Join Slack
Powered by
Hello everyone, allow me to present my new applica...
# feed
u
Hello everyone, allow me to present my new application, it is a translator and corrector boost with AI who works completely offline, it is completely plug and play, it uses the power of your hardware, it does not need install ollama, or anything, supports the voice input and output and even image translation. In fact I am not satisfied with translators type google translation and I find myself asking myself to translate this, translate that, and I find it is a shame of spending tokens for nothing and to have a dedicated interface, so I created Edge translator ! I am really satisfied with results, under the choice between Gemma 2 or 4b and LiteRt for exceptional performances ! github.com/NucleusFramework/EdgeTranslator
🎉 5
🚀 5
K 8
s
Awesome! I'll try it! Suggestion: Also create a MSIX and push it to the Microsoft Store for visibility. People searching for something like this should find it there. I paid $19 registration fee, but now it’s completely free or charge.
👍 1
u
Thank you very much! Yes I have the intention of publishing it on the stores after 🙂 (I used my application to write this message)
👍 1
On Android, we can use the npu which makes the application very fast even on a smartphone, I have to finish some things, especially at the level of the UI but in reality it's super cool to have an application like that when we are in a foreign country.
s
Tested it on Windows. Works & feels great! Just a small nitpick. Look at the edge. 😄
u
Yes, I just corrected it :)
🚀 1
what kind of pc do you have? did the hardware acceleration work well?
s
You're crushing Ashampoo LocaLingo with this and they sell it for 12€. 😄
u
I'm testing it. 🙂
s
My machine is pretty competent; I got it this year. AMD Ryzen 9800X3D, 32 GB RAM, RTX 5070 Ti. It's more a gaming machine than an Office PC. 😄
😁 1
u
and you see your GPU which is used during the translation?
s
Ok, I'll give it a longer text. For long texts, I like to use Alice in Wonderland for testing because it's in the public domain.
alice.txt
u
you are limited to 8000 characters :)
s
It's using my GPU... and a lot of RAM. 😄
➕ 1
Translation from English to German took 42 seconds.
For local processing acceptable.
Why the 8000 character limit? Because of Gemma context window, or for performance reasons?
u
the context is of 64k but I notice that after the performances are very degraded
but it's true that the usage is not to translate entire documents, it's just to talk like I am currently 🙂
s
How do deal with longer texts? Maybe split it and translate the parts before merging it back?
u
Yes, I thought about it, maybe for a v2, for the moment it's mainly for my chat sessions
👍 1
s
I think I found a bug. I had Alice in Wonderland in my source text field... I remove it from the source text field and replaced it with a simple "Hello". It went on and again answered with the Alice in Wonderland translation. I even clicked on the "X" to clear everything. Now the chat window does no longer react and RAM stays at 29 GB.
u
I am going to verify
👍 1
s
The OCR also works great. You made a really useful tool. ❤️
🙏 1
u
look at the code you are going to laugh, I am going to nothing at all
I'm just sending this to the model
s
I guess it got overwhelmed. Yesterday I managed to toast my DeepSeek session, too. (if you saw my post in #C09222272 😄)
u
try to keep the model in memory always, I think the bug comes from there,
s
Do you start a fresh session once the user clears the chat window? (Either by clicking "x" or removing everything with select + backspace) I think you should.
u
yes obviously
s
In that case it's strange that it remembers Alice.
try to keep the model in memory always, I think the bug comes from there,
I think the "on demand" option is a good thing, because it really takes all my memory.
u
Yes, of course, it's just that I think the bug comes from here, but yes, it's essential
t
@Stefan Oltmann Just to clarify, the 42 seconds English to German translation was for the whole document, or just cut it at around 8000 characters because of the limit?
u
It's for the 800 characters, the whole document would be much longer, but once again for use like mine, i.e. for majority chat sessions, for writing emails, the translation takes a few seconds, for 3 lines (250 characters, it takes 2.5 seconds on my MacBook Air M4)
t
yeah, thats understandable, I was just wondering on the "performance" side if it was 8000 character translated in 42 seconds, or more 🙂
s
Yes, the 8000, because the editor window cut it off.
u
no, even with an RTX 5090 that doesn't seem possible to me
s
In version 2 we'll see Elie cutting text in paragraphs, translate the parts and merging them back to a big document.
😅 1
🤣 1
no, even with an RTX 5090 that doesn't seem possible to me
Is 42 seconds too long for my RTX 5070?
u
or add an Ollama cloud backend, because even with the free formula for translation, it would be enough for almost unlimited use 🙂
42 seconds seems very correct for 8000 characters, after I imagine that the litert drivers are not yet optimized for Nvidia graphics cards, it looks like it's much more optimized for Intel chips and Apple Silicon
s
Ran on CPU. Took 295 seconds.
I'm not good at math, but I guess GPU is faster.
😂 2
u
in any case, the GraalVM native compilation works even for apps of this kind, it's cool 🙂
Personally, I don't like leaving my apps open; for me, it's very important that they open instantly.
m
If I have to translate a longer text section by section, how can I then ensure the consistency of the translation? That affects both the choice of the context and the linguistic level and the style.
u
You cannot, that is why it is not at all the primary goal of this application, and I don't know if it is worth the trouble, but in reality your question is about any llm, no llm can take an entire book in context
s
Depends on the book and context. 😄 1 million context limit is a thing 😉
But not for local, that's true.
u
even 1 m of context is not all sufficient, you fill them with a long document
s
Mind that Alice in Wonderland is also a book. 😉
It's around 150k characters. Tokens consist of multiple characters. So maybe 100k tokens? Should fit into some contexts.
u
yes, but the more you increase the context, the more it hallucinates
s
True.
m
Yes, but that context can be reduced dramatically by removing all the unimportant filler words.
Also, when I use the correction feature, then I would like to be able to see what was actually corrected. (highlighting the difference)
👍 2
d
@אליהו הדס did you activate Multi-Token Prediction in LiteRT-LM? That should speed it up at least twice.
m
@אליהו הדס this is really interesting. I’m curious what the biggest pain points have been building the local-LLM part of the app — especially around LiteRT-LM, KMP integration, startup time, memory and model execution. I’m one of the SKaiNET developers, an open-source Kotlin/KMP AI/ML stack, and I’m seeing more interest in a new class of smart apps where AI/ML becomes part of the application architecture itself — local inference, embeddings, smaller models, etc. — rather than something hidden behind a backend API. I’d be very interested in your experience: what works well today, where you’re hitting friction, and what you wish were easier. Happy to discuss here or separately if useful.
s
Long-term, in terms of privacy and availability, local LLMs are the way to go. I hate sending my data to the cloud and only do it because, for certain tasks like coding agents, my machine isn't powerful enough (yet). What can be done locally should be done locally.
👍 1
If that fkn Nvidia H200 would cost 2k € instead of 40k+ € I'd already have one.
m
@Stefan Oltmann Exactly. I think we’re seeing a broader paradigm shift toward local / edge AI. Not necessarily “run the next frontier model on every developer machine,” but increasingly capable smaller models combined with much stronger local hardware. We’re already seeing chips and boards with dedicated NPUs aimed specifically at generative/on-device AI ( Synaptics’ Coralboard or Arduino VENTUNO Q), while small LLMs are getting surprisingly capable at the same time. That combination is what I find really interesting: suddenly a lot more AI/ML functionality can become part of the application itself instead of being a remote service. And that’s also why I’m very interested in the pain points developers hit when actually building these kinds of apps(runtime, memory, model formats, hardware acceleration, KMP integration, etc). And yes, a €2k H200 would solve quite a few “local inference” problems 😄
👍 1
u
@אליהו הדס did you activate Multi-Token Prediction in LiteRT-LM? That should speed it up at least twice.
I've already had a lot of trouble making this stable, especially on Nvidia chips; I even had to use an old version 0.14 because of github.com/google-ai-edge/LiteRT-LM/issues/3230#…. I'm a bit afraid to enable experimental flags, have you tried them? Is it stable? For now, I find it fast enough for me, but if it's stable, it's worth it yes.
❤️ 1
but I should probably leave the possibility of activating it
👍 1
d
Yeah, I guess stability is hardware-dependent. Though I believe MTP is low risk.
u
on 8000 characters on my MacBook Air M4, I went from 230s to 170s, so yes it's interesting, I need to test on my nvidia card
👍 1
m
@אליהו הדס Your app is absolutely beautiful! You have a new PR with a small fix 🙂
u
Thank you very much, sorry for not having time to reply to you again, I am writing my response (and translating it with my app hehe)
Basically, I don't like material on desktop that much, but yes, I am happy with the result
> @אליהו הדס this is really interesting. I’m curious what the biggest pain points have been building the local-LLM part of the app — especially around LiteRT-LM, KMP integration, startup time, memory and model execution. > I’m one of the SKaiNET developers, an open-source Kotlin/KMP AI/ML stack, and I’m seeing more interest in a new class of smart apps where AI/ML becomes part of the application architecture itself — local inference, embeddings, smaller models, etc. — rather than something hidden behind a backend API. > I’d be very interested in your experience: what works well today, where you’re hitting friction, and what you wish were easier. Happy to discuss here or separately if useful. @Michal Harakal To start answering you, apart from the teething bugs of litert, it's just a lib to integrate into your app, and after that, it's as simple to use as a standard LLM api; I just send a prompt and stream the response, no agentic stuff, nothing complex in my app. I think currently we can't do much with local LLMs, their context is too small, and the power is too limited, but I think it's going to change drastically as soon as we can easily use the NPU, we'll be able to do some very nice things, and yes, I loved @Stefan Oltmann phrase that everything that can be done locally must be done locally.
❤️ 2
👍 1
d
I had to add "/usr/lib64/libOpenCL.so1" to support Fedora. Would you like a PR? (Shouldn't you use 
pkg-config
btw rather than hardcoded directories?) Also requiring 
patchelf
is a bit non-standard... 😊 The app is great anyway!! Love it!
❤️ 1
I have some of the libLiteRt libraries that are not found though. Have you seen that in the latest versions they finally provided versioned binaries for all platforms?
Also some progress indicator is needed for slow machines.
u
I had to add "/usr/lib64/libOpenCL.so1" to support Fedora. Would you like a PR? (Shouldn't you use 
pkg-config
btw rather than hardcoded directories?) Also requiring 
patchelf
is a bit non-standard... 😊
Yes, for now I haven't taken care of Fedora at all and I accepted what my agent gave me so that it works easily 🙂 , but I will gladly accept a PR
👍 1
I have some of the libLiteRt libraries that are not found though. Have you seen that in the latest versions they finally provided versioned binaries for all platforms?
I am stuck with 0.14.0 because of github.com/google-ai-edge/LiteRT-LM/issues/3230#…, but if it unblocks Fedora easily, we can only use 0.14.0 on Windows; it's probably the best thing to do.
> Also some progress indicator is needed for slow machines. What do you mean?
d
I will try the newer versions to see if it works for me.
👍 1
>> Also some progress indicator is needed for slow machines. >> > What do you mean? > Well, when using CPU on my old 2015 MBP 13", the time to first token is a few seconds... 😆 So it is difficult to know whether the app is processing.
u
Does hardware acceleration work on your MBP?
d
and does the graphical acceleration work on your Intel MBP?
Nope, because of the missing 
libLiteRtTopKOpenClSampler.so
library.
u
wow on Windows NVIDIA without MTP 41s and with 19s
❤️ 1
I reboot on Linux and I'm testing
s
The 41s matches my number.
u
oh yes, I published an update with the mtp
🚀 1
s
Your app made me curious about what else my PC could do, and that’s how I discovered Voicebox. Qwen3-TTS 1.7B voice cloning is amazing. Right now, I’m generating audio in my own voice for all the classic German fairy tales, so my kids can listen to them.
🤯 1
That teaches me two things: 1. Being a professional audiobook narrator is an even more endangered job than being a software developer. 2. If I get a call from my mom and the number is suppressed, it could absolutely be a scam. For anything important, I should insist on meeting in person.
Can I upload my own voice for the translation TTS? 😄
OK, the best audiobook narrators, who emphasize their reading to match what’s happening in the text, will stay a bit longer.
u
Can I upload my own voice for the translation TTS?
that would be a very funny feature
s
I don't know which format Voicebox uses. Is there a standard for a voice?
u
I tested on Linux, 70s without MTP, 40s with it, so the drivers are much less optimized without surprise.
I don't know which format Voicebox uses.
Is there a standard for a voice?
I don't know
s
I needed to upload a sample 30 second audio and needed to provide the text. The things it creates are a match.
Now I'm thinking about creating tutorial videos for my tools with my own voice.
I'm not good at live recording audio, so this is way better.
4 Views
Open in Slack
PreviousNext
Powered by