My team is heavily exploring local models for deve...

# ai

[Anonymike]

My team is heavily exploring local models for development currently and having pretty decent luck with the Qwen Coder models, but we're hoping for more models with localized focus towards the languages we use, like Kotlin. I see Jetbrains has a Hugging Face repository with some small models trained specifically for Kotlin, but it appears somewhat neglected and I'm hoping someone here can shed any light on Jetbrains interest in coding models or training a model focused less on Python and more on Kotlin, which we use for 95% of our work. Would love to get your thoughts in the thread!
https://huggingface.co/collections/JetBrains/mellum

[Pavel Gorgulov]

Hi!
Thanks for bringing this up, this is really interesting.
I don't know yet what the plans are for future Mellum releases, but I'd also be curious to learn more about what you're looking for.
There are kotlin related datasets available on hf as well and benchmarks on dpai (github). Have you considered fine-tuning models yourselves?
Also, when you say local models, are you mainly looking at models that can run on local machines, around 8b, or are larger models also an option for you?

It would also be great to hear more about your main use cases

[Anonymike]

Hi @Pavel Gorgulov and thanks for the response. TLDR; Yes to all of that lol.
We're ultimately looking to self-host our primary AI infrastructure and tools. We already build models for things like face verification, assisting background screeners with quality assurance on background checks, document recognition and extraction, etc, so we're familiar with highly functional models. As we've studied agent workflows and interaction, we believe the core issues are mainly:
1. Trying to do too much with too little, ultimately creating a lot of mistakes along the way.
2. Abusing the use of one large model when often we have a thorough context where we could distribute tasks out to different models based on purpose (code (kotlin/rust/go) vs language dissection vs specific decisioning). Skills help, but they stack up in context very quickly and usually only a subset are needed for a primary task (ex: kotlin coding vs research tasks). Koog handles this well with subgraphs where we can disallow a code agent from randomly searching the web. We'd rather offer the code agent tools to dissect kotlin projects and gradle builds with better use of context and far less repetitive iteration for common needs.
Regarding our experience and use cases:
1. We already host up to 30b parameter models for our team and some locally as well. We've had the best luck with Qwen Coder Next. We've also tested many different harnesses and are looking to build our own (likely with Koog).
a. Most existing harnesses are too generalized to efficiently use the model to accomplish a task. For example: If we ask an agent to review a project and do something, it will have to go down a list of 10 languages / test commands to determine that the project is a kotlin project using gradle. On a local model this is an easy efficiency win if fixed, but on a remote model that's a ton of tokens eaten for information a simple tool could provide on init.
2. We're open to fine-tuning language models, but pretty new to that task, so even a good example would go along way next to a project like Mellum or even just a suggested path from JB's experience.
3. We are privacy focused and while we already do a lot with AI, we find even 3rd party providers make many mistakes, due very little to manage context vs spend more tokens, and send way more information than we'd like to other parties. We currently restrict 3rd party tools (even hosted models through Junie) to projects that we would be willing to open-source or isolated components.
4. Regarding benchmarks, I'd personally like to see more language/application category related benchmarks on open models.
Does that help? I know its a lot, but I'd love to continue this conversation and turn it into something meaningful. Mellum is kind of a hint at it, but has JB considered an open, Jetbrains created model, where we can subscribe to fund training and the team making a better open software development model, with the ability for us to self-host/extend upon?

Feel free to DM me as well!