Topical: what would it take to have KotlinDL drive the inference for Metas recently released LLaMA language model? ...and thereby integrate its abilities into Kotlin Apps. We can observe that the pytorch based source for llama.cpp and vanilla-llama is not so huge, can we expect to achieve similar via KotlinDL ONNX integration? Or can someone tell me I'm barking up the wrong tree?
03/22/2023, 11:21 AM
Hi, I integrate ONNX in londogard-nlp-toolkit
This allows to run all ONNX models directly from huggingface hub 🙂
KotlinDL doesn’t support NLP last I checked
03/22/2023, 12:13 PM
Thanks; how does that work, are there a well-known set of NLP API's that ONNX (and other frameworks like PyTorch) supports and KotlinDL just doesn't expose? Any idea how big would the work to support be? Looks like the feature request is here
03/22/2023, 12:23 PM
Not sure, over all DJL is a cool library to work with to use models for all tasks.
But it's java-first
KotlinDL wraps tensorflow to build/inference models and onnx to inference.
Currently only vision is supported natively.
That's all I know unfortunately
But see my code in Londogard-nlp-toolkit to understand how big of a task it is with DJL -- pretty simple