@ursus What you said is virtually same as saying you need a library optimized with the vector API to handle your tasks. Since the vector API is not final (will not for a relatively long time due to the lack of value objects), not so many libraries out there are using it. I only see Apache Lucene using it quite actively internally.
I'm using it in my project, and it's, as you said, a bit verbose, but worth leveraging. In my experience, it's usual to require the vector API only in the core, small areas in your code.
If you don't want to bother with writing code with the vector API and what you need is something like data frame API, then you should wait a bit more until the major libraries leverage the vector API in their implementation.