Has anyone managed to use the yolov4 model from th...

# kotlindl

[James Yox]

Has anyone managed to use the yolov4 model from the onnx model zoo in KotlinDL? I have been trying to get it to work similar to this PR

https://github.com/Kotlin/kotlindl/pull/253

I think this PR is also pretty out of date at this point but it helped direct me a bit.

I am struggling with post processing (I think?). The model documentation contains a lot of python code that I am pretty unfamiliar with. The Kotlin code in that PR makes it look pretty simple but they didn't seem to get usable results and that's exactly where I am stuck. I am not sure if this PR is even correct. I have stuff coming out of the model but it just seems like random gibberish. Lots of negative values, etc... AFAIK I should be getting relative coordinates that are between 0 and 1.

I understand generally what I need to do. Structure the flat data into an appropriately shaped tensor (1, 52, 52, 3, 85) then parse the bounding boxes... I can create a tensor from the data of that shape but when I try to grab the info out It doesn't seem to be a valid representation of bounding boxes... I could be missing many things though since this is my first foray into trying to use anything beyond a super high level library for ML so I could be missing something very obvious.

Model Documentation: https://github.com/onnx/models/tree/main/vision/object_detection_segmentation/yolov4

[roman.belov]

сс @zaleslaw

[zaleslaw]

Dear @James Yox I feel your pain. I experimented a lot with the implementation of all these postprocessing and preprocessing code (https://github.com/onnx/models/tree/main/vision/object_detection_segmentation/yolov4), replacing NumPy ops with our Kotlin library Multik, but discovered that there were not enough operators to port all these NumPy ops.

The mentioned PR is outdated and was not finished correctly and saved for history.

I have not checked yet, but probably new YOLO models have simpler pre/post-processing code. It's easy to port to onnx the model if it has an Apache / MIT license

Unfortunately, personally, I could not finish this work due to the limitation of resources. This is why we shipped this model and waiting for help from the community.

If you are ready to propose something or implement post-processing, I'm ready to test and review, feel free to open PR. If you find another new model, we could include it in our Zoo

[James Yox]

At least I wasn't just missing something obvious 🙂. I'll take a look at some of the other yolo models. If they are easier maybe I can get somewhere faster. If not, then I guess I will have to go line by line through the python post processing and try to get Kotlin to do the same thing. Might be fun 😂

Multik is new to me! Looks interesting and very useful for post processing. Anyway if I get something working I'll be sure to spin up a PR. I really want to get the latest and greatest object recognition models working in Kotlin

[zaleslaw]

Great, we also could create issues for missed operations and highly likely it could be implemented by request

[James Yox]

As far as I can tell yolov7 does have much simpler post processing. This github repo appears to be using an onnx version of the model and it looks quite simple in comparison:

https://github.com/ibaiGorordo/ONNX-YOLOv7-Object-Detection/blob/main/yolov7/YOLOv7.py

[altavir]

@zaleslaw Just out of curiosity, which operators are missing?

[James Yox]

I'm pleased to report I was able to get YOLOv7 working in Kotlin 😄 . I have simple post processing implemented using multiK. It was a lot easier than yolov4. I am still working on making it a little nicer. For instance, I want to add non maximum suppression to filter down the bounding boxes. I think that's something that would be valuable in KotlinDL for other models as well (it seems a bit specialized to ML to be included in multiK). I'll try to get a WIP PR done this weekend that we can iterate on and serve as a discussion point for this. At the very least I think it would be a helpful example for others wanting to run onnx models.