https://kotlinlang.org logo
#science
Title
# science
y

Yoav Luft

09/02/2020, 7:36 AM
Hey @altavir! We’ve been talking a little bit about on discourse about DataForge
a

altavir

09/02/2020, 7:37 AM
Hi, I was just writing to you with a proposal to discuss it here.
What specific tasks do you have for remote execution? Is it filtering or something more complicated?
Just for the context, we are currently finalizing results of different work made by JBR interns this summer. The prototype allows to pass functions as parameters without interpreting the code or passing compiled bytecode. Just by the means of reverse call. I am not sure it is relevant in this case.
y

Yoav Luft

09/02/2020, 12:30 PM
Sorry for my low latency 🙂 I’m answering between assignments.
The simplest transformation is linear scaling, but I already know from users of previous iterations of this project that it is not enough. Event-wise mathematical transformation of events seems to be relevant also, as well as windowed operations (looking back at X last events, or events in last Y seconds). For more complex usages I found state machines to be very useful, but this seems to be out scope. Step functions on the other hand will probably also be useful, and maybe later we’ll want to record readings and train classifiers based on them but this is far future ATM
a

altavir

09/03/2020, 9:43 AM
But the question is whether it is required to do those transformations on-site. I will try to draw a diagram of how we going to solve it right now for devices.
y

Yoav Luft

09/03/2020, 11:32 AM
I don’t understand what you mean by “on-site”, can you clarify or give me an example?
a

altavir

09/03/2020, 3:55 PM
Sorry, a bit tied up with the beginning of a new teaching year. I will try to produce some pictures until the end of the week.
y

Yoav Luft

09/04/2020, 12:02 PM
I’ve read about it a little bit, by on-site you mean as in the physical location where the devices are? In that case then yes, it’s both on-site and “real-time”.
a

altavir

09/04/2020, 1:48 PM
OK, I finally got some time to draw pictures. Here is what you are trying to do. Stateful device controller with additional channel for configuration. The communication between the device and controller is on-site and therfore super-fast. Still, you need to create a custom communication channel for each device.
Now this is the general scheme we are currently aiming at. The idea is that we have scaleable central event bus and each device, analyzer and user has an input stream and an output stream. So when event leaves the device, it goes to analyzer which has custom logic and could be replaced as a whole, and the analyzer sends the event to the user. This way, we won't get real-time analyzer data, it will have high throughput, but higher latency, but it is stateless and scaleable.
The main question is how real time is real time and what are the requirements.
y

Yoav Luft

09/05/2020, 10:31 AM
ok, I see. So we’re working for tangible interactions, meaning that our latency needs to be good enough such that users won’t feel it. I don’t have exact numbers and I guess it might be different from device to device (our auditory system, for example, is more sensitive to varied latency), but anything under 50ms will be good, and under 20ms will be f*ing amazing. For some interactions higher latency might not be a problem but probably not much higher then 100ms.
a

altavir

09/05/2020, 10:33 AM
Well, the standard network latency is about 20 ms. And cold scripting could be worse. So I do not think that sending sctipts to be compiled on-site is a good idea. You probably need some kind of serializeable transformations.
y

Yoav Luft

09/05/2020, 10:36 AM
Will scripts be compiled on every execution, or could there be a way to cache that? Script will change very rarely during an interaction
a

altavir

09/05/2020, 10:37 AM
Scripts could be pre-compiled and then the starting time will be small. Still I would use it only if everything else fails
y

Yoav Luft

09/05/2020, 10:38 AM
I see. So the data forge approach is writing modules that could be wired through the user application using some serialized representation?
a

altavir

09/05/2020, 10:40 AM
Yes, since we are doing highly customizable and scaleable framework, we can't rely on scripts. We use event-based approach. Actually the same goes for polyglot communication solutions we are working right now. Passing byte-code or scripts is just not reliable.
y

Yoav Luft

09/05/2020, 10:41 AM
ok! Thanks, I’m convinced! And I hope that I’ll get the time to test data forge soon
a

altavir

09/05/2020, 10:42 AM
It is in a prototype stage yet, this is why we need all the input we can get.
Right now I recomment to create a serializeable form for your transformation like
Copy code
sealed class Transformation

class Scale(val scale: Double): Transformation
And pass it via regular json to your device controller.
y

Yoav Luft

09/05/2020, 10:55 AM
and somewhere else I will have parser that will build computable expressions from that?
a

altavir

09/05/2020, 10:56 AM
Yes, something like that. If you have only limited number of transformations, it would work the best. Of course if you will need to introduce new types of transformations without changing the client code, it will be much more complicated.
y

Yoav Luft

09/05/2020, 11:04 AM
but I think for now it would be enough
2 Views