https://kotlinlang.org logo
Join Slack
Powered by
If anyone needs an automatic differentiation engin...
# datascience
e
If anyone needs an automatic differentiation engine in Kotin you can use this gist of mine (I don’t have time to turn it into an actual library). It is not the most optimal implementation in terms of performance, but it is quite easy-to-use in terms of API and has very concise and simple implementation + Kotlin DSL FTW. https://gist.github.com/elizarov/1ad3a8583e88cb6ea7a0ad09bb591d3d I’ts backwards-mode (back-propagation), so you get derivative of one output value by all input values in one pass. That is the thing you’d need to do gradient descent. Use it like this:
Copy code
val x = D(2) // define variable(s) and their values
val y = grad { sqr(x) + 5 * x + 3 } // write formulae in grad context
assertEquals(17.0, y.x) // the value of result (y)
assertEquals(9.0, x.d)  // dy/dx
👍 3
a
I have autodiff in kmath for Jvm via commons math with similar design. I can add it as mpp implementation with appropriate remark about author.
👍 1
e
Yeah. I don’t like “stringly-typed” designs. The one in my gist is 100% type-safe and fully composable.
a
Basic DerivativeStructure works exactly like your example (only with infinite derivatives). The example shows forward declaration of expression, which could use forward declaration and define parameters in a separate place. I have not found a way to do so without string keys.
e
I don’t see why would you need forward declarations. You can just declare functions with parameters, call them, and have engine compute derivatives as a side-effect, too.
And no, this is completely different approach. You seem to explicitly build expression tree and then compute symbolic derivatives?
a
What I wanted is to be able to generate function and parameters in different places. Currently I can do both using either DerivativeStructureField or ExpressionContext.
e
My engine does not. It is an algorithmic diff engine. It computes the result and derivatives in one go, without explicitly building an expression tree. It is compatible with any logic (ifs, when, loops) and scales to computations of extremely big sizes (line neural networks)
You can compare performance on some fairly large expression to get a feel of a difference.
a
I do understand what it does. Exactly the same as DerivativeStructure in commons math. I can use your implementation for mpp without loading commons math.
It does not create the expression tree.
e
commons-math derivativestructure is too inefficient. Compare, for example, how much abstraction its definition of 
sin()
function uses and how concrete and concise mine is:
Copy code
fun AD.sin(x: D): D = derive(D(sin(x.x))) { z ->
    x.d += z.d * cos(x.x)
}
a
Sorry, I am on phone. I will write in a little bit more detail, when I get home. I will be happy to add your implementation and then compare performance.
e
Actually, I have not even started to optimize my code. I brought it to the state of the most beatiful API, not maximal performance.
a
I need mpp API anyway
Your API is almost identical to kmath.
e
Note, though, mine is hard-coded for backwards-mode. There is no support for forward-mode. So it’s only efficient when your functions map R^n to R^m and n >> m
K API
a
Yep. Kotlin just asks for this design style
Finally got home. The problem with your implementation compared with Commons-math is that it does not allow neither higher derivatives (and second derivatives is quite usual use case) nor, partial derivatives. So it is strictly univariate first-derivative engine. The function 
grad
is a bit confusing since gradient is usually used for scalar multivariate function and generates vector, not scalar. It is still nice to have, I will generalize it for different types of object using 
ExtendedField
so it will be applicable to complex numbers and (in future) to vectors of numbers.
e
Indeed. It is tailored for gradient descent. It computes gradient of result with respect to all input parameters.
a
@elizarov I've integrated the code in kmath, but I really did not like accessing the derivatives via mutable state of input, so I've changed API to store derivatives only during actual calculation and not to leak it outside. The experimental code is here: https://github.com/mipt-npm/kmath/blob/autodiff-experiment/kmath-core/src/commonMain/kotlin/scientifik/kmath/operations/AutoDiff.kt. I've also added actual gradient and divergence calculation to the result. My solution will probably have minor overhead due to derivative lookup, but I think it will be rather small.
9 Views
Open in Slack
PreviousNext
Powered by