Julia reported release of their own DataFrame: <https://dataframes.juliadata.org/stable/>. Right now...

# datascience

[altavir]

Julia reported release of their own DataFrame: https://dataframes.juliadata.org/stable/. Right now they are probably the main competition to emerging Kotlin for Data ecosystem, so we whould study it. 🧵

[altavir]

DataFrane builders

df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"])

While such things are not possible in a type-safe language, we can do things, that are similar yet much more safe:

val a by symbol
val b by symbol
val df = DataFrame{
  a(1..4)
  b("M", "F", "F", "M")
}

column constructors here could be done via member extension like Symbol.invoke(). Currently they all must be pre-defined in the builder, but with Multi-receivers we can add them as extensions.

[altavir]

Symbolic accessors

julia> df.A

Currently there is an ongoing effort by the Kotlin DataFrame team to do that via staged compilation, but in KMath we found another way. We can use pre-defined symbol objects as identifiers. so we could do things like this:

val A by symbol
df[A]

One could use column definitions instead and get a type-safe accessor.

[altavir]

Row-based and column-based tables are easy to support. We can even use type-safe row accessors.

[altavir]

Accessor by expression

julia> df[(df.A .> 500) .& (300 .< df.C .< 400), :]

Well, it is hard to do that and I am pretty sure we should not do it this way. It is better to create a DataFrameQuery object and create a builder for it like this:

df.query{
  a{it>500}
  c{it in 300..400}
}

[altavir]

Data mutability
I am pretty sure that data in the table should be immutable and when someone changes something, the new zero-copy with one replaced column should be created.

[altavir]

IO
I think the koltin ecosystem is quite rich in that regard, all we need is to provide convinien constructors for the DataFrame and make in an interface to allow 3rd party implementations.

[Ilya Muradyan]

Symbolic accessors

AFAIK it is already implemented this way
https://github.com/nikitinas/dataframe/blob/bc9feae6d8200499bf7d7a0fbb45bb0491b90e[…]rc/test/kotlin/org/jetbrains/dataframe/person/DataFrameTests.kt

[Ilya Muradyan]

Accessor by expression

Nice point, but simple (DataFrame) -> Boolean filter is generally enough

[Ilya Muradyan]

About mutability and IO - it is also done this way in DataFrame

[altavir]

I know, but since DataFrame has not any public discusstions yet, I decided to write the points here.

[altavir]

By the way, I would like to have some kind of discussion on DataFrame API. I have some concerns about how it is right now.

[altavir]

cc @roman.belov @Anatoly Nikitin

[holgerbrandl]

I am late to the party, but thanks for the pointer!