Added bi-directional integration with DataFrame to...

# datascience

[altavir]

Added bi-directional integration with DataFrame to Tables.kt: https://github.com/mipt-npm/tables-kt/blob/master/tables-kt-dataframe/src/test/kotlin/space/kscience/dataforge/dataframe/DataFrameTableTest.kt. Working with DataFrame outside of notebook is hard

[Hampus Londögård]

As there's no README attached in repo, what does Tables.kt bring that DataFrame does not? Or Krangl for that matter?

[altavir]

Yeah, it is pretty experimental. Tables.kt has a different target. Like KMath it is a glue library that aims to provide a lightweight, but universal multipatform API to work with table data. It is not intended for heavy pandas-like column operations, but instead it provides clean interfaces and integrations with different sources. For example, we already have Exposed - based integration with DB. Also it is multiplatform.

I've actually tried to deprecate it in favor of DataFrame, but found out that DataFrame does not cover our use cases in terms of flexibility.

[altavir]

And I forgot to mention that DataFrame supports only column-based tables, whereas Tables.kt supports row-based tables (our primary use case) and will support spreadsheets in future

[Hampus Londögård]

Multiplatform is very cool!
By row based you mean each row is encoded as a data class? Rather than each col being encoded?

[altavir]

Not necessary data class (Map is better in this case), but mostly yes, it is about inner representation. If you take a loooong text file and read it line by line, you want a row-based implementation. Tables.kt also supports non-random-access row iterators, which allows to do streaming processing of rows (not possible with pandas/DataFrame)

[altavir]

I've added a readme: https://github.com/mipt-npm/tables-kt

Right now it should be possible to load a table from a database and convert it to DataFrame.

[altavir]

But I have not tested it

[Hampus Londögård]

Sounds great!

I wish we had a column first DataFrame that actually only read columns for say parquet. And used arrow-format 👍

[altavir]

As far as I know, this is planned for DataFrame itself.