When I try to pivot a column on a 100K unique rows...

# datascience

[Dumitru Preguza]

When I try to pivot a column on a 100K unique rows dataset I get this error:

The problem is found in one of the loaded libraries: check library converters (fields callbacks)
Error compiling code:
@DataSchema
interface _DataFrameType1 {
    @ColumnName("00000")
    val `00000`: String
    @ColumnName("00001")
    val `00001`: String
    @ColumnName("00002")
....

Is it a know issue ?

All I'm trying to achieve is to have this notation in the end:

tryToPivot["00002"] == "59835" // treat the dataframe as a map

[Dumitru Preguza]

Here is my dataframe:

val tryToPivot = numbersDF.add("the_mask") {
    index().toString().padStart(5, '0')
}.pivot("the_mask").values {
    value
}

[Dumitru Preguza]

I guess it's a too large schema/dto that compiler tries to ingest

[Dumitru Preguza]

Basically it will generate a class with 100K fields

[Nikita Klimenko [JB]]

Hi! I think least we can do is to provide better error message and maybe a way to disable codegen callback to avoid this error. As to make this work.. probably compiling interface with 100K fields won't work well in any case =(
But, just to be sure, i think this error might not completely break cell execution. You should be able to call your resulting dataframe without typesafe wrapper
I'll create an issue

[Dumitru Preguza]

Ok, thank you!