https://kotlinlang.org logo
Title
r

razvandragut

07/30/2021, 9:28 PM
hi, the following throws exceptions
fun main() {
    withSpark {
        dsOf(
            listOf(mapOf("a" to "b", "x" to "y")),
            listOf(mapOf("a" to "b", "x" to "y")),
            listOf(mapOf("a" to "b", "x" to "y"))
        ).showDS().printSchema()
    }
}
j

Jolan Rensen [JetBrains]

07/31/2021, 5:59 PM
I don't think spark can even handle datasets of lists of maps. No idea how that would even be encoded in a Column/Row structure. Any particular reason you need it?
r

razvandragut

08/06/2021, 2:42 AM
spark can handle it if, for example, I return a list of maps from a UDF & I specify a custom/explicit DataType. The use case is to store a list of
attribute sets: [ { attr1: v1, attr2: v2}, {attrA: va, attrB:vb, attrX:vx} ]
where the attribute names/keys can be different between the sets and even unknown
a

asm0dey

08/27/2021, 5:47 PM
Wowzer
Completely missed this
@razvandragut fixed it, ging to release soon-ish
:party-parrot: 1
r

razvandragut

10/03/2021, 12:58 PM
Thanks !! 😉