So I downgraded to jdk11 and made sure hive 3.2.2 ...

# kotlin-spark

[Eric Rodriguez]

So I downgraded to jdk11 and made sure hive 3.2.2 is the only version in the deps. and everything works as expected both in local[*] and against a cluster on docjer-compose.

But then I began to save delta forma instead of parquet and now I get

cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.ScalaUDF.f of type scala.Function1 in instance of org.apache.spark.sql.catalyst.expressions.ScalaUDF``

when I run against the cluster (works on lcoal). Any idea how I may debug this?

[Jolan Rensen [JB]]

Looks like some lambda can't be serialized properly. You can try to set the kotlin compiler option: freeCompilerArgs.add("-Xlambdas=class") to see if that makes a difference. That has helped me in the past

[Eric Rodriguez]

will check and let u know

[Eric Rodriguez]

how do I add freeCompilerArgs in maven?

[Jolan Rensen [JB]]

ah like this https://kotlinlang.org/docs/maven.html#specify-compiler-options

[Jolan Rensen [JB]]

so <arg>-Xlambdas=class</arg>

[Eric Rodriguez]

ok, so it didn't helped actually, but now I'm kinda convinced the issue is my docker-compoe stand-alone cluster has jdk17 while I'm compiling for jdk11 (which is used on dataproc)

[Jolan Rensen [JB]]

That's also very much possible. There may have been a change in the JDK to how lambdas are stored and serilized

[Eric Rodriguez]

found the issue, i forgot to remove 1 LocalDate from the parameters:

sports.toDS()
            .let { addDate(it, now.year, now.monthValue, now.dayOfMonth) }

instead of extracting the values and passing those:

val now = LocalDate.now()
    val year = now.year
    val month = now.monthValue
    val day = now.dayOfMonth
sports.toDS()
            .let { addDate(it, year, month, day) }