Hi, when trying the run a small example defined in...

# kotlin-spark

[holgerbrandl]

Hi, when trying the run a small example defined in https://github.com/holgerbrandl/kalasim/blob/master/modules/sparksim/src/main/kotlin/org/kalasim/exaples/spark/DistributedSim.kt#L26 it fails with

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (192.168.217.128 executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
... very long stacktrace  ....

I was under the impression that all serialization should be very simply in the expression, and every job is returning a simple string.
What am I doing wrong?

[Jolan Rensen [JB]]

I'll try to run it myself in a bit. But I can already recommend moving SimConfig outside the function. Spark has some trouble detecting inner classes.

[Jolan Rensen [JB]]

Okay after a lot of tries I cannot seem to run it on a master running on localhost. It works for local[*] fine. But after a lot of different setups, running it on the master get's me stuck at SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
So... I can't reproduce it 😕 Could you maybe find an example with local[*] where you get the same error?

[holgerbrandl]

Thanks for your support. I'll try to narrow down the problem. I'm quite new to spark but I will give my best. My spark is actually running in a local Linux VM that's why I was using a spark://host:port url to configure the spark session. I could also spawn a spark cluster in azure to test with a second system.

[Jolan Rensen [JB]]

I was also trying to run it on linux with start-master.sh. It worked (with spark 3.3.0, scala 2.12), but it kept hanging at that message. But that's probably just my setup and not relevant to the code. I use spark mostly for small stuff, so then running it locally in the IDE is fine, but then not everything is serialized, so I might not catch all problems that way.