A modern programming language that makes developers happier.

kotlinlang

Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure if anyone has been able to figure out. When importing that dependency my code get's a class not found error but works fine from running it in IntelliJ. When opening the Jar I can see it in the class path so stumped on what the issue is. Even weirder is I have a file that works perfectly fine with spark-core and the RDD api, however it breaks with the same error when I import the  spark-sql library. As far as I understand that library is dependent on core and should be packaging it as well.

Got the issue solved in the Data Science group. I had been creating my own fat jar by Baeldong's instructions (section 3) <https://www.baeldung.com/gradle-fat-jar>

This caused some sort of issue that's unknown to me. I fixed it by using the application plugin as well as the shadowJar plugin (same article but section 6). Running ./gradlew shadowJar created a fat jar that worked and was usable by spark-submit.