Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure ...

# datascience

[Thomas Vander Wal]

Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure if anyone has been able to figure out. When importing that dependency my code get's a class not found error but works fine from running it in IntelliJ. When opening the Jar I can see it in the class path so stumped on what the issue is. Even weirder is I have a file that works perfectly fine with spark-core and the RDD api, however it breaks with the same error when I import the spark-sql library. As far as I understand that library is dependent on core and should be packaging it as well.

[altavir]

It seems to be a build problem. Could you provide the build file?

[Thomas Vander Wal]

Here is the build file for the app that runs fine with core. I should specify that the spark-sql works when running with the play button in the gutter, but my errors are being thrown when trying to use spark-submit.

[Thomas Vander Wal]

plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.3.11'
}

group 'test'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
    compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '2.4.0'
//    compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '2.4.0'

}

compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}


jar {
//    zip64 = true
    manifest {
        attributes 'Main-Class': 'FirstApplicationKt'
    }
    from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}

[altavir]

You should not use manual jar packaging. Use application plugin instead. Also you can try to use both dependencies Gradle will decide what to take.

[Thomas Vander Wal]

OK I will play around with that. I added that in to create the fat jar and to get around the NoClassDef error, but that must not have been the correct solution. Thanks!

[altavir]

Use application and try to use Gradle run task instead of idea. It is more reliable.

[Thomas Vander Wal]

@altavir Thanks man! I'm new to most of these things but that got me on the right track. the gradle run task allowed me to run it from the command line but still had issues with spark submit. I used a combination of that and shadowJar to create the fat jar and it worked like a charm!

[altavir]

You are welcome. Not my ID though.

[Thomas Vander Wal]

Yikes it's been a long day 😅

[altavir]

It is morning where I am.