https://kotlinlang.org logo
Title
t

Thomas Vander Wal

02/02/2019, 4:09 AM
Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure if anyone has been able to figure out. When importing that dependency my code get's a class not found error but works fine from running it in IntelliJ. When opening the Jar I can see it in the class path so stumped on what the issue is. Even weirder is I have a file that works perfectly fine with spark-core and the RDD api, however it breaks with the same error when I import the spark-sql library. As far as I understand that library is dependent on core and should be packaging it as well.
a

altavir

02/02/2019, 5:18 AM
It seems to be a build problem. Could you provide the build file?
t

Thomas Vander Wal

02/02/2019, 5:47 AM
Here is the build file for the app that runs fine with core. I should specify that the spark-sql works when running with the play button in the gutter, but my errors are being thrown when trying to use spark-submit.
plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.3.11'
}

group 'test'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
    compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '2.4.0'
//    compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '2.4.0'

}

compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}


jar {
//    zip64 = true
    manifest {
        attributes 'Main-Class': 'FirstApplicationKt'
    }
    from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}
a

altavir

02/02/2019, 5:50 AM
You should not use manual jar packaging. Use application plugin instead. Also you can try to use both dependencies Gradle will decide what to take.
t

Thomas Vander Wal

02/02/2019, 5:53 AM
OK I will play around with that. I added that in to create the fat jar and to get around the NoClassDef error, but that must not have been the correct solution. Thanks!
a

altavir

02/02/2019, 5:54 AM
Use application and try to use Gradle run task instead of idea. It is more reliable.
t

Thomas Vander Wal

02/02/2019, 6:41 AM
@altavir Thanks man! I'm new to most of these things but that got me on the right track. the gradle run task allowed me to run it from the command line but still had issues with spark submit. I used a combination of that and shadowJar to create the fat jar and it worked like a charm!
a

altavir

02/02/2019, 6:42 AM
You are welcome. Not my ID though.
t

Thomas Vander Wal

02/02/2019, 6:43 AM
Yikes it's been a long day 😅
a

altavir

02/02/2019, 6:45 AM
It is morning where I am.