Hi everyone, we at JetBrains are curious to find o...
# kotlin-spark
j
Hi everyone, we at JetBrains are curious to find out what you use the Kotlin Spark API for. Do you have any problems with it? What would you like to see in upcoming versions? Don't hesitate to let us know 🙂
t
Hi! Thanks for asking 🙂 My company builds some custom Spark operators, and I would love to do it in Kotlin! One feature that would make my life easier is an easy way to create a Scala
seq
. Something like
seqOf(...)
would be great!
j
Hi @Tianyu Zhu! Can you give an example for where you would need a Scala Seq? With the kotlin spark api, we normally try to mirror the Java API as close as possible (but applying kotlin-specific features like lambda's, operator overloads etc.). However I as far as I know Scala Seqs are not used anywhere in the Java API. Everything can be done using a simple List.
t
Sure! For example, I want to create a custom SparkSQL Expression that uses
org.apache.spark.sql.catalyst.expressions.LambaFunction
. To construct that object, I need a
Seq
.
j
I see! Have not come across that yet. Would something like this work?:
Copy code
fun <T> seqOf(vararg elements: T): scala.collection.Seq<T> = scala.collection.JavaConverters.asScalaBuffer(elements.asList()).toSeq()
https://github.com/Kotlin/kotlin-spark-api/issues/163 created an issue for it 🙂 feel free to continue the chat there
t
I've been using
WrappedArray.make(elements)
but I think your code should work too!