https://kotlinlang.org logo
#datascience
Title
# datascience
i

iamsteveholmes

06/24/2022, 5:18 PM
Hey Folks! Spark Jupyter question here. I got spark working using
spark-streaming
and `withSpark`and I’m happily crunching through big jobs. However I noticed the resources are set to a single core and 1gb of memory which might be why my jobs are taking a long time. Is there a way to configure the environment to upgrade those values? Also, any recommendations for what you might set am i7 Macbook Pro to?
a

altavir

06/25/2022, 7:10 AM
I am not familiar with how Spark is configured, but the jupyter itself depends on command line arguments as it is described here: https://github.com/Kotlin/kotlin-jupyter#usage (use -Xmx argument).
The multicore is governed by the framework itself
r

roman.belov

06/27/2022, 5:05 PM
cc @Jolan Rensen [JetBrains]
j

Jolan Rensen [JetBrains]

06/28/2022, 10:33 AM
@iamsteveholmes Do you mean the resources of Spark or of Jupyter?
i

iamsteveholmes

06/28/2022, 3:14 PM
Spark
j

Jolan Rensen [JetBrains]

06/28/2022, 3:20 PM
Alright, yeah then you can give the props as a map to the withSpark() command :) all spark environment variables should work through that. It's the same as defining options in the normal spark session builder
i

iamsteveholmes

06/28/2022, 8:20 PM
Thank you so much! Can't wait to play with that.
4 Views