Hey folks! Using Datalore or a Jupyter notebook and spark is there anyway to access local files? I’m getting:

No FileSystem for scheme "file"
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"

You can't access files on your computer since DataLore does not know anything about it. You can add files to your datalore workspace though.

I’ve added the file to the workspace in datalore and then I try to read it as follows:

val equipment = spark.read().json("allEquipment.json" )

But I still get the same error. Have you been able to read a file from the workspace using spark?

Not to the workspace. To attached files. It works differently from local jupyter since each notebook has its own isolated environment.

This is what I have:

You should ask @Pasha Finkelshteyn, but if you know how to use spark with regular file, notebook is not different.

@altavir thank you for mentioning me! But all the experimenting is done by @Jolan Rensen [JB] But it looks to me that you don't need to use scheme to access files entirely. But I know nothing about filesystem in datalore

It is a regular linux local filesystem

@Jolan Rensen [JetBrains] @Pasha Finkelshteyn I would love to see any samples of accessing files using spark on datalore or jupyter notebooks. My experience with Spark was years ago (when spark was new-ish) in Scala and I set up my own environments.

I am not familiar with spark, but probably something like this should do: https://spark.apache.org/docs/latest/sql-data-sources-text.html

Yeah that’s what I’ve been following. If you look at my example line it follows their example. 🤔

Oh that's weird, it appears I get

No FileSystem for scheme "file"
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"

as well. Thanks for letting us know! Looks like a bug

Thank you so much! I really appreciate all your help!