https://kotlinlang.org logo
Title
p

Peter

04/16/2022, 4:00 PM
Used
JupyterReplTestCase
since a long time to automate some regression testing of notebooks and it always worked fine. But now I’m trying slightly different setup with a lib.json file containing some imports (before this was handled by
JupyterIntegration
) and I cannot get it to work. In my notebook:
%use roboquant@url[<https://roboquant.org/roboquant.json>]
In my unit test:
class NotebookTester() : JupyterReplTestCase(ReplProvider.forLibrariesTesting(listOf("roboquant"))) { ... }
But if I try this, the imports that are defined in the roboquant.json are ignored. Anyone know what I’m doing wrong or how I can get the imports in the roboquant.json files to be used for my unit test (other then specifying them explicitly in my notebooks) ?
i

Ilya Muradyan

04/16/2022, 7:06 PM
It doesn't work only in the unit test or in the notebook too?
It seems you need to write a specific library resolver for this case: it should ignore
dependencies
and
repositories
in all (or only in specific) JSON descriptors. I can help you with this, but there is a couple of questions to be sure I understand your case correctly: 1. How do you use JSON descriptor in test? I mean, what do you write in
eval("%use ...")
? 2. How does your JSON descriptor for the test look like?
p

Peter

04/16/2022, 9:01 PM
It works in the notebook standalone, just not in the unit test.
It is fully automated, so it runs each cell and compares it against the pre-recorded output.
i

Ilya Muradyan

04/16/2022, 9:04 PM
Aha, so it's the same
%use roboquant@url[<https://roboquant.org/roboquant.json>]
in your test
p

Peter

04/16/2022, 9:04 PM
The unit test code
private class NotebookTester(lib: String) : JupyterReplTestCase(ReplProvider.forLibrariesTesting(listOf(lib))) {

 
    fun validateNotebook(notebookPath: String) {
        val notebookFile = File(notebookPath)
        val notebook = JupyterParser.parse(notebookFile)

        for (cell in notebook.cells.filterIsInstance<CodeCell>()) {
            val cellResult = exec(cell.source)
            val result = if (cellResult is MimeTypedResult) cellResult.entries.first().value else cellResult.toString()

            if (cell.outputs.isNotEmpty()) {
                val firstOutput = cell.outputs.first()
                if (firstOutput is ExecuteResult && firstOutput.data.isNotEmpty()) {
                    val value = firstOutput.data.entries.first()
                    assertEquals(value.value, result)
                }
            }
        }

    }

}
Indeed, that is the code. Used to be
%use @https://roboquant.org/roboquant.json
But then the Jupyter Test case doesn’t know the library name (roboquant) and complains about “@default” being unknown.
i

Ilya Muradyan

04/16/2022, 10:14 PM
If you only use descriptors from URLs, following REPL provider should suit you. You should pass it to
JupyterReplTestCase
constructor.
%use @https://roboquant.org/roboquant.json
will also work. Dependencies will be taken from classpath, imports and other things will be loaded from descriptor
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.buildJsonObject
import org.jetbrains.kotlinx.jupyter.ReplForJupyter
import org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl
import org.jetbrains.kotlinx.jupyter.api.libraries.LibraryResolutionInfo
import org.jetbrains.kotlinx.jupyter.common.getHttp
import org.jetbrains.kotlinx.jupyter.common.jsonObject
import org.jetbrains.kotlinx.jupyter.defaultRepositories
import org.jetbrains.kotlinx.jupyter.dependencies.ResolverConfig
import org.jetbrains.kotlinx.jupyter.libraries.AbstractLibraryResolutionInfo
import org.jetbrains.kotlinx.jupyter.libraries.ByNothingLibraryResolutionInfo
import org.jetbrains.kotlinx.jupyter.libraries.ResolutionInfoProvider
import org.jetbrains.kotlinx.jupyter.libraries.SpecificLibraryResolver
import java.io.File
import java.net.URL

object RoboquantReplProvider : ReplProvider {
    override fun invoke(classpath: List<File>): ReplForJupyter {
        return ReplForJupyterImpl(
            resolutionInfoProvider,
            classpath,
            resolverConfig = ResolverConfig(defaultRepositories, urlEditingResolver),
            isEmbedded = true
        ).apply {
            eval { librariesScanner.addLibrariesFromClassLoader(currentClassLoader, this) }
        }
    }

    private val urlEditingResolver = SpecificLibraryResolver(AbstractLibraryResolutionInfo.ByURL::class) { info, _ ->
        val response = getHttp(info.url.toString())
        val json = response.jsonObject
        val newJson = buildJsonObject {
            for ((key, value) in json.entries) {
                if (key == "repositories" || key == "dependencies") continue
                put(key, value)
            }
        }
        Json.encodeToString(newJson)
    }

    private val resolutionInfoProvider = object : ResolutionInfoProvider {
        override var fallback: LibraryResolutionInfo
            get() = ByNothingLibraryResolutionInfo
            set(value) {}

        override fun get(string: String): LibraryResolutionInfo {
            return AbstractLibraryResolutionInfo.ByURL(URL(string))
        }
    }
}
👍 1
p

Peter

04/17/2022, 5:37 AM
Thanks for the help, very much appreciated. Will give it a try.
Just cut&paste your code snippet and use that new RoboquantReplProvider and works like a charm. Thank you so much for spending your time in the weekend to help me out 🙏 🙏🙏 P.S really enjoy the combination of Kotlin + Jupyter. In case you are curious what is currently build, click this link and run the notebook: Run Notebook on MyBinder
🔥 3