Hello! For the most part I’m loving doing idiomati...
# skrape-it
e
Hello! For the most part I’m loving doing idiomatic scraping with Skrape{it}! However, I’m having trouble getting the js-rendered functionality working. The example at https://docs.skrape.it/docs/dsl/extract-client-side-rendered-data looks like it’s for a previous version, since
mode = Mode.DOM
is no longer available. According to the docs on Github, it looks like all I should need to do is pass
BrowserFetcher
to the skrape function as an argument, but that doesn’t seem to do the trick. I tried setting jsExecution to true, e.g. something like this:
Copy code
skrape(BrowserFetcher) {
    request { url = urlToScrape }
}
extract {
    htmlDocument(html = responseBody, baseUri = baseUri, jsExecution = true) {
        val t = title { findFirst { text } }
        i { "Got title:$t" }
    }
}
…which looks like it’s trying to do something, but I’m getting an error loading one of the included js files:
Copy code
java.io.IOException: Unable to download JavaScript from '<https://cdn>...Stub.js' (status 400).
	at com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1075)
Not sure why it’s a 400, as the file in question seems to download fine with a browser or other http clients. Any assistance is much appreciated…