Fetching Webview HTML DOM using Coroutines

Question

I write this code to use it to fetch DOM of HTML page loaded in webview after fetching I want to process the DOM using DukTape Library with remote JavaScript file (to handle any future changes of the DOM structure without forcing user to update the application) the code is for Kotlin Android project:

First the Fragment Hosting Duktape Code:

val duktape = Duktape.create()
// Bind the RetrofitBridge instance to Duktape

val webView = binding.unvisibleWebview
val retrofitBridge = RetrofitBridge(webView)
duktape.set("retrofitBridge", AndroidNetworking::class.java, retrofitBridge)

try {
    // Execute JavaScript code
    val jsCode = """var result = retrofitBridge.webViewRequest("https://jsonplaceholder.typicode.com/posts"); result; // Return the result to Kotlin"""

    val result = duktape.evaluate(jsCode)
    Log.d("ResponseHtml", result.toString())
    println(result) // Print the output
} catch (e: Exception) {
    e.printStackTrace()
} finally {
    duktape.close() // Clean up resources
}

Duktape AndroidNetworking Interface:

interface AndroidNetworking {
    fun getRequest(url: String): String
    fun postRequest(url : String): String
    fun webViewRequest(url: String): String
}

webViewRequest Implementation:

override fun webViewRequest(url: String): String {
    var resultHtml: String? = null
    
    WebViewClientRequest(webView, url) { html ->
        resultHtml = html
    }
    return resultHtml ?: ""
}

WebViewClientRequest class:

class WebViewClientRequest(
    private val webView: WebView,
    private val url: String,
) {

    init {
        val webSettings = webView.settings
        webSettings.javaScriptEnabled = true
        webSettings.userAgentString =
            "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36"

        val cookieManager = CookieManager.getInstance()
        cookieManager.setAcceptCookie(true)
        cookieManager.setAcceptThirdPartyCookies(webView, true)

        webView.addJavascriptInterface(JsInterface(), "HTMLOUT")
        webView.webViewClient = WebClientFetchData()

        webView.loadUrl(url)
    }

    inner class WebClientFetchData : WebViewClient() {
        override fun onPageFinished(view: WebView?, url: String?) {
            super.onPageFinished(view, url)
            // Inject JavaScript to fetch the HTML content
            webView.loadUrl(
                "javascript:window.HTMLOUT.showHTML(''+document.getElementsByTagName('html')[0].innerHTML+'');"
            )
        }

        override fun onReceivedError(view: WebView?, request: WebResourceRequest?, error: WebResourceError?) {
            super.onReceivedError(view, request, error)
            println("Error loading page: $error")
        }
    }

    inner class JsInterface {
        @JavascriptInterface
        fun showHTML(html: String) {
            println(html)
        }
    }
}

What I want to implement is to not returning webViewRequest until DOM is scraped without changing webViewRequest to suspend function in order to Work with dukeTape

I tried to use runBlocking no use also I tried the following working but I can't pass data to DukTape

runBlocking {
    WebViewClientRequest(webView, url) { html ->
        resultHtml = html
    }
}

CoroutineScope(Dispatchers.Main).launch {
    WebViewClientRequest(webView, url) { html ->
        resultHtml = html
    }
}

Fetching Webview HTML DOM using Coroutines

Answers (0)

Related Questions