CODEWITHSUNDEEP
rusthttp-postnoscriptreqwest
lolozen
lolozen

Reputation: 426

In rust how do I deal with page that ask for javascript?

I'm trying to make a HTTP request in rust using reqwest to get data from a website, similarly to what this command would do

~$ curl -X POST -H "Content-Type: application/json" -d '{"video":""}' watchparty.me/createRoom -L
{"name":"/supreme-cherries-answer"}
~$

so here is what I tried:

use reqwest::header;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct Room {
    name: String
}

async fn get_room_name() {

    //new client with cookies
    let client = reqwest::Client::builder()
        .cookie_store(true)
        .build()
        .unwrap();

    let body = r#"{
        "video": ""
    }"#;

    let res = client
        .post("https://watchparty.me/createRoom")
        .header(header::USER_AGENT, "mozilla/5.0")
        .header(header::CONTENT_TYPE, "application/json")
        .header(header::ACCEPT, "application/json")
        .body(body)
        .send()
        .await
        .unwrap();

    match res.status() {
        reqwest::StatusCode::OK => {
            //make sure response is json and not an error
            println!("Room name: {:?}", &res.text().await.unwrap());
        },
        _ => println!("Error: {}", res.status())
    }
}

#[tokio::main]
async fn main() {
    get_room_name().await;
}

thing is that with this code I get this as a response

Room name: "<!doctype html>
<html lang=\"en\">
   <head>
      <meta charset=\"utf-8\"/>
      <link rel=\"icon\" href=\"/favicon.ico\"/>
      <meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"/>
      <meta name=\"theme-color\" content=\"#000000\"/>
      <meta name=\"description\" content=\"Watch together with friends. Chat and react in real time to the same stream, whether it's YouTube, your own video, or a virtual browser.\"/>
      <meta property=\"og:image\" content=\"/screenshot4.png\"/>
      <link rel=\"apple-touch-icon\" href=\"/favicon.png\"/>
      <link rel=\"manifest\" href=\"/manifest.json\"/>
      <title>WatchParty</title>
      <script defer=\"defer\" src=\"/static/js/main.437dd445.js\"></script>
      <link href=\"/static/css/main.9154545e.css\" rel=\"stylesheet\">
   </head>
   <body>
      <script>!function(){var a=\"fbclid\";if(-1!==location.search.indexOf(\"fbclid=\")){var e=\"\";try{var c=new URL(location);c.searchParams.delete(a),e=c.href}catch(a){var l=new RegExp(\"[?&]fbclid=.*$\");e=location.search.replace(l,\"\"),e=location.pathname+e+location.hash}history.replaceState(null,\"\",e)}}()</script>
      <noscript>You need to enable JavaScript to run this app.</noscript>   <----- this is the part that annoys me
      <div id=\"root\"></div>
   </body>
   <script async src=\"https://www.googletagmanager.com/gtag/js?id=UA-45337794-6\"></script><script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag(\"js\",new Date),gtag(\"config\",\"UA-45337794-6\")</script>
</html>
"

I don't know how I can make it so my code get the same result as my curl query (of course without using the curl crate), does anyone have an idea ?

EDIT: I modified a bit my code regarding some comment that were done but the result ended up to be the same

Upvotes: 1

Views: 666

Answers (1)

Jishan Shaikh
Jishan Shaikh

Reputation: 1806

This is because https://watchparty.me/createRoom is redirecting once. Checked from the -i flag of the CURL:

$ curl -X POST -H "Content-Type: application/json" -d '{"video":""}' watchparty.me/createRoom -Li

HTTP/1.1 301 Moved Permanently
Date: Thu, 04 May 2023 15:24:47 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Thu, 04 May 2023 16:24:47 GMT
Location: https://www.watchparty.me/createRoom
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=MuWcCoj86EW9kl2orX6SpVlY%2B%2Fgwb6DR5637tSa6IFfs16RUyAYQXnGVZ9mrJHjkebAQTf789SlAboTzu03kmtO0v22WCJryTv3iKhI92WUyLEuYWyHOX4qp5S8Y8kjl"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c21c0aec93585d6-BOM
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP/2 200 
date: Thu, 04 May 2023 15:24:49 GMT
content-type: application/json; charset=utf-8
content-length: 38
x-powered-by: Express
access-control-allow-origin: *
etag: W/"26-umy5SJpIYT3XtIZYL82SfSgo4zs"
vary: Accept-Encoding
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=MNv3ogdbLprH82OGlphP%2Bdau%2BFbB%2By2gjhsYwi16qQEEtaTVYvPrupsx5YGRUHKENqFVv%2BQ3GWn5bxRS3lb40jEABv8IiQeWsfzYvS%2F3r5eaGhjE3f5v6PPLUDU0S%2Fj3Hue4uA%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7c21c0b8cbaad393-CDG
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

{"name":"/unbecoming-pizzas-preserve"}

To update the code: replace your URL: https://watchparty... to https://www.watchparty.... Here's tested code:

use reqwest::header;
// use serde::{Deserialize, Serialize};
// We don't need Room struct and serde actually
// struct Room {
//     name: String
// }

async fn get_room_name() {
    let client = reqwest::Client::builder()
        .build()
        .unwrap();

    let body = r#"{
        "video": ""
    }"#;

    let res = client
        .post("https://www.watchparty.me/createRoom")
        .header(header::USER_AGENT, "mozilla/5.0")
        .header(header::CONTENT_TYPE, "application/json")
        .header(header::ACCEPT, "application/json")
        .body(body)
        .send()
        .await
        .unwrap();

    match res.status() {
        reqwest::StatusCode::OK => {
            println!("Room name: {:?}", &res.text().await.unwrap());
        },
        _ => println!("Error: {}", res.status())
    }
}

#[tokio::main]
async fn main() {
    get_room_name().await;
}

The output of which:

$ cargo run
   Compiling soTesting v0.1.0 (/home/cpp23g/rustProjects/soTesting)
    Finished dev [unoptimized + debuginfo] target(s) in 3.41s
     Running `target/debug/soTesting`
Room name: "{\"name\":\"/tasty-animal-heap\"}"

In fact, we can cleanly rewrite it as (tested ofcourse):

use reqwest;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();
    let res = client.post("https://www.watchparty.me/createRoom")
        .header("Content-Type", "application/json")
        .body(r#"{"video":""}"#)
        .send()
        .await?;
    println!("{}", res.text().await?);
    Ok(())
}

Upvotes: 2

Related Questions