CODEWITHSUNDEEP
rustrayon
Nane
Nane

Reputation: 403

How to iterate over an array with rayon?

I am new to rust, I am trying to get responses from an array of URLs using rayon

use rayon::prelude::*;
use reqwest::*;

fn main() -> Result<()> {
    let urls = vec!["https://example.com"];
    urls.par_iter().for_each(|url: &&str| ->  Result<()> {
        println!("Hello... {:?}", url);
        let resp = reqwest::blocking::get(url)?.text()?;
        println!("{:#?}", resp);
        Ok(())
    });
    Ok(())
}

but I am getting this error

error[E0271]: type mismatch resolving `<[closure@src\main.rs:252:30: 257:6] as FnOnce<(&&str,)>>::Output == ()`
   --> src\main.rs:252:21
    |
252 |     urls.par_iter().for_each(|url| ->  Result<()> {
    |                     ^^^^^^^^ expected enum `std::result::Result`, found `()`
    |
    = note:   expected enum `std::result::Result<(), reqwest::Error>`
            found unit type `()`

error[E0277]: the trait bound `&&str: IntoUrl` is not satisfied
   --> src\main.rs:254:20
    |
254 |         let resp = reqwest::blocking::get(url)?.text()?;
    |                    ^^^^^^^^^^^^^^^^^^^^^^ the trait `IntoUrl` is not implemented for `&&str`
    | 
   ::: mod.rs:106:15
    |
106 | pub fn get<T: crate::IntoUrl>(url: T) -> crate::Result<Response> {
    |               -------------- required by this bound in `reqwest::blocking::get`
    |
    = help: the following implementations were found:
              <&'a str as IntoUrl>

Upvotes: 0

Views: 1715

Answers (2)

Chayim Friedman
Chayim Friedman

Reputation: 70870

The second error is the easier: IntoUrl is not implemented for &&str, but it is for &str. So all you need is to dereference the pointer:

urls.par_iter().for_each(|url: &&str| -> Result<()> {
    let url = *url;

Or

urls.par_iter().for_each(|&url: &&str| -> Result<()> {

The first is trickier. The problem is that for_each() expects a closure that returns (), and giving it a closure that returns Result does not work.

Because you want to continue with other requests if a request fails, you cannot use rayon's try_for_each() since it will stop all pending requests on failure. Instead, you want to just ignore the failure, perhaps log it. Using unwrap() is a bad idea: not just it will stop other requests, it will also abort your program! Never use unwrap(), unless you're 100% sure the code will never fail (it's also good to comment why if it isn't obvious). You can give up on the ? operator and just be explicit:

urls.par_iter().for_each(|&url| {
    println!("Hello... {:?}", url);
    if let Ok(resp) = reqwest::blocking::get(url) {
        if let Ok(resp) = resp.text() {
            println!("{:#?}", resp);
        }
    }
});

But while you can improve this code a bit, it is still ugly. try blocks will improve the situation; until they land on stable, there is a common idiom for a block with multiple potential failure points:

urls.par_iter().for_each(|&url| {
    _ = (|| -> Result<()> {
        println!("Hello... {:?}", url);
        let resp = reqwest::blocking::get(url)?.text()?;
        println!("{:#?}", resp);
        Ok(())
    })();
});

This uses closure's ability to use the ? operator to simplify the code.

Note however that rayon is not intended for IO-bound tasks: use async framework like tokio.

Upvotes: 2

Chandan
Chandan

Reputation: 11797

for_each take closure/lambda whose return type is unit not Result
? operator cannot be used inside function/closure/lambda which return unit -> ()
* is being used to dereference and accessing the underlying string

src/main.rs

use rayon::prelude::*;

fn main() {
    let urls = vec!["https://example.com"];
    urls.par_iter().for_each(|url: &&str| {
        println!("Hello... {:?}", *url);
        let resp = reqwest::blocking::get(*url).unwrap().text();
        println!("{:#?}", resp);
    });
}

Cargo.toml

[package]
name = "s71275260"
version = "0.1.0"
edition = "2021"

# see more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
rayon = "1.5.1"
reqwest = { version = "0.11", features = ["blocking"] }

Upvotes: 1

Related Questions