CODEWITHSUNDEEP
rustrayon
Saqib Ali
Saqib Ali

Reputation: 4408

Executing external commands in parallel and capturing the output in an array in Rust

I have the following while loop that runs generate_user_key for each of the file in the file_array, and outputs the result. I would like to parallelize this such that an array of the generated keys is returned, and the process is executed in parallel instead of sequential to make it faster.

use std::process::Command;

//file_array definition here

let mut i = 0;
while (i<100) {
  let generated_key = Command::new("generate_user_key")
                       .arg(file_array[i])
                       .output()
                       .expect("generate_user_key command failed to start");
  println!("stdout: {}", String::from_utf8_lossy(&generated_key.stdout));
  i=i+1;
}

What is the best way to implement this in rust?

Upvotes: 4

Views: 1437

Answers (3)

Jmb
Jmb

Reputation: 23264

This should be easy to do with rayon. E.g. something like this (untested since I don't have your generate_user_key):

use rayon::prelude::*;
let keys = (0..100).into_par_iter().map (|_| {
        Command::new("generate_user_key")
            .arg(file_array[i])
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();

or better:

use rayon::prelude::*;
let keys = file_array.par_iter().map (|f| {
        Command::new("generate_user_key")
            .arg(f)
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();

Upvotes: 0

Chandan
Chandan

Reputation: 11797

If you want to loop over the array items using rayon then you can simply create into_par_iter and work on array items

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = arr.into_par_iter().flat_map(|value| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", value)])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", value, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

You can also use range to loop over and use the counter as array index

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = (0..arr.len()).into_par_iter().flat_map(|idx| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", arr[idx])])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", idx, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

Example using thread

use std::thread;
use std::time::Duration;

fn main() {
    let mut threads = vec![];
    for idx in 0..arr.len() {
        threads.push(thread::spawn(move || -> Vec<_> {
            let output = Command::new("sh")
                    .args(["-c", &format!("echo -n {}", idx)])
                    .output()
                    .expect("failed to execute process");
            println!("Index: {}, Output: {:?}", idx, output.stdout);
            thread::sleep(Duration::from_millis(1));
            output.stdout
        }));
    }

    let result = threads.into_iter().flat_map(|c| c.join().unwrap()).collect::<Vec<_>>();

    println!("{:?}", result);
}

Upvotes: 2

Locke
Locke

Reputation: 8944

When all else fails, throw threads at the problem. It almost certainly isn't the correct approach, but it works.

let mut join_handles = Vec::new();

for _ in 0..100 {
    join_handles.push(thread::spawn(|| {
        let generated_key = Command::new("generate_user_key")
                              .arg(file_array[i])
                              .output()
                              .expect("generate_user_key command failed to start");

        String::from_utf8_lossy(&generated_key.stdout)
    }));
}

let outputs = join_handles.into_iter().map(Result::unwrap).collect::<Vec<_>>();

Edit: The correct solution is probably using Command::spawn to start the processes without blocking. The OS can then handle running them in parallel and you can then collect the outputs.

Upvotes: -1

Related Questions