CODEWITHSUNDEEP
arraysf#
App2015
App2015

Reputation: 993

process subarrays asynchronously and reduce the results to a single array

Input If input is in the form of array of arrays.

let items = [|
        [|"item1"; "item2"|]
        [|"item3"; "item4"|]
        [|"item5"; "item6"|]
        [|"item7"; "item8"|]
        [|"item9"; "item10"|]
        [|"item11"; "item12"|]
    |]

Asynchronous action that returns asynchronous result or error

let action (item: string) : Async<Result<string, string>> =
     async {
        return Ok (item + ":processed")
     }

Attempt process one subarray at a time in parallel

let result = items
            |> Seq.map (Seq.map action >> Async.Parallel)
            |> Async.Parallel // wrong? process root items sequentially
            |> Async.RunSynchronously

Expectations:

a) Process one subarray at a time in parallel, then process the second subarray in parallel and so on. (In other words sequential processing for the root items and parallel processing for subitems)

b) Then collect all the results and merge them into a singly dimensioned results array while maintaining the order.

c) Preferably using built-in methods provided by Array, Seq, List, Async etc. instead of any custom operators (that'd be last resort)

d) Optional - If it's not possible to have something within the chain, then as a last resort perhaps convert the result subarrays into single array at the end and return to the caller, if that leads to a cleaner and minimalistic approach which I prefer.

Attempt 2

let result2 = items
            |> Seq.map (Seq.map action >> Async.Parallel)
            |> Async.Parallel // wrong? is it processing root items sequentially
            |> Async.RunSynchronously
            |> Array.collect id


Array.iter (fun (item: Result<string, string>) ->
    match item with
    | Ok r -> Console.WriteLine(r)
    | Error e -> Console.WriteLine(e)
) result2

Edit

let action (item: string) : Async<Result<string, string>> =
     async {
        return Ok (item + ":processed")
     }

let items = [| "item1"; "item2"; "item3"; "item4"; "item5"; "item6"; "item7"; "item8"; "item9"; "item10"|]

let result = items
          |> Seq.chunkBySize 2
          |> Seq.map (Seq.map action >> Async.Parallel)
          |> Seq.map Async.RunSynchronously
          |> Seq.toArray
          |> Array.collect id

Upvotes: 3

Views: 140

Answers (1)

ekim boran
ekim boran

Reputation: 1819

let result = items |> Array.map ( Array.map action >> Async.Parallel)
                   |> Array.map Async.RunSynchronously
                   |> Array.collect id

Edit: Note that majority of operations defined on Seq can be found in array and vice versa. If you initially have an array you can use array operation all the way down.

let items = [| "item1"; "item2"; "item3"; "item4"; "item5"; "item6"; "item7"; "item8"; "item9"; "item10"|]

let result = items 
  |> Array.chunkBySize 2 
  |> Array.map (Array.map action >> Async.Parallel >> Async.RunSynchronously)          
  |> Array.concat

Upvotes: 3

Related Questions