CODEWITHSUNDEEP
f#parallel-processing
Terry C
Terry C

Reputation: 99

F# Array.Parallel hanging

I have been struggling with parallel and async constructs in F# for the last couple days and not sure where to go at this point. I have been programming with F# for about 4 months - certainly no expert - and I currently have a series of calculations that are implemented in F# (asp.net 4.5) and are working correctly when executed sequentially. I am running the calculations on a multi-core server and since there are millions of inputs to perform the same calculation on, I am hoping to take advantage of parallelism to speed it up.

The calculations are extremely data parallel - basically the exact calculation on different input data. I have tried a number of different avenues and I continually run into the same issue - it seems as if the parallel looping never gets to the end of the input data set. I have tried TPL, ConcurrentQueues, Parallel.Array.map/iter and all the same result: the program starts out fine and then somewhere in the middle (indeterminate) it just hangs and never completes. For simplicity I actually removed the calculation from the program and I am just calling a print method, and Here is where the code is currently at:

let runParallel = 

    let ids = query {for c in db.CustTable do select c.id} |> Seq.take(5) 

    let customerInputArray= getAllObservations ids

    Array.Parallel.iter(fun c -> testParallel c)  customerInputArray

    let key = System.Console.ReadKey()
    0

A few points... I limited the results above to only 5 just for debugging. The actual program does not apply the Take(5). The testParallel method is just a printfn "test".

The customerInputArray is a complex data type. It is a tuple of lists that contain records. So I am pretty sure my problem must be there...but I added exception handling and no exception is getting raised, so have no idea how to go about finding the problem.

Any help is appreciated. Thanks in advance.

EDIT: Thanks for the advice...I think it is definitely deadlock. When I remove all of the printfn, sprintfn, and string concat operations, it completes. (of course, I need those things in there.)

Is printfn, sprintfn, and string ops not thread-safe?

Another EDIT: Iteration always stops on the last item..So if my input array has 15 items, the processing stops on item 14, or seems to never get to item 15. Then everything just hangs. Does not matter what the size of the input array is..Any ideas what can be causing this? I even switched over to Parallel.ForEach (instead of Array.Parallel) and same behavior.

Upvotes: 1

Views: 436

Answers (1)

Terry C
Terry C

Reputation: 99

Update on the situation and how I resolved this issue.

I was unable to upload code from my example due to my company's firewall policy, so in the end my question did not have enough details. I failed to mention that I was using a type provider which was important information in this situation. But here is what I figured out.

I am using the F# type provider for SQL Server and was passing around its Service Types which I suspect are not thread-safe. When I replaced the ServiceTypes with plain old F# Records, the code worked fine - no more deadlocks and everything completed without error.

Upvotes: 2

Related Questions