CODEWITHSUNDEEP
parallel-processingelixir
Lewis Jones
Lewis Jones

Reputation: 236

Elixir System.cmd in parallel map erroring with timeout

I have a parallel map in Elixir using Task.async. I am using it with System.cmd to parallel ruby rbenv install a list of ruby version strings.

The script runs and installs the ruby versions. However it does not exit the Task once the ruby versions have been installed and errors with a Task.await timeout error.

I have tried passing a simple IO.puts function into the parallel map and behaves correctly, exiting when the work is done and not raising an error. What am I missing in my System.cmd to ensure that each process is ended once each rbenv process is finished.

# My parallel map
def pmap(collection, func) do
  collection
  |> Enum.map(&(Task.async(fn -> func.(&1) end)))
  |> Enum.map(&Task.await/1)
end

# System.cmd is being passed into the map like this
def parallel_install(ruby_versions) do
  pmap(ruby_versions, &(System.cmd("rbenv", ["install", &1])))
end

Output with Error:

rbenv: /Users/lewis.jones/.rbenv/versions/2.4.4 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.5.1 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.1.10 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.3.7 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.2.10 already exists
** (exit) exited in: Task.await(%Task{owner: #PID<0.73.0>, pid: #PID<0.82.0>, 
ref: #Reference<0.100168651.1374158856.61975>}, 5000)
   ** (EXIT) time out
   (elixir) lib/task.ex:491: Task.await/2
   (elixir) lib/enum.ex:1270: Enum."-map/2-lists^map/1-0-"/2
   (elixir) lib/code.ex:376: Code.require_file/2

Upvotes: 0

Views: 469

Answers (1)

Dogbert
Dogbert

Reputation: 222188

Task.await defaults to a timeout of 5 seconds. I'm guessing rbenv install takes longer than that. You can either increase the timeout or set it to infinity.

300 seconds:

|> Enum.map(&Task.await(&1, 300_000))

or infinity:

|> Enum.map(&Task.await(&1, :infinity))

Upvotes: 1

Related Questions