CODEWITHSUNDEEP
concurrencyprocesserlangidioms
user235273
user235273

Reputation:

Should I spawn multiple process for a fun or spawn one and send multiple messages?

Say I have a list of file names (.zip). I want to extract the files concurrently (parallel), and for each extracted file, process them in parallel. My current code is some what like:

start() ->
    % ..
    Pid = spawn_link(fun process/0),
    Pid ! {self(), process_stats},
    receive
        {Pid, Response} -> Response;
        {'EXIT', Pid, Reason} -> ..
    end.

process() ->
    receive
        {Pid, process_stats} ->
            Apps = get_apps(),
            Archive = spawn_link(fun archive/0),
            lists:foreach(fun({Id, AppId}) ->
                              Archive ! {self(), {extract, ?STATS_DIR++AppId++".zip", Pid}}
                          end, Apps),
            process();
        {Pid, {onextract, FilesList, Reply}} ->
            Reply ! {self(), ok},  %
            process();            
    end.

archive() ->
    receive
        {Pid, {extract, FilePath, Reply}} -> % Passing in Reply so that the receive block can send back message to the initiator. But it looks odd.
            {ok, FilesList} = zip:extract(FilePath, [{cwd, ?STATS_DIR}]),
            Pid ! {self(), {onextract, FilesList, Reply}},
            archive()
    end.

get_apps() -> ok. % returns a list of file names.

So, my question is I'm spawing only one process Archive and sending multiple messages. In doing so will the messages be processed concurrently? The Inside Erlang VM paper says that there are multiple run queues for each scheduler, so I'm assuming that messages can be processed concurrently? Or in order to process messages concurrently do I have to spawn multiple processes and send them one message each? Like in 

        lists:foreach(fun({Id, AppId}) ->
                          Archive = spawn_link(fun archive/0),
                          Archive ! {self(), {extract, ?STATS_DIR++AppId++".zip"}}, % Here I don't have to send the Reply Pid as the receive statement is within.
                          receive
                              {Archive, {onextract, FilesList}} -> ok. % Is it ok to have nested receive blocks within a process?
                          end
                      end, Apps),

Is it ok to have a process to have nested receive blocks? Which approach is more appropriate here?

Upvotes: 1

Views: 182

Answers (1)

Peer Stritzinger
Peer Stritzinger

Reputation: 8372

Erlang schedulers schedule processes not messages. If you want to have something happening concurrently you need multiple processes.

The flow inside processes is sequential, so your implementation of archive/0 is receiving a message, then extracting, then replying and after recursing back it receives the next message.

Since spawning a process is cheap there is nothing wrong with having a process per file. No need to send a message to start the processing, you can pass the file name in a closure. Here e.g. with a list comprehension:

[ spawn_link(fun() -> {ok, Fs} = zip:extract(Path, Options), Pid ! {repl, Fs} end) ||
      Path <- List ],
 %% collect replies

Or even easier use rpc:pmap and let it do all the work.

Upvotes: 4

Related Questions