CODEWITHSUNDEEP
unixgnu-parallelulimit
arunb2w
arunb2w

Reputation: 1196

concurrent max process limit when gnuparallel used in unix

I am using gnu parallel to fire some concurrent process. Below is the command am using.

parallel 'node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES' ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6 | grep "Query Executed"|wc -l

As per expectation it should fire 120 parallel process concurrently. But when I did execute the ps command I see only 90 process running. Below is the output.

[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node
2951      66965 196290 15 07:50 pts/25   00:00:00 perl /usr/bin/parallel node  /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6
2951      67082  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 1
2951      67087  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 2
2951      67088  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 3
2951      67093  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 4
2951      67098  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 5
2951      67103  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 6
2951      67108  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 1
2951      67113  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 2
2951      67118  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 3
2951      67123  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 4
2951      67128  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 5
2951      67133  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 6
2951      67138  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 1
2951      67143  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 2
2951      67148  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 3
2951      67153  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 4
2951      67158  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 5
2951      67163  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 6
2951      67168  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 1
2951      67173  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 2
2951      67178  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 3
2951      67183  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 4
2951      67188  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 5
2951      67193  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 6
2951      67198  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 1
2951      67203  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 2
2951      67208  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 3
2951      67213  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 4
2951      67218  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 5
2951      67223  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 6
2951      67228  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 1
2951      67233  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 2
2951      67238  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 3
2951      67243  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 4
2951      67248  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 5
2951      67253  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 6
2951      67258  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 1
2951      67263  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 2
2951      67268  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 3
2951      67273  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 4
2951      67278  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 5
2951      67283  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 6
2951      67288  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 1
2951      67293  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 2
2951      67298  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 3
2951      67303  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 4
2951      67308  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 5
2951      67313  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 6
2951      67318  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 1
2951      67323  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 2
2951      67328  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 3
2951      67333  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 4
2951      67338  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 5
2951      67343  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 6
2951      67348  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 1
2951      67353  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 2
2951      67358  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 3
2951      67363  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 4
2951      67368  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 5
2951      67373  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 6
2951      67378  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 1
2951      67383  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 2
2951      67388  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 3
2951      67393  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 4
2951      67398  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 5
2951      67403  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 6
2951      67404  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 1
2951      67409  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 2
2951      67418  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 3
2951      67423  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 4
2951      67428  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 5
2951      67433  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 6
2951      67438  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 1
2951      67443  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 2
2951      67448  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 3
2951      67453  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 4
2951      67458  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 5
2951      67463  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 6
2951      67468  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 1
2951      67473  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 2
2951      67478  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 3
2951      67483  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 4
2951      67488  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 5
2951      67493  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 6
2951      67498  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 1
2951      67503  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 2
2951      67508  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 3
2951      67513  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 4
2951      67523  66905  0 07:50 pts/0    00:00:00 grep node
[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node|wc -l
90

Any reason why only 90 processes got fired concurrently?

And when I checked the ulimt -u it has the limit as 2000 so really not able to understand why it only spawned 90 parallel process alone.

[prod PTEST1] parunkarthick@dev118:~> ulimit -u
2000

Any help would be really appreciated!!!

Upvotes: 1

Views: 3316

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207465

By default, GNU Parallel will use a degree of parallelism that matches the number of CPU cores you have, which you can test with:

parallel --number-of-cores

If you want to run a different number of parallel jobs, use the -j parameter, E.g. for 200 parallel jobs, use:

parallel -j 200 ...

Or, if you want around 50% more jobs in parallel than you have cores, use a percentage:

parallel -j 150% ...

By the way, your command might be a bit less unwieldy with the following syntax if you are using bash:

parallel  'node  /home/XYZ.js F_CHARGES' ::: {1..20} ::: {1..6}

Upvotes: 1

Related Questions