CODEWITHSUNDEEP
bashfile-descriptor
JorgeeFG
JorgeeFG

Reputation: 5941

Bash running out of File Descriptors

I was working on a project, and I want to contribute with the solution I found:

The code is of the kind:

while true
do
     while read VAR
     do
        ......
     done < <(find ........ | sort)
     sleep 3
done

The errors in the log where:

/dir/script.bin: redirection error: cannot duplicate fd: Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: line 26: <(find "${DIRVAR}" -type f -name '*.pdf' | sort): Too many open files
find: `/somedirtofind': Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: line 26: <(find "${DIRVAR}" -type f -name '*.pdf' | sort): ambiguous redirect

I noticed with command:

ls -l /proc/3657(pid here)/fd that the file descriptors were constantly increasing.

Using Debian 7, GNU bash, version 4.2.37(1)-release (i486-pc-linux-gnu)

Upvotes: 5

Views: 5679

Answers (2)

a113nw
a113nw

Reputation: 1412

I was having the same issue and played around with the things you suggested. You said:

But executing the script with ./test.sh & (in background), works without problems too.

So what worked for me is to run in the background and just wait for it to finish each time:

while true
do
     while read VAR
     do
        ......
     done < <(find ........ | sort) &
     wait
done

Another thing that worked was to put the code creating the descriptor into a function without running on background:

function fd_code(){
     while read VAR
     do
        ......
     done < <(find ........ | sort)
}

while true
do
     fd_code
done

Upvotes: 0

JorgeeFG
JorgeeFG

Reputation: 5941

The solution that worked for me is:

while true
do
     find ........ | sort | while read VAR
     do
     done
     sleep 3
done

Which is, avoiding the subshell at the end, there must be some kind of leak.

Now I don't see file descriptors in the process directory when doing ls

Mail to bugtracker:

The minimum reproduceable code is:

#!/bin/bash
function something() {
  while true
  do
    while read VAR
    do
      dummyvar="a"
    done < <(find "/run/shm/directory" -type f | sort)
    sleep 3
  done
}
something &

Which fails with many pipes fd open.

Changing the While feed to this:

#!/bin/bash
function something() {
  find "/run/shm/directory" -type f | sort | while true
  do
    while read VAR
    do
      dummyvar="a"
    done
    sleep 3
  done
}
something &

Works completely normal.

However, removing the call as function in background:

#!/bin/bash
while true
do
  while read VAR
  do
    dummyvar="a"
  done < <(find "/run/shm/debora" -type f | sort)
  sleep 3
done

But executing the script with ./test.sh & (in background), works
without problems too.

Upvotes: 4

Related Questions