How to pass a very very long list of arguments to a program?

Question

I'm coding and running programs to which I need to pass a long list of data files for analysis, sometimes a few hundred thousands. The problem is that the argument list can be so long that the system (Unix) refuses to run it, outputting:

bash: ./yourProgram: Argument list too long

Is there a environment variable I can change to bypass this obstacle?

The only solution I can think of is writing my program list in a separate file (using ls ... >) and then reading each file line by line. Would you know of anything simpler?

ps: my programs are written in C++ if it matters

Answer 1

How to pass a very very long list of arguments to a program?

1. place the arguments in a file

2. redirect the file to the standard input of the application, on startup

   bash$ echo "arg1 arg2 arg3 ... argn" >> inputs.txt
   bash$ ./yourProgram  < inputs.txt
   

This has the advantage of storing your arguments (so that for a subsequent execution, you only need to run the second line).

Answer 2

I would just pass it as stdin..

echo "file1 file2 file3" | ./program

Answer 3

Better to have an environment variable defined with values as space delimited list of items, for example, define as

export MYLIST=a b ab cd ef

Within your program, use getenv("MYLIST") to get the value as char *, and tokenize to get individual values