CODEWITHSUNDEEP
shellcommand-linezsh
user1692342
user1692342

Reputation: 5237

zsh: argument list too long: sudo

I've a command which I need to run in which one of the args is a list of comma separated ids. The list of ids is over 50k. I've the stored the list of ids in a file and I'm running the command in the following way:

sudo ./mycommand --ids `cat /tmp/ids.txt`

However I get an error zsh: argument list too long: sudo This I believe is because the kernel has a max size of arguments it can take. One option for me is to manually split the file into smaller pieces (since the ids are comma separated I can't just break it evenly) and then run the command each time for each file.

Is there a better approach?

ids.txt file looks like this:

24342,24324234,122,54545,565656,234235

Upvotes: 1

Views: 3956

Answers (1)

Jonathan Leffler
Jonathan Leffler

Reputation: 754530

Converting comments into a semi-coherent answer.

The file ids.txt contains a single line of comma-separated values, and the total size of the file can be too big to be the argument list to a program.

Under many circumstances, using xargs is the right answer, but it relies on being able to split the input up in to manageable chunks of work, and it must be OK to run the program several times to get the job done.

In this case, xargs doesn't help because of the size and format of the file.

It isn't stated absolutely clearly that all the values in the file must be processed in a single invocation of the command. It also isn't absolutely clear whether the list of numbers must all be in a single argument or whether multiple arguments would work instead. If multiple invocations are not an issue, it is feasible to reformat the file so that it can be split by xargs into manageable chunks. If need be, it can be done to create a single comma-separated argument.

However, it appears that these options are not acceptable. In that case, something has to change.

  • If you must supply a single argument that is too big for your system, you're hosed until you change something — either the system parameters or your program.

Changing the program is usually easier than reconfiguring the o/s, especially if you take into account reconfiguring upgrades to the o/s.

One option worth reviewing is changing the program to accept a file name instead of the list of numbers on the command line:

sudo ./mycommand --ids-list=/tmp/ids.txt

and the program opens and reads the ID numbers from the file. Note that this preserves the existing --ids …comma,separated,list,of,IDs notation. The use of the = is optional; a space also works.

Indeed, many programs work on the basis that arguments provided to it are file names to be processed (the Unix filter programs — think grep, sed, sort, cat, …), so simply using:

sudo ./mycommand /tmp/ids.txt

might be sufficient, and you could have multiple files in a single invocation by supplying multiple names:

sudo ./mycommand /tmp/ids1.txt /tmp/ids2.txt /tmp/ids3.txt …

Each file could be processed in turn. Whether the set of files constitutes a single batch operation or each file is its own batch operation depends on what mycommand is really doing.

Upvotes: 2

Related Questions