CODEWITHSUNDEEP
unixxargs
John Smith
John Smith

Reputation: 117

xargs does not read line by line properly

Here is my current commands which i use:

xargs -n 1 -P 100 php b <links

I have a script 'b' and a file with links in 'links' and sometimes I don't know why this doesn't work correctly and add to every line symbol "?" like this:

root     19714  0.0  1.0  19880  5480 pts/2    R+   11:19   0:00 php b http://www.google.com?
root     19715  0.0  0.9  19524  4892 pts/2    R+   11:19   0:00 php b http://www.alexa.com?
root     19716  0.0  1.0  19880  5496 pts/2    R+   11:19   0:00 php b http://www.amazon.com?

see to the end of the ps-aux result line is "?" symbol and my file doesn't contain that... but not all the times... what could be the problem?

Upvotes: 2

Views: 1198

Answers (1)

Jonathan Leffler
Jonathan Leffler

Reputation: 754480

Converting comments into an answer

Could your links file have CRLF (Windows or DOS style) line endings? In which case, the ? represents the CR (carriage return)… And the fix is to convert the DOS file into a Unix file somewhere along the line. For example:

tr -d '\r' < links | xargs -n 1 -P 100 php b

I build the links file with cat "[another file]" | sort | uniq > links. So I don't think it is a Windows/DOS style issue.

Does [another file] have DOS-format line endings? If so, links will too; the command line will preserve the CRLF line endings if they're in the input.

Also, the use of cat is often unnecessary (UUoC or Useless Use of Cat), and in this case uniq too is not needed; you could use

sort -u "[another file]" > links

[another file] is like this:

line1\r\nline2\r\nline3\r\n

The \r\n is one of the ways the CRLF line endings are reported on Unix. The \r notation is used for carriage return, CR, in C and other languages; the \n notation is used for newline (NL, aka line feed or LF).

BTW: If I paste the same file with copy/paste via vi or nano it works.

The editors may be converting the file format for you automatically. See: How to convert the ^M linebreak to normal linebreak in a file opened in vim? Alternatively, if you are copying and pasting, then the copy may well not see, and therefore not copy, the CR characters.

I tried with tr -d … and it works.

Good; that at least means this is a reasonable explanation, and it all makes sense. If you don't need the list of links separately, you can use:

sort -u "[another file]" | tr -d '\r' | xargs -n 1 -P 100 php b

This does it all in one command line with no intermediate files to clean up. You could add tee links | before xargs if you do want the files list too, of course.

Upvotes: 2

Related Questions