hexdump and xxd produce different outputs on red hat and ubuntu/mint

Question

Firstly , I am mentioning whatever happened on my linux mint/ubuntu system ...

udit@udit-Dabba ~/ah $ xxd -r -p input.txt output.txt

udit@udit-Dabba ~/ah $ cat input.txt 
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08 
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29 

udit@udit-Dabba ~/ah $ cat output.txt
` z2P��~�)

udit@udit-Dabba ~/ah $ hexdump -C output.txt 

00000000  60 00 00 00 00 14 06 20  00 00 00 00 00 00 00 00  |`...... ........|
00000010  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 02  00 15 00 15 7a 32 b1 08  |............z2..|
00000030  00 00 00 00 50 02 ff ff  7e 95 00 00 00 00 02 29  |....P...~......)|
00000040

udit@udit-Dabba ~/ah $ od -x output.txt 
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0000 0200 1500 1500 327a 08b1
0000060 0000 0000 0250 ffff 957e 0000 0000 2902
0000100

Now when I tried same thing on red hat , some strange behaviour is occuring.

[root@cc]$ xxd -r -p input.txt output.txt

[root@cc]$ cat input.txt 
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08 
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29 

[root@cc]$ cat output.txt
` P~)[root@cc]$ hexdump -C output.txt 
00000000  60 00 00 00 00 14 06 20  00 00 00 00 00 00 00 00  |`...... ........|
00000010  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 50 02 ff ff  7e 95 00 00 00 00 02 29  |....P...~......)|
00000030

[root@cc]$ od -x output.txt 
0000000 0060 0000 1400 2006 0000 0000 0000 0000
0000020 0000 0000 0000 0100 0000 0000 0000 0000
0000040 0000 0000 0250 ffff 957e 0000 0000 2902
0000060

I am not able to find out why this discrepency occcuring between outputs of red hat and linux mint/ubuntu.

If possible tell me some awk/shell to perform the same in place of xxd -r -p

Pleae help me !!

EDIT:

On linux mint/ubuntu system ..

 udit@udit-Dabba ~/ah $ hexdump -C input.txt
 00000000  36 30 20 30 30 20 30 30  20 30 30 20 30 30 20 31  |60 00 00 00 00 1|
 00000010  34 20 30 36 20 32 30 20  30 30 20 30 30 20 30 30  |4 06 20 00 00 00|
 00000020  20 30 30 20 30 30 20 30  30 20 30 30 20 30 30 20  | 00 00 00 00 00 |
 00000030  0a 30 30 20 30 30 20 30  30 20 30 30 20 30 30 20  |.00 00 00 00 00 |
 00000040  30 30 20 30 30 20 30 31  20 30 30 20 30 30 20 30  |00 00 01 00 00 0|
 00000050  30 20 30 30 20 30 30 20  30 30 20 30 30 20 30 30  |0 00 00 00 00 00|
 00000060  20 0a 30 30 20 30 30 20  30 30 20 30 30 20 30 30  | .00 00 00 00 00|
 00000070  20 30 30 20 30 30 20 30  32 20 30 30 20 31 35 20  | 00 00 02 00 15 |
 00000080  30 30 20 31 35 20 37 41  20 33 32 20 42 31 20 30  |00 15 7A 32 B1 0|
 00000090  38 20 0a 30 30 20 30 30  20 30 30 20 30 30 20 35  |8 .00 00 00 00 5|
 000000a0  30 20 30 32 20 46 46 20  46 46 20 37 45 20 39 35  |0 02 FF FF 7E 95|
 000000b0  20 30 30 20 30 30 20 30  30 20 30 30 20 30 32 20  | 00 00 00 00 02 |
 000000c0  32 39 20 0a 0a 0a 0a 0a                           |29 .....|
 000000c8

On red hat machine ..

 [root@cc]$ hexdump -C input.txt
 00000000  36 30 20 30 30 20 30 30  20 30 30 20 30 30 20 31  |60 00 00 00 00 1|
 00000010  34 20 30 36 20 32 30 20  30 30 20 30 30 20 30 30  |4 06 20 00 00 00|
 00000020  20 30 30 20 30 30 20 30  30 20 30 30 20 30 30 0a  | 00 00 00 00 00.|
 00000030  30 30 20 30 30 20 30 30  20 30 30 20 30 30 20 30  |00 00 00 00 00 0|
 00000040  30 20 30 30 20 30 31 20  30 30 20 30 30 20 30 30  |0 00 01 00 00 00| 
 00000050  20 30 30 20 30 30 20 30  30 20 30 30 20 30 30 20  | 00 00 00 00 00 |
 00000060  0a 30 30 20 30 30 20 30  30 20 30 30 20 30 30 20  |.00 00 00 00 00 |
 00000070  30 30 20 30 30 20 30 32  20 30 30 20 31 35 20 30  |00 00 02 00 15 0|
 00000080  30 20 31 35 20 37 41 20  33 32 20 42 31 20 30 38  |0 15 7A 32 B1 08|
 00000090  20 0a 30 30 20 30 30 20  30 30 20 30 30 20 35 30  | .00 00 00 00 50|
 000000a0  20 30 32 20 46 46 20 46  46 20 37 45 20 39 35 20  | 02 FF FF 7E 95 |
 000000b0  30 30 20 30 30 20 30 30  20 30 30 20 30 32 20 32  |00 00 00 00 02 2|
 000000c0  39 20 20 0a                                       |9  .|
 000000c4       

Answer 1

A late answer, but hopefully may help to somebody else reading this:

xxd had buggy handling of spaces in "-r -p" mode. As it scanned the characters, if it hit 2 consecutive non-hex characters, it would throw away everything until the next newline or end of file. If the second of the 2 consecutive characters was itself the newline character (i.e., the line ended with a space followed by a newline), the net effect would be to discard the entire next line.

I am pretty certain this is what caused Udit's problem. (I think the hexdump he later added of input.txt did not actually reflect the original state of that file.)

It looks like xxd is primarily maintained as part of the vim distribution. I looked at the xxd source included with vim 7.3, and it has the buggy handling problem fixed. Unfortunately nobody bothers to update the version #, so it still reports the same xxd 1.10 version string.

Try creating this test file (with 1 blank space at the end of the first line and none on the second; should be a 22 byte file on Linux/Unix; if you run "xxd -p" on it, it should report 3230200a34652034662035340a34322034312034340a):

20 
4e 4f 54
42 41 44

If you run "xxd -r -p" on this, with a version of xxd exhibiting the buggy handling, you should see it print BAD. With a newer (fixed) version of xxd, it'll print NOTBAD.

If you run into this buggy handling and updating to a newer version of xxd is not an option, you can avoid this problem with some sed preprocessing, e.g.,

$ sed -e 's/ *//g' -e '/^$/d' input.txt | xxd -r -p > output.txt

Answer 2

I'd say that you have some invisible character in the input.txt file that makes the third line look invalid somehow. And maybe the invalid character is encoding-dependent, and that's why it works in one machine and not in other.

It would be helpful to see the result of hexdump -C input.txt to check that.

Answer 3

I have no idea whats happening, but the third line is being eaten on red hat.

This is what you asked for. Reverse hexdump with awk+bash.

echo 'ibase=16' | cat - hdump | tr ' ' \; | bc | awk '{printf("%c",$0)}'

replace hdump with input.txt

echo 'ibase=16' | cat - input.txt | tr ' ' \; | bc | awk '{printf("%c",$0)}' > output.txt

codes were found here: http://www.unix.com/shell-programming-scripting/132294-reverse-hexdump-without-xxd.html#post302404631

Answer 4

$ cat foo
60 00 00 00 00 14 06 20 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 02 00 15 00 15 7A 32 B1 08 
00 00 00 00 50 02 FF FF 7E 95 00 00 00 00 02 29

$ cat foo | xxd -r -p | hexdump -C
00000000  60 00 00 00 00 14 06 20  00 00 00 00 00 00 00 00  |`...... ........|
00000010  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 02  00 15 00 15 7a 32 b1 08  |............z2..|
00000030  00 00 00 00 50 02 ff ff  7e 95 00 00 00 00 02 29  |....P...~......)|
00000040