CODEWITHSUNDEEP
linuxbashshellunixterminal
Pebba
Pebba

Reputation: 11

uniq terminal command doesnt work?

I'am trying to learn how to use the terminal command uniq. I'am using mac, bash shell (unix).

this is my textfile "terminal.txt":

this is a line  
this is a line  
this is a line

this is also a line  
this is also a line 

this is not a line

I found an example on using uniq here http://www.computerhope.com/unix/uuniq.htm so this is how I intended using the command. I have not copied the text I wrote it manually into the textfile. However when I type:

uniq terminal.txt

I get this error message:

uniq: terminal.txt: Illegal byte sequence

after some googling I found that I should write LC_ALL=C in front of uniq: 

LC_ALL=C uniq terminal.txt

But when I type this I get this output:

??t

This is not the correct output. I don't know what I'm doing wrong, I can't find a answer on google either.

I there anyone out there who know what I'm doing wrong?

update:

this is the result of od -c terminal.txt:

0000000  377 376   t  \0   h  \0   i  \0   s  \0      \0   i  \0   s  \0
0000020       \0   a  \0      \0   l  \0   i  \0   n  \0   e  \0      \0
0000040   \n  \0   t  \0   h  \0   i  \0   s  \0      \0   i  \0   s  \0
0000060       \0   a  \0      \0   l  \0   i  \0   n  \0   e  \0      \0
0000100   \n  \0   t  \0   h  \0   i  \0   s  \0      \0   i  \0   s  \0
0000120       \0   a  \0      \0   l  \0   i  \0   n  \0   e  \0  \n  \0
0000140   \n  \0   t  \0   h  \0   i  \0   s  \0      \0   i  \0   s  \0
0000160       \0   a  \0   l  \0   s  \0   o  \0      \0   a  \0      \0
0000200    l  \0   i  \0   n  \0   e  \0      \0  \n  \0   t  \0   h  \0
0000220    i  \0   s  \0      \0   i  \0   s  \0      \0   a  \0   l  \0
0000240    s  \0   o  \0      \0   a  \0      \0   l  \0   i  \0   n  \0
0000260    e  \0      \0  \n  \0  \n  \0   t  \0   h  \0   i  \0   s  \0
0000300       \0   i  \0   s  \0      \0   n  \0   o  \0   t  \0      \0
0000320    a  \0      \0   l  \0   i  \0   n  \0   e  \0  \n  \0        
0000336

and this is the result of file terminal.txt:

terminal.txt: Little-endian UTF-16 Unicode text

and cat terminal.txt:

??this is a line

this is a line 

this is a line


this is also a line 

this is also a line 


this is not a line

Upvotes: 0

Views: 2566

Answers (1)

Julian
Julian

Reputation: 2907

Your file is encoded in UTF-16 which uniq isn't able to handle.

To convert the file to UTF-8 which it can, do the following:

iconv -f utf-16 -t utf-8 terminal.txt > terminal2.txt

The uniq command should then work on your newly created file.

In order to resolve this going forward you should work to understand your editor settings and change them so your files are stored as UTF-8 instead of UTF-16.

You may also want to get familiar with the simple editors you can run on the command line such as vi/vim, emacs, or nano, although these editors take some getting used to at first. These editors generally create simple text files which won't have the problems caused by the editor you used to create your file.

Upvotes: 4

Related Questions