CODEWITHSUNDEEP
bashdockerredis
mrdoubtful
mrdoubtful

Reputation: 529

Convert Redis Mass Insertion Protocol Format independent of OS

I am trying to perform Redis mass insertion using the command cat data.txt | redis-cli --pipe as mentioned in https://redis.io/topics/mass-insert.

The data format on macOS has to be converted so that mass insertion could be performed with cat ${FILE} | perl -i -p -e 's|[\r\n]+|\r\n|g' | redis-cli --pipe.

However, the above command does not work on a Linux environment (or a docker environment with the container built from an alpine based image). Instead, the following command has to performed cat ${FILE} | sed 's/\r*$/\r/' | redis-cli --pipe.

Is there a command that would work in both environments?

EDIT: Attached the following:

  1. Redis Mass Insertion script on Alpine Linux: https://gist.github.com/francjohny/f2b13b4cfc147e07e52824ec88ba3781

  2. Redis Mass Insertion script on Mac OS: https://gist.github.com/francjohny/b57756a1e0124dd562959ca5ece2a32b

  3. Redis Protocol Format data file: https://gist.github.com/francjohny/0c21f32d9902809b215f4e92f5e6a9f1

  4. ➜ head ouput.rpf| xxd - Mac OS : https://gist.github.com/francjohny/e1a646ab44e7edd7374d28e9ca400711

  5. ➜ head ouput.rpf| xxd - Alpine Linux: https://gist.github.com/francjohny/252904928ded4c045448d12b205228df

Upvotes: 1

Views: 492

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207660

Updated Answer

From the data you have added, it seems you just have linefeeds separating your lines, whereas Redis requires carriage return followed by linefeed. So basically, you want the equivalent of the unix2dos program, which is not included in macOS. However, macOS does include Perl, so you should be able to use:

perl -pe 's/\n/\r\n/' data.rpf | redis-cli --pipe

It works fine on my Mac.

Original Answer

You appear to have mixed line endings in your various environments. I would imagine this Perl would replace any number of carriage returns and line feeds in any mixture with a single carriage return and linefeed like Redis requires:

perl -pe 's|[\r\n]*|\r\n|' data.txt | redis-cli ...

If not, please answer my question in the comments.

Upvotes: 1

Related Questions