CODEWITHSUNDEEP
sortingrandomlarge-files
Maetela
Maetela

Reputation: 31

how to sort in random a large file

I have a big txt file ( list of coupon separate by a comma) the coupon are in an order list, I need to randomise it. But my file is too big ( around 16GB), the unix command sort cannot manage it - memory issue). any idea how I can randomised it in another way?

Upvotes: 3

Views: 193

Answers (1)

user515430
user515430

Reputation: 316

Here is what I would do. For each entry generate four random bits. All entries with 0000 as the four bits are copied into a temporary file f0000, all entries with 0001 as the four bits are copied into a temporary file f0001, etc.

Now shuffle f000 and write the shuffled entries to the output file fs. Shuffle f0001 and append the shuffled entries to fs, etc.

It is fairly easy to see that the resulting output file, fs, is a shuffle of the original file.

If four bits are not enough, go to five bits or six bits. You get the idea.

Upvotes: 2

Related Questions