CODEWITHSUNDEEP
linuxmysqllarge-files
geo
geo

Reputation: 1001

Edit very large sql dump/text file (on linux)

I have to import a large mysql dump (up to 10G). However the sql dump already predefined with a database structure with index definition. I want to speed up the db insert by removing the index and table definition.

That means I have to remove/edit the first few lines of a 10G text file. What is the most efficient way to do this on linux?

Programs that require loading the entire file into RAM will be an overkill to me.

Upvotes: 19

Views: 27592

Answers (4)

Ofir Hadad
Ofir Hadad

Reputation: 1900

sed 's/OLD_TEXT/NEW_TEXT/g' < oldfile > newfile

or

cat file | sed 's/OLD_TEXT/NEW_TEXT/g' > newfile

Upvotes: 2

sierrasdetandil
sierrasdetandil

Reputation: 461

joe is an editor that works well with large files. I just used it to edit a ~5G SQL dump file. It took about a minute to open the file and a few minutes to save it, with very little use of swap (on a system with 4G RAM).

Upvotes: 7

Chas. Owens
Chas. Owens

Reputation: 64939

Perl can read the file line by line:

perl -pi.bak -e 's/^create index/--create index/'

Upvotes: -1

rmmh
rmmh

Reputation: 7095

Rather than removing the first few lines, try editing them to be whitespace.

The hexedit program can do this-- it reads files in chunks, so opening a 10GB file is no different from opening a 100KB file to it.

$ hexedit largefile.sql.dump
tab (switch to ASCII side)
space (repeat as needed until your header is gone)
F2 (save)/Ctrl-X (save and exit)/Ctrl-C (exit without saving)

Upvotes: 39

Related Questions