CODEWITHSUNDEEP
pythoncompressionbzip
Marcin
Marcin

Reputation: 7994

parsing large compressed xml files, python

file  = BZ2File(SOME_FILE_PATH)
p = xml.parsers.expat.ParserCreate()
p.Parse(file)

Here's code that tries to parse xml file compressed with bz2. Unfortunately it fails with a message:

TypeError: Parse() argument 1 must be string or read-only buffer, not bz2.BZ2File

Is there a way to parse on the fly compressed bz2 xml files?

Note: p.Parse(file.read()) is not an option here. I want to parse a file which is larger than available memory, so I need to have a stream.

Upvotes: 2

Views: 3113

Answers (3)

Nick
Nick

Reputation: 321

Just use p.ParseFile(file) instead of p.Parse(file).

Parse() takes a string, ParseFile() takes a file handle, and reads the data in as required.

Ref: http://docs.python.org/library/pyexpat.html#xml.parsers.expat.xmlparser.ParseFile

Upvotes: 5

Joe Koberg
Joe Koberg

Reputation: 26759

Can you pass in an mmap()'ed file? That should take care of automatically paging the needed parts of the file in, and avoid memory overflow. Of course if expat builts a parse tree, it might still run out of memory.

http://docs.python.org/library/mmap.html

Memory-mapped file objects behave like both strings and like file objects. Unlike normal string objects, however, these are mutable. You can use mmap objects in most places where strings are expected; for example, you can use the re module to search through a memory-mapped file.

Upvotes: 0

Amber
Amber

Reputation: 527378

Use .read() on the file object to read in the entire file as a string, and then pass that to Parse?

file  = BZ2File(SOME_FILE_PATH)
p = xml.parsers.expat.ParserCreate()
p.Parse(file.read())

Upvotes: 1

Related Questions