Reputation: 123
I'm trying to uncompress a file from www.dukascopy.com using lzma (files seems to be binary, it's in their .bi5 format). It's forex data for the EUR/USD pair. Excuse me, but I'm not very familiar with how to read binary files. Did I decompress it right? If so, how should I convert my resulting data that I have decompressed.
Test file: https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5
My test code:
import lzma
with open('/Users/me/Downloads/dukascopy_php_scripts_v0.27/EURUSD/2008/05/16/00h_ticks.bi5','rb') as f:
infile = f.read()
decompressor = lzma.LZMADecompressor()
print(lzma.decompress(infile))
Which prints:
(starting with).....b'\x00\x00\x16\xd4\x00\x02X\xf6\x00\x02X\xe6@\x19\x99\x9a?\xe6ff\x00\x00 (\x00\x02X\xf7\x00\x02X\xe3A\x10\x00\x00@\x19\x99\x9a\x00\x00)c\x00\x02X\xed\x00\x02X\xe8@\xc0\x00\x00@\x9333\x00\x00.\x87\x00\x02X\xe9\x00\x02X\xe4@\x99\x99\x9a@\xc0\x00\x00\x00\x00.\xfa\x00\x02X\xe8\x00\x02X\xe3A\x10\x00\x00@\x19\x99\x9a\x00\x000N\x00\x02X\xe8\x00\x02X\xe3Aq\x99\x9a@\x99\x99\x9a\x00\x000\xc0\x00\x02X\xee\x00\x02X\xe4@\xc0\x00\x00?\xe6ff\x00\x001<\x00\x02X\xf4\x00\x02X\xeaA\x14\xcc\xcd@\x19\x99\x9a\x00\x001\xb3\x00\x02X\xf9\x00\x02X\xef@\xac\xcc\xcd@\xe6ff\x00\x002&\x00\x02X\xfb\x00\x02X\xf6@\x99\x99\x9a@\x19\x99\x9a\x00\x00T\xd7\x00\x02Y\x03\x00\x02X\xef@\x19\x99\x9a?\x99\x99\x9a\x00\x00U\x1b\x00\x02X\xfa\x00\x02X\xf5A\x14\xcc\xcd?\x80\x00\x00\x00\x00U?\x00\x02X\xf6\x00\x02X\xf1@\x99\x99\x9aAS33\x00\x00Uc\x00\x02X\xf3\x00\x02X\xe9@\x99\x99\x9a@\xac\xcc\xcd\x00\x00U\xef\x00\x02X\xe8\x00\x02X\xde@\x86ffA#33\x00\x00V\x13\x00\x02X\xdd\ ......(ending with).....
\x00\x02Y+\x00\x02Y!?\x80\x00\x00@\x99\x99\x9a'
Upvotes: 2
Views: 2544
Reputation: 19050
You can iteratively decompress the file without reading it entirely into memory like this:
Example:
#!/usr/bin/env python
URL = "https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5"
from requests import get
from lzma import LZMADecompressor
decompressor = LZMADecompressor()
response = get(URL, stream=True)
with open("foo.bin", "wb") as f:
for chunk in response.iter_content(chunk_size=1024):
if chunk:
result = decompressor.decompress(chunk)
if result != b"":
f.write(result)
We can then confirm and verify the output by comparing a download of the file and decompression with the lzma
command line tool:
Output:
$ python foo.py
$ curl -q -o - https://dl.dropboxusercontent.com/u/11283578/00h_ticks.bi5 | lzma -d - > test.bin
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11600 100 11600 0 0 7509 0 0:00:01 0:00:01 --:--:-- 7508
$ shasum foo.bin
c36d2314e43f3c71ad85e75e748b4f54edc8d74a foo.bin
$ shasum test.bin
c36d2314e43f3c71ad85e75e748b4f54edc8d74a test.bin
Since I don't have local access to the same file as you nor do I have the same environment the above example downloads the file you put on Dropbox and decompresses it iteratively with Python 3.4 and the lzma.LZMADecompressor
library/class.
Upvotes: 1