CODEWITHSUNDEEP
pythoncsv
PouJa
PouJa

Reputation: 289

How to write bytes to csv file as byte strings not integers?

There is a list of bytes objects (each one is 4 bytes) that is returned as output of one code and I want to save it into a .csv file using CSV module and read it back later in another script. Here is the code that I have learnt from python's official documentation:

import struct
import csv

k   = 0x100000
rng = range(0, k)
x1 = [b''] * k
x = 0xffffffff

for i in rng:
    x1[i]   = struct.pack("<L", x)
    x -= 1

print(x1[0])              # b'\xff\xff\xff\xff'

List = x1

with open("test.csv", 'w', newline='') as rF:
    wr = csv.writer(rF, dialect='excel')
    for i in List:
        wr.writerow(i)

When looking inside the created test.csv using notepad, instead of a column of byte strings I see 4 columns of 8-bit integers. Few first lines of test.csv are:

255,255,255,255
254,255,255,255
253,255,255,255
252,255,255,255
251,255,255,255
250,255,255,255
       .
       .
       .

What am I doing wrong that this is happening? Is there a way to get a csv file with one column of byte strings? for example:

b'\xff\xff\xff\xff'
b'\xfe\xff\xff\xff'
b'\xfd\xff\xff\xff'
          .
          .
          .

Actually I do not care how are my bytes stored in a csv. I just care to have them back into a list of bytes using csv.reader in another script and want the loading process be the quickest possible.

Upvotes: 1

Views: 4670

Answers (1)

Srce Cde
Srce Cde

Reputation: 1824

This will do.

import pandas as pd
import struct

k   = 0x100000
rng = range(0, k)
x1 = [b''] * k
x = 0xffffffff

for i in rng:
    x1[i]   = struct.pack("<L", x)
    x -= 1

df = pd.DataFrame()
df["data"] = x1
df.to_csv("test.csv", index=False, header=None)

This will output file in bytes. Sample output

b'\xff\xff\xff\xff'
b'\xfe\xff\xff\xff'
b'\xfd\xff\xff\xff'
b'\xfc\xff\xff\xff'
b'\xfb\xff\xff\xff'

You can use pandas instead of csv, to read the file back.

df = pd.read_csv("test.csv")

Alternative

with open("test.csv", "wb") as f:
    for i in x1:
        f.write(i)
        f.write('\n'.encode('utf-8'))

# Reading file
y = []
with open("test.csv", "rb") as f:
    for i in f.readlines():
        y.append(i.replace('\n'.encode('utf-8'), "".encode("utf-8")))
pprint(y[:10])

Output

[b'\xff\xff\xff\xff',
 b'\xfe\xff\xff\xff',
 b'\xfd\xff\xff\xff',
 b'\xfc\xff\xff\xff',
 b'\xfb\xff\xff\xff',
 b'\xfa\xff\xff\xff',
 b'\xf9\xff\xff\xff',
 b'\xf8\xff\xff\xff',
 b'\xf7\xff\xff\xff',
 b'\xf6\xff\xff\xff']

Upvotes: 2

Related Questions