CODEWITHSUNDEEP
pythoncsvpython-2.7scrapy
md1hunox
md1hunox

Reputation: 3955

export scraped data to CSV format with rowwise headers

Scrapy writes data into a CSV file column wise by default ie. one field /column. How do I write the data Row wise, ie. horizontally wherein first cell of a row will be name of the field.

I went through the scrapy documentation, but there i'snt anything using which I can change the format in which data in CSV is written

Update:

How I'm getting right now:

products,price
TV,25000
refrigerator,15000

This is how I want it:

products,tv,refrigerator
price,25000,15000

Why I want it that way?
I'm continuing the data extraction process done by someone else earlier, and 65% process has been finished. So this is to maintain consistency with the format.

Upvotes: 0

Views: 289

Answers (1)

Chris Barker
Chris Barker

Reputation: 2399

If the CSV file is large and you want to avoid loading the data into Python lists or dictionaries, you can do this:

infile = "/path/to/input_file.csv"
outfile = "/path/to/output_file.csv"
with open(infile, 'r') as source:
    num_fields = len(source.readline().split(','))
    source.seek(0) # Go back to beginning
    with open(outfile, 'w') as dest:
        for n in range(num_fields):
            for input_line in source:
                dest.write(input_line.split(',')[n] + ',')
            source.seek(0)
            dest.write('\b\n') # remove trailing comma

Otherwise you can just load everything up:

infile = "/path/to/input_file.csv"
outfile = "/path/to/output_file.csv"
with open(infile, 'r') as source:
    data = [line.strip().split(',') for line in source]
with open(outfile, 'w') as dest:
    for n in range(len(data[0])):
        dest.writeline(','.join(line[n] for line in data))

Upvotes: 1

Related Questions