CODEWITHSUNDEEP
pythoniteration
Apricot
Apricot

Reputation: 3021

pipe one file at a time python

I have more than 10000 json files which I have to convert to for further processing. I am using the following code:

import json
import time
import os
import csv
import fnmatch

tweets = []
count = 0

search_folder = ('/Volumes/Transcend/Axiom/IPL/test/')

for root, dirs, files in os.walk(search_folder):
    for file in files:
        pathname = os.path.join(root, file)

for file in open(pathname):
    try:
        tweets.append(json.loads(file))
    except:
        pass

count = count + 1

This iterates over just one file and stops. I tried adding while True: before for file in open(pathname): and it just doesn't stop nor it creates the csv files. I want to read one file at a time, convert it to csv, then move on to the next file. I tried adding count = count + 1 at the end of the file after completing converting the csv. Still it stops after converting the first file. Can someone help please?

Upvotes: 0

Views: 64

Answers (1)

Cyphase
Cyphase

Reputation: 12022

Your indentation is off; you need to put the second for loop inside the first one.

Separate from your main problem, you should use a with statement to open the file. Also, you were reusing the variable name file, which you shouldn't be using anyway since it's the name of a built-in. I also made a few other minor edits.

import json
import os

tweets = []
count = 0

search_folder = '/Volumes/Transcend/Axiom/IPL/test/'

for root, dirs, filenames in os.walk(search_folder):
    for filename in filenames:
        pathname = os.path.join(root, filename)
        with open(pathname, 'r') as infile:
            for line in infile:
                try:
                    tweets.append(json.loads(line))
                except:  # Don't use bare except clauses
                    pass

        count += 1

Upvotes: 1

Related Questions