CODEWITHSUNDEEP
pythoncsv
Fahad Khan
Fahad Khan

Reputation: 67

Python - Create multiple folders from CSV file

I want to create multiple folders/directories (if they don't exist) using information from a CSV file.

I have the information from csv as follows:

    Column0 Column1 Column2 Column3
    51  TestName1   0   https://siteAdress//completed/file.txt
    53  TestName2   0   https://siteAdress//completed/file.txt
    67  TestName1   2   https://siteAdress//uploads/file.txt
    68  TestName1   2   https://siteAdress//uploads/file.txt

I want to iterate column3, if it contains 'uploads' then it should make a folder with the corresponding jobname mentioned on column1 then create 'input' folder and within it create respective file.txt file, and if column3 contains 'completed' then it should make 'output' folder (within that same jobname folder next to input folder) and then within it the 'file.txt' file. And do this for all jobs mentioned in column1.

Something like this:

TestName1/input/file.txt
TestName1/output/file.txt
TestName1/output2/file.txt

TestName2/input/file.txt
TestName2/output/file.txt

Note: Most of data will contain multiple output folders for each jobname. In this case it should create as many output folders as are mentioned in the csv file.

So far, I have done this:

import csv, os
#reads from csv file
with open('limitedresult.csv') as csvfile:
    readCSV = csv.reader(csvfile, delimiter = ',')
    for row in readCSV:
        print(row)

Your help would be highly appreciated, please let me know if the question is still confusing and I will try to explain in more detail.

Upvotes: 1

Views: 1763

Answers (1)

Martin Evans
Martin Evans

Reputation: 46759

The following approach should help get you started:

  1. Open the CSV file and skip the header row.
  2. Read a row, splitting it into named columns.
  3. If the file_url contains input, use a sub folder of input, etc.
  4. Create a folder based on output_root and the sub folder name.
  5. Use a Python Counter to keep track of the number of times each sub folder is used.
  6. Add the current sub folder count to the folder name and create any necessary output folders.
  7. Use the Python requests library to download the text file from the website.
  8. Extract the filename from the URL and use this to write the file contents.

The script is as follows:

from collections import Counter
import requests
import csv
import os

output_root = r'/myroot'
output_counter = Counter()

with open('limitedresult.csv', newline='') as csvfile:
    readCSV = csv.reader(csvfile)
    header = next(readCSV)

    for number, test, col2, file_url in readCSV:
        if 'completed' in file_url:
            sub_folder = 'input'
        elif 'uploads' in file_url:
            sub_folder = 'output' 
        else:
            sub_folder = None
            print('Invalid URL -', file_url)

        if sub_folder:
            output_folder = os.path.join(output_root, test, sub_folder)
            output_counter.update([output_folder])
            output_folder += str(output_counter[output_folder])
            os.makedirs(output_folder, exist_ok=True)
            data = requests.get(file_url)
            file_name = os.path.split(file_url)[1]

            with open(os.path.join(output_folder, file_name), 'w') as f_output:
                f_output.write(data.text)

Note, you may need to install requests, this can usually be done using pip install requests.

Upvotes: 1

Related Questions