Saving multiple csv files with diferent names that change dependent on variable value - python

Question

first time asking something here, although I have been using this quite a bit and its been really helpfull. I have already search for it but I cannot find an answer.

My program will be constainly receiving car data and I want to save it to multiple csv files. I want to save for every lap. So for Lap1, I would have something like car_data_1.csv, for the second lap car_data_2.csv and so on.

I know how to do it by force, creating every file with the name and appending the results to that specific file. But since I do not know how many laps I will have, it does not seem to me the right way and best way to do it.

So in the code below, when I create the line with open ('car_data_(LapNumber).csv', 'w') as csv_file: I want the program to create a file with the name car_data_1, since I will start on Lap 1. So LapNumber (inside paranthesis in that line just to show you) is the name of my variable that will change and I would like to save each csv with data respective to each lap.

import socket
import csv

'''UDP SET UP'''

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
UDP_IP = 'IP'
UDP_PORT = Port

sock.bind((UDP_IP, UDP_PORT))

'''CSV SET UP'''

fieldnames = ["On","Time","LapNumber","Engine","Speed","Power","BestLap","Distance"]

LapNumber=1

with open(f'car_data_{LapNumber}.csv','w') as csv_file:
    csv_writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
    csv_writer.writeheader()

while True:

    data, addr = sock.recvfrom(1024)  # buffer size is 1024 bytes
    values = struct.unpack("<iIffffff",data)

    LapNumber = values[2]

    with open(f'car_data_{LapNumber}.csv','a') as csv_file:
        csv_writer =csv.DictWriter(csv_file, fieldnames=fieldnames)
        #csv_writer.writeheader()

        info = {
            "On"    :values[0],
            "Time"  :values[1],
            "LapNumber" :values[2],
            "Engine"    :values[3],
            "Speed" :values[4],
            "Power" :values[5],
            "BestLap"   :values[6],
            "Distance"  :values[7],         
        }

        csv_writer.writerow(info)

I understand that maybe I will have to change the structure because I think that I will have to create the csv inside a conditional while or something like that because if I create the csv and the append the values inside the while it will be constainly overwriting the values on the csv an I will end up just with the last set of values (before crossing the line for another lap).

What are your suggestions? I have been thinking to add some for and if so let me know what you guys think is the best options. This project is recent and I have been taking it step by step to make sure things work, and now I want to better the program.

Thank you for your help, please let me know if you need any more information. I am using windows, python 3.8 and pycharm.

Answer 1

You need to use a format variable in the name of the file

with open('car_data_{}.csv'.format(LapNumber),'w')

Now the variable LapNumber is substituted into the string, in place of the {}

or, if you are using Python 3.6 or later, use f-strings

with open(f'car_data_{LapNumber}.csv','w')

Your current approach creates the filename 'car_data_(LapNumber.csv)' and does not actually substitute the variable into the string.

For the second part of your question, relating to writing the headers - you want to keep a reference to the current lap you are writing to (which itself is a proxy for the file you are writing to). When the lap changes you know you are writing to a new file, and need to add a header.

Something like;

# Initialise writing_lap to lap number 1
writing_lap = 1
while True:

    data, addr = sock.recvfrom(1024)  # buffer size is 1024 bytes
    values = struct.unpack("<iIffffff",data)

    # Read the current lap
    current_lap = values[2]

    with open(f'car_data_{writing_lap}.csv','a') as csv_file:
        csv_writer =csv.DictWriter(csv_file, fieldnames=fieldnames)
        if current_lap > writing_lap:
             # If the current_lap has increased, write the headers
             csv_writer.writeheader()
             # And remember to update the writing_lap variable
             writing_lap = current_lap

        info = {
            "On"    :values[0],
            "Time"  :values[1],
            "LapNumber" :values[2],
            "Engine"    :values[3],
            "Speed" :values[4],
            "Power" :values[5],
            "BestLap"   :values[6],
            "Distance"  :values[7],         
        }

        csv_writer.writerow(info)

Answer 2

As wstk have answered your first question, your second one in the comment is about adding the header to new file. (I cannot comment because not having enough reputation)

To add the header to every new csv file, you can check when the LapNumber change and add the header right there.

For example, inside the while loop:

#LapNumber = values[2] --> you just need to move this line as below
#instead of putting LapNumber inside {}, you can just put values[2], and use 
#LapNumber for checking the condition of writing the header

with open(f'car_data_{values[2]}.csv','a') as csv_file:
    csv_writer =csv.DictWriter(csv_file, fieldnames=fieldnames)
    if values[2] != LapNumber:
        csv_writer.writeheader()
        LapNumber = values[2]

Now your script only write a header whenever it creates new file.