CODEWITHSUNDEEP
pythoncsv
zyy
zyy

Reputation: 1584

How can I read csv file only after finding a certain pattern with Python?

So I have several csv files that represent some data, each of which may have different lines of initial comments

table_doi: 10.17182/hepdata.52402.v1/t7
name: Table 7
...
ABS(YRAP), < 0.1
SQRT(S) [GeV], 1960
PT [GEV], PT [GEV] LOW, PT [GEV] HIGH, D2(SIG)/DYRAP/DPT [NB/GEV]
67, 62, 72, 6.68
...
613.5, 527, 700, 1.81E-07

I would like to read in only the relevant data and their headers as well, which start from the line

PT [GEV], PT [GEV] LOW, PT [GEV] HIGH, D2(SIG)/DYRAP/DPT [NB/GEV]

Therefore the strategy I would think of is to find the pattern PT [GEV] and start reading from there.

However, I am not sure how to achieve this in Python, could anyone help me on that?

Thank you in advance!

By the way, the function I currently have is

import os
import glob
import csv

def read_multicolumn_csv_files_into_dictionary(folderpath, dictionary):
    filepath = folderpath + '*.csv'
    files = sorted(glob.glob(filepath))
    for file in files:
        data_set = file.replace(folderpath, '').replace('.csv', '')
        dictionary[data_set] = {}
        with open(file, 'r') as data_file:
            data_pipe = csv.DictReader(data_file)
            dictionary[data_set]['pt'] = []
            dictionary[data_set]['sigma'] = []
            for row in data_pipe:
                dictionary[data_set]['pt'].append(float(row['PT [GEV]']))
                dictionary[data_set]['sigma'].append(float(row['D2(SIG)/DYRAP/DPT [NB/GEV]']))
    return dictionary

which only works if I manually delete those initial comments in the csv files.

Upvotes: 2

Views: 3293

Answers (5)

I_Al-thamary
I_Al-thamary

Reputation: 4033

Try this where it will be searching for the row that contains PT [GEV] and if it finds the contains, it will change the m to be true and start to append the rest of date to the list :

import csv

contain= 'PT [GEV]'
List=[]
m=false
with open('Users.csv', 'rt') as f:
     reader = csv.reader(f, delimiter=',') 
     for row in reader:
          for field in row:
              if field == contain:
              m=true
          if m==true:
             List.append(row)

Upvotes: 1

Ruchit Dalwadi
Ruchit Dalwadi

Reputation: 331

checkout startswith. Also, you can find detailed explanation here. https://cmdlinetips.com/2018/01/3-ways-to-read-a-file-and-skip-initial-comments-in-python/

Upvotes: 2

Meeko
Meeko

Reputation: 85

I would just create a help function to get your csv reader to the first record:

def remove_comments_from_file():

    file_name = "super_secret_file.csv"
    file = open(file_name, 'rU')

    csv_read_file = csv.reader(file)        

    for row in csv_read_file:
        if row[0] == "PT [GEV]"
            break

    return csv_read_file

Something along those lines, when the csv reader is returned, it will start at your first record (in this example - 67, 62, 72, 6.68)

Upvotes: 0

blhsing
blhsing

Reputation: 106901

You can use the file.tell method to save the file pointer position while you read and skip the lines until you find the header line, at which point you can use the file.seek method to reset the file pointer back to the beginning of the header line so that csv.DictReader can parse the rest of the file as valid CSV:

with open(file, 'r') as data_file:
    while True:
        position = data_file.tell()
        line = next(data_file)
        if line.count(',') == 3: # or whatever condition your header line satisfies
            data_file.seek(position) # reset file pointer to the beginning of the header line
            break
    data_pipe = csv.DictReader(data_file)
    ...

Upvotes: 1

Chris
Chris

Reputation: 29742

Assuming every file has a line that startswith PT [GEV]:

import os
import pandas as pd

...
csvs = []
for file in files:
    with open(file) as f:
        for i, l in enumerate(f):
            if l.startswith('PT [GEV]'):
                csvs.append(pd.read_csv(file, skiprows = i))
                break
df = pd.concat(csvs)

Upvotes: 2

Related Questions