CODEWITHSUNDEEP
pythonnumpy
narayan.p
narayan.p

Reputation: 105

Import Multiple Text files (Large Number) using numpy and Post Processing

This forum has been extremely helpful for a python novice like me to improve my knowledge. I have generated a large number of raw data in text format from my CFD simulation. My objective is to import these text files into python and do some postprocessing on them. This is a code that I have currently. 

import numpy as np
from matplotlib import pyplot as plt
import os

filename=np.array(['v1-0520.txt','v1-0878.txt','v1-1592.txt','v1-3020.txt','v1-5878.txt'])

for i in filename:
    format_name= i
    path='E:/Fall2015/Research/CFDSimulations_Fall2015/ddn310/Autoexport/v1'
    data= os.path.join(path,format_name)
    X,Y,U,V,T,Tr = np.loadtxt(data,usecols=(1,2,3,4,5,6),skiprows=1,unpack = True) # Here X and Y represents the X and Y coordinate,U,V,T,Tr represents the Dependent Variables
    plt.figure(1)
    plt.plot(T,Y)
    plt.legend(['vt1a','vtb','vtc','vtd','vte','vtf'])
    plt.grid(b=True)

Is there a better way to do this, like importing all the text files (~10000 files) at once into python and then accessing whichever files I need for post processing (maybe indexing). All the text files will have the same number of columns and rows.

I am just a beginner to Python.I will be grateful if someone can help me or point me in the right direction.

Upvotes: 1

Views: 1279

Answers (1)

hpaulj
hpaulj

Reputation: 231375

Your post needs to be edited to show proper indentation.

Based on a quick read, I think you are:

reading a file, making a small edit, and write it back
then you load it into a numpy array and plot it

Presumably the purpose of your edit is to correct some header or value.

You don't need to write the file back. You can use content directly in loadtxt.

content = content.replace("nodenumber","#nodenumber") # Ignoring Node number column
data1=np.loadtxt(content.splitlines())
Y=data1[:,2]
temp=data1[:,5]

loadtxt accepts any thing that feeds it line by line. content.splitlines() makes a list of lines, which loadtxt can use.

the load could be more compact with:

Y, temp = np.loadtxt(content.splitlines(), usecols=(2,5), unpack=True)

With usecols you might not even need the replace step. You haven't given us a sample file to test.

I don't understand your multiple file needs. One way other you need to open and read each file, one by one. And it would be best to close one before going on to the next. The with open(name) as f: syntax is great for ensuring that a file is closed.

You could collect the loaded data in larger lists or arrays. If Y and temp are identical in size for all files, they can be collected into larger dimensional array, e.g. YY[i,:] = Y for the ith file, where YY is preallocated. If they can vary in size, it is better to collect them in lists.

Upvotes: 1

Related Questions