Iterating on a file and comparing values using python

Question

I have a section of code that opens files containing information with wavenumber and intensity like this:

500.21506 -0.00134
500.45613 0.00231
500.69720 -0.00187
500.93826 0.00129
501.17933 -0.00049
501.42040 0.00028
501.66147 0.00114
501.90253 -0.00036
502.14360 0.00247

My code attempts to parse the information between two given wavelengths: lowwav and highwav. I would like to print only the intensities of the wavenumbers that fall between lowwav and highwav. My entire code looks like:

import datetime
import glob
path = '/Users/140803/*' 
files = glob.glob(path)

for line in open('sfit4.ctl', 'r'): 
            x = line.strip()
            if x.startswith('band.1.nu_start'):
                a,b = x.split('=')
                b = float(b)
                b = "{0:.3f}".format(b)
                lowwav = b
            if x.startswith('band.1.nu_stop'):
                a,b = x.split('=')
                b = float(b)
                b = "{0:.3f}".format(b)
                highwav = b 

with open('\\_spec_final.t15', 'w') as f:
    with open('info.txt', 'rt') as infofile:
        for count, line in enumerate(infofile): 
            lat = float(line[88:94]) 
            lon = float(line[119:127])
            year = int(line[190:194])
            month = int(line[195:197])
            day = int(line[198:200])
            hour = int(line[201:203])
            minute = int(line[204:206])
            second = int(line[207:209])
            dur = float(line[302:315])
            numpoints = float(line[655:660])
            fov = line[481:497] # field of view?
            sza = float(line[418:426])
            snr = 0.0000 
            roe = 6396.2 
            res = 0.5000
            lowwav = float(lowwav)
            highwav = float(highwav)
            spacebw = (highwav - lowwav)/ numpoints

            d = datetime.datetime(year, month, day, hour, minute, second)
            f.write('{:>12.5f}{:>12.5f}{:>12.5f}{:>12.5f}{:>8.1f}'.format(sza,roe,lat,lon,snr)) # line 1
            f.write("\n")
            f.write('{:>10d}{:>5d}{:>5d}{:>5d}{:>5d}{:>5d}'.format(year,month,day,hour,minute,second)) # line 2
            f.write("\n")
            f.write( ('{:%Y/%m/%d %H:%M:%S}'.format(d)) + "UT Solar Azimuth:" + ('{:>6.3f}'.format(sza)) + " Resolution:" + ('{:>6.4f}'.format(res)) + " Duration:" + ('{:>6.2f}'.format(dur))) # line 3
            f.write("\n")
            f.write('{:>21.13f}{:>26.13f}{:>24.17e}{:>12f}'.format(lowwav,highwav,spacebw,numpoints)) # line 4
            f.write("\n")


            with open(files[count], 'r') as g:
                for line in g:
                    wave_no, tensity = [float(item) for item in line.split()]
                    if lowwav <= wave_no <= highwav :
                        f.write(str(tensity) + '\n')


g.close()        
f.close()
infofile.close()

Right now, everything works fine except the last part where I compare wavelengths and print out the intensities corresponding to wavelengths between lowwav and highwav. No intensities are printing into the output file.

Answer 1

Split each line by white space, unpack the split list to two names wavelength and intensity.

[line.split() for line in r] makes

500.21506 -0.00134
500.45613 0.00231

to

[['500.21506', '-0.00134'], ['500.45613', '0.00231']]

This listcomp [(wavelength, intensity) for wavelength,intensity in lol if low <= float(wavelength) <= high] returns

[('500.21506', '-0.00134'), ('500.45613', '0.00231')]

If you join them back [' '.join((w, i)) for w,i in [('500.21506', '-0.00134'), ('500.45613', '0.00231')] you get ['500.21506 -0.00134', '500.45613 0.00231']

Use listcomp to filter out wavelength. And join wavelength and intensity back to string and write to file.

with open('data.txt', 'r') as r, open('\\_spec_final.t15', 'w') as w:
    lol = (line.split() for line in r)
    intensities = (' '.join((wavelength, intensity)) for wavelength,intensity in lol if low <= float(wavelength) <= high)
    w.writelines(intensities)

If you want to output to terminal do print(list(intensities)) instead of w.writelines(intensities)
Contents of data.txt;

500.21506 -0.00134
500.45613 0.00231
500.69720 -0.00187
500.93826 0.00129
501.17933 -0.00049
501.42040 0.00028
501.66147 0.00114
501.90253 -0.00036
502.14360 0.00247

Output when low is 500 and high is 50`;

['500.21506 -0.00134', '500.45613 0.00231']

Answer 2

The problem is that when you iterate over the file g you are effectively moving its "file pointer". So the second loop finds the file at the beginning and doesn't produce any value.

Secondly, you are producing all these nums lists, but every iteration of the lop shadows the previous value, making it unreachable.

Either you want to collected all the values and then iterate on those:

with open(files[count], 'r') as g:
    all_nums = []
    for line in g:
        all_nums.append([float(item) for item in line.split()])

    for nums in all_nums:
        if (lowwav - nums[0]) < 0 or (highwav - nums[0]) > 0 :
            f.write(str(nums[1]))
            f.write('\n')
        else: break 

Or just do everything inside the first loop (this should be more efficient):

with open(files[count], 'r') as g:
    for line in g:
        nums = [float(item) for item in line.split()]
        if (lowwav - nums[0]) < 0 or (highwav - nums[0]) > 0 :
            f.write(str(nums[1]))
            f.write('\n')
        else: break 

Also note that the break statement will stop the processing of the values when the condition is false for the first time, you probably want to remove it.

---

This said, note that your code prints all values where nums[0] that either are bigger than lowwav, or smaller than highwav, which means that if lowwav < highwav every number value will be printed. You probably want to use and in place of or if you want to check whether they are between lowwav and highwav. Moreover in python you could just write lowwav < nums[0] < highwav for this.

I would personally use the following:

with open(files[count], 'r') as g:
    for line in g:
        wave_no, intensity = [float(item) for item in line.split()]
        if lowwav < wave_no < highwav:
            f.write(str(intensity)+'\n')