pandas.read_csv gives FileNotFound error inside a loop

Question

pandas.read_csv is working properly when used as a single statement. But it is giving FileNotFoundError when it is being used inside a loop even though the file exists.

for filename in os.listdir("./Datasets/pollution"):
    print(filename) # To check which file is under processing
    df = pd.read_csv(filename, sep=",").head(1)

These above lines are giving this following error.

pollutionData184866.csv <----- The name of the file is printed properly.
Traceback (most recent call last):
  File "/home/parnab/PycharmProjects/FinalYearProject/locationExtractor.py", line 13, in <module>
    df = pd.read_csv(i, sep=",").head(1)
  File "/usr/lib/python3.6/site-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/lib/python3.6/site-packages/pandas/io/parsers.py", line 389, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/lib/python3.6/site-packages/pandas/io/parsers.py", line 730, in __init__
    self._make_engine(self.engine)
  File "/usr/lib/python3.6/site-packages/pandas/io/parsers.py", line 923, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/usr/lib/python3.6/site-packages/pandas/io/parsers.py", line 1390, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas/parser.pyx", line 373, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:4184)
  File "pandas/parser.pyx", line 667, in pandas.parser.TextReader._setup_parser_source (pandas/parser.c:8449)
FileNotFoundError: File b'pollutionData184866.csv' does not exist

But when I am doing

filename = 'pollutionData184866.csv'
df = pd.read_csv(filename, sep=',')

It is working fine.

What am I doing wrong?

Answer 1

os.listdir("./Datasets/pollution") returns a list of files without a path and according to the path "./Datasets/pollution" you are parsing CSV files NOT from the current directory ".", so changing it to glob.glob('./Datasets/pollution/*.csv') should work, because glob.glob() returns a list of satisfying files/directories including given path

Demo:

In [19]: os.listdir('d:/temp/.data/629509')
Out[19]:
['AAON_data.csv',
 'AAON_data.png',
 'AAPL_data.csv',
 'AAPL_data.png',
 'AAP_data.csv',
 'AAP_data.png']

In [20]: glob.glob('d:/temp/.data/629509/*.csv')
Out[20]:
['d:/temp/.data/629509\\AAON_data.csv',
 'd:/temp/.data/629509\\AAPL_data.csv',
 'd:/temp/.data/629509\\AAP_data.csv']