CODEWITHSUNDEEP
pythonpandas
Priit Mets
Priit Mets

Reputation: 495

How import multiple text files to one dataframe in python?

I found here how to import multiple text files to one data frame. However, it gives an error. Files are with the names as footballseason1,footballseason2,footballseason3 ... (until footballseason5000)

import pandas as pd
import datetime as dt
import os, glob
os.chdir("~/Downloads/data")
filenames = [i for i in glob.glob("*.txt")]

FileNotFoundError: [Errno 2] No such file or directory: '~/Downloads/data'

However, if I try to import one file, everything is working and the directory is found

df = pd.read_csv("~/Downloads/data/footballseason1.txt", sep=",")

Could you help to fix the problem? and are there any ways to do it without changing directory and simply do all the steps using the path where all files are located?

Upvotes: 1

Views: 1319

Answers (4)

tdy
tdy

Reputation: 41327

Python's os does not understand ~ by default, so it needs to be expanded manually:

filenames = [i for i in glob.glob(os.path.expanduser("~/Downloads/data/*.txt"))]

Upvotes: 2

Nk03
Nk03

Reputation: 14949

Via pathlib ->

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
df = pd.concat([
    pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
])

With added check - >

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
if inp_path.exists():
    df = pd.concat([
        pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
    ])
else:
    print('input dir doesn\'t exist please check path')

Upvotes: 0

Rudraaksh KB
Rudraaksh KB

Reputation: 1

Importing Data from Multiple files Now let’s see how can we import data from multiple files from a specific directory. There are many ways to do so, but I personally believe this is an easier and simpler way to use and also to understand especially for beginners. 1)First, we are going to import the OS and glob libraries. We need them to navigate through different working directories and getting their paths. import os import glob 2) We also need to import the pandas library as we need to work with data frames. import pandas as pd 3) Let’s change our working directory to the directory where we have all the data files. os.chdir(r"C:\Users\HARISH\Path_for_our_files") 4) Now we need to create a for loop which iterates through all the .csv file in the current working directory filenames = [i for i in glob.glob("*.csv")]

Upvotes: -1

imdevskp
imdevskp

Reputation: 2223

You can use python's list comprehension and pd.concat like below

df = pd.concat([pd.read_csv(i, sep=',') for i in glob.glob("~/Downloads/data/*.txt", recursive=True)])

Upvotes: 0

Related Questions