CODEWITHSUNDEEP
pythonpandasdatetimeparsing
Tao
Tao

Reputation: 397

Pandas read_csv() parse multiple datetime formats

I would like to read multiple data files that have different DateTime formats. But How can I parse these formats together in one go?

dateparse_1 = lambda x: pd.datetime.strptime(x, "%d/%m/%Y %H:%M:%S.%f") 
dateparse_2 = lambda x: pd.datetime.strptime(x, "%Y-%m-%d %H:%M:%S.%f")
dateparse_3 = lambda x: pd.datetime.strptime(x, "%Y-%m-%d %H:%M:%S.")

for f in all_filenames:
    df = pd.read_csv(f,encoding='latin-1',low_memory=False, index_col='TimeStamp', parse_dates=True, date_parser = dateparse_1 or dateparse_2 or dateparse_3)

Upvotes: 2

Views: 419

Answers (1)

d125q
d125q

Reputation: 1666

You can combine dateparse_{1,2,3} by trying them until one succeeds. For example,

def combine_date_parsers(date_parsers):
    def combined_date_parser(value):
        for date_parser in date_parsers:
            try:
                return date_parser(value)
            except ValueError:
                pass
        else:
            raise ValueError(value)
    return combined_date_parser


dateparse_1 = lambda x: pd.datetime.strptime(x, "%d/%m/%Y %H:%M:%S.%f")
dateparse_2 = lambda x: pd.datetime.strptime(x, "%Y-%m-%d %H:%M:%S.%f")
dateparse_3 = lambda x: pd.datetime.strptime(x, "%Y-%m-%d %H:%M:%S.")

date_parser = combine_date_parsers([
    dateparse_1,
    dateparse_2,
    dateparse_3,
])

pd.read_csv(
    f,
    encoding='latin-1',
    low_memory=False,
    index_col='TimeStamp',
    parse_dates=True,
    date_parser=date_parser,
)

Upvotes: 2

Related Questions