CODEWITHSUNDEEP
pythondatepandaslinear-regression
jeangelj
jeangelj

Reputation: 4498

python linear regression predict by date

I want to predict a value at a date in the future with simple linear regression, but I can't due to the date format.

This is the dataframe I have: 

data_df = 
date          value
2016-01-15    1555
2016-01-16    1678
2016-01-17    1789
...  

y = np.asarray(data_df['value'])
X = data_df[['date']]
X_train, X_test, y_train, y_test = train_test_split             
(X,y,train_size=.7,random_state=42)

model = LinearRegression() #create linear regression object
model.fit(X_train, y_train) #train model on train data
model.score(X_train, y_train) #check score

print (‘Coefficient: \n’, model.coef_)
print (‘Intercept: \n’, model.intercept_) 
coefs = zip(model.coef_, X.columns)
model.__dict__
print "sl = %.1f + " % model.intercept_ + \
     " + ".join("%.1f %s" % coef for coef in coefs) #linear model

I tried to convert the date unsuccessfully

data_df['conv_date'] = data_df.date.apply(lambda x: x.toordinal())

data_df['conv_date'] = pd.to_datetime(data_df.date, format="%Y-%M-%D")

Upvotes: 33

Views: 79763

Answers (6)

Jony
Jony

Reputation: 51

I'm diving into the different options given here and I just wanted to summarize them. It takes time to write a full answer but this is what I've researched.

Examples reference

I took the same date with different data types following the requirements of each method. Maybe I'm missing other options.

t = pd.Timestamp('2021-09-03 00:00:00')    
   # Timestamp('2021-09-03 00:00:00')   pandas._libs.tslibs.timestamps.Timestamp
t2 = dtt.date(2021, 9, 3)          
   #  datetime.date(2021, 9, 3)     datetime.date

Pandas methods

pandas.to_numeric(arg, errors='raise', downcast=None)
# argscalar, list, tuple, 1-d array, or Series
Example
st3 = pd.to_numeric(df_example.index, downcast='integer')
st3[0]  
1630627200000000000

Python/Pandas with the same outcome

  • pandas.Timestamp.toordinal
  • Python-date.toordinal
    Note: The Proleptic Gregorian ordinal gives the number of days elapsed from the date 01/Jan/0001. And here ordinal is called Proleptic since the Gregorian calendar itself is followed from October 1582.Aug 23, 2021. 
# I checked it out and I found a 215 days difference
hoy = dtt.date.today()   # datetime.date(2022, 8, 3)
hoy.toordinal()  # 738370   - (365 * 2022)  =  340
hoy.timetuple()  # tm_yday=215  ---> 340 - 215 = 125
Example
t2.toordinal()
    738036
pd.Timestamp.toordinal(a) 
    738036

Python methods

Example
time.mktime(t2.timetuple())     
1630638000.0

Upvotes: 0

Kubra Altun
Kubra Altun

Reputation: 405

It is really important to differentiate the data types that you want to use for regression/classification.

When you are using time series, that is another case but if you want to use time data as a numerical data type as your input, then you should transform your data type from datetime to float (if your data_df['conv_date] is a datetime object, if not then you should first transform it by using; data_df['conv_date'] = pd.to_datetime(data_df.date, format="%Y-%M-%D") )

I agree with Thomas Vetterli's answer. It is useful to be careful what kind of time data you are using.

If you are only using year and month data then dt.datetime.toordinal would be enough to use;

>>import datetime
>>data_df['conv_date'] = pd.to_datetime(data_df.date, format="%Y-%M-%D")
>>data_df['conv_date'] = data_df['conv_date'].map(datetime.datetime.toordinal)
737577

But if you want to use also the hour, minute and second information then time.mktime() suits better;

>>import time
>>data_df['conv_date'] = pd.to_datetime(data_df.date, format="%Y-%M-%D")
>>data_df['conv_date'] = data_df['conv_date'].apply(lambda  var: time.mktime(var.timetuple()))
1591016041.0

Also 1591016044.0 is another exemplary output from my data, it varies with changes in seconds.

Upvotes: 0

Thomas Vetterli
Thomas Vetterli

Reputation: 856

When using

dt.datetime.toordinal

be careful that it only converts dates values and does not take into account minutes, seconds etc.. For a complete answer on generating ordinals from full datetime objects you can use something like:

df['Datetime column'].apply(lambda x: time.mktime(x.timetuple()))

Upvotes: 1

Siraj S.
Siraj S.

Reputation: 3751

convert:

1) date to dataframe index 

df = df.set_index('date', append=False)

2) convert datetime object to float64 object

df = df.index.to_julian_date()

run the regression with date being the independent variable.

Upvotes: 6

Chandan
Chandan

Reputation: 772

Linear regression doesn't work on date data. Therefore we need to convert it into numerical value.The following code will convert the date into numerical value:

import datetime as dt
data_df['Date'] = pd.to_datetime(data_df['Date'])
data_df['Date']=data_df['Date'].map(dt.datetime.toordinal)

Upvotes: 42

Quazi Marufur Rahman
Quazi Marufur Rahman

Reputation: 2623

Liner regression works on numerical data. Datetime type is not appropriate for this case. You should remove that column after separating it to three separate columns (year, month and day).

Upvotes: 2

Related Questions