CODEWITHSUNDEEP
python-3.xpandasstatsmodels
Sanoj
Sanoj

Reputation: 1467

Getting ValueError: The indices for endog and exog are not aligned

I am getting above error when I am running an iteration using FOR loop to build multiple models. First two models having similar data sets build fine. While building third model I am getting this error. The code where error is thrown is when I call sm.logit() using Statsmodel package of python:

y = y_mort.convert_objects(convert_numeric=True)

#Building Logistic model_LSVC
print("Shape of y:", y.shape, " &&Shape of X_selected_lsvc:", X.shape)
print("y values:",y.head())
logit = sm.Logit(y,X,missing='drop')

The error that appears:

Shape of y: (9018,)  &&Shape of X_selected_lsvc: (9018, 59)
y values: 0    0
1    1
2    0
3    0
4    0
Name: mort, dtype: int64
ValueError                                Traceback (most recent call last)
<ipython-input-8-fec746e2ee99> in <module>()
    160     print("Shape of y:", y.shape, " &&Shape of X_selected_lsvc:", X.shape)
    161     print("y values:",y.head())
--> 162     logit = sm.Logit(y,X,missing='drop')
    163     # fit the model
    164     est = logit.fit(method='cg')

D:\Anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py in __init__(self, endog, exog, **kwargs)
    399 
    400     def __init__(self, endog, exog, **kwargs):
--> 401         super(BinaryModel, self).__init__(endog, exog, **kwargs)
    402         if (self.__class__.__name__ != 'MNLogit' and
    403                 not np.all((self.endog >= 0) & (self.endog <= 1))):

D:\Anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py in __init__(self, endog, exog, **kwargs)
    152     """
    153     def __init__(self, endog, exog, **kwargs):
--> 154         super(DiscreteModel, self).__init__(endog, exog, **kwargs)
    155         self.raise_on_perfect_prediction = True
    156 

D:\Anaconda3\lib\site-packages\statsmodels\base\model.py in __init__(self, endog, exog, **kwargs)
    184 
    185     def __init__(self, endog, exog=None, **kwargs):
--> 186         super(LikelihoodModel, self).__init__(endog, exog, **kwargs)
    187         self.initialize()
    188 

D:\Anaconda3\lib\site-packages\statsmodels\base\model.py in __init__(self, endog, exog, **kwargs)
     58         hasconst = kwargs.pop('hasconst', None)
     59         self.data = self._handle_data(endog, exog, missing, hasconst,
---> 60                                       **kwargs)
     61         self.k_constant = self.data.k_constant
     62         self.exog = self.data.exog

D:\Anaconda3\lib\site-packages\statsmodels\base\model.py in _handle_data(self, endog, exog, missing, hasconst, **kwargs)
     82 
     83     def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
---> 84         data = handle_data(endog, exog, missing, hasconst, **kwargs)
     85         # kwargs arrays could have changed, easier to just attach here
     86         for key in kwargs:

D:\Anaconda3\lib\site-packages\statsmodels\base\data.py in handle_data(endog, exog, missing, hasconst, **kwargs)
    564     klass = handle_data_class_factory(endog, exog)
    565     return klass(endog, exog=exog, missing=missing, hasconst=hasconst,
--> 566                  **kwargs)

D:\Anaconda3\lib\site-packages\statsmodels\base\data.py in __init__(self, endog, exog, missing, hasconst, **kwargs)
     74         # this has side-effects, attaches k_constant and const_idx
     75         self._handle_constant(hasconst)
---> 76         self._check_integrity()
     77         self._cache = resettable_cache()
     78 

D:\Anaconda3\lib\site-packages\statsmodels\base\data.py in _check_integrity(self)
    450                 (hasattr(endog, 'index') and hasattr(exog, 'index')) and
    451                 not self.orig_endog.index.equals(self.orig_exog.index)):
--> 452             raise ValueError("The indices for endog and exog are not aligned")
    453         super(PandasData, self)._check_integrity()
    454 

ValueError: The indices for endog and exog are not aligned

The y matrix and X matrix have shape of (9018,),(9018, 59). Therefore any mismatch in dependent and independent variables doesn't appear. Any idea?

Upvotes: 11

Views: 57041

Answers (7)

RajendraW
RajendraW

Reputation: 29

ValueError: The indices for endog and exog are not aligned

Above error is basically due to index mismatch in both X & y datasets while cleaning and preparation.

I removed this error by removing the indices of both X & y datasets as: y_train = y_train.reset_index(drop=True) X_train = X_train.reset_index(drop=True)

Pls provide your valuable feedback

Upvotes: 0

Qasim Arthuna
Qasim Arthuna

Reputation: 11

do y_train.values.ravel(). Actually shape of y_train is in 2D array. So you need to convert it into 1D array. hope it works for you.

Upvotes: 0

Monika
Monika

Reputation: 31

This error may also come due to wrong usage of API

Correct:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=0.7, test_size=0.3, random_state=100
)

Incorrect:

X_train, y_train, X_test, y_test = train_test_split(
    X, y, train_size=0.7, test_size=0.3, random_state=100
)

Upvotes: 3

Anurag Patel
Anurag Patel

Reputation: 21

It may be due to different indices in x and y. This may happen when we initially removed some values from dataframe and perform some operations on x after separating x and y. The indices in y will contain the missing indices from original dataframe while x will have continuous indices. It's best to do dataframe.reset_index(drop = True) before separating x and y.

Upvotes: 2

Ashish Anand
Ashish Anand

Reputation: 2743

The error message indicates that you have endog and exog with different shape. This is common error in python which can be easily solved by using 'reshape' function on dependent variable to align it with independent variable's shape.

y_train.values.reshape(-1,1)

Above lines means:- We have provided column as 1 but rows as unknown i.e. we got a single column with as many rows as X.

Lets take a example:-

z = np.array([[1, 2], [ 3, 4]])
print(z.shape)    # (2, 2)

Now we will use reshape(-1,1) function on this array. We can see new array has 4 row and 1 column.

new_z= z.reshape(-1,1)
print(new_z)        #array([[1],[2],[3], [4]])
print(new_z.shape)  #(4, 1)

Upvotes: 4

Rok Povsic
Rok Povsic

Reputation: 4885

Try converting y into a list before the sm.Logit() line.

y = list(y)

Upvotes: 21

Andreas Hsieh
Andreas Hsieh

Reputation: 2140

Have you checked if you have Nan in your data? You can use np.isNan(X) and np.isNan(y). I saw you turned on the option drop so I suspect if you have Nan in your data then that will change the shape of your input.

Upvotes: 0

Related Questions