Reputation: 2582

Check which columns in DataFrame are Categorical

I want to a simple and generic way to find which columns are categorical in my DataFrame, when I don't manually specify each column type, unlike in this SO question. The df is created with:

import pandas as pd
df = pd.read_csv("test.csv", header=None)

e.g.

           0         1         2         3        4
0   1.539240  0.423437 -0.687014   Chicago   Safari
1   0.815336  0.913623  1.800160    Boston   Safari
2   0.821214 -0.824839  0.483724  New York   Safari

UPDATE (2018/02/04) The question assumes numerical columns are NOT categorical, @Zero's accepted answer solves this.

**As @Sagarkar's comment points out Programming Data Types and Statistical Data Types (Categorical/Ordinal/Nominal/Numerical) types are orthogonal concepts, thus mapping between them isn't straightforward.

@Jeff's answer specifies the precise manner to manually specify the statistical data types to individual df columns.

Upvotes: 68

Answers (22)

Taha Mahjoubi

Reputation: 402

This always worked pretty well for me:

categorical_columns = list(set(df.columns) - set(df.describe().columns))

Upvotes: 0

Venkat

Reputation: 3

You need to check the columns with slightly less than 10 characters, or you need to check columns with unique values that are slightly more than 10 characters manually.

def find_cate(df):
cols=df.columns
i=0
for col in cols:
    if len(df[col].unique())<=10:
        print(col,len(df[col].unique()))
        i=i+1
print(i)

Upvotes: 0

Lakshman Dunna

Reputation: 11

categorical_values  = (df.dtypes == 'object')
categorical_variables = categorical_variables =[categorical_values.index[ind] 
for ind, val in enumerate(categorical_values) if val == True]

In the first line of code, we obtain a series which gives information regarding all the columns. The series gives information on which column is an object type and which column is not of the object type by representing it with a Boolean value.

In the second line, we use a list comprehension using enumeration(iterating through index and value), so that we could easily find the column which is of categorical type and append it to the categorical_variables list

Upvotes: 1

dCrystal

Reputation: 604

Use pandas.DataFrame.select_dtypes. There are categorical dtypes that can be found by 'categorical' flag. For Strings you might use the numpy object dtype

More Info: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.select_dtypes.html

Example:

import pandas as pd
df = pd.DataFrame({'Integer': [1, 2] * 3,'Bool': [True, False] * 3,'Float': [1.0, 2.0] * 3,'String': ['Dog', 'Cat'] * 3})
df

Out[1]:    
    Integer Bool    Float   String
0   1       True    1.0     Dog
1   2       False   2.0     Cat
2   1       True    1.0     Dog
3   2       False   2.0     Cat
4   1       True    1.0     Dog
5   2       False   2.0     Cat

df.select_dtypes(include=['category', object]).columns

Out[2]:
Index(['String'], dtype='object')

Upvotes: 2

Ranjita Shetty

Reputation: 625

# Get categorical and numerical variables

numCols = X.select_dtypes("number").columns
catCols = X.select_dtypes("object").columns
numCols= list(set(numCols))
catCols= list(set(catCols))

Upvotes: 6

Shikhar Omar

Reputation: 444

You can get the list of categorical columns using this code:

dfName.select_dtypes(include=['object']).columns.tolist()

And intuitively for numerical columns:

dfName.select_dtypes(exclude=['object']).columns.tolist()

Upvotes: 25

Khaled Ihitt

Reputation: 1

You can get the list of categorical columns using this code :

categorical_columns = (df.dtypes == 'object')

get categorical columns names:
object_cols = list(categorical_columns[categorical_columns].index)

Upvotes: 0

user96265

Reputation: 558

Sklearn gives you a one liner (or a 2 liner if you want to use it on many DataFrames). Lets say your DataFrame object is df then:

## good example in https://scikit-learn.org/stable/auto_examples/ensemble/plot_stack_predictors.html
from sklearn.compose import make_column_selector
        
cat_cols = make_column_selector(dtype_include=object) (df)
print (cat_cols)
## OR to use with many DataFrames, create one _selector object first
num_selector = make_column_selector(dtype_include=np.number) 
num_cols = num_selector (df)
print (num_cols)

Upvotes: 0

Bhanu Singh

Reputation: 11

This code will get all categorical variables:

cat_cols = [col for col in df.columns if col not in df.describe().columns]

Upvotes: 1

Michele Piccolini

Reputation: 2953

You don't need to query the data if you are just interested in which columns are of what type.

The fastest method (when %%timeit-ing it) is:

df.dtypes[df.dtypes == 'category'].index

(this will give you a pandas' Index. You can .tolist() to get a list out of it, if you need that.)

This works because df.dtypes is a pd.Series of strings (its own dtype is 'object'), so you can actually just select for the type that you need with normal pandas querying.

You don't have your categorical types as 'category' but as simple strings ('object')? Then just:

df.dtypes[df.dtypes == 'object'].index

Do you have a mix of 'object' and 'category'? Then use isin like you would do normally to query for multiple matches:

df.dtypes[df.dtypes.isin(['object','category'])].index

Upvotes: 6

Hemanth Harikrishnan

Reputation: 17

df.select_dtypes(exclude=["number"]).columns

This will help you to directly display all the non numerical rows

Upvotes: 0

Chethan

Reputation: 41

First we can segregate the data frame with the default types available when we read the datasets. This will list out all the different types and the corresponding data.

for types in data.dtypes.unique():
    print(types)
    print(data.select_dtypes(types).columns)

Upvotes: 1

d2a2d

Reputation: 1196

Often columns get pandas dtype of string (or "object") or category. Better to include both incase the columns you look for don't get listed under category dtype.

dataframe.select_dtypes(include=['object','category']).columns.tolist()

Upvotes: 5

Aztec-3x5

Reputation: 155

I have faced similar obstacle where categorizing variables was a challenge. However I came up with some approaches based on the nature of the data. This would give a general and flexible answer to your issue as well as to future data.

Normally while categorization of data is done on the basis of its datatype which sometimes may result in wrong analysis. (Usually done by df.select_dtypes(include = ['object', 'category'])

Approach:

The approach is of viewing the data not on a column level but on a row level. This approach would give the number of distinct values which would automatically distinguish categorical variables from numerical types.
That is if count of unique values in a row exceed more than certain number of values (This is for you to decide how many categorical variables you presume in your column)

for eg: if ['Dog', 'Cat', 'Bird', 'Fish', 'Reptile'] makes up for five unique categorical values for a particular column and if number of distinct values don't exceed more than those five unique categorical values in that column then that column falls under categorical variables.

elif ['Dog', 'Cat', 'Bird', 'Fish', 'Reptile'] makes up for five unique categorical values for a particular column and if number of distinct values exceed more than those five unique categorical values in that column then they fall under numerical variables.

if [col for col in df.columns if len(df[col].unique()) <=5]:
     cat_var = [col for col in df.columns if len(df[col].unique()) <=5]  
elif [col for col in df.columns if len(df[col].unique()) > 5]:
     num_var = [col for col in df.columns if len(df[col].unique()) > 5]

# where 5 : presumed number of categorical variables and may be flexible for user to decide.

I have used if and elif for better illustration. There is no need for that you can directly go for lines inside the condition.

Upvotes: 2

Gucci148

Reputation: 2157

select categorical column names

cat_features=[i for i in df.columns if df.dtypes[i]=='object']

Upvotes: 6

Hamza Chennaq

Reputation: 66

# Import packages
import numpy as np
import pandas as pd

# Data
df = pd.DataFrame({"Country" : ["France", "Spain", "Germany", "Spain", "Germany", "France"], 
                   "Age" : [34, 27, 30, 32, 42, 30], 
                   "Purchased" : ["No", "Yes", "No", "No", "Yes", "Yes"]})
df

Out[1]:
  Country Age Purchased
0  France  34        No
1   Spain  27       Yes
2 Germany  30        No
3   Spain  32        No
4 Germany  42       Yes
5  France  30       Yes

# Checking data type
df.dtypes

Out[2]: 
Country      object
Age           int64
Purchased    object
dtype: object

# Saving CATEGORICAL Variables
cat_col = [c for i, c in enumerate(df.columns) if df.dtypes[i] in [np.object]]
cat_col
Out[3]: ['Country', 'Purchased']

Upvotes: 0

Sudhir Tiwari

Reputation: 41

numeric_var = [key for key in dict(df.dtypes)
                   if dict(pd.dtypes)[key]
                       in ['float64','float32','int32','int64']] # Numeric Variable

cat_var = [key for key in dict(df.dtypes)
             if dict(df.dtypes)[key] in ['object'] ] # Categorical Varible

Upvotes: 4

ankit2saxena

Reputation: 21

This will give an array of all the categorical variables in a dataframe.

dataset.select_dtypes(include=['O']).columns.values

Upvotes: 0

pds

Reputation: 2582

The way I found was updating to Pandas v0.16.0, then excluding number dtypes with:

df.select_dtypes(exclude=["number","bool_","object_"])

Which works, providing no types are changed and no more are added to NumPy. The suggestion in the question's comments by @Jeff suggests include=["category"], but that didn't seem to work.

NumPy Types: link

Numpy Types

Upvotes: 55

Jeff

Reputation: 129018

For posterity. The canonical method to select dtypes is .select_dtypes. You can specify an actual numpy dtype or convertible, or 'category' which not a numpy dtype.

In [1]: df = DataFrame({'A' : Series(range(3)).astype('category'), 'B' : range(3), 'C' : list('abc'), 'D' : np.random.randn(3) })

In [2]: df
Out[2]: 
   A  B  C         D
0  0  0  a  0.141296
1  1  1  b  0.939059
2  2  2  c -2.305019

In [3]: df.select_dtypes(include=['category'])
Out[3]: 
   A
0  0
1  1
2  2

In [4]: df.select_dtypes(include=['object'])
Out[4]: 
   C
0  a
1  b
2  c

In [5]: df.select_dtypes(include=['object']).dtypes
Out[5]: 
C    object
dtype: object

In [6]: df.select_dtypes(include=['category','int']).dtypes
Out[6]: 
A    category
B       int64
dtype: object

In [7]: df.select_dtypes(include=['category','int','float']).dtypes
Out[7]: 
A    category
B       int64
D     float64
dtype: object

Upvotes: 31

Zero

Reputation: 76947

You could use df._get_numeric_data() to get numeric columns and then find out categorical columns

In [66]: cols = df.columns

In [67]: num_cols = df._get_numeric_data().columns

In [68]: num_cols
Out[68]: Index([u'0', u'1', u'2'], dtype='object')

In [69]: list(set(cols) - set(num_cols))
Out[69]: ['3', '4']

Upvotes: 64

Liam Foley

Reputation: 7822

Use .dtypes

In [10]: df.dtypes
Out[10]: 
0    float64
1    float64
2    float64
3     object
4     object
dtype: object

Upvotes: 2

Check which columns in DataFrame are Categorical

Answers (22)

Related Questions