Reputation: 2118

How to convert a Scikit-learn dataset to a Pandas dataset

How do I convert data from a Scikit-learn Bunch object to a Pandas DataFrame?

from sklearn.datasets import load_iris
import pandas as pd
data = load_iris()
print(type(data))
data1 = pd. # Is there a Pandas method to accomplish this?

Upvotes: 173

Answers (30)

Sudhir Kumar Joon

Reputation: 1

from sklearn.datasets import load_iris
iris = load_iris(as_frame=True)
iris = iris['frame']
iris.head()

Upvotes: 0

Egelbets

Reputation: 336

By far the simplest solution:

from sklearn.datasets import load_iris
iris = load_iris(as_frame=True)
df = iris["frame"] # will also contain the target column

More information: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html

Upvotes: 1

normanius

Reputation: 9762

So many answers, so much noise... The following is simple and uses pd.Categorical for the target variable.

import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()

df = pd.DataFrame(data=iris.data, columns=iris.feature_names)
df["species"] = pd.Categorical.from_codes(iris.target, iris.target_names)

#      sepal_length  sepal_width  petal_length  petal_width    species
# 0             5.1          3.5           1.4          0.2     setosa
# 1             4.9          3.0           1.4          0.2     setosa
# 2             4.7          3.2           1.3          0.2     setosa
# 3             4.6          3.1           1.5          0.2     setosa
# 4             5.0          3.6           1.4          0.2     setosa
# ..            ...          ...           ...          ...        ...
# 145           6.7          3.0           5.2          2.3  virginica
# 146           6.3          2.5           5.0          1.9  virginica
# 147           6.5          3.0           5.2          2.0  virginica
# 148           6.2          3.4           5.4          2.3  virginica
# 149           5.9          3.0           5.1          1.8  virginica
# 
# [150 rows x 5 columns]

To extract the integer codes of the target variable, use the cat accessor.

df.species.cat.codes

# 0      0
# 1      0
# 2      0
# 3      0
# 4      0
#       ..
# 145    2
# 146    2
# 147    2
# 148    2
# 149    2
# Length: 150, dtype: int8

Upvotes: 1

Oloyede Abdulganiyu

Reputation: 21

A more simpler and approachable manner I tried

import pandas as pd
from sklearn import datasets

iris = load_iris()

X= pd.DataFrame(iris['data'], columns= iris['feature_names'])
y = pd.DataFrame(iris['target'],columns=['target'])
df = X.join(y)

Upvotes: 1

SamithaP

Reputation: 100

This is an easy way and works with majority of datasets in sklearn

import pandas as pd
from sklearn import datasets

# download iris data set
iris = datasets.load_iris()

# load feature columns to DataFrame
df = pd.DataFrame(data=iris.data, columns=iris.feature_names)

# add a column to df called 'target_c' then asign the target data of iris data
df['target_c'] = iris.target

# view final DataFrame
df.head()

Upvotes: 1

Liquidgenius

Reputation: 708

Many of the solutions are either missing column names or the species target names. This solution provides target_name labels.

@Ankit-mathanker's solution works, however it iterates the Dataframe Series 'target_names' to substitute the iris species for integer identifiers.

Based on the adage 'Don't iterate a Dataframe if you don't have to,' the following solution utilizes pd.replace() to more concisely accomplish the replacement.

import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris['data'], columns = iris['feature_names'])
df['target'] = pd.Series(iris['target'], name = 'target_values')
df['target_name'] = df['target'].replace([0,1,2],
['iris-' + species for species in iris['target_names'].tolist()])

df.head(3)

	sepal length (cm)	sepal width (cm)	petal length (cm)	petal width (cm)	target_name
0	5.1	3.5	1.4	0.2	iris-setosa
1	4.9	3.0	1.4	0.2	iris-setosa
2	4.7	3.2	1.3	0.2	iris-setosa

Upvotes: 8

justin4480

Reputation: 1431

from sklearn.datasets import load_iris
import pandas as pd

data = load_iris()
df = pd.DataFrame(data=data.data, columns=data.feature_names)
df.head()

This tutorial maybe of interest: http://www.neural.cz/dataset-exploration-boston-house-pricing.html

Upvotes: 127

Richard Jarram

Reputation: 1004

Plenty of good responses to this question; I've added my own below.

import pandas as pd
from sklearn.datasets import load_iris

df = pd.DataFrame(
    # load all 4 dimensions of the dataframe EXCLUDING species data
    load_iris()['data'],
    # set the column names for the 4 dimensions of data
    columns=load_iris()['feature_names']
)

# we create a new column called 'species' with 150 rows of numerical data 0-2 signifying a species type. 
# Our column `species` should have data such `[0, 0, 1, 2, 1, 0]` etc.
df['species'] = load_iris()['target']
# we map the numerical data to string data for species type
df['species'] = df['species'].map({
    0 : 'setosa',
    1 : 'versicolor',
    2 : 'virginica'   
})

df.head()

sepal-df-head

Breakdown

For some reason the load_iris['feature_names] has only 4 columns (sepal length, sepal width, petal length, petal width); moreover, the load_iris['data'] only contains data for those feature_names mentioned above.
Instead, the species column names are stored in load_iris()['target_names'] == array(['setosa', 'versicolor', 'virginica'].
On top of this, the species row data is stored in load_iris()['target'].nunique() == 3
Our goal was simply to add a new column called species that used the map function to convert numerical data 0-2 into 3 types of string data signifying the iris species.

Upvotes: 1

BhishanPoudel

Reputation: 17154

New Update

You can use the parameter as_frame=True to get pandas dataframes.

If as_frame parameter available (eg. load_iris)

from sklearn import datasets
X,y = datasets.load_iris(return_X_y=True) # numpy arrays

dic_data = datasets.load_iris(as_frame=True)
print(dic_data.keys())

df = dic_data['frame'] # pandas dataframe data + target
df_X = dic_data['data'] # pandas dataframe data only
ser_y = dic_data['target'] # pandas series target only
dic_data['target_names'] # numpy array

If as_frame parameter NOT available (eg. load_boston)

from sklearn import datasets

fnames = [ i for i in dir(datasets) if 'load_' in i]
print(fnames)

fname = 'load_boston'
loader = getattr(datasets,fname)()
df = pd.DataFrame(loader['data'],columns= loader['feature_names'])
df['target'] = loader['target']
df.head(2)

Upvotes: 20

Raimundo Jimenez

Reputation: 121

You can use pd.DataFrame constructor, giving a numpy array (data) and a list of the names of the columns (columns). To have everything in one DataFrame, you can concatenate the features and the target into one numpy array with np.c_[...] (note the square brackets and not parenthesis). Also, you can have some trouble if you don't convert the feature names (iris['feature_names']) to a list before concatenation:

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()

df = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                     columns= list(iris['feature_names']) + ['target'])

Upvotes: 1

user3151256

Reputation: 173

This is easy method worked for me.

boston = load_boston()
boston_frame = pd.DataFrame(data=boston.data, columns=boston.feature_names)
boston_frame["target"] = boston.target

But this can applied to load_iris as well.

Upvotes: 9

Ankit Mathanker

Reputation: 1

from sklearn.datasets import load_iris
import pandas as pd

iris_dataset = load_iris()

datasets = pd.DataFrame(iris_dataset['data'], columns = 
           iris_dataset['feature_names'])
target_val = pd.Series(iris_dataset['target'], name = 
            'target_values')

species = []
for val in target_val:
    if val == 0:
        species.append('iris-setosa')
    if val == 1:
        species.append('iris-versicolor')
    if val == 2:
        species.append('iris-virginica')
species = pd.Series(species)

datasets['target'] = target_val
datasets['target_name'] = species
datasets.head()

Upvotes: 0

Joe

Reputation: 51

Here's another integrated method example maybe helpful.

from sklearn.datasets import load_iris
iris_X, iris_y = load_iris(return_X_y=True, as_frame=True)
type(iris_X), type(iris_y)

The data iris_X are imported as pandas DataFrame and the target iris_y are imported as pandas Series.

Upvotes: 3

Paul Rougieux

Reputation: 11399

Otherwise use seaborn data sets which are actual pandas data frames:

import seaborn
iris = seaborn.load_dataset("iris")
type(iris)
# <class 'pandas.core.frame.DataFrame'>

Compare with scikit learn data sets:

from sklearn import datasets
iris = datasets.load_iris()
type(iris)
# <class 'sklearn.utils.Bunch'>
dir(iris)
# ['DESCR', 'data', 'feature_names', 'filename', 'target', 'target_names']

Upvotes: 11

Union find

Reputation: 8150

The API is a little cleaner than the responses suggested. Here, using as_frame and being sure to include a response column as well.

import pandas as pd
from sklearn.datasets import load_wine

features, target = load_wine(as_frame=True).data, load_wine(as_frame=True).target
df = features
df['target'] = target

df.head(2)

Upvotes: 2

dheinz

Reputation: 1086

As of version 0.23, you can directly return a DataFrame using the as_frame argument. For example, loading the iris data set:

from sklearn.datasets import load_iris
iris = load_iris(as_frame=True)
df = iris.data

In my understanding using the provisionally release notes, this works for the breast_cancer, diabetes, digits, iris, linnerud, wine and california_houses data sets.

Upvotes: 4

AdN

Reputation: 161

I took couple of ideas from your answers and I don't know how to make it shorter :)

import pandas as pd
from sklearn.datasets import load_iris
iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris['feature_names'])
df['target'] = iris['target']

This gives a Pandas DataFrame with feature_names plus target as columns and RangeIndex(start=0, stop=len(df), step=1). I would like to have a shorter code where I can have 'target' added directly.

Upvotes: 1

Dhiraj Himani

Reputation: 141

Basically what you need is the "data", and you have it in the scikit bunch, now you need just the "target" (prediction) which is also in the bunch.

So just need to concat these two to make the data complete

  data_df = pd.DataFrame(cancer.data,columns=cancer.feature_names)
  target_df = pd.DataFrame(cancer.target,columns=['target'])

  final_df = data_df.join(target_df)

Upvotes: 2

Shilp Baroda

Reputation: 1

One of the best ways:

data = pd.DataFrame(digits.data)

Digits is the sklearn dataframe and I converted it to a pandas DataFrame

Upvotes: 0

Manideep Pullalachervu

Reputation: 39

import pandas as pd
from sklearn.datasets import load_iris
iris = load_iris()
X = iris['data']
y = iris['target']
iris_df = pd.DataFrame(X, columns = iris['feature_names'])
iris_df.head()

Upvotes: 0

Himanshu Poddar

Reputation: 7779

Whatever TomDLT answered it may not work for some of you because

data1 = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                 columns= iris['feature_names'] + ['target'])

because iris['feature_names'] returns you a numpy array. In numpy array you can't add an array and a list ['target'] by just + operator. Hence you need to convert it into a list first and then add.

You can do

data1 = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                 columns= list(iris['feature_names']) + ['target'])

This will work fine tho..

Upvotes: 0

Jeff Hernandez

Reputation: 2123

This snippet is only syntactic sugar built upon what TomDLT and rolyat have already contributed and explained. The only differences would be that load_iris will return a tuple instead of a dictionary and the columns names are enumerated.

df = pd.DataFrame(np.c_[load_iris(return_X_y=True)])

Upvotes: 1

Nilav Baran Ghosh

Reputation: 1349

TOMDLt's solution is not generic enough for all the datasets in scikit-learn. For example it does not work for the boston housing dataset. I propose a different solution which is more universal. No need to use numpy as well.

from sklearn import datasets
import pandas as pd

boston_data = datasets.load_boston()
df_boston = pd.DataFrame(boston_data.data,columns=boston_data.feature_names)
df_boston['target'] = pd.Series(boston_data.target)
df_boston.head()

As a general function:

def sklearn_to_df(sklearn_dataset):
    df = pd.DataFrame(sklearn_dataset.data, columns=sklearn_dataset.feature_names)
    df['target'] = pd.Series(sklearn_dataset.target)
    return df

df_boston = sklearn_to_df(datasets.load_boston())

Upvotes: 88

niraj

Reputation: 18208

Other way to combine features and target variables can be using np.column_stack (details)

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

data = load_iris()
df = pd.DataFrame(np.column_stack((data.data, data.target)), columns = data.feature_names+['target'])
print(df.head())

Result:

   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)     target
0                5.1               3.5                1.4               0.2     0.0
1                4.9               3.0                1.4               0.2     0.0 
2                4.7               3.2                1.3               0.2     0.0 
3                4.6               3.1                1.5               0.2     0.0
4                5.0               3.6                1.4               0.2     0.0

If you need the string label for the target, then you can use replace by convertingtarget_names to dictionary and add a new column:

df['label'] = df.target.replace(dict(enumerate(data.target_names)))
print(df.head())

Result:

   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)     target  label 
0                5.1               3.5                1.4               0.2     0.0     setosa
1                4.9               3.0                1.4               0.2     0.0     setosa
2                4.7               3.2                1.3               0.2     0.0     setosa
3                4.6               3.1                1.5               0.2     0.0     setosa
4                5.0               3.6                1.4               0.2     0.0     setosa

Upvotes: 6

Victor Tong

Reputation: 1007

Took me 2 hours to figure this out

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()
##iris.keys()


df= pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                 columns= iris['feature_names'] + ['target'])

df['species'] = pd.Categorical.from_codes(iris.target, iris.target_names)

Get back the species for my pandas

Upvotes: 19

daguito81

Reputation: 151

Just as an alternative that I could wrap my head around much easier:

data = load_iris()
df = pd.DataFrame(data['data'], columns=data['feature_names'])
df['target'] = data['target']
df.head()

Basically instead of concatenating from the get go, just make a data frame with the matrix of features and then just add the target column with data['whatvername'] and grab the target values from the dataset

Upvotes: 15

Mukul Aggarwal

Reputation: 1585

This works for me.

dataFrame = pd.dataFrame(data = np.c_[ [iris['data'],iris['target'] ],
columns=iris['feature_names'].tolist() + ['target'])

Upvotes: 6

user1969453

Reputation:

Working off the best answer and addressing my comment, here is a function for the conversion

def bunch_to_dataframe(bunch):
  fnames = bunch.feature_names
  features = fnames.tolist() if isinstance(fnames, np.ndarray) else fnames
  features += ['target']
  return pd.DataFrame(data= np.c_[bunch['data'], bunch['target']],
                 columns=features)

Upvotes: 1

TomDLT

Reputation: 4505

Manually, you can use pd.DataFrame constructor, giving a numpy array (data) and a list of the names of the columns (columns). To have everything in one DataFrame, you can concatenate the features and the target into one numpy array with np.c_[...] (note the []):

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

# save load_iris() sklearn dataset to iris
# if you'd like to check dataset type use: type(load_iris())
# if you'd like to view list of attributes use: dir(load_iris())
iris = load_iris()

# np.c_ is the numpy concatenate function
# which is used to concat iris['data'] and iris['target'] arrays 
# for pandas column argument: concat iris['feature_names'] list
# and string list (in this case one string); you can make this anything you'd like..  
# the original dataset would probably call this ['Species']
data1 = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                     columns= iris['feature_names'] + ['target'])

Upvotes: 209

HakunaMaData

Reputation: 1321

There might be a better way but here is what I have done in the past and it works quite well:

items = data.items()                          #Gets all the data from this Bunch - a huge list
mydata = pd.DataFrame(items[1][1])            #Gets the Attributes
mydata[len(mydata.columns)] = items[2][1]     #Adds a column for the Target Variable
mydata.columns = items[-1][1] + [items[2][0]] #Gets the column names and updates the dataframe

Now mydata will have everything you need - attributes, target variable and columnnames

Upvotes: 0

How to convert a Scikit-learn dataset to a Pandas dataset

Answers (30)

Breakdown

New Update

If as_frame parameter available (eg. load_iris)

If as_frame parameter NOT available (eg. load_boston)

Related Questions