K-means clustering on 3 dimensions with sklearn

Question

I'm trying to cluster data using lat/lon as X/Y axes and DaysUntilDueDate as my Z axis. I also want to retain the index column ('PM') so that I can create a schedule later using this clustering analysis. The tutorial I found here has been wonderful but I don't know if it's taking the Z-axis into account, and my poking around hasn't resulted in anything but errors. I think the essential point in the code is the parameters of the iloc bit of this line:

kmeans_model = KMeans(n_clusters=k, random_state=1).fit(A.iloc[:, :])

I tried changing this part to iloc[1:4] (to only work on columns 1-3) but that resulted in the following error:

ValueError: n_samples=3 should be >= n_clusters=4

So my question is: How can I set up my code to run clustering analysis on 3-dimensions while retaining the index ('PM') column?

Here's my python file, thanks for your help:

from sklearn.cluster import KMeans
import csv
import pandas as pd

# Import csv file with data in following columns:
#    [PM (index)] [Longitude] [Latitude] [DaysUntilDueDate]

df = pd.read_csv('point_data_test.csv',index_col=['PM'])

numProjects = len(df)
K = numProjects // 3    # Around three projects can be worked per day


print("Number of projects: ", numProjects)
print("K-clusters: ", K)

for k in range(1, K):
    # Create a kmeans model on our data, using k clusters.
    #   Random_state helps ensure that the algorithm returns the
    #   same results each time.
    kmeans_model = KMeans(n_clusters=k, random_state=1).fit(df.iloc[:, :])

    # These are our fitted labels for clusters --
    #   the first cluster has label 0, and the second has label 1.
    labels = kmeans_model.labels_

    # Sum of distances of samples to their closest cluster center
    SSE = kmeans_model.inertia_

print("k:",k, " SSE:", SSE)

# Add labels to df
df['Labels'] = labels
#print(df)

df.to_csv('test_KMeans_out.csv')

Answer 1

It seems the issue is with the syntax of iloc[1:4].

From your question it appears you changed:

kmeans_model = KMeans(n_clusters=k, random_state=1).fit(df.iloc[:, :])

to:

kmeans_model = KMeans(n_clusters=k, random_state=1).fit(df.iloc[1:4])

It seems to me that either you have a typo or you don't understand how iloc works. So I will explain.

You should start by reading Indexing and Selecting Data from the pandas documentation.

But in short .iloc is an integer based indexing method for selecting data by position.

Let's say you have the dataframe:

 A    B    C
 1    2    3
 4    5    6
 7    8    9
10   11   12  

The use of iloc in the example you provided iloc[:,:] selects all rows and columns and produces the entire dataframe. In case you aren't familiar with Python's slice notation take a look at the question Explain slice notation or the docs for An Informal Introduction to Python. The example you said caused your error iloc[1:4] selects the rows at index 1-3. This would result in:

 A    B    C
 4    5    6
 7    8    9
10   11   12 

Now, if you think about what you are trying to do and the error you received you will realize that you have selected fewer samples form your data than you are looking for clusters. 3 samples (rows 1, 2, 3) but you're telling KMeans to find 4 clusters, which just isn't possible.

What you really intended to do (as I understand it) was to select all rows and columns 1-3 that correspond to your lat, lng, and z values. To do this just add a colon as the first argument to iloc like so:

df.iloc[:, 1:4]

Now you will have selected all of your samples and the columns at index 1, 2, and 3. Now, assuming you have enough samples, KMeans should work as you intended.