CODEWITHSUNDEEP
rdbscan
Archana Jalaja Surendran
Archana Jalaja Surendran

Reputation: 137

DBSCAN Clustering returning single cluster with noise points

I am trying to perform DBSCAN clustering on the data https://www.kaggle.com/arjunbhasin2013/ccdata. I have cleaned the data and applied the algorithm.

data1 <- read.csv('C:\\Users\\write\\Documents\\R\\data\\Project\\Clustering\\CC GENERAL.csv')

head(data1)

data1 <- data1[,2:18]

dim(data1)

colnames(data1)

head(data1,2)

#to check if data has empty col or rows
library(purrr)
is_empty(data1)

#to check if data has duplicates
library(dplyr)
any(duplicated(data1))

#to check if data has NA values
any(is.na(data1))
data1 <- na.omit(data1)
any(is.na(data1))
dim(data1)

Algorithm was applied as follows.

#DBSCAN
data1 <- scale(data1)

library(fpc)
library(dbscan)

set.seed(500)
#to find optimal eps
kNNdistplot(data1, k =  34)
abline(h = 4, lty = 3)

enter image description here

The figure shows the 'knee' to identify the 'eps' value. Since there are 17 attributes to be considered for clustering, I have taken k=17*2 =34.

db <- dbscan(data1,eps = 4,minPts = 34)
db

The result I obtained is "The clustering contains 1 cluster(s) and 147 noise points."

No matter whatever values I change for eps and minPts the result is same.

Can anyone tell where I have gone wrong?

Thanks in advance.

Upvotes: 0

Views: 3259

Answers (2)

Michael Hahsler
Michael Hahsler

Reputation: 3075

A typical problem with using DBSCAN (and clustering in general) is that real data typically does not fall into nice clusters, but forms one connected point cloud. In this case, DBSCAN will always find only a single cluster. You can check this with several methods. The most direct method would be to use a pairs plot (a scatterplot matrix):

plot(as.data.frame(data1))

Since you have many variables, the scatterplot pannels are very small, but you can see that the points are very close together in almost all pannels. DBSCAN will connect all points in these dense areas into a single cluster. k-means will just partition the dense area.

Another option is to check for clusterability with methods like VAT or iVAT (https://link.springer.com/chapter/10.1007/978-3-642-13657-3_5).

library("seriation")

## calculate distances for a small sample
d <- dist(data1[sample(seq(nrow(data1)), size = 1000), ])

iVAT(d)

You will see that the plot shows no block structure around the diagonal indicating that clustering will not find much.

To improve clustering, you need to work on the data. You can remove irrelevant variables, you may have very skewed variables that should be transformed first. You could also try non-linear embedding before clustering.

Upvotes: 0

Acturio
Acturio

Reputation: 84

You have two options:

  1. Increase the radius of your center points (given by the epsilon parameter)
  2. Decrease the minimum number of points (minPts) to define a center point.

I would start by decreasing the minPts parameter, since I think it is very high and since it does not find points within that radius, it does not group more points within a group

Upvotes: 0

Related Questions