Cluster unseen points using Spectral Clustering

Question

I am using Spectral Clustering method to cluster my data. The implementation seems to work properly. However, I have one problem - I have a set of unseen points (not present in the training set) and would like to cluster these based on the centroids derived by k-means (Step 5 in the paper). However, the k-means is computed on the k eigenvectors and therefore the centroids are low-dimensional.

Does any-one knows a method that can be used to map an unseen point to a low-dimension and compute the distance between the projected point and the centroids derived by k-means in step 5.

Answer 1

Late answer, but here's how to do it in R. I've been searching it myself, but I finally managed to code it myself.

##Let's use kernlab for all kernel stuff
library(kernlab)

##Let's generate two concentric circles to cluster
r1 = 1 + .1*rnorm(250) #inner
r2 = 2 + .1*rnorm(250) #outer
q1 = 2*pi*runif(500) #random angle distribution
q2 = 2*pi*runif(500) #random angle distribution

##This is our data now
data = cbind(x = c(r1*cos(q1),r2*cos(q2)), y = c(r1*sin(q1),r2*sin(q2)))

##Let's take a sample to define train and test data
t = sample(1:nrow(data), 0.95*nrow(data))
train = data[t,]
test = data[-t,]

##This is our data
plot(train, pch = 1, col = adjustcolor("black", alpha = .5))
points(test, pch = 16)
legend("topleft", c("train data","test data"), pch = c(1,16), bg = "white")


##The paper gives great instructions on how to perform spectral clustering
##so I'll be following the steps
##Ng, A. Y., Jordan, M. I., & Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 2, 849-856.
##Pg.2 http://papers.nips.cc/paper/2092-on-spectral-clustering-analysis-and-an-algorithm.pdf
#1. Form the affinity matrix
k = 2L #This is the number ofo clusters we will train
K = rbfdot(sigma = 300) #Our kernel
A = kernelMatrix(K, train) #Caution choosing your kernel product function, some have higher numerical imprecision
diag(A) = 0
#2. Define the diagonal matrix D and the laplacean matrix L
D = diag(rowSums(A))
L = diag(1/sqrt(diag(D))) %*% A %*% diag(1/sqrt(diag(D)))
#3. Find the eigenvectors of L
X = eigen(L, symmetric = TRUE)$vectors[,1:k]
#4. Form Y from X
Y = X/sqrt(rowSums(X^2))
#5. Cluster (k-means)
kM = kmeans(Y, centers = k, iter.max = 100L, nstart = 1000L)
#6. This is the cluster assignment of the original data
cl = fitted(kM, "classes")
##Projection on eigen vectors, see the ranges, it shows how there's a single preferential direction
plot(jitter(Y, .1), ylab = "2nd eigenfunction", xlab = "1st eigenfunction", col = adjustcolor(rainbow(3)[2*cl-1], alpha = .5))

##LET'S TRY TEST DATA NOW
B = kernelMatrix(K, test, train) #The kernel product between train and test data

##We project on the learned eigenfunctions
f = tcrossprod(B, t(Y))
#This part is described in Bengio, Y., Vincent, P., Paiement, J. F., Delalleau, O., Ouimet, M., & Le Roux, N. (2003). Spectral clustering and kernel PCA are learning eigenfunctions (Vol. 1239). CIRANO.
#Pg.12 http://www.cirano.qc.ca/pdf/publication/2003s-19.pdf

##And assign clusters based on the centers in that space
new.cl = apply(as.matrix(f), 1, function(x) { which.max(tcrossprod(x,kM$centers)) } ) #This computes the distance to the k-means centers on the transformed space

##And here's our result
plot(train, pch = 1, col = adjustcolor(rainbow(3)[2*cl-1], alpha = .5))
points(test, pch = 16, col = rainbow(3)[2*new.cl-1])
legend("topleft", c("train data","test data"), pch = c(1,16), bg = "white")

The output pictures

Answer 2

Use the same assignment rule used in the clustering method from Step 5 to assign the new data points. For example, k-means uses some distance metric d to assign the data points in the original learning dataset to some cluster. Just use this same metric to assign the unseen point to one of your final clusters. So, add a new Step 7.

7. Assign an unseen point P to cluster j if d(P, x_j) is a minimum over all possible clusters x_i.