How to adjust kMeans clustering sensitivity?

Question

I have the following dataset:

        node        bc cluster
1    russian  0.457039       1
48       man  0.286875       1
155    woman  0.129939       0
3        bit  0.092721       0
5      write  0.065424       0
98       age  0.064347       0
97     escap  0.062675       0
74      game  0.062606       0

Then I perform kMeans clustering by bc value to separate the nodes into two different groups. Right now with the code below I get the result above (the clustering result is in the cluster column).

    bc_df = pd.DataFrame({"node": bc_nodes, "bc": bc_values})
    bc_df = bc_df.sort_values("bc", ascending=False)
    km = KMeans(n_clusters=2).fit(bc_df[['bc']])
    bc_df.loc[:,'cluster'] = km.labels_
    print(bc_df.head(8))

Which is pretty good, but I would like it to work slightly differently and to select the first 4 nodes into the first cluster and then the other ones in the 2nd one, because they are more similar to each other.

Can I do some adjustment to kMeans or maybe you know another algorithm in sklearn that can do that?

Answer 1

Just choose the threshold yourself.

It's not appropriate to hack the algorithm until you get the desired result.

If you want the first five terms to be a cluster, then just label them as you like. Don't pretend it is a clustering result.

Answer 2

What it looks like you are wanting is a clustering on 1-dimensional data. One way to to work this out is to use the Jenks Natural Breaks (google it to get an explanation of it).

I didn't write this function (lots of credit goes to @Frank with his solution here)

Given your dataframe:

import pandas as pd

df = pd.DataFrame([
['russian',  0.457039],
['man',  0.286875],
['woman',  0.129939],
['bit',  0.092721],
['write',  0.065424],
['age',  0.064347],
['escap',  0.062675],
['game',  0.062606]], columns = ['node','bc'])

Code with Jenks Natural Break Function:

def get_jenks_breaks(data_list, number_class):
    data_list.sort()
    mat1 = []
    for i in range(len(data_list) + 1):
        temp = []
        for j in range(number_class + 1):
            temp.append(0)
        mat1.append(temp)
    mat2 = []
    for i in range(len(data_list) + 1):
        temp = []
        for j in range(number_class + 1):
            temp.append(0)
        mat2.append(temp)
    for i in range(1, number_class + 1):
        mat1[1][i] = 1
        mat2[1][i] = 0
        for j in range(2, len(data_list) + 1):
            mat2[j][i] = float('inf')
    v = 0.0
    for l in range(2, len(data_list) + 1):
        s1 = 0.0
        s2 = 0.0
        w = 0.0
        for m in range(1, l + 1):
            i3 = l - m + 1
            val = float(data_list[i3 - 1])
            s2 += val * val
            s1 += val
            w += 1
            v = s2 - (s1 * s1) / w
            i4 = i3 - 1
            if i4 != 0:
                for j in range(2, number_class + 1):
                    if mat2[l][j] >= (v + mat2[i4][j - 1]):
                        mat1[l][j] = i3
                        mat2[l][j] = v + mat2[i4][j - 1]
        mat1[l][1] = 1
        mat2[l][1] = v
    k = len(data_list)
    kclass = []
    for i in range(number_class + 1):
        kclass.append(min(data_list))
    kclass[number_class] = float(data_list[len(data_list) - 1])
    count_num = number_class
    while count_num >= 2:  # print "rank = " + str(mat1[k][count_num])
        idx = int((mat1[k][count_num]) - 2)
        # print "val = " + str(data_list[idx])
        kclass[count_num - 1] = data_list[idx]
        k = int((mat1[k][count_num] - 1))
        count_num -= 1
    return kclass






# Get values to find the natural breaks    
x = list(df['bc'])

# Calculate the break values. 
# I want 2 groups, so parameter is 2.
# If you print (get_jenks_breaks(x, 2)), it will give you 3 values: [min, break1, max]
# Obviously if you want more groups, you'll need to adjust this and also adjust the assign_cluster function below.
breaking_point = get_jenks_breaks(x, 2)[1]

# Creating group for the bc column
def assign_cluster(bc):
    if bc < breaking_point:
        return 0
    else:
        return 1

# Apply `assign_cluster` to `df['bc']`    
df['cluster'] = df['bc'].apply(assign_cluster)

Output:

print (df)
      node        bc  cluster
0  russian  0.457039        1
1      man  0.286875        1
2    woman  0.129939        1
3      bit  0.092721        0
4    write  0.065424        0
5      age  0.064347        0
6    escap  0.062675        0
7     game  0.062606        0

Answer 3

The first two values always end up in another class than the ones starting at index 3, because they lie below the mean of ~0.152703. Since your question can also be interpreted as a simple two class problem you could also separate the two classes by using the median of ~0.0790725:

idx = df['bc'] > df['bc'].median()

Now you can use this indices to select your two classes, that are separated by the median:

df[idx]

Gives

        node        bc  cluster

  1  russian  0.457039        1
 48      man  0.286875        1
155    woman  0.129939        0
  3      bit  0.092721        0

And

df[~idx]

gives

     node        bc  cluster

 5  write  0.065424        0
98    age  0.064347        0
97  escap  0.062675        0
74   game  0.062606        0