CODEWITHSUNDEEP
pythonpandaslistdataframecompare
ohai
ohai

Reputation: 183

Compare Value in Dataframe and List

I was doing LDA Model.

from helper import *  
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import gensim
from gensim.models import ldamodel
import gensim.corpora

#sp = split.astype('str')
text_list =  [i.split() for i in text]
#text_list = split[0]

train_headlines = [i.split() for i in text_list[0]];
num_topics = 10
id2word = gensim.corpora.Dictionary(train_headlines)
corpus = [id2word.doc2bow(text) for text in train_headlines]
lda = ldamodel.LdaModel(corpus=corpus, id2word=id2word, num_topics=num_topics)

def get_lda_topics(model, num_topics):
    word_dict = {}
    for i in range(num_topics):
        words = model.show_topic(i, topn = 50);
        word_dict['Topic # ' + '{:02d}'.format(i+1)] = [i[0] for i in words]
        #print(word_dict)
    return pd.DataFrame(word_dict)

topics_lda = get_lda_topics(lda, num_topics)

And the result is dataframe #1 contains word, like this :


    Topic # 01   Topic # 02   Topic # 03   Topic # 04   Topic # 05   Topic # 06   Topic # 07   Topic # 08
0     mendapat    Kompascom          ini        resmi        dalam     jaringan         baru    KOMPAScom
1        dunia       secara        fakta         IFCN         lain      Network       selain           Di
2       selain       selain      Lembaga       secara    Kompascom       secara       secara      penguji
3          ada      Network    Kompascom    Kompascom          ini           49    Kompascom        dunia

and I want to check if the contents of dataframe # 1 is also in list # 2. The following is list #2:

['dalam', 'database', 'dilihat', 'sini', 'atau', 'bisa', 'hoaks', 'fakta', 'di', 'KOMPAScom']
['liputan6com', 'mafindo', 'itu', 'tirtoid', 'tempoco', 'lima', 'turnbackhoaxid', 'adalah', 'lembaga', 'dan']
['lembaga', 'checking', 'fact', '49', 'jaringan', 'ada', 'international', 'tersertifikasi', 'network', 'penguji']
['sumber', 'dijadikan', 'beritanya', 'pendanaan', 'partisan', 'non', 'sikap', 'dasar', 'transparan', 'mengutip']

So, the ouput will be:

'mendapat':0, 'Kompascom':0, 'ini': 0, 'resmi':0, 'dalam':1, 'jaringan':1, 'baru':0, 'KOMPAScom':1,.....

1 if the text is in the dataframe and list, 0 if the text is only in one dataframe or list. Can anyone can help? Thanks. Any help is appreciated.

Upvotes: 1

Views: 111

Answers (2)

AirSquid
AirSquid

Reputation: 11938

Gather all of the values in your dataframe in a set, and compare to the set of comparison items.

import pandas as pd
df = pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]})

print(df)

# use set comprehension to "flatten" the nested list that comes from df.values
df_values = {item for sublist in df.values for item in sublist}
print(df_values)
compares = {1,4,9}

# find the items that are in both
print(compares & df_values)

# find the items that are only in the compares set
print(compares - df_values)

Produces:

   a  b
0  1  4
1  2  5
2  3  6

{1, 2, 3, 4, 5, 6}

{1, 4}

{9}

Upvotes: 0

U13-Forward
U13-Forward

Reputation: 71610

Try this code with isin:

l = ['dalam', 'database', 'dilihat', 'sini', 'atau', 'bisa', 'hoaks', 'fakta', 'di', 'KOMPAScom', 'liputan6com', 'mafindo', 'itu', 'tirtoid', 'tempoco', 'lima', 'turnbackhoaxid', 'adalah', 'lembaga', 'dan', 'lembaga', 'checking', 'fact', '49', 'jaringan', 'ada', 'international', 'tersertifikasi', 'network', 'penguji', 'sumber', 'dijadikan', 'beritanya', 'pendanaan', 'partisan', 'non', 'sikap', 'dasar', 'transparan', 'mengutip']
print(dict(zip(df.values.flatten().tolist(), df.isin(l).astype(int).values.flatten().tolist())))

Output:

{'mendapat': 0, 'Kompascom': 0, 'ini': 0, 'resmi': 0, 'dalam': 1, 'jaringan': 1, 'baru': 0, 'KOMPAScom': 1, 'dunia': 0, 'secara': 0, 'fakta': 1, 'IFCN': 0, 'lain': 0, 'Network': 0, 'selain': 0, 'Di': 0, 'Lembaga': 0, 'penguji': 1, 'ada': 1, '49': 1}

Upvotes: 2

Related Questions