CODEWITHSUNDEEP
python-3.xdataframedask
elfinorr
elfinorr

Reputation: 189

Update of dask's dataframe

I'm new to dask so could you help me please? I have a csv-file like this:

id,popularity,hashtag,seen
0,100,#footbal,0
1,200,#2017,0
2,300,#1,0

and somehow i managed to get a dask dataframe hashtags_to_update:

id  seen
0   118
2   136

I'd like to merge a data from hashtags_to_update with data from csv-file to get:

id,popularity,hashtag,seen
0,100,#footbal,118
1,200,#2017,0
2,300,#1,136

For now I'm doing the following

hashtags_df = dd.read_csv('path/to/csv/file').set_index('id')
hashtags_df["seen"] = hashtags_df["seen"].add(hashtags_to_update["seen"], fill_value=0).astype('int64')
hashtags_df.compute().to_csv('output.csv', sep=',')

But as far as I know there are some problems when the data contains strings which are casted as python's objects, so there will be no parallelism because of GIL.

Is there anything you could advice me to do? Thank you in advance.

Upvotes: 2

Views: 681

Answers (1)

moshevi
moshevi

Reputation: 5943

you can use multiprocessing (thus avoiding the GIL).

there are several ways:

setup a client (by default it will ensure multiprocessing):

from dask.distributed import Client
client = Client()

or 

import dask.multiprocessing
dask.config.set(scheduler='processes')  # overwrite default with multiprocessing scheduler

according to the documentation the Former is recommended.

more info:

Client

dask.config.set

Upvotes: 1

Related Questions