CODEWITHSUNDEEP
amazon-s3luigi
Dimitris Poulopoulos
Dimitris Poulopoulos

Reputation: 1159

How to write a pickle file to S3, as a result of a luigi Task?

I want to store a pickle file on S3, as a result of a luigi Task. Below is the class that defines the Task:

class CreateItemVocabulariesTask(luigi.Task):
    def __init__(self):
        self.client = S3Client(AwsConfig().aws_access_key_id,
                               AwsConfig().aws_secret_access_key)
        super().__init__()

    def requires(self):
        return [GetItem2VecDataTask()]

    def run(self):
        filename = 'item2vec_results.tsv'
        data = self.client.get('s3://{}/item2vec_results.tsv'.format(AwsConfig().item2vec_path),
                               filename)
        df = pd.read_csv(filename, sep='\t', encoding='latin1')
        unique_users = df['CustomerId'].unique()
        unique_items = df['ProductNumber'].unique()
        item_to_int, int_to_item = utils.create_lookup_tables(unique_items)
        user_to_int, int_to_user = utils.create_lookup_tables(unique_users)

        with self.output()[0].open('wb') as out_file:
            pickle.dump(item_to_int, out_file)
        with self.output()[1].open('wb') as out_file:
            pickle.dump(int_to_item, out_file)
        with self.output()[2].open('wb') as out_file:
            pickle.dump(user_to_int, out_file)
        with self.output()[3].open('wb') as out_file:
            pickle.dump(int_to_user, out_file)

    def output(self):
        files = [S3Target('s3://{}/item2int.pkl'.format(AwsConfig().item2vec_path), client=self.client),
                 S3Target('s3://{}/int2item.pkl'.format(AwsConfig().item2vec_path), client=self.client),
                 S3Target('s3://{}/user2int.pkl'.format(AwsConfig().item2vec_path), client=self.client),
                 S3Target('s3://{}/int2user.pkl'.format(AwsConfig().item2vec_path), client=self.client),]
        return files

When I run this task I get the error ValueError: Unsupported open mode 'wb'. The items I try to dump into a pickle file are just python dictionaries.

Full traceback:

Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\luigi\worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "C:\Anaconda3\lib\site-packages\luigi\worker.py", line 140, in _run_get_new_deps
    task_gen = self.task.run()
  File "C:\Users\user\Documents\python workspace\pipeline.py", line 60, in run
    with self.output()[0].open('wb') as out_file:
  File "C:\Anaconda3\lib\site-packages\luigi\contrib\s3.py", line 714, in open
    raise ValueError("Unsupported open mode '%s'" % mode)
ValueError: Unsupported open mode 'wb'

Upvotes: 2

Views: 1280

Answers (1)

matagus
matagus

Reputation: 6206

This is an issue that only happens on python 3.x as explained here. In order to use python 3 and write a binary file or target (ie using 'wb' mode) just set format parameter for S3Target to Nop. Like this:

S3Target('s3://path/to/file', client=self.client, format=luigi.format.Nop)

Notice it's just a trick and not so intuitive nor documented.

Upvotes: 1

Related Questions