Remote ray call not detecting function arguments

Question

When executing a @ray.remote function, the following exception is being thrown, even though I am supplying all the arguments set in the function definition:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ray/signature.py in extend_args(function_signature, args, kwargs)
    208                     raise Exception("No value was provided for the argument "
    209                                     "'{}' for the function '{}'.".format(
--> 210                                         keyword_name, function_name))
    211 
    212     no_positionals = len(arg_is_positionals) == 0 or not arg_is_positionals[-1]

Exception: No value was provided for the argument 'phones' for the function 'compile_file'.

Edit: A minimal example of my remote function definition and remote call is as follows:

import ray
ray.init(num_cpus=4, num_gpus=1, include_webui=False) #initialize ray with 4 CPUs

@ray.remote
def compile_file(self, rgx_patts, phones): # method for my Case class

    self._phones = self.phonelist(rgx_patts, phones)

def compile_all(inputDirectory='C/TODOS', phones = ['10002000']):  
   d = {}
   file_lst = pdfLister(inputDirectory, termin)
   for i, file in enumerate(file_lst):
      doc = Case(file)
      doc.compile_file.remote(rgx_patts, phones) # exception thrown here
      d[i] = doc
   case_dic = {k: ray.get(dic_id) for k, dic_id in d.items()}
   return case_dic

Edit: Full exception below

   ---------------------------------------------------------------------------
   Exception                             Traceback (most recent call last)
   <timed exec> in <module>()

   ~/compile_files.py in compile_all(pckle, inputDirectory,   pickle_op_file, termin, rgx_patts, ceav_phones)
    111                             prm._rgx_patts, prm._ceav_phones)
    114             d[i] = doc
    115         ceav_dic = {k: ray.get(dic_id) for k, dic_id in d.items()}

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ray/remote_function.py in remote(self, *args, **kwargs)
    103     def remote(self, *args, **kwargs):
    104         """This runs immediately when a remote function is called."""
--> 105         return self._submit(args=args, kwargs=kwargs)
    106 
    107     def _submit(self,

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ray/remote_function.py in _submit(self, args, kwargs, num_return_vals, num_cpus, num_gpus, resources)
    118         kwargs = {} if kwargs is None else kwargs
    119         args = ray.signature.extend_args(self._function_signature, args,
--> 120                                          kwargs)
    121 
    122         if num_return_vals is None:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ray/signature.py in extend_args(function_signature, args, kwargs)
    208                     raise Exception("No value was provided for the argument "
    209                                     "'{}' for the function '{}'.".format(
--> 210                                         keyword_name, function_name))
    211 
    212     no_positionals = len(arg_is_positionals) == 0 or not arg_is_positionals[-1]

   Exception: No value was provided for the argument 'ceav_phones' for the function 'compile_file'.

Answer 1

In your example, compile_file seems to be the method of a class, but the class definition is not provided.

I'd suggest either moving everything into the compile_file function so that it is a standalone function, e.g.,

import ray

ray.init(num_cpus=4, num_gpus=1)

class Case(object):
    def __init__(self, file):
        pass

    def compile_file(self):
        pass

@ray.remote
def compile_file_helper(file):
    case = Case(file)
    case.compile_file()

value_ids = []
for file in ['file1', 'file2', 'file3']:
    value_ids.append(compile_file_helper.remote(file))

values = ray.get(value_ids)

Another alternative is to make the Case class an actor. For example,

# This assumes you've already called "import ray" and "ray.init()".

@ray.remote
class Case(object):
    def __init__(self, file):
        pass

    def compile_file(self):
        pass

# Create one actor per file (more generally, you could create a pool
# of actors and have each actor compile multiple files).
value_ids = []
for file in ['file1', 'file2', 'file3']:
    case = Case.remote(file)
    value_ids.append(case.compile_file.remote())

values = ray.get(value_ids)

In this example, we're only invoking one method on each actor, however, this approach really only makes sense if you are going to invoke multiple methods per actor.

Answer 2

The reason for the exception is that your remote function expects three arguments: self, rgx_patts, and phones, since that's how it was declared:

@ray.remote
def compile_file(self, rgx_patts, phones): 

Note that you only passed two arguments to it in your call:

doc.compile_file.remote(rgx_patts, phones)

The first argument is interpreted as being self, the second argument is interpreted as being rgx_patts, and there is no argument passed for phones, hence the exception. A possible workaround is to pass the object (doc, in your example) as your first argument, such that the call would become:

doc.compile_file.remote(doc, rgx_patts, phones)

Which looks a bit wonky, but I think it should work - and in fact it works for me in a similar scenario, although I can't test it for your case since your code sample is not self contained. If you go with this solution, and doc is the same very time, you might want to use ray.put() to serialize doc only once, instead of every time you call that method. To do that, you could do doc_id = ray.put(doc) (only once) and then replace doc with doc_id in the above code (doc_id is the object id for the serialized doc).

Note that another solution is to convert your class into an Actor, which seems appropriate if your class has a "state" that needs to be stored in memory in between calls to the method.