how to pass arguments to a python fabric custom task

Question

I cannot figure out how to actually pass arguments to a fabric custom task.

I have a bunch of tasks that all need to do the same setup, so I was hoping to subclass the task and have the base class do the setup and then run the specific subtasks. Both the setup code and the subtasks need access to some arguments that are passed in from the command-line to the task. I also need to be able to set default values for the arguments.

Original Attempt

My original attempt shows what I am trying to do without any sub classes. This code works correctly. The code below is in file tmp1.py:

from fabric.api import task

def do_setup(myarg):
    ''' common setup for all tasks '''
    print "in do_setup(myarg=%s)" % myarg
    # do setup using myarg for something important

@task
def actual_task1(myarg='default_value', alias='at'):
    print "In actual_task1(myarg=%s)" % myarg
    do_setup(myarg)
    # do rest of work ...

@task
def actual_task2(myarg='default_value', alias='at'):
    print "In actual_task2(myarg=%s)" % myarg
    do_setup(myarg)
    # do rest of work ...

I run it from the command-line without any args and correctly see the default for myarg of 'default_value'

fab -f ./tmp1.py actual_task1

Prints:

In actual_task1(myarg=default_value)
in do_setup(myarg=default_value)
Done.

Then I call it with myarg='hello' and see that 'hello' gets passed through correctly

fab -f ./tmp1.py actual_task1:myarg='hello'

It outputs:

In actual_task1(myarg=hello)
in do_setup(myarg=hello)
Done.

Attempt with a custom task

My next attempt is to make a common task to encapsulate the setup part. This is copied from http://docs.fabfile.org/en/1.5/usage/tasks.html The code below is in the file tmp2.py:

from fabric.api import task
from fabric.tasks import Task

def do_setup(myarg):
    ''' common setup for all tasks '''
    print "in do_setup(myarg=%s)" % myarg
    # do setup using myarg for something important

'''
Attempt to make a common task to encapsulate the setup part
copied from http://docs.fabfile.org/en/1.5/usage/tasks.html
'''
class CustomTask(Task):
    def init(self, func, myarg, args, *kwargs):
        super(CustomTask, self).init(args, *kwargs)
        print("=> init(myarg=%s, args=%s, kwargs=%s" % (myarg, args, kwargs))
        self.func = func
        self.myarg = myarg
        print "in init: self.func=",self.func,"self.myarg=",self.myarg

    def run(self, *args, **kwargs):
        return self.func(self.myarg, *args, **kwargs)

@task(task_class=CustomTask, myarg='default_value', alias='at')
def actual_task1():
    print "In actual_task1(myarg=%s)" % myarg
    # do rest of work ...

When run, there are 2 problems:

1. __init__ gets "default_value" instead of "Hello"

2. It complains that actual_task1() expects 0 arguments

I run it this way:

fab -f ./tmp2.py actual_task1:myarg="Hello"

Prints:

=> init(myarg=default_value, args=(), kwargs={'alias': 'at'}
in init: self.func= self.myarg= default_value
Traceback (most recent call last):
File "/home/xxx/Documents/pyenvs/xxx/local/lib/python2.7/site-packages/fabric/main.py", line 743,      in main args, *kwargs
File "/home/xxx/Documents/pyenvs/xxx/local/lib/python2.7/site-packages/fabric/tasks.py", line 405,  in execute results[''] = task.run(args, *new_kwargs)
File "/home/xxx/test_fab/tmp2.py", line 21, in run
return self.func(self.myarg, args, *kwargs)
TypeError: actual_task1() takes no arguments (1 given)

I spent quite a bit of time trying to make this work but I cannot seem to solve the default_value issue. I must be missing something?

I would appreciate some help figuring out how to make this sample program run. The second version with the custom task needs to behave just like the original version I showed.

Thank you for any help with this issue.

Answer 1

Fixed example with setup:

from fabric.api import task
from fabric.tasks import Task

def do_setup(foo, verbose):
    ''' common setup for all tasks '''
    print "IN do_setup(foo=%s, verbose=%s)" % (foo, verbose)
    # do setup using foo and verbose...

class CustomTask(Task):
    def __init__(self, func, *args, **kwargs):
        ''' 
        The special args like hosts and roles do not show up in
        args, and kwargs, they are stripped already.

        args and kwargs may contain task specific special arguments
        (e.g. aliases, alias, default, and name) to customize the
        task. They are set in the @task decorator and cannot be passed
        on the command-line. Note also that these special task
        arguments are not passed to the run method.

        Non-special arguments (there are none in this example) are
        set in the task decorator. These other arguments are not
        passed to the run method and cannot be overridden from the
        command-line.

        Note that if you pass any "task specific special arguments" or
        "non-special arguments declared in the task decorator" from the
        command-line, they are treated as different arguments and the
        command-line values are passed to the run method but not to
        this method.
        '''
        super(CustomTask, self).__init__(*args, **kwargs)
        print "IN __init__(args=%s, kwargs=%s)" % (args, kwargs)
        self.func = func

    def run(self, foo='foo_default_val', verbose='verbose_default_val', 
            *args, **kwargs):
        ''' 
        The arguments to this method will be:
        1) arguments from the actual task (e.g. foo and verbose). This method
        is where you set a default value for the arguments from the
        actual_task, not on the actual_task.
        2) task specific arguments from the command-line
        (e.g. actual_task:bar='xxx'). This example is not expecting any,
        so it strips them and does not pass them to the
        actual_function one (e.g. it calls self.func with only foo
        and verbose and does not pass args and kwargs)
        '''
        print "IN run(foo=%s, verbose=%s, args=%s, kwargs=%s)" % \
                  (foo, verbose, args, kwargs)
        do_setup(foo, verbose)
        return self.func(foo, verbose)

@task(task_class=CustomTask, alias="RUNME")
def actual_task(foo, verbose):
    print 'IN task actual_task(foo=%s, verbose=%s)' % (foo, verbose)

Run with only host specified on the command-line:

fab -f ./example_with_setup.py actual_task:host='hhh'

IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=foo_default_val, verbose=verbose_default_val, args=(), kwargs={})
IN do_setup(foo=foo_default_val, verbose=verbose_default_val)
IN task actual_task(foo=foo_default_val, verbose=verbose_default_val)

Run specifying foo on the commandline:

fab -f ./example_with_setup.py actual_task:host='hhh',foo='bar'

IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=bar, verbose=verbose_default_val, args=(), kwargs={})
IN do_setup(foo=bar, verbose=verbose_default_val)
IN task actual_task(foo=bar, verbose=verbose_default_val)

Run specifying both foo and verbose on the command-line:

fab -f ./example_with_setup.py actual_task:host='hhh',foo='bar',verbose=True

IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=bar, verbose=True, args=(), kwargs={})
IN do_setup(foo=bar, verbose=True)
IN task actual_task(foo=bar, verbose=True)

Answer 2

This is the fixed example.

# fixed the example from http://docs.fabfile.org/en/1.8/usage/tasks.html
from fabric.api import task
from fabric.tasks import Task

class CustomTask(Task):
    def __init__(self, func, myarg1, *args, **kwargs):
        ''' 
        The special args like hosts and roles do not show up in
        args, and kwargs, they are stripped already.

        args and kwargs may contain task specific special arguments
        (e.g. aliases, alias, default, and name) to customize the
        task. They are set in the @task decorator and cannot be passed
        on the command-line. Note also that these special task
        arguments are not passed to the run method.

        Non-special arguments (in this example myarg1) are set in the task
        decorator. These other arguments are not passed to the run
        method and cannot be overridden from the command-line. 

        Note that if you pass any "task specific special arguments" or
        "non-special arguments declared in the task decorator" from the
        command-line, they are treated as different arguments and the
        command-line values are passed to the run method but not to
        this method.
        '''
        super(CustomTask, self).__init__(*args, **kwargs)
        print "IN __init__(myarg1=%s, args=%s, kwargs=%s)" % \
            (myarg1, args, kwargs)
        self.func = func
        self.myarg1 = myarg1

    def run(self, myarg2='default_value2', *args, **kwargs):
        ''' 
        The arguments to this method will be:
        1) arguments from the actual task (e.g. myarg2). This method
        is where you set a default value for the arguments from the
        actual_task, not on the actual_task.
        2) task specific arguments from the command-line
        (e.g. actual_host:foo='foo'). This example is not expecting
        any, so it strips them and does not pass them to the
        actual_function (e.g. it calls self.func with only myarg2 and
        does not pass args and kwargs)
        '''
        print "IN run(myarg2=%s, args=%s, kwargs=%s)" % \
            (myarg2, args, kwargs)
        return self.func(myarg2)

@task(task_class=CustomTask, myarg1='special_value', alias='RUNME')
def actual_task(myarg2):
    print "IN actual_task(myarg2=%s)" % myarg2

Run with only hosts specified on the command-line:

fab -f ./fixed_example actual_task:hosts="hhh"

IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=default_value2, args=(), kwargs={})
IN actual_task(myarg2=default_value2)

Run specifying myarg2 on the command-line:

fab -f ./fixed_example actual_task:hosts="hhh",myarg2="good_value"

IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=good_value, args=(), kwargs={})
IN actual_task(myarg2=good_value)

Bad run specifying myarg1 and alias on the command-line. Notice that init gets the values specified in the task decorator and not the values from the command-line. Notice that run gets myarg1 and alias as arguments now.

fab -f ./fixed_example actual_task:hosts="hhh",myarg1="myarg1_from_commandline",alias="alias_from_commandline"

IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=default_value2, args=(), kwargs={'alias': 'alias_from_commandline', 'myarg1': 'myarg1_from_commandline'})
IN actual_task(myarg2=default_value2)

Answer 3

In the custom class section, the function actual_task1 doesn't actually take arguments, so the only valid way to invoke your fabric file is:

fab -f ./tmp2.py actual_task1

Furthermore, I don't think you're actually calling do_setup in either CustomTask or actual_task1