CODEWITHSUNDEEP
pythonpython-3.xdoit
Harry
Harry

Reputation: 3412

Python doit: Create subtasks using file generated from previous task

With Python doit I want to generate sub-tasks by reading a list of files from a file (that was generated in an earlier task), then yielding actions for each of those files:

def task_generates_list_of_files():

    def generate_file():

        with open('list_of_files.txt', 'w') as fo:

            for x in range(20):
                fo.write(f'thing_{x}.txt\n')

    return {
        'actions': [generate_file],
        'targets': ['list_of_files.txt']
    }

def task_generate_subtasks_using_file():

    with open('list_of_files.txt', 'r') as fo:

        for filename in fo:

            filename = filename.strip()

            yield {
                'name': filename,
                'actions': [f'program_to_run {filename}'],
                'file_dep': ['list_of_files.txt']
            }

However, because list_of_files.txt doesn't exist when the tasks are being set up by doit, it raises a FileNotFoundError exception.

I have seen this answer but am not clear that getargs can work when generating subtasks, because I wouldn't be able to loop through the list of files until they were injected into a Python action, at which point I can't yield any sub-tasks. Doing that results in:

Python Task error:...
It must return:
False for failed task.
True, None, string or dict for successful task
returned <generator object...

Upvotes: 2

Views: 1372

Answers (1)

Harry
Harry

Reputation: 3412

In the end, I'd missed something I hadn't seen before in the doit documentation: delayed tasks. You can use create_after to defer creation of a task until after a given task is executed.

In my case this allows list_of_files.txt to be generated by task_generates_list_of_files before list_of_files.txt is read within the task definition of task_generate_subtasks_using_file:

from doit import create_after

# Show output
DOIT_CONFIG = {
    'verbosity': 2
}

def task_generates_list_of_files():

    def generate_file():

        with open('list_of_files.txt', 'w') as fo:

            for x in range(20):
                fo.write(f'thing_{x}.txt\n')

    return {
        'actions': [generate_file],
        'targets': ['list_of_files.txt']
    }

@create_after('generates_list_of_files')
def task_generate_subtasks_using_file():

    with open('list_of_files.txt', 'r') as fo:

        for filename in fo:

            filename = filename.strip()

            yield {
                'name': filename,
                'actions': [f'echo {filename}'],
                'file_dep': ['list_of_files.txt']
            }

Upvotes: 1

Related Questions