Ansible: Copy multiple files from multiple sources to multiple destinations in single command

Question

I am trying to copy some configuration files from /tmp to /opt. Here first I am recursively searching for the files in /tmp and /opt directories and storing it in the variable tmp_file_path and code_file_path respectively, which has an attribute files.path that I need to use in the source and destination for copy

- name: Find files in tmp
  find:
    paths: /tmp/
    file_type: file
    recurse: yes
    patterns:
      - file1
      - file2
      - file3
  register: tmp_file_path
- debug:
    var: tmp_file_path

- name: Find files in code
  find:
    paths: /opt/
    file_type: file
    recurse: yes
    patterns: 
      - file1
      - file2
      - file3
  register: code_file_path
- debug:
    var: code_file_path

Here the source file paths can be /tmp/folder1/file1, /tmp/folder2/file2, /tmp/folder13/file3. Destinations can be /opt/folderA/file1, /opt/folderB/file3, /opt/folderC/file3

As of now I have managed to write the task as below

- name: Copy files from tmp to code directory
  copy:
    src: "{{item.path}}"
    dest: "{{item.path}}"
    remote_src: yes
  with_items:
    - { "{{ tmp_file_path.files }}", "{{ code_file_path.files }}" }

The copy has to be done in a single command so that I do not end up hardcoding the paths for source and destination. Can anyone help me with achieving this?

Below piece of code worked for recursively copying files from code_file_path to /tmp

- name: Copy files from code directory to tmp
  copy:
    src: "{{item.path}}"
    dest: /tmp/
    remote_src: yes
  with_items: "{{code_file_path.files}}"

Answer 1

You may achieve this with this condition : every filenames are unique in the way that if 2 files have the same name in your list from /opt directory, then their content should be the same.

If it's the case, then you may use the with_nested loop and using a conditionnal when on filename.

For example:

- name: Copy files from tmp to code directory
  copy:
    src: "{{ item.0 }}"
    dest: "{{ item.1 }}"
    remote_src: yes
  when: 
    - item.0|basename == item.1|basename
  with_nested:
    - "{{ tmp_file_path.files|map(attribute='path')|list }}"
    - "{{ code_file_path.files|map(attribute='path')|list }}"

The only "problem" with this solution is that you'll run the loop a lot of times...

You may also want to use the loop synthax over with_ synthax and use some loop_control in order to choose what is printed: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html

---

If you have the jinja.ext.do extension loaded (in your ansible.cfg: jinja2_extensions = jinja2.ext.do) you may then contruct a dict with your paths:

- name: Make a dict
  set_fact:
    files_dict: |
      {%- set out_dict = dict() -%}
      {%- for tmp_file in tmp_file_path.files|map(attribute='path')|list -%}
      {%-   do out_dict.update({tmp_file|basename: {'tmp': tmp_file}}) -%}
      {%- endfor -%}
      {%- for opt_file in code_file_path.files|map(attribute='path')|list -%}
      {%-   do out_dict[opt_file|basename].update({'opt': opt_file}) -%}
      {%- endfor -%}
      {{ out_dict }}

- name: Copy from dict
  copy:
    src: "{{ item.value.tmp }}"
    dest: "{{ item.value.opt }}"
    remote_src: yes
  with_dict: "{{ files_dict }}"

Answer 2

Try this

- name: Copy files from tmp to code directory
  copy:
    src: "{{ item.0 }}"
    dest: "{{ item.1 }}"
    remote_src: yes
  with_together:
    - "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
    - "{{ code_file_path.files|map(attribute='path')|list|sort }}"

Try with debug first

    - debug:
        msg:
          - "src: {{ item.0 }}"
          - "dest: {{ item.1 }}"
      with_together:
        - "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
        - "{{ code_file_path.files|map(attribute='path')|list|sort }}"

It might be useful to test the sanity first

    - debug:
        msg: The numbers of files do not match
      when: tmp_file_path.files|map(attribute='path')|list|length !=
            code_file_path.files|map(attribute='path')|list|length

---

Q: "This did not work as expected because the files which I am searching in code_file_path has multiple sub-directories. It is sorting on the entire file path returned by code_file_path.files rather than just the file name."

A: Create a list with both paths and names. Then sort the list by the name. For example

    - set_fact:
        code_files: "{{ code_files|default([]) +
                        [{'path': item, 'name': item|basename}] }}"
      loop: "{{ code_file_path.files|map(attribute='path')|list }}"

    - debug:
        msg:
          - "src: {{ item.0 }}"
          - "dest: {{ item.1.path }}"
      with_together:
        - "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
        - "{{ code_files|sort(attribute='name') }}"

---

Example

shell> tree tmp
tmp
├── file1
├── file2
└── file3

shell> tree opt
opt
├── bar
│   └── file2
├── baz
│   └── file3
└── foo
    └── file1

The tasks below

    - set_fact:
        code_files: "{{ code_files|default([]) +
                        [{'path': item, 'name': item|basename}] }}"
      loop: "{{ code_file_path.files|map(attribute='path')|list }}"

    - debug:
        var: code_files|sort(attribute='name')

    - debug:
        msg:
          - "src: {{ item.0 }}"
          - "dest: {{ item.1.path }}"
      with_together:
        - "{{ tmp_file_path.files|map(attribute='path')|list|sort }}"
        - "{{ code_files|sort(attribute='name') }}"

give

    "code_files|sort(attribute='name')": [
        {
            "name": "file1", 
            "path": "/export/test/opt/foo/file1"
        }, 
        {
            "name": "file2", 
            "path": "/export/test/opt/bar/file2"
        }, 
        {
            "name": "file3", 
            "path": "/export/test/opt/baz/file3"
        }
    ]

    "msg": [
        "src: /export/test/tmp/file1", 
        "dest: /export/test/opt/foo/file1"
    ]

    "msg": [
        "src: /export/test/tmp/file2", 
        "dest: /export/test/opt/bar/file2"
    ]

    "msg": [
        "src: /export/test/tmp/file3", 
        "dest: /export/test/opt/baz/file3"
    ]