CODEWITHSUNDEEP
urlencodinghashansiblejinja2
BrainStone
BrainStone

Reputation: 3195

Ansible get hash of binary file from URL

I'm currently trying to get the hash of a binary file stored at an URL.
So I first tried:

hash: {{ lookup('url', my_url) | hash('sha1') }}

This does indeed generate a hash, but not the one I expected.
To verify I made a simple template file with only {{ lookup('url', my_url) }} to what the differences were. Looking at that file it seems like the contents of the URL are interpreted as text and encoded to a different encoding before being output in the file. I haven't quite understood the pattern, but bytes that have the highest bit set get severely managed in the output.
Additionally setting the split_lines parameter to false, doesn't fix the issue. However it results in a few bytes being different in the resulting file. Changing all commas to newlines. Kinda makes sense but also kinda doesn't as with that option being set to true (the default) the result isn't printed as a list.

I also can't quite use the uri builtin, as I need to do that for a dynamic list of URLs.

Minimal example (role):

tasks/main.yml:

- name: "Example"
  template:
    src: example.yml.j2
    dest: example.yml
    lstrip_blocks: true
    trim_blocks: true

defaults/main.yml:

url_list: ["http://example.com/file1.zip", "http://example.com/file2.zip"]

templates/example.yml.j2:

hashes:
  {% for url in url_list %}
  - {{ lookup('url', url) | hash('sha1') }}
  {% endfor %}

Upvotes: 1

Views: 509

Answers (1)

Vladimir Botka
Vladimir Botka

Reputation: 68124

Q: "The url_list varies from host to host."

A: For example, given the files

example.com> for i in *.zip; do sha1 $i; done
SHA1 (file1.zip) = af16fa8e5f3ee18d1c1cf20f2d3816b8b2141c02
SHA1 (file2.zip) = 39b0436794261f51793b1bb772c4731c35850c73
SHA1 (file3.zip) = 2cb291abfed23e3b98009cd6eca754d05dd99559
SHA1 (file4.zip) = d00308e4fb7b924eb4adaa06a859cabec7c4cbae

example.com> for i in *.zip; do md5 $i; done
MD5 (file1.zip) = 937e4d2ea0df44e995d796b0a32ad548
MD5 (file2.zip) = b5c3bb57f8b59815467c89c6beacc41f
MD5 (file3.zip) = e6627cd2113334374cb3e9b0129ff372
MD5 (file4.zip) = 52f421b3608b9cc7df1f6a38d437db97

and the host_vars

shell> cat host_vars/test_11
url_list:
  - http://example.com:8080/file1.zip
  - http://example.com:8080/file2.zip

shell> cat host_vars/test_12
url_list:
  - http://example.com:8080/file3.zip
  - http://example.com:8080/file4.zip

Use get_url, e.g.

- hosts: test_11,test_12
  tasks:
    - tempfile:
        state: directory
      register: tempfile
    - get_url:
        url: "{{ item }}"
        dest: "{{ tempfile.path }}"
      loop: "{{ url_list }}"
      register: result

The results comprise md5 by default

    - set_fact:
        hashes: "{{ result.results|map(attribute='md5sum')|list }}"

gives

ok: [test_11] => 
  hashes:
  - 937e4d2ea0df44e995d796b0a32ad548
  - b5c3bb57f8b59815467c89c6beacc41f
ok: [test_12] => 
  hashes:
  - e6627cd2113334374cb3e9b0129ff372
  - 52f421b3608b9cc7df1f6a38d437db97

If you want a different hash, get it, e.g.

    - command:
        cmd: "sha1 -q {{ item }}"
      loop: "{{ result.results|map(attribute='dest')|list }}"
      register: hresult
    - set_fact:
        hashes: "{{ hresult.results|map(attribute='stdout')|list }}"

gives

ok: [test_11] => 
  hashes:
  - af16fa8e5f3ee18d1c1cf20f2d3816b8b2141c02
  - 39b0436794261f51793b1bb772c4731c35850c73
ok: [test_12] => 
  hashes:
  - 2cb291abfed23e3b98009cd6eca754d05dd99559
  - d00308e4fb7b924eb4adaa06a859cabec7c4cbae

Upvotes: 3

Related Questions