Ansible

How to Download Files Using the Ansible get_url Module

As a Linux user, you are probably familiar with tools that allow you to download files from remote servers using HTTP, HTTPS, and FTP protocols, such as wget and cURL. When performing automation tasks, you will occasionally need to download files and packages to remote hosts. While you can use wget and cURL on the remote hosts to perform this function, you can also bypass the required installation to use these tools using the Ansible get_url module.

This tutorial shows you how to use the Ansible get_url command to download files from remote servers.

Before we begin, the Ansible get_url command requires the following prerequisites:

  1. The remote server from which the file(s) are to be downloaded should have direct access to the host server.
  2. The host server should support both HTTP and HTTPS protocols.

About the get_url Module

Before you learn about how to implement the get_url module, first, there are a few things worth understanding about this module. The get_url module supports the following features:

  • Checksum download and validation from server
  • HTTP, HTTPS, and FTP servers
  • Proxy servers, which can be specified by the <protocol>_proxy directive
  • Setting a timeout for get requests
  • Web crawling
  • Basic web authorization

Next, we will show you how to use the get_url module to download files with various protocols and configurations.

Using the get_url Module to Obtain Files

The following are some examples that you can implement using the get_url module when downloading files from a remote server.

Download Files from HTTP/HTTPS Server with Direct URL

Consider the following playbook that creates a directory in the ~/.local and uses the get_url module to download the Debian MySQL package.

NOTE: You can find MYSQL server packages in the resource provided below:

https://linkfy.to/mysql-packages

- hosts: all  

- name: Download Debian MySQL server using HTTP/HTTPS  

  tasks:
 - name: Make directory mysql-server in /home/user/.local    

  file:

    path: ~/.local/mysql-server        

    state: directory        

    mode: 0777      

  - name: GET MySQL-server packages      

      get_url:

        url: “https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar”        

       dest: ~/.local/mysql-server
      mode: 0777
      checksum: md5:5568e206a187a3b658392520540f556e

The above playbook starts by creating a directory in $HOME/.local/mysql-server, where the downloaded package should be stored. If you are creating a directory in a privileged directory, remember to use the ‘become’ directive.

Next, the playbook calls the get_url module and specifies the URL from which to download the package, followed by the destination directory to which to store the file. The final section of the playbook specifies the md5 checksum to check the validity of the file.

NOTE: In the above playbook, we hardcoded the checksum, but you can specify the URL at which the checksum is hosted.

After you run the playbook, you will obtain an output indicating the success or failure of the tasks.

$ ansible-playbook download_mysql_server.yml   PLAY [Download Debian MySQL server using HTTP/HTTPS] ************************************************************************************************************************************************************
  TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [35.222.210.12]
  TASK [Make directory mysql-server in /home/user/.local] ***************************************************************************************************************************************************************
35.222.210.12              : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Once the process is completed, you can log into the host and verify whether the file exists.

Download Files with sha256 Checksum

You can also download files and verify them using the sha256 checksum, as shown in the example playbook below:

- hosts: all  

- name: Download Debian MySQL server using HTTP/HTTPS  

  tasks:

  - name: Make directory mysql-server in /home/user/.local    

    file:

      path: ~/.local/mysql-server      

      state: directory      

      mode: 0777      

  - name: GET MySQL-server packages    

    get_url:

      url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar

      dest: ~/.local/mysql-server
     mode: 0777
     checksum: sha256:b5bb9d8014a0f9b1d61e21e796d78eefdf1352f23cd32812f4850b878ae4944c

To use an URL instead of hardcoding the checksum, consider the following example:

- hosts: all  

- name: Download Debian MySQL server using HTTP/HTTPS  

  tasks:
  - name: Make directory mysql-server in /home/user/.local    

     file:

       path: ~/.local/mysql-server      

       state: directory      

       mode: 0777      

   - name: GET MySQL-server packages    

     get_url:
      url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar
      dest: ~/.local/mysql-server
      mode: 0777
      checksum: sha256:https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar.sha265

Download Files with Timeout on Failure to Respond

In some cases, you may have a specified resource URL that can take longer to respond or is unavailable. This can cause the connection to close before the server responds, as the default timeout is usually 10 seconds. To explicitly specify the timeout value, use the timeout: <value> directive.

Consider the following playbook:

- hosts: all  

- name: Download Debian MySQL server with timeout  

  tasks:
  - name: Make directory mysql-server in /home/user/.local    

   file:

     path: ~/.local/mysql-server      

     state: directory      

     mode: 0777      

   - name: GET MySQL-server packages    

       get_url:

         url: https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar
       dest: ~/.local/mysql-server
       mode: 0777
       checksum: sha256:https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.22-1debian10_amd64.deb-bundle.tar.sha265
       timeout: 30

The above playbook specifies the timeout as 30 seconds, and the connection will be dropped if the server does not respond within the set time.

Download Files with Authentication

To download a file to a server that requires authentication, you will need to specify the login values.

Consider the following playbook:

- hosts: all    

- name: Download File with authentication      

  become: yes      

  get_url:
      url: http://102.15.192.120/backups/database.tar.gz
      dest: /backups
      username: user
      password: '{{pass}}'
      mode: 0777
      timeout: 5

Download Files from Local File Path

To download a file from a local file path, you can use the file:// URI scheme, followed by the path to the file.

Consider the following playbook:

- hosts: webservers  

- name: Download File from local file path    

  become: yes    

  get_url:

    url: file:///backups/secure/config.tar.gz

    dest: /dev/null

Download FTP Files

Downloading FTP files is very similar to the process described in the previous section. All you need to do is specify ftp:// as the protocol to the server.

To download a secure file, you must also add the login information, as shown previously.

Consider the following playbook:

- hosts: all  

  tasks:
  - name: Download file from FTP server    

  become: yes    

    get_url:
      url: ftp://192.168.11.101
      dest: /backups
      mode: 0777

Ansible get_url Module Options

The get_url module also supports various options that you can use to specify and monitor file download and management. The options available with the get_url module include the following:

  • Backup: The backup option, a Boolean of yes and no, allows you to specify whether to create a backup copy of the file to be downloaded.
  • Group: The group option specifies the group that has ownership of the downloaded file; this option is like the chown command in Unix.
  • Headers: The headers option is used to specify custom HTTP headers in a dictionary in hash format.
  • http_agent: The http_agent option specifies the HTTP agent.
  • Owner: The owner option specifies the owner of the download files.
  • Seuser: The seuser option sets the user in the SELinux file context.
  • use_proxy: The use_proxy option sets whether a proxy should be used. If this option is set to false, all proxies will be ignored, even if you have one specified in the target host.

Conclusion

This article discussed the get_url module in Ansible in detail and showed you how to use this module to download files from various sources. You can use this article to reference how to download files in Ansible using the get_url module.

About the author

John Otieno

John Otieno

My name is John and am a fellow geek like you. I am passionate about all things computers from Hardware, Operating systems to Programming. My dream is to share my knowledge with the world and help out fellow geeks. Follow my content by subscribing to LinuxHint mailing list