This guide will implement task queues in Redis using the Python Redis Queue library.
What is Redis Queue?
Python Redis Queue or RQ is a simple yet powerful Python library that works with Redis to perform task ques and execution in the background using workers. RQ is straightforward to use for beginners but still very powerful for large projects.
The feature of queuing tasks is essential when working with functions and code that tends to block program execution. An example of such code is network requests.
Let us discuss how we can use this tool.
Environment Setup
Before we can proceed, you need to ensure you have a good environment. For this, you will require a running version of the Redis server, Python 3, and Pip installed.
We will illustrate the installation and setup on a Ubuntu system.
Start by updating the packages and install Redis server using the commands shown below:
sudo apt-get install redis -y
Once completed, start the Redis server using the command:
The next step is to install Python3 and pip on our system. Feel free to skip to the next sections if you have Python installed.
Next, use pip to install the RQ library.
The above command will download and install the RQ library, and we can start using it.
Working with the Redis Queue
To illustrate using the RQ library, we will use a simple HTTP request. In our example, we will create a simple function that makes an API call to ipify.org and get our current IP address. The function makes an HTTP request to the server, meaning it is a blocking function.
Create a simple python file and call it IP.py. Next, enter the code as:
def get_ip(url):
response = requests.get(url).json()
return response
print(get_ip("https://api.ipify.org?format=json"))
The above code will return your current IP address. You will notice that the request takes a few seconds to resolve and the server to respond. This means that the rest of the code is blocked until this block is executed.
An example response from the code above is as shown:
To prevent the function from blocking the program execution, we can pass it to RQ, which can be processed as an asynchronous task.
We can do this by importing the RQ library, creating a simple queue, and queue our blocking function.
Create a simple python file and call it print_ip. Enter the code as shown:
from rq import Queue
from IP import get_ip
q = Queue(connection=Redis())
result = q.enqueue(get_ip, "https://api.ipify.org?format=json")
Save and close the file.
We need to run a worker in our working directory to process the enqueued tasks in the background.
A worker is a Python process that runs in the background to execute blocking tasks in the code. RQ uses the functionality of workers to perform enqueued tasks.
To execute the code in the previous example, open a new terminal window and navigate to your working directory (where the python code is located).
Next, execute the command below to start the worker.
The command above should start the worker as shown:
If you do not need a scheduler, you can remove the –with-scheduler option.
Once the worker is running, execute the code:
You should now see the information about the tasks printed in the worker window as shown:
To get precise information about the non-blocking feature of this function, you can try adding a bunch of print statements after it.
You will notice that the print statements are printed immediately after the file is executed despite the requests taking a while to process.
Conclusion
This guide walks you through the basics of working with the Redis Queue. Although we use simple examples in this guide, it will hopefully provide you with a starting point to implement more complex options. Consider reading the RQ documentation to learn more.
