How to compress files with zstd from the command line or bash script

In the modern world, we interact and work with data everywhere and every day. Depending on the needs at hand and the data requiring processing, storage, and computing power can quickly become limited resources.

To solve this problem, developers developed compression algorithms and tools capable of compressing data in real-time, reducing the size and processing power required. One of these tools is Zstandard, commonly known as Zstd.

Zstd is a free, open-source, real-time compression algorithm developed by Yann Collect, an employee at Facebook. Zstd is very fast and offers outstanding compression ratios. It is a lossless compression algorithm written in C but has API implementations in other popular programming languages such as Python, Java, C#, JavaScript, and many more. It also provides in-memory compression and decompression functions.

To check if Zstd supports your desired language, check the resource provided below:

https://facebook.github.io/zstd/

If you wish to look at benchmarking information about Zstd, use the link below:

https://github.com/facebook/zstd

This tutorial will show you how to compile and install the Zstd tool in Linux, then use it to perform data compression and decompression.

How to Install Zstd

To use Zstd, we need to install it by compiling from the sources. Depending on the system you are running and the configuration, you may need to install dependencies and tools to perform the compilation successfully.

Start by updating your system and installing GNU make using the command:

Once we have all the tools installed, we can download the source files and compile them. Start by navigating where you have read, write, and execute permissions. ~/Desktop

Next, use wget to download the files into the directory.

Now unarchive the download file and navigate into the directory using the commands as:

The final steps are to install Zstd by using make and make install inside the Zstd directory.

Once the compilation and installation complete successfully, you can start using Zstd on your system to compress and decompress files.

How to Use Zstd

Zstd does not defer from popular compression and decompression methods at all. Although the underlying technology and implementation are different from other tools, compressing a file with Zstd is similar to tar and gzip syntax.

How to compress a file

To compress a file, call the zstd command followed by the -z flag, which tells zstd to do the compression, and finally, the name of the file to compress.

For example, the command below compresses the system-backup file into a .zst file.

The command output as shown below:

You should see a file with .zst extension as:

Once the command executes, the file gets compressed and creates a filename .zst file that you can decompress.

Compress and remove the source file

As you can see from the above command, the source file does not get removed by default upon compression. You can specify to remove the source files by using the –rm flag:

Specifying the –rm flag automatically removes the source file.

Get file info

To display related information about the Zstd compressed file, you can use the -l flag followed by the file name. The displayed information includes the file size, compression ratio, and the file checksum

Specify the compression level

To explicitly specify the compression level, use the – where the level is a value ranging from 1 – 19. The default compression level is 3. You can also unlock higher compression levels, i.e., level 20 – 22.

NOTE: The higher the compression level, the higher memory usage.

For example, to compress a file with a compression level of 10, use the command:

Specify Compression Speed.

Zstd also allows you to set the compression speed ranging from 1 – to. The compression speed is inversely proportional to the compression ratio. The default compression speed is 1, and the higher the value, the faster the compression speed.

For example, to use the maximum compression speed, use the command:

Specify Compression Format

You can also specify the compression format to use if you do not like the default zst compression. Formats include zstd, gzip, xz, lzma, and lz4.

Use the –format flag and specify the format as:

Compress file list

Suppose you have a list of files you would like to compress all at once. Zstd allows you to pass a file containing a list of files and compresses them recursively.

For example, a file list.txt containing the files

Once you save the file, you can pass the list to Zstd with all other options to perform on the files.

This command will compress all the files specified in the text file and remove them upon completion.

Decompress a file

To decompress a file, you can use the -d flag with the zstd command or simply use the unzstd command to decompress.

For example:

Verify file integrity

To test the integrity of a zst compressed file, use the -t flag as shown in the command below:

Conclusion

As we can see from the examples, Zstd is a powerful compression algorithm with numerous use cases. To learn more about how it works and its implementation, check the man page and experiment.

Thanks for reading.