How to parse nginx access logs

If you want to optimize your web server, it is essential to understand Nginx access logs. Logging is the single critical action you take before an issue encounters. It also plays its part in detecting defects in web development. Nginx access logs comprise detailed information about the access requests of the users. To analyze and monitor your web server, you have to parse Nginx access logs.

How to parse Nginx access logs

As a Linux user, you can use two methods for parsing your Nginx access. You can either utilize the Linux commands or use a log analyzer tool. Execution of the Linux commands will parse the Nginx access logs for the specified function, whereas you can get a complete report when you parse Nginx access logs with any analyzer tool. This write-up will demonstrate both methods for parsing the access logs of Nginx. So, let’s start!

How to parse Nginx access logs for getting IP addresses list

This section will show you how to get the client IP addresses using a Linux command. The provided method of parsing Nginx access logs can be useful for you if you want to know who has connected to the server or when you want to check out the IP addresses associated with the known troublemakers.

Considering that your Nginx access logs are saved in the “/var/log” directory, executing the below-given command will parse Nginx access logs to retrieve a list of the IP addresses of the users that have been accessed in the log file:

$ sudo cat /var/log/nginx/access.log | awk '{ print $1}' | sort | uniq -c | sort

How to parse Nginx access logs for getting accessed file list

In a situation when you want to check out the list of the files that the users on your server access, write out below-given commands in your terminal and hit “Enter”:

$ sudo cat /var/log/nginx/access.log | awk '{ print $7}' | sort | uniq -c | sort

How to parse Nginx access logs for counting requests per second

A coding genius can create a script for reading the Nginx access logs file, parsing the timestamps, and counting the number of requests per second. However, you can perform this whole operation by executing a simple Linux command which is:

$ sudo cat /var/log/nginx/access.log | awk '{print $4}' | uniq -c | sort -rn | head

The output of the above-given command will sort out all of the requests while putting a timestamp with each of them. Here, the first number represents the request count:

How to parse Nginx access logs for getting response codes

A three-digit HTTP Response Status Code is returned when a web server receives a request from a website user. This code indicates the event which is going to happen. For instance, a “301” response code means “Permanently moved”, whereas “200” states, “Okay, here is the content you requested”. You can parse your Nginx access logs for getting the response codes. To do so, execute the below-given in your Linux terminal:

How to parse Nginx access logs using online analyzer tools

It does not matter; if you use Nginx as a static content server, a load balancer, or a web server, you surely want to look at your access logs to see how well it is doing. There are a few options for log analyzer tools for Nginx, such as Goaccess, Visitors, ELK Stack. You can choose an analyzer tool according to your requirements.

However, to demonstrate the procedure of parsing Nginx access logs using an online analyzer tool, we selected Goaccess. To use Goaccess for parsing Nginx access logs, you have to install it first on your system:

$ sudo apt install goaccess

Using Goaccess analyzer tool : Goaccess comprises a real-time monitoring feature as well as an interactive terminal viewer. It was built in the C language, that’s why it is fast and utilizes an on-disk B+Tree database to handle logs gradually. The goal behind designing the Goaccess analyzer tool was to provide something that quickly performs log analysis on the terminal.

Goaccess generates metrics after every 200 milliseconds. As a result, you can have a precise picture of what’s going on with the traffic in real-time. This feature is handy to find out the reason behind the cases when something is not working correctly or unexpected traffic spikes. Goaccess will then determine whether the problem is with the Nginx application or with the network.

Instead of manually checking the Nginx access log for finding the issue, you can utilize the most critical parsing information of Nginx access logs by executing this command:

$ goaccess /var/log/nginx/access.log

Now, choose a log format and press “Enter” to continue:

After choosing a log format, you will be able to view the dashboard of the Goaccess analyzer, which will show you the information related to Unique visitors per day, Requested Files:

Goaccess will also provide you the information about the Static request, Not Found URLs:

You can also check out the Hostnames and IPs of your website visitors and also about their operating system, Browsers, and the Time duration they have spent on the site:

Scroll down through the Goaccess dashboard view the information related to the Referrers URLs, Referring Sites, HTTP Status Codes, and Key phrases from the Google Search Engine:

In the last section, we will see the Geo Location data linked with our web server:

Conclusion

Access logs provide you information you need to figure out what your Nginx is up to. You can parse Nginx access logs to monitor, analyze, and optimize your web server. If you are a Linux user, then you can parse the Nginx logs using commands or the Nginx analyzer tools. The analyzer tool offers you a complete report about your web server performance while the execution of the commands used to parse Ngnix access logs will only show you the output of specified action. This write-up showed you how to parse Ngnix access logs using Linux commands and Goaccess analyzer tool.