Background

Nginx logs are important information for website O&M. They are used to record server activities and request information. Nginx servers usually output Nginx logs to local files. In the traditional methods, when O&M personnel query logs, they need to log in to the server and use tools in the operating system to view files. Since the logs are distributed on various servers, limitations such as error-prone CLI operations, and server permissions result in low log search efficiency. This section takes Nginx logs as an example to introduce the process of log collection, query, analysis and alarm.

Advantages

Compared with traditional log methods, Cloud Log Service has the following advantages:

Data is stored centrally, eliminating the need to log in to multiple servers for queries, which is particularly important under a microservice architecture;
Quick retrieval of logs, freeing you from cumbersome CLI operations and improving troubleshooting efficiency;
Detect abnormal logs in real time, with alarms available to accelerate fault response.

Procedure

Step 1: Create log projects and log units

Activate Cloud Log Service, log in to the log service console and create log projects and log units.

Create log projects: A log project is a resource unit for managing the log service. Usually, the logs of an independent project/business can be mapped to a log project.
Create log units: Log unit is the basic unit for collecting, storing, retrieving, and analyzing log data. Log data is managed as log units. Usually, logs from one application/service can be collected into one log unit. A separate log unit can be created in this case for Nginx logs.

Step 2: Configure log collection rules

Configure a host group:

A host group is a set of cloud servers for which you want to collect logs. It is a virtual group. Cloud Log Service manages all cloud servers that require log collection through the collector as host groups. Here you need to create a host group for the cloud servers where the Nginx log files are located.

Install the log collector:

Before configuring collection rules, you need to install the log collector on the target cloud servers. Install the collector according to the guidance on the page.

Create collection rules:

Collection rules refer to the policies used by the collector for collector logs, including the file collection paths, and log structuring modes, which can be created and configured in the console.

Configure field indexes:

Index is a data storage structure that can reduce the query time of log data. After the corresponding index on the logs is created, the data with the index field can be queried and analyzed accordingly.

Step 3: Query Nginx logs

When O&M personnel need to query Nginx logs, they can log in to the Cloud Log Service console to perform query operations. In the Nginx log unit created in step 1, click Retrieve Log to enter the retrieval page, enter the corresponding query statement, and set the query time range, and then you can view the query results at the bottom of the page. The following are common query cases for Nginx logs:

To query requests with the 200 status code, enter status=400 to query:
To query requests with status codes other than 200, enter NOT status=200 to query:
To query requests with a response time of more than 0.5 seconds, enter request_time>0.5 to query:
To query requests whose request method is GET and whose response status code is 400, enter request_method = 'GET' and status = 400 to query.

Step 4: Perform statistical analysis on Nginx logs

Cloud Log Service supports multi-dimensional analysis on Nginx logs and displays the results in charts.

To see the distribution of each request state in a pie chart, enter the analysis statement: select "status",count(*) as "num" from log group by "status" order by "num":
To count the top three access sources, enter the analysis statement: select count(1) as pv , http_referer group by http_referer order by pv desc limit 3

Cloud Log Service

Analysis Based on Business Logs