Cloud Log Service supports the ingestion of logs from Elastic Cloud Servers (ECSs). You can create collection rules on the Cloud Log Service console, and configure the path of the ECS logs to be collected in log units, and the collector will collect logs based on the collection rules. When the collection is completed, you can query and analyze the logs in real time on the Cloud Log Service console.

Before You Begin

You have created log projects and log units. For details, see Creating Log Projects and Log Units.

Procedure

Select ECS–Text Logs for the Cloud Log Service ingestion method, and follow the steps below to complete the ingestion configuration.

Select the target log unit.

• Log in to the Cloud Log Service console, click on the name of the target log project in the Log Project module on the console overview page, and enter the Log Project page.

• Under the target log unit on the left, click Data Ingestion .

• Select ECS–Text Logs as the data to be ingested.

Select cloud servers.

• If you don't have a host group available yet, perform the following operations:

• Click New Host Group to switch to the host group creation tab.

• Enter the host group name.

• In the list of activated ECSs, select the target ECSs that you need to collect from logs to build a host group for central collection and management.

• Click Next.

• If you already have an available host group, click Select from Existing Host Groups, select the target host group, and click

• Next.

Create a collection configuration.

specific collection rules, see Collection Rule Configuration for details.

When the collection rules are configured, click Save and Enable to create the collection configuration and start collecting logs. Click Save and Do Not Enable and the collection configuration will only be created but will not be applied. You can enable it later on the Data Ingestion Management page.

Query and analysis configuration.

In this step, you can configure the index fields. For details, see Index Configurations.

Finish.

On the completion page, click Retrieve Log to switch to the log retrieval and analysis page. For details, see Querying and Analyzing Logs.

Collection Rule Configuration

You need to configure the collection rules in the log ingestion process. The collector will collect logs of your cloud servers based on the collection rules you configure.

The specific configuration parameters of the collection configuration are described as follows:

• Collection rule name: You need to customize the collection rule name. The name should contain 3 to 200 characters of letters, numbers, underlines (_) and hyphens (-), and must start and end with a letter or number.

• Collection path configuration: Fill in the log path and log file name for the logs you want to collect. The collector will monitor all log files matching the file name rules in these directories (including sub-directories) and collect the log data into Cloud Log Service. You can configure the collection path for:

• Collecting a single log file in a single directory. Example:

/var/logs/a.log

In this example, the collector monitors the log file named "a.log" in the "/var/log/" directory.

• Wildcard character mode is supported for fuzzy matching of directories and file names. The wildcard character (*) means matching multiple arbitrary characters, and the wildcard character (?) means matching a single arbitrary character. See below for the examples.

Collection Path	Description
/var/logs/*/a.log	As long as there is "a.log" in any directory in the "/var/logs/" directory, the log will be collected.
/var/logs/service-*/a.log	As long as there is "a.log" in any directory starting with "service-" in the "/var/logs/ directory", the log will be collected.
/var/logs/service/a*.log	As long as there is a log-type log file starting with "a" in the "/var/logs/service" directory, the log will be collected.
/var/logs/*.log	Log files with the suffix ".log" in the "/var/logs/" directory will be collected.

• Logs can be collected recursively. "/**/" represents the current directory and all its subdirectories. Example:

/var/log/**/a.log

The collector will monitor log files named "a.log" in all subdirectories in the path with the prefix "/var/log/".

Description

Only one double asterisk ( ) can be contained in a collection path. For example, "/var/logs/ /test/**/a.log" is not acceptable.
A collection path cannot begin with a double asterisk ( ), such as "/ /logtest/a.log" to avoid collecting system files.
• Collection Policy
• Full: Collect data from the first line of the target file.
• Incremental: Collect data from the target file after the collection configuration is applied. If the target file is not updated at the time when the collection configuration is applied, logs in the file will not be collected.
• Split Mode
You can select the splitting mode to achieve structuring parsing of logs. Currently, the collector provides various structuring parsing methods. See below:

Parsing Methods	Description
Single-line Full Text	Single-line full text means that the log contains only one line of content. When logs are collected, a line break indicates the end of a log, that is, in log files, one log is separated from another with a line break. The log data itself will not be structured, and the log fields will not be extracted. Each log has a default field (message), and the collector stores log contents in the message. For details, seeLog Structuring Parsing.
Multi-line Full Text	Multi-line full text means that a piece of complete log data may span multiple lines. You need to specify the first-line regular expression for matching. When a certain line of log matches the preset regular expression, it is considered to be the beginning of a log, and the next beginning of a line is considered the end identifier of this log. The log content is also stored in the message field. For details, seeLog Structuring Parsing.
Single-line Regular	Single-line regular mode is used to process structured logs. For logs containing one line of content, you need to specify a regular expression and the collector will extract multiple values from a complete log based on the regular expression. For details, seeLog Structuring Parsing.
Multi-line Regular	The multi-line regular mode is used to process structured logs. For logs containing multiple lines of content, you need to specify a line beginning regular expression to match the beginning of the logs, and another regular expression to extract multiple values. For details, seeLog Structuring Parsing.
Single-line Delimiter	The single-line delimiter mode supports splitting a log into multiple values using delimiters to achieve structured processing. This mode is only applicable to single-line logs, and each complete log ends with a line break. For details, seeLog Structuring Parsing.
JSON	Supports parsing JSON logs of the Object type and extracts JSON log content as key-value pairs, that is, the key of the first layer of Object is used as a key and the value of the first layer of Object is used as a value. For details, seeLog Structuring Parsing.

Cloud Log Service

Ingesting ECS–Text Logs