This section is an overview of data processing on Cloud Log Service.

Cloud Log Service provides scalable and high-availability data processing services. Data processing services can be used for log normalization, enrichment, transfer, desensitization and filtering.

Processes

The log processing service processes logs in the following three steps.

A consumer group reads delimited logs from a source log unit.
Cloud Log Service processes each delimited log based on a processing rule.
Cloud Log Service writes the processed log to a target log unit. After the log is processed, you can view the processed log in the target log unit.

Product Features

Data normalization: Log data often comes from different system components, applications, or devices, so its formats and structures may vary, causing difficulties in analyzing, querying, and visualizing log data. Data normalization can extract fields from logs of different formats and convert these logs to a consistent format for subsequent processing and analysis.
Data desensitization: Desensitize sensitive information (such as passwords, mobile phone numbers, and addresses) in log data.
Data filtering: Filter logs of key businesses or services for subsequent key analysis and other scenarios.

Processing Syntax

The processing DSL (Domain Specific Language) is a Python-compatible script used for Cloud Log Service data processing. It provides more than 200 built-in functions, including process control functions, event operation functions, and field operation functions.

Cloud Log Service

Overview