This section describes the terms that are related to the data processing feature.

Terms

ETL

Extract, transform, and load (ETL) is a process during which data is extracted from business systems, cleansed, transformed, and loaded. This process unifies and standardizes data from different sources. Cloud Log Service can load data from a source log unit, transform data, and then write the transformed data to a target log unit. Cloud Log Service can also load data from OSS and other log units.

Event, Data, and Log

In data processing, events and data are represented by logs. For example, the event time is equivalent to the log time, and the e_drop_fields function discards log fields.

Log Time

The log time indicates the point in time at which an event occurs. The log time is also known as the event time. The log time is indicated by the reserved field time in Cloud Log Service. The value of this field is extracted from the time information in logs. The value is a UNIX timestamp representing the number of seconds that have elapsed since the epoch time January 1, 1970, 00:00:00 UTC. Data type: integer. Unit: seconds.

Log Receiving Time

The log receiving time indicates the point in time at which a log is received by a server of Cloud Log Service. By default, this time is not saved in logs. However, if you turn on Log Public IP for a log unit, this time is recorded in the log tag field receive_time . In the data processing, the complete name of this field is tag :receive_time. The value is a UNIX timestamp representing the number of seconds that have elapsed since the epoch time January 1, 1970, 00:00:00 UTC. Data type: integer. Unit: seconds.

Log Tag

Logs have tags for you to differentiate one log from another. Each tag field is prefixed with " tag :". Cloud Log Service supports two types of tags, which are:

Custom tags: tags that you add when you call the API PutLogs operation to write data.
System tags: tags that are added by Cloud Log Service, including client_ip and receive_time .

Configuration-related Terms

Source Log Unit

In data processing, the data processing feature reads data from a source log unit for processing.

You can configure only one source log unit for a data processing task, but you can configure the same source log unit for different processing tasks.

Target Log Unit

In data processing, the data processing feature writes processed data to target log units.

You can configure one or more target log units for a data processing task. Data can be written to target log units with a static or dynamic configuration. For the specific configuration methods, see Distributing Data to Multiple Target Log Units.

DSL

The domain specific language (DSL) for Cloud Log Service is a Python-compatible scripting language, and is used for data transformation in Cloud Log Service. The DSL for Cloud Log Service is built on Python. It provides more than 200 built-in functions to simplify common data processing tasks. It also allows you to use custom Python extensions. For more information, see Introduction to Languages.

Processing Rules

A processing rule is a data processing script that is orchestrated by using the DSL for Cloud Log Service.

Processing Task

A data processing task is the minimum scheduling unit of data processing. You must configure a source log unit, one or more target log units, a transformation rule, a transformation time range, and other parameters for a data processing task.

Cloud Log Service

Basic Terms