Cloud Log Service or Self-built ELK, Which Should I Choose?

Compared with self-built ELK (Elasticsearch/Logstash/Kibana), eSurfing Cloud's Cloud Log Service has the following advantages:

• One-stop and fully managed service reduces O&M investment.

• One set of services shared by multiple tenants, which can improve resource utilization compared with the multiple sets of ELKs you need to build.

• A high compression ratio lowers storage costs.

The following is a comparison of some functions.

Features	Comparison
Cloud service log collection	ELK: Uses open source collectors such as logstash or filebeat to collect logs. Cloud Log Service: Uses a collector to collect logs and provides a collection configuration guidance page, making it easy to get started.
Multi-language SDK log collection	ELK: No. Cloud Log Service: Multi-language logs can be reported directly to Cloud Log Service. (Coming soon)
Log Structuring Parsing	ELK: Implements structuring parsing of customized logs based on the collector. Cloud Log Service: Enables structuring parsing logs. You can use regular expressions, JSON, and delimiters to parse logs.
SQL functions	ELK: Only supports basic SQL statistics functions. Cloud Log Service: Offers a rich set of extended functions, such as aggregation, string, date, and time functions besides basic SQL functions, which broaden the range of use cases.
Charts	Cloud Log Service provides a variety of visualized charts such as tables, line charts, pie charts, bar charts, and flow charts.
Log Alarms	ELK: Log alarm is not available. Cloud log service: Out-of-the-box and quasi-real-time log keyword and SQL alarms are available. Supports multiple alarm channels
Log transfer	ELK: Logs cannot be directly transferred to ZOS. Cloud Log Service: Logs can be transferred to eSurfing Cloud ZOS through simple configuration on the console.
Log processing	ELK: Log processing is not available. Cloud Log Service: Provides scalable and high-availability data processing services, and supports data normalization, desensitization, and filtering.

To sum up, it is recommended that you use the fully managed Cloud Log Service instead of the self-built ELK.

Usage Suggestions

Cloud Server Scenarios

Scenario

If your application is deployed on an eSurfing Cloud server, you can use Cloud Log Service to collect logs centrally.

Usage Suggestions

• Log collection method: Collection through the collector. You need to install ICAgent on the cloud server, and create collection rules in the Cloud Log Service console to configure a path for Elastic Cloud Server (ECS) logs to be collected into a log unit. The collector will collect logs according to the collection rules. When the collection is completed, you can query and analyze the logs in real time on the Cloud Log Service console. For specific operations, see Ingesting ECS–Text Logs.

• Log project planning: Put the logs of an application system in a log project, and you can use the name of the application system as the name of the log project.

• Log unit planning:

o If there are no fixed rules for your logs and the logs cannot be structurally parsed, you can collect logs of similar components into the same log unit, such as Java components, PHP components, and Python components, which makes it easier for you to manage logs of multiple components; if the number of your components is relatively small (e.g., less than 20), you can collect the logs of each component into different log units.

o If your logs can be structurally parsed (e.g., Nginx logs), it is recommended that you collect logs in the same format into the same log unit, so that you can use the SQL analysis conveniently to display the analysis in charts.

How Do I Migrate Logs from a Third-Party Cloud to eSurfing Cloud?

This section describes how to migrate logs from a third-party cloud to eSurfing Cloud.

If you use the log service of a third-party cloud vendor and have a large amount of log data on the vendor's object storage, you can perform the following operations to migrate the logs to eSurfing Cloud:

Standard logs (for search and analysis): stored for 7 to 14 days for application O&M. These logs do not need to be migrated to eSurfing Cloud. Directly enable Cloud Log Services on eSurfing Cloud. After your services run on eSurfing Cloud for 14 days, the logs stored by the third-party cloud vendor will be aged. For details, see Getting Started with Cloud Log Service.
Archived logs: stored for more than 180 days for security compliance. These logs are usually transferred to ZOS. Use the migration function of ZOS to migrate files stored in the third-party vendor's object storage to eSurfing Cloud ZOS. For detailed operations, see Migrating Data from a Third-party Cloud to eSurfing Cloud ZOS.

Cloud Log Service

Log Management