Operation Scenarios

Upload a large number of files in batches to the ZOS, for example, uploading system log files of an application.

Upload a large number of GB-sized files to the ZOS, for example, uploading HD video files.

Optimization Suggestion

① Use object names with random prefixes

Upload a large number of files in batches to the ZOS, especially in MB-sized or less mass storage scenarios. It is recommended to use object names with random prefixes so that the business requests can be evenly distributed in multiple partitions to optimize the scale-out performance.

Example:

For example, before this, objects are uploaded in the following form to archive log files of an application:

/log/20220610-01.log.tar.gz

/log/20220610-02.log.tar.gz

/log/20220610-03.log.tar.gz

/log/20220610-04.log.tar.gz

...

/log/20220611-01.log.tar.gz

/log/20220611-02.log.tar.gz

/log/20220611-03.log.tar.gz

/log/20220611-04.log.tar.gz

It is recommended to add a hexadecimal hash prefix of more than 3 digits to object names:

/8bc-log/20220610-01.log.tar.gz

/95d-log/20220610-02.log.tar.gz

/6c5-log/20220610-03.log.tar.gz

/da3-log/20220610-04.log.tar.gz

...

/a7c-log/20220611-01.log.tar.gz

/2bc-log/20220611-02.log.tar.gz

/ea7-log/20220611-03.log.tar.gz

/1d8-log/20220611-04.log.tar.gz

② Upload large files concurrently in mutipart

When uploading GB-sized or even TB-sized files (such as over-sized image installation packages and HD video files), it is recommended to upload them in multipart.

It costs a great deal to limit the file size to 5 GB in each upload process because you have to transmit the entire file once the transmission is interrupted due to network exception. Therefore, it is recommended to upload large files in multipart, and this will also improve performance.

Some S3-compatible client tools have the ability to upload a file in multipart by default. For example, the S3 Browser uploads large files, by default, in 8 MB parts:

Developers are recommended to upload large files by multipart to the API interface.

Zettabyte Object Storage

Best Practice for ZOS Performance Optimization