Before You Begin

• Activate the ZOS service.

• Obtained the Access Key (AK), Secret Key (SK), and endpoint for Internet access from the ZOS console.

• Installed Python environment and downloaded the corresponding version of the ZOS official Python SDK. In this example, sdk_python3.X.tar is used. For more information, see Developer Documentation.

Environment Configuration

Extract sdk_python3.X.tar, and open the package required for importing through the command line in the extracted sdk file folder.

pip3 install -r requirements.txt

Copy the service-2.sns.sdk-extras.json and service-2.s3.sdk-extras.json files from the sdk file folder to the C:\Users\Your username.aws\models\s3\2006-03-01 file folder. If this file folder does not exist, create it.

Uploading Files in Segments

Scenario

• Large file uploads: Segmented uploads can enhance efficiency when handling large files. If a segment fails to upload due to reasons such as a network fault, you only need to upload that specific segment again, eliminating the necessity of re-uploading the entire file.

• Unstable network environment: In an unstable network environment, characterized by intermittent connectivity or limited bandwidth, segmented uploads mitigate the risk of upload failures caused by network interruptions or timeouts. In case of an interruption during the upload, you only need to re-upload the affected segment rather than the entire file.

• Uncertain file sizes: In cases of uncertain file sizes, such as often encountered in the video surveillance industry, segmented uploads offer a solution for initiating the upload process.

Operation Steps

Create a Python file and introduce the session module of the boto3 package within it.
```
from boto3.sessionimport Session
```

Configure the AK, SK, and Internet access endpoint for accessing ZOS.

access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#

Obtain the client for performing ZOS operations.

session =Session(access_key, secret_key)
s3_client =session.client('s3', endpoint_url=url)

Start the segmented upload and obtain the upload ID.

# Enter the name of the bucket to which the object will be uploaded and the object name. The bucket must exist.
dict_cmp = s3_client.create_multipart_upload(Bucket='?', Key='?')
uploadID = dict_cmp['UploadId'] # Record the uploadID, which will be used for the subsequent upload.

Read the segments to be uploaded.

# Create a list to collect all file segments.
chunks = []
# Read file segments.
with open("Enter the address of your file here", 'rb') as f:
    while True:
      chunk = f.read(5 * 1024 * 1024) # Enter the segment size here. In this example, the size is 5M. The minimum segment size is 5M.
      if not chunk:
        break
      chunks.append(chunk)

Upload all the segments and record the ETag and number of each segment.

# Create a list to record the ETag and number of each uploaded segment.
parts = []
i = 0
for chunk in chunks:
    i += 1
    # Upload a segment.
    dict_up = s3_client.upload_part(Bucket='?', Body=chunk,
                                    Key='?', PartNumber=i,
                                    UploadId=uploadID)
    # Record the ETag and number of this segment.
    part = {
        'ETag': dict_up['ETag'],
        'PartNumber': i,
    }
    parts.append(part)

Incorporate all the segments and store them into ZOS.

# The upload succeeds after all the segments are incorporated.
s3_client.complete_multipart_upload(
        Bucket='?',
        Key='?',
        MultipartUpload={
            'Parts': parts
        },
        UploadId=uploadID,
    )

Entire Code

from boto3.session import Session
 
access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#
 
 
key = '?' # File name after the file is uploaded to the bucket
bucketName = '?' # Name of the bucket to which the file will be uploaded
 
try:
    session = Session(access_key, secret_key)
    s3_client = session.client('s3', endpoint_url=url)
    # Start the segmented upload and obtain the upload ID.
    dict_cmp = s3_client.create_multipart_upload(Bucket=bucketName, Key=key)
    uploadID = dict_cmp['UploadId']# Record the uploadID, which will be used for the subsequent upload.
 
    filePos = "Enter the file path here."
    size = 5 * 1024 * 1024 #Enter the segment size here. In this example, the size is 5M. The minimum segment size is 5M.
 
    # Create a list to collect all file segments.
    chunks = []
    # Read file segments.
    with open(filePos, 'rb') as f:
        while True:
            chunk = f.read(size)
            if not chunk:
                break
            chunks.append(chunk)
    # Create a list to collect the ETag and number of each uploaded segment.
    parts = []
    i = 0
    for chunk in chunks:
        i += 1
        # Upload a segment.
        dict_up = s3_client.upload_part(Bucket=bucketName, Body=chunk,
                                        Key=key, PartNumber=i,
                                        UploadId=uploadID)
        # Record the ETag and number of this segment.
        part = {
            'ETag': dict_up['ETag'],
            'PartNumber': i,
        }
        parts.append(part)
 
    # Incorporate all the segments to complete the upload.
    s3_client.complete_multipart_upload(
        Bucket=bucketName,
        Key=key,
        MultipartUpload={
            'Parts': parts
        },
        UploadId=uploadID,
    )
    print('The file upload succeeds.')
except BaseException:
    print("Exceptions occur during the upload. Check the parameters and try again.")
finally:
    # Close the file stream.
    f.close()

Appending

Scenario

For regularly uploaded objects, appending to their contents is not supported. If the content of an object needs to change, you can only achieve this by uploading a new object with the same name. This inconvenience is particularly noticeable in scenarios where frequent rewriting occurs, such as with logs and video surveillance. Appending to existing objects enhances both scalability and the efficiency of file uploading.

Operation Steps

Create a Python file and introduce the session module of the boto3 package within it.
```
from boto3.sessionimport Session
```

Configure the AK, SK, and Internet access endpoint for accessing ZOS.

access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#

Obtain the client for performing ZOS operations.

session =Session(access_key, secret_key)
s3_client =session.client('s3', endpoint_url=url)

Read the file and record its MD5 value.

with open('Enter the path of the file to be uploaded here', 'rb') as file:
    data = file.read()
    md5 = hashlib.md5(data).digest()
    md5 = base64.b64encode(md5)

Upload the object. Upload the object for the first time. Upload the object for the first time.

s3_client.put_object(Bucket='Enter the bucket to which the object will be uploaded',
                     Metadata=dict(m1='m1'),
                     Body=data,
                     Key='Enter the key value of the object after being uploaded',
                     ContentMD5=str(md5, 'utf-8'),
                     Append=True,# Enable the appending feature
                     AppendPosition=0)# Specify the location where appending starts

Append to the uploaded object.

# Obtain information about the object to which data is appended
response = s3_client.head_object(Bucket="bucketblocks", 
                                 Key="mac.txt")
# Obtain the length of the current object
appendPos = response['ContentLength']

s3_client.put_object(Bucket='Enter the bucket to which the object will be uploaded',
                     Metadata=dict(m1='m1'),
                     Body=data,
                     Key='Enter the key value of the object after being uploaded',
                     ContentMD5=str(md5, 'utf-8'),
                     Append=True,# Enable the appending feature
                     AppendPosition=appendPos)# Specify the location where appending starts

Entire Code

from boto3.session import Session
import hashlib
import base64
 
access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#
 
# Name of the bucket to which the file will be uploaded
bname = 'Enter the name of the bucket to which the fill will be uploaded.'
# Key value of the file stored in the bucket
key = 'Enter the key value of the file being uploaded to the bucket.'
# The location where appending starts, which is 0 by default
appendPos = 0
 
try:
    # Read the file and record its MD5 value.
    with open('Enter the path of the file to be uploaded here', 'rb') as file:
        data = file.read()
        md5 = hashlib.md5(data).digest()
        md5 = base64.b64encode(md5)
    # Obtain the client for performing ZOS operations.
    session = Session(access_key, secret_key)
    s3_client = session.client("s3", endpoint_url=url)
    # Obtain information about all the objects in the bucket.
    response = s3_client.list_objects(Bucket=bname)
    # Traverse the key values of all the objects.
    for obj in response['Contents']:
        # The bucket has an object with the same key. Check whether appending is allowed.
        if obj['Key'] == key:
            res = s3_client.head_object(Bucket=bname, Key=key)
            if res['ResponseMetadata']['HTTPHeaders']['x-rgw-object-type'] != 'Appendable':
                print('This object is not an Appendable type. Appending to the object is not allowed.')
                raise Exception
            #Appending to the object in the bucket is allowed. Update the location where appending starts.
            appendPos = res['ContentLength']
    # Complete the appending.
    s3_client.put_object(Bucket=bname,
                         Metadata=dict(m1='m1'),
                         Body=data,
                         Key=key,
                         ContentMD5=str(md5, 'utf-8'),
                         Append=True,
                         AppendPosition=appendPos)
    if appendPos == 0:
        print('The initial upload succeeds.')
    else:
        print('The appending succeeds.')
 
except BaseException:
    print("The parameter configurations are invalid. Adjust them and try again.")
finally:
    # Close the file stream.
    file.close()

Resumable Transfer

Scenario

• Large file uploads: When uploading large files, such as videos, backup files, and software packages, which may take considerable time, exceptions may occur, such as network interruptions, tool malfunctions, or manual interventions. Resumable uploads enable you to resume uploading an interrupted file from the point of interruption rather than restarting the upload of the entire file.

• Unstable network environment: In an unstable network environment, like a mobile network or with limited bandwidth, uploading large files may encounter interruptions or timeouts. Resumable uploads resolve issues in these scenarios.

• Long-duration uploads: Certain upload tasks may extend over a prolonged period, heightening the risk of unforeseen interruptions. Resumable uploads divide an upload task into segments, preserving upload progress to allow for resumption after interruptions.

• Client or server failures: In the event of client or server failures during an upload, resumable uploads enable you to reconnect and resume uploading from the point of interruption, mitigating the severity of the impact on the upload task.

Operation Steps

Create a Python file and introduce the session module of the boto3 package within it.
```
from boto3.sessionimport Session
```

Configure the AK, SK, and Internet access endpoint for accessing ZOS.

access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#

Obtain the client for performing ZOS operations.

session =Session(access_key, secret_key)
s3_client =session.client('s3', endpoint_url=url)

Query the segments that have not been fully uploaded or have been abandoned, and obtain their upload ID and key.

response = s3_client.list_multipart_uploads(Bucket='Name of the bucket to which the object will be uploaded')
uploadInfo = response['Uploads']

Obtain information about uploaded segments using the upload ID and key from the previous step.

response = s3_client.list_parts(Bucket='Name of the bucket to which the object will be uploaded', Key='Key value',
                         UploadId='Upload ID')
finishedPartsInfo = response['Parts']
# Create a list to record the numbers of uploaded segments.
finishedPartNumbers = []
# Record the segment sizes.
partSize = 0
# Create a list to collect the ETag and number of each uploaded segment.
parts = []

for finishedPart in finishedPartsInfo:
    finishedPartNumbers.append(finishedPart['PartNumber'])
    if partSize < finishedPart['Size']:
        partSize = finishedPart['Size']
    part = {
        'ETag': finishedPart['ETag'],
        'PartNumber': finishedPart['PartNumber']
    }
    parts.append(part)

Upload unfinished segments of the object and record their ETags.

# Create a list to collect all file segments.
chunks = []
# Read file segments.
with open("Enter the address of your file here", 'rb') as f:
    while True:
        chunk = f.read(5*1024*1024)  # Enter the segment size here. In this example, the size is 5M. The minimum segment size is 5M.
        if not chunk:
            break
        chunks.append(chunk)

i = 0
for chunk in chunks:
    i += 1
    # This segment has been uploaded. Skip it.
    if i in finishedPartNumbers:
        continue
    # Upload a segment.
    response = s3_client.upload_part(Bucket=bucketName, Body=chunk,
                                    Key=key, PartNumber=i,
                                    UploadId='Upload ID')
    # Record the ETag and number of this segment.
    part = {
        'ETag': response['ETag'],
        'PartNumber': i,
    }
    parts.append(part)

Incorporate all the segments and store them into ZOS.

# The upload succeeds after all the segments are incorporated.
s3_client.complete_multipart_upload(
        Bucket=bucketName,
        Key=key,
        MultipartUpload={
            'Parts': parts
        },
        UploadId=uploadID,
   )

Entire Code

from boto3.session import Session
 
access_key = "Enter your AK here."#
secret_key = "Enter you SK here."#
url = "Enter you endpoint here."#
 
bucketName = '?' # Name of the bucket to which the file will be uploaded
filePos = "?" # Location of the file to be uploaded
 
try:
    session = Session(access_key, secret_key)
    s3_client = session.client('s3', endpoint_url=url)
    # Query the segments that have not been fully uploaded or have been abandoned, and obtain their upload ID and key.
    response = s3_client.list_multipart_uploads(Bucket=bucketName)
    uploadInfo = response['Uploads']
    if uploadInfo is None:
        print(bucketName + 'There are no objects that require resumable uploads in the bucket.')
        raise BaseException
    # Obtain the Upload ID and key of the object that requires resumable uploads. In this example, the first object is used.
    uploadId = uploadInfo[0]['UploadId']
    key = uploadInfo[0]['Key']
 
    # Obtain information about uploaded segments.
    response = s3_client.list_parts(Bucket=bucketName, Key=key, UploadId=uploadId)
    finishedPartsInfo = response['Parts']
    # Create a list to record the numbers of uploaded segments.
    finishedPartNumbers = []
    # Record the segment sizes.
    partSize = 0
    # Create a list to collect the ETag and number of each uploaded segment.
    parts = []
    for finishedPart in finishedPartsInfo:
        finishedPartNumbers.append(finishedPart['PartNumber'])
        if partSize < finishedPart['Size']:
            partSize = finishedPart['Size']
        part = {
            'ETag': finishedPart['ETag'],
            'PartNumber': finishedPart['PartNumber']
        }
        parts.append(part)
 
 
    # Create a list to collect all file segments.
    chunks = []
    # Read file segments.
    with open(filePos, 'rb') as f:
        while True:
            chunk = f.read(partSize)  # Enter the segment size here. In this example, the size is 5M. The minimum segment size is 5M.
            if not chunk:
                break
            chunks.append(chunk)
 
    i = 0
    for chunk in chunks:
        i += 1
        # This segment has been uploaded. Skip it.
        if i in finishedPartNumbers:
            continue
        # Upload a segment.
        response = s3_client.upload_part(Bucket=bucketName, Body=chunk,
                                         Key=key, PartNumber=i,
                                         UploadId=uploadId)
        # Record the ETag and number of this segment.
        part = {
            'ETag': response['ETag'],
            'PartNumber': i,
        }
        parts.append(part)
 
    # Incorporate all the segments to complete the upload.
    s3_client.complete_multipart_upload(
        Bucket=bucketName,
        Key=key,
        MultipartUpload={
            'Parts': parts
        },
        UploadId=uploadId
    )
    print('The file upload succeeds.')
except BaseException:
    print("Exceptions occur during the upload. Check the parameters and try again.")
finally:
    f.close()

Zettabyte Object Storage

Segmented Upload of Large Files