Python: Upload Large Files S3 Fast
I Am Trying to Upload Programmatically an Very Large File up to 1Gb on S3. as I Found That Aws S3 Supports Multipart Upload for Large Files, and I Found Some...
I am trying to upload programmatically an very large file up to 1GB on S3. As I found that AWS S3 supports multipart upload for large files, and I found some Python code to do it. (link )
My point: the speed of upload was too slow (almost 1 min).
Is there any way to increase the performance of multipart upload. Or any good library support S3 uploading
2 Answers
Leave my answer here for ref, the performance increase twice with this code:
import boto3
from boto3.s3.transfer import TransferConfig
s3_client = boto3.client('s3')
S3_BUCKET = 'mybucket'
FILE_PATH = '/path/to/file/'
KEY_PATH = "/path/to/s3key/"
def uploadFileS3(filename):
config = TransferConfig(multipart_threshold=1024*25, max_concurrency=10,
multipart_chunksize=1024*25, use_threads=True)
file = FILE_PATH + filename
key = KEY_PATH + filename
s3_client.upload_file(file, S3_BUCKET, key,
ExtraArgs={ 'ACL': 'public-read', 'ContentType': 'video/mp4'},
Config = config,
Callback=ProgressPercentage(file)
)
uploadFileS3('upload.mp4')
Special thank to @BryceH for suggestion. Although solution did increase the performance of S3 uploading, but I still open to receive any better solution. Thanks
1 minute for 1 GB is quite fast for that much data over the internet. You should consider S3 transfer acceleration for this use case.
Alexander Ross has covered the video game industry for a decade, writing deep dives on game design, esports tournaments, VR developments, and gaming culture.
