124. S3 Performance

In this episode, we discuss some tips and tricks for optimizing performance when working with Amazon S3 at scale. We start by giving an overview of how S3 works, highlighting the distributed nature of the service and how data is stored redundantly across multiple availability zones for durability. We then dive into specific tips like using multipart uploads and downloads, spreading the load across key namespaces, enabling transfer acceleration, and using S3 byte-range fetches. Overall, we aim to provide developers building S3-intensive applications with practical guidance to squeeze the most performance out of the service. 💰 SPONSORS 💰 AWS Bites is brought to you by fourTheorem an AWS consulting partner with tons of experience with S3. If you need someone to work with to optimise your S3-based workloads, check out at ⁠⁠⁠⁠⁠⁠fourtheorem.com⁠⁠⁠⁠⁠⁠! In this episode, we mentioned the following resources: Our previous episode on S3 best practices: https://awsbites.com/83-bucket-list-of-s3-best-practices “Deep dive on Amazon S3” (re:Invent talk from 2021): https://www.youtube.com/watch?v=FJJxcwSfWYg More recent re:Invent talk on Amazon S3 (with updated data) - We discovered this one just after the recording: https://youtu.be/sYDJYqvNeXU Multi-part upload user guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html Code examples from the SDK (high-level and low-level APIs): https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html Node.js official helper library (@aws-sdk/lib-storage): https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-lib-storage/ Example on how you can implement byte-range fetches: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_Scenario_UsingLargeFiles_section.html s3-getobject-accelerator library: https://github.com/widdix/s3-getobject-accelerator aws-c-s3 library: https://github.com/awslabs/aws-c-s3 S3 storage lens: https://aws.amazon.com/s3/storage-lens/ Docs on S3 Transfer Accelerator: https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html Performance Guidelines for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html Performance Design Patterns for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠