Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement]: Checksum S3 uploads #55

Open
estensen opened this issue Nov 19, 2023 · 0 comments
Open

[Enhancement]: Checksum S3 uploads #55

estensen opened this issue Nov 19, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@estensen
Copy link
Contributor

Use case and current behavior

AWS uses checksum for uploading files, but another checksum can be added and stored as metadata to be absolutely sure that the data that was supposed to be stored is stored. It's best practice for migrations of data and while you're not doing that here it might be good to have some extra protection.

Enhancement

Checksums can be specified on s3.PutObjectInput

Solution proposal

Either use SHA256 as checksum or CRC32C. Since blobs are keyed by it can be used, but it's a bit less performant than CRC32C.

Additional Information

https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
https://aws.amazon.com/getting-started/hands-on/amazon-s3-with-additional-checksums/?ref=docs_gateway/amazons3/checking-object-integrity.html

@estensen estensen added the enhancement New feature or request label Nov 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant