Performance optimizations

[jpmckinney]

I/O options

• Write the LZ4 files alongside the directories. If an LZ4 file exists, skip writing.
• Calculating the checksum and bytes during tarfile creation might be faster.
• Writing the tarfile to boto3's socket will be faster, if possible.
• A single PUT, which can be used for files up to 5GB, will be better I/O but maybe worse network than a multipart upload, since uploads can be parallelized.
• I don't know if boto3 calculates ContentMD5. MD5 is slow (though there is a SIMD version). If we can skip ContentMD5, we can instead check integrity using the ETag.

Note that if we need to use multipart uploads, we need to consider incomplete uploads.

Network options

• Faster if using an AWS server. See also:
  • https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance.html
  • https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/
  • https://aws.amazon.com/premiumsupport/knowledge-center/s3-optimize-transfer/

To measure performance, we can report the running time and the bytes read, written, transferred (and cached).