AWSException: RequestTimeTooSkewed - s3.put_object

[LuizFDuarte]

I am using Amazon s3 to store some binary files (the size of the files vary from 5KB to 10MB), sometimes the method throws this error

MethodError(convert, (Dict, AWS.AWSExceptions.AWSException: RequestTimeTooSkewed -- The difference between the request time and the current time is too large.

HTTP.ExceptionRequest.StatusError(403, "PUT", "/bucket/key", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 04 Jan 2023 20:41:17 GMT
Server: AmazonS3
Connection: close

[Message Body was streamed]""")

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><RequestTime>20230104T200955Z</RequestTime><ServerTime>2023-01-04T20:41:18Z</ServerTime><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds>...

These requests are being made from a docker container running on an EKS cluster. However, sometimes I receive the same error running the code in my computer.

[quinnj]

Can you share a bit more about your workflow? Are you making lots of requests concurrently? Do you have a sense for how long individual requests take or if some are taking especially long? For context, I've seen this kind of error in heavily cpu-saturated scenarios where lots of requests are made concurrently and there can be a gap between the time the request is signed (w/ a specific timestamp) and when the request is actually sent on the wire (and thus when s3 receives the timestamp). This is just a factor of Julia's task scheduling switching between tasks and if a request is signed, then we switch to another task, then don't switch back to the signed request for a long time and then send the request, it can produce this error.

It can sometimes be helpful to do some basic client-side rate-limiting to try and avoid this scenario by using a Base.Semaphore and tuning it accordingly.

[quinnj]

(also tangentially related is one of the goals of CloudStore.jl is I'm trying to ensure the request signing happens at the very last step before we actually send the request on the wire to hopefully minimize these kind of issues; we switched a production system that was running into these errors semi-frequently to use CloudStore and haven't seen these errors since)