changefeedccl: add kafka sink config options for idempotence, retries, and max.inflight.requests.per.connection

[andyyang890]

We should allow users to customize the following Kafka producer options (corresponding sarama options in brackets) as they have an impact on whether events may become not strictly ordered after retries:

• enable.idempotence (Producer.Idempotent)
• retries (Producer.Retry.Max)
• max.in.flight.requests.per.connection (Net.MaxOpenRequests)

Namely, the sarama defaults for these options (enable.idempotence = false, retries = 50, max.in.flight.requests.per.connection = 5), which we use, could potentially allow events to become not strictly ordered during retries.

From the Kafka documentation about retries (https://kafka.apache.org/documentation/#producerconfigs_retries):

Allowing retries while setting enable.idempotence to false and max.in.flight.requests.per.connection to greater than 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.

Jira issue: CRDB-37801

[blathers-crl]

cc @cockroachdb/cdc

[andyyang890]

I investigated this a bit and it seems like for the version of sarama we're on (v1.42.1), this still leads to out-of-order issues. Based on IBM/sarama#2803, it seems like this is a bug in newer versions of sarama. We could consider downgrading sarama or switching off of the AsyncProducer.

[rharding6373]

Since exposing these options won't fix the underlying out of order issue, we are not going to be implementing this in the foreseeable future. Closing this issue.