DLQ with start_timestamp is not working correctly

[ShivanshuBagga2]

Hi,

• Version: 6.0.0
• Operating System: CentOS 7

Following is my configuration for DLQ input plugin-

input{
 dead_letter_queue {
    path => "/opt/shivanshu/setup/logstash-6.0.0/data/dead_letter_queue"
    start_timestamp => "2026-02-08T06:10:21"
    #commit_offsets => false
    pipeline_id => "main"
    tags => "dlq"
 }
}


filter{
}

output
{

if!("dlq" in [tags]){
file{
gzip => true
path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/logstash_%{+YYYY-MM-dd}.log.gz"
}
}else{
file{
gzip => true
path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/dlq-logstash_%{+YYYY-MM-dd}.log.gz"
}
}

I have some events stored in DLQ and wanna process events only those are added recently. So I use start_timestamp , but even after apply this I'm getting all events . So for testing purpose I changed its value to 2026-02-08T06:10:21, but even in this case I'm getting all events .

Here is my one sample event-

{"Locale":"Desiredresponselocalization215950","Message":{},"EventType":"Usage215940","ResourceName":"/api/startSessionForPerson##","Time":"2018-02-08T01:10:21.595-0500","UserAction":"userActionData1518070221594","SubjectType":"YYTU","ResponsePayload":"responsePayloadData1518070221594","@version":1,"host":"127.0.0.1","UserAgent":"userAgentData1518070221594","ClientId":695,"logger_name":"DemoApplication","CorrelationId":"4861cb44-0921-4da7-21594082ec-e86dcc05be35215940","SubjectId":"59bc52da6fb36c95771797f1215950","ElapsedTime(ms)":1964106054,"SessionId":"DB8BA6C6F897136A533696E84D681AB4215950","ModelId":"modelIdData1518070221594","RequestPayload":"requestPayloadData1518070221594","Status":"MOVED_PERMANENTLY","AdminId":"08340215940","ParentSpanID":"e86dcc05be35-0921-5431cb44-82ec-4da7215950","level":"INFO","Lifecycle":"qw","count":4,"Host":"aaaaaaa.hewitt.com","ResourceVersion":5.5,"Severity":"severityData1518070221594","RoleCd":"roleCdData1518070221594","SrcIP":"10.100.19.23","tags":["dlq"],"@timestamp":"2018-02-08T06:10:21.638Z","ConsumerRefID":"ddbdb1bc-b826-4265-9163-3e81e3712e64215940","SystemInstanceId":"5431cb44-8654-4da7-82ec-e86dcc05be35215950","port":42742,"thread_name":"pool-1-thread-2","level_value":20000,"ConsumerId":"Portal_44","ResourceInstance":"42272_CL1_TBAWS_QC_M1215950","@metdata":{"ip_address":"127.0.0.1"},"externalServiceName":"externalServiceNameData1518070221594","TestCfg":"JJ#UY","RequestType":"POST","SpanID":"5431cb44-0921-4da7-82ec-e86dcc05be35215950","EffectiveDate":"02-08-2018"}

You can find same question at here

[yaauie]

The start_timestamp directive is not a filter on any attribute of the Event (such as your Time or @timestamp values); it allows the DLQ reader to skip over items that were written to the DLQ earlier than the given time.

Do you have items that were written to the DLQ earlier than the given time that are showing up? If so, can you share a tarball of the DLQ directory? If you need to share privately, you can encrypt it with my public key, which can be found at https://keybase.io/yaauie

[ppf2]

This is reproduced using a copy of the DLQ data folder from an end user.

The bug appears to be that if start_timestamp is specified to be larger than the latest DLQ entry time, it will ignore the start_timestamp completely.

For example, if the latest DLQ entry_time is 2020-02-27T22:20:46.299Z in the DLQ data folder, specifying 2020-02-27T22:20:46.299Z (same as latest DLQ entry_time) as the start_timestamp correctly filters it down to just the last entry. Specifying 2020-02-27T22:20:46.300Z (just 1 ms later) results in the DLQ ignoring start_timestamp and returning everything.